We live in a complex world, and it’s impossible to be an expert on everything that impacts our lives. In many domains, we have to trust the expertise of others to guide our decisions. Yet not all experts hold rational beliefs, and many people who are framed as experts in media are not actually experts. How do we separate the wheat from the chaff, focusing on high-quality sources of information and ignoring low-quality sources?
Over the years I’ve observed the behavior of various public figures in my areas of expertise, and I feel that I’ve developed pretty good nose for quickly sniffing out the credibility of public experts. In this post, I’ve attempted to take my intuitions and put them in writing so I can pass at least some of them on to others.
Here is a list of some of the heuristics I use when I’m assessing the credibility of people who are framed as experts. None of these are ironclad rules, but together they can help you get a quick sense of whether to listen to someone:
- Does he have training or experience in the subject? This is particularly relevant when someone disagrees with expert consensus. Sometimes people get a platform just because they have a novel or interesting idea, even if that idea is unconvincing to a knowledgeable person. It is of course possible that the non-expert is right and the experts are wrong, but it’s unlikely. This is just a heuristic, since in some areas the experts are truly not knowledgeable. For example, economists are barely better than chance at predicting the short-term behavior of the economy, but this doesn’t prevent some of them from prognosticating about it.
- Does she try to explain everything with one idea? The work of Philip Tetlock, PhD and others has shown that people who have one big idea to explain everything (“hedgehogs” or ideologues) are very bad at accurately modeling the world, predicting outcomes, and recommending effective actions. These people are often selected for media attention because they are clear and confident about their beliefs. The world is a complex place, and people who are able to model that complexity in their minds have better information than those who aren’t. Look for people who tend to use multi-factor explanations.
- Does he show nuance? Related to #2. There are usually exceptions and nuances. Is this person able to build them into his mental model?
- Is she able to change her mind? Inability to change one’s mind is a hallmark of an irrational belief system. Can you find examples of this person changing her mind in the past when presented with new evidence or a better interpretation of existing evidence?
- Does he lean toward conspiracy explanations and/or blaming the government? Large-scale conspiracies are improbable and the government is a favorite punching bag for cranks.
- Does she cite evidence that isn’t representative of the literature as a whole? This is common among experts, less common among the best experts, and very common among non-experts. It’s also often easy to spot with a little effort. Do a quick Google Scholar search for a recent meta-analysis on the topic, ideally from the Cochrane Collaboration. Are the conclusions of the meta-analysis consistent with the evidence she is citing, or is she cherry-picking individual studies that support her position?
- Does he portray experts who disagree with him as corrupt, stupid, or dishonest? Corruption, irrationality, and dishonesty do exist among experts, but if someone makes these claims without presenting clear evidence for them, it’s a bad sign. Accusing opposing experts of “lies”, “scams”, and other similar language is a red flag that the person making the claim is not objective. Conflicts of interest are common among experts, and good to keep in mind, but it’s useful to consider possible conflicts of interest of the person pointing them out as well.
- Does the topic relate to personal identity, or is it otherwise highly political or controversial? Domains such as religion, politics, and nutrition relate strongly to peoples’ personal identities. In this context, beliefs are often driven by group affiliation rather than rational consideration of evidence. The likelihood that someone is providing high-quality information in these domains is lower than in less controversial areas like physics or neuroscience.
What did I miss?
Is this person heavily subsidized by some profit-making group?
Stephan, the problem is:
1) In any scientific field, some of the most experienced experts were wrong in the past
2) In any scientic field, there are many issues with single or main factor explanations
3) “Nuance” is often used as a defensive strategy by industry groups
4) Changing your mind without solid reason is not scientific
5) There have of course been many conspiracies, cigarettes and sugar being prime examples
6) The literature is full of (sponsored) junk; being ‘representative’ is a classic industry strategy
7) Physcis (big bang) and neuroscience (free will) are highly controversial topics; most scientific fields are; not acknowledging this is probably the best indicator of a non-expert.
Hi Sam,
Here are my responses:
1) Experts are wrong sometimes, but they are more often right than non-experts. The point is not to be perfect (impossible) but to improve batting average.
2) This is sometimes true. Maybe a better heuristic would be to watch out for someone who has single-factor explanations when others have multi-factor explanations for the same phenomenon.
3) Nuance may be sometimes used as a defensive strategy by industry groups, but that doesn’t invalidate it as a useful heuristic. Nuance is a typical characteristic of good models.
4) Changing your mind without good reason isn’t scientific, as you said, but throughout a person’s career if the evidence never merited a change of mind that would be pretty extraordinary! Scientific evidence is constantly evolving so in practice there will generally be valid reasons for a person to change his mind at some point. That’s what you want to look for.
5) My point wasn’t to argue that conspiracies never happen, only that they are much more often invoked than actually occur. Also I don’t think sugar qualifies as a conspiracy. See this paper in Science magazine discussing the shortcomings of that view. http://science.sciencemag.org/content/359/6377/747.full
6) If this is what you believe then you must not place much value on the scientific literature in general. I can’t help you there.
7) Most of physics and neuroscience are not controversial. You selected non-representative subtopics within those fields that are controversial. See point #6.
Most of physics isn’t controversial? You may be surprised to hear that everything we thought we knew about particle physics is coming into question. The new particles predicted by the Standard Model are not being found, and people are getting worried.
The really big problem is Higgs boson. It’s lighter than it should be. Not just a bit lighter, millions of billions of times lighter. It’s an absolute disaster.
https://orbitermag.com/is-particle-physics-in-crisis/
“In recent years, physicists have been watching the data coming in from the Large Hadron Collider (LHC) with a growing sense of unease. ….. none of the more exotic particles and interactions that theorists hoped to see has been forthcoming. ….
“….why is the Higgs boson so light? In experiments it weighed in at 125 times the mass of a proton. But calculations using the theory implied that it should be much bigger – roughly ten million billion times bigger, in fact. …..”
I read the entire essay. I can tell that you did not. The author makes the case that the current approach used by other high energy physicsts is ineffective at analysing data from CERN. He then proceeded to propose and advicate for a different method of analysis.
You can tell I did not read the entire essay? Actually I read it twice.
A mathematician friend here in Oxford asked a physicist he knows whether it was true and he said yes. My friend was astonished, just like you are. It seems nobody wants to talk about it.
So you cherry picked one non-scientific article from the Web to decide that all of physics as we know it is wrong?
I can do that too: https://www.wired.com/2015/11/physicists-are-desperate-to-be-wrong-about-the-higgs-boson/
https://www.google.ca/amp/s/phys.org/news/2016-01-higgs-particles-heavier-earlier.amp
https://www.scientificamerican.com/article/higgs-boson-looks-standard-but-upgraded-lhc-may-tell-a-different-tale1/
I see over and over that Theoretical Physicists are disappointed that nothing is disproving the Standard Model of particle physics. They want to be wrong, so they can figure out why, modify, and improve their model. Everything with the LHC is pretty much going exactly as planned (and that’s disappointing), because the experts in the field are much better at predicting real world phenomenon than non-experts (astrology, psychics, religion, etc). Sure, they’re wrong sometimes, but they’re more often right.
“you cherry picked one non-scientific article from the Web”
The article was written by a physicist who works on these things.
“Ben Allanach is a professor in the department of applied mathematics and theoretical physics at the University of Cambridge. Along with other members of the Cambridge Supersymmetry Working Group, his research focuses on collider searches for new physics.”
well, they found the standard Higgs. It decays like a Higgs should decay. The Standard Model is complete and there is no need for other particles. But here is the thing: there is no need for other high energy accelerators. So they are ramping up the propaganda for the next big one. You guys claim to be familiar with the medical sciences, and you have never seen things like these?
You are right. Actually we are at the cusp of changes to physics of the magnitude made by Einstein. We already know that GR and QM don’t work well together. There is something wrong when you expect that the visible mass is much smaller than the expected mass of a galaxy. Particularly when there is a simple empirical equation (MOND) that fits the data of all galaxy scale structures without any extra mass, and with a single universal parameter. There are some new emergent gravity theories under development which are likely to change the whole theoretical landscape. Also read Lost in Maths by Sabine Hossenfelder. Particle Physics is really in grave crisis, if you follow Sabine.
Yes they are lost in the suffocating prison of mathematics.
They need to start with the structure of the atom. They threw away the Bohr atom with nothing sensible to replace it. And they need to resurrect the ether. Obviously you can’t have light propagating in empty space if it really is empty.
If the ether is a sea of tiny particles, it has the potential to explain quite a lot. EM waves would be waves in this sea, and gravity would be the local ether density. Matter particles would be aggregates or ‘quanta’ of ether particles, and magnetism their spin.
The “there was no sugar conspiracy” article is refuted in the same journal: http://science.sciencemag.org/content/360/6388/501.1 . Industry influence was and is significant, and to some extent hidden or non-transparent.
The literature is seriously corrupted, both due to industry influence and shoddy science in general. In fields such as Psychology, about a third of the results are not reproducible. When it comes to drugs, many studies are run by manufacturers.
And yes, vast parts of physics (particle physics, cosmology and much more) and neuroscience are highly controversial. In neuroscience, there is not even a theory of how the brain (let alone the mind) works, how drugs such as anti-depressants work, how IQ works etc.
Hi Sam,
A conspiracy is “a secret plan by a group to do something unlawful or harmful”. There was nothing secret about the sugar industry funding researchers. Furthermore, it was the same thing other industries were doing. That’s the point of the Science article. By definition, there was no conspiracy.
It was simply a case of an industry group funding a researcher whose perspective they liked, same as the meat and dairy industries funded researchers like John Yudkin and Ronald Krauss because their work cast doubt on the harms of saturated fat. I don’t understand why people get so paranoid when the sugar industry funds researchers but don’t seem to care when other industry sectors do the same thing. I agree industry funding can be a corrupting influence, but that doesn’t make it a conspiracy.
In obesity:
has the person been obese himself and now isn’t, or is he obese still despite all his “knowledge?” If he has been and now isn’t he deserves more of a listen. If they are still obese, they are talking through their hats!
Do they promote “eat less, move more?” If they do, they are in denial.
Do they dismiss hormones and brain? If they do they are a waste of space.
If they deny that food can be addictive then forget them. Anything can be addictive.
If they say detox, hang out with Deepak Chopra or the like… well..!
In regards to obesity, I pay attention to what Rena Wing concludes about her study population.
http://www.nwcr.ws
Behaviors that prevent weight regain are more important to me than abstract arguments about hormones and food addiction. For instance, participants in the NWCR are generally physically active, weigh themselves often and eat breakfast.
Does the person believe there is an easy solution to problems that have plagued mankind since the beginning? Like government should just do x.
Is the person respected and praised by their ideological opponents – this helps filter out people engaged in true academic debate from the cranks
Does their “science” ultimately refer back to that done by an advocacy group?
Would the person face strong social criticism, media criticism, and deplatforming by major institutions if they said something differently? Think ‘politically incorrect’ statements.
Making statements with an absence of qualifiers and overgeneralization that leads to overstating their case beyond the evidence base e.g. “Additives in processed foods cause gut inflammation” as opposed to a statement using qualifiers and specifics: “Some studies have found that the consumption of some commonly used emulsifiers increases gut inflammation in genetically susceptible rats and intestinal cell line models”.
ray peat falls in most of these categories but, based on my experiments, i think he holds most of the truth. any thoughts about that?
Hi Coconut,
I looked into Ray Peat a while back and didn’t find his arguments credible. Really far-out theories that weren’t well supported by the evidence he cited, or the lit as a whole. I think there’s a reason why he fits into the categories above. That’s my opinion.
Dear doctor Guyenet,
Thank you very much for your answer. i think, as you have pointed out, that many experimental and observational studies clearly disprove Ray Peat claims. But i have to admit that this way of eating works very well for me. i’m eating one chocolate bar (100g) a day, fruit juice, cheese and very limited pufa and no starch. the results are (again, for me) outstanding: leaner than i was on very low carb diet and i’m not overweight anymore (with all blood test perfect). i still dont understand why and i’d like to know if you think it’s dangerous. i have experienced before very low carb and very low carb mediteranean diet. i think the decisive factors of leaness for me were: salt, sugar ans stearic acid. i can’t explain otherwise. again any thought? thank you very much for your work.
I will discount 1) (training on the subject). When medicine is as corrupt as it is, training is unneeded and most often associated with big Pharma favoring advice. For example my doctor tries to lower my cholesterol to levels where life expectancy decreases. Add to that that some of the biggest impacts on nutrition have come from a dentist or anthropologists.
Let us come to 2). The one big idea is evolution. I am going to say that nutritional theories which are inconsistent with evolutionary history should be discarded. Of course we are in the middle of a very rapid (300 generations or less so far) evolution. The genes most sensitive to agricultural diets have been eliminated, but to this day several tens of percent of humans are born with some intolerance to common foods. It is this partially incomplete evolution that muddies the waters a lot.
Paraphrasing Dobzhansky, there’s no going back on the linear evolution path glib. History shows us where we’re headed. A couple recent observations. First, basketry carries over intact from Turkey to North America…as does agriculture, pyramid building, etc. etc. etc. Collective social memory preserved the advance long before writing. Second, Asian depictions of elk in petroglyphs disappeared in the Bronze Age as dependence on hunting also disappeared.
We are opportunistic scavengers with incredible memories.
we are mostly talking enzymatic and metabolic “memory” here. If you don’t have phytase, you don’t have it.
The social memory stopped the Eurasian settlers from reverting to paleo genetics. While the Salish were grain-free and low carb eaters for 10,000 years, they immediately started eating starchy carbs when they could get them. Women who formerly carried 100 lb clam baskets had no problem carrying 100 lb flour sacks…and flour is also much simpler than drying and storing fish.
Farther south, starchy carbs (corn/quinoa/potatoes) were developed rapidly as staple foods.
Anyone who unironically cites Gary Taubes as an authority on nutrition
Anyone who pretends the Blue Zones don’t exist
Anyone who thinks a magical macronutrient ratio can achieve fat loss without a caloric deficit
As the Voodoo Donuts slogan says, the magic is in the hole…
Optimal macronutrient ratios are a secondary sweet spot, once calories eaten are balanced with calories expended to maintain weight. IMO that’s somewhere around 15%P/30%F/55%C.
Another one, related to being willing to change one’s mind: Is the expert generally the first to correct themselves when they make an error? Or when their predictions turn out to be wrong? (economics and political blogs in particular)
8. Is the person’s name Gary Taubes?
Is his name Gary Taubes?
I like to read your blog and think you’re onto something real but this was a little obvious. BTW I’m a calories in-calories out person.