Stephan J. Guyenet, PhD

The science of body weight and health

Archives for August 2018

Quickly assessing the credibility of public experts

by

We live in a complex world, and it’s impossible to be an expert on everything that impacts our lives.  In many domains, we have to trust the expertise of others to guide our decisions.  Yet not all experts hold rational beliefs, and many people who are framed as experts in media are not actually experts.  How do we separate the wheat from the chaff, focusing on high-quality sources of information and ignoring low-quality sources?

Over the years I’ve observed the behavior of various public figures in my areas of expertise, and I feel that I’ve developed pretty good nose for quickly sniffing out the credibility of public experts.  In this post, I’ve attempted to take my intuitions and put them in writing so I can pass at least some of them on to others.

Here is a list of some of the heuristics I use when I’m assessing the credibility of people who are framed as experts.  None of these are ironclad rules, but together they can help you get a quick sense of whether to listen to someone:

  1. Does he have training or experience in the subject?  This is particularly relevant when someone disagrees with expert consensus.  Sometimes people get a platform just because they have a novel or interesting idea, even if that idea is unconvincing to a knowledgeable person.  It is of course possible that the non-expert is right and the experts are wrong, but it’s unlikely.  This is just a heuristic, since in some areas the experts are truly not knowledgeable.  For example, economists are barely better than chance at predicting the short-term behavior of the economy, but this doesn’t prevent some of them from prognosticating about it.
  2. Does she try to explain everything with one idea?  The work of Philip Tetlock, PhD and others has shown that people who have one big idea to explain everything (“hedgehogs” or ideologues) are very bad at accurately modeling the world, predicting outcomes, and recommending effective actions.  These people are often selected for media attention because they are clear and confident about their beliefs.  The world is a complex place, and people who are able to model that complexity in their minds have better information than those who aren’t.  Look for people who tend to use multi-factor explanations.
  3. Does he show nuance?  Related to #2.  There are usually exceptions and nuances.  Is this person able to build them into his mental model?
  4. Is she able to change her mind?  Inability to change one’s mind is a hallmark of an irrational belief system.  Can you find examples of this person changing her mind in the past when presented with new evidence or a better interpretation of existing evidence?
  5. Does he lean toward conspiracy explanations and/or blaming the government?  Large-scale conspiracies are improbable and the government is a favorite punching bag for cranks.
  6. Does she cite evidence that isn’t representative of the literature as a whole?  This is common among experts, less common among the best experts, and very common among non-experts.  It’s also often easy to spot with a little effort.  Do a quick Google Scholar search for a recent meta-analysis on the topic, ideally from the Cochrane Collaboration.  Are the conclusions of the meta-analysis consistent with the evidence she is citing, or is she cherry-picking individual studies that support her position?
  7. Does he portray experts who disagree with him as corrupt, stupid, or dishonest?  Corruption, irrationality, and dishonesty do exist among experts, but if someone makes these claims without presenting clear evidence for them, it’s a bad sign.  Accusing opposing experts of “lies”, “scams”, and other similar language is a red flag that the person making the claim is not objective.  Conflicts of interest are common among experts, and good to keep in mind, but it’s useful to consider possible conflicts of interest of the person pointing them out as well.
  8. Does the topic relate to personal identity, or is it otherwise highly political or controversial?  Domains such as religion, politics, and nutrition relate strongly to peoples’ personal identities.  In this context, beliefs are often driven by group affiliation rather than rational consideration of evidence.  The likelihood that someone is providing high-quality information in these domains is lower than in less controversial areas like physics or neuroscience.

What did I miss?

A more complete picture of the “collapse” of NuSI, and corrections to my previous post on it

by

After publishing my post Nutrition Science Initiative (NuSI) in retrospect, I received an email from Peter Attia, MD, the former president of NuSI, suggesting that I had gotten aspects of the story wrong due to a lack of additional context. Attia and I spoke twice by phone, and I think he has a point. I also had an additional conversation with Kevin Hall, PhD, one of the key researchers NuSI funded, for additional context from his perspective.

Salaries

NuSI executives didn’t set their own compensation, as I and others have implied. Compensation was set by the Compensation Committee of NuSI’s board, which did not include NuSI executives or employees. All overhead (i.e., all non-research dollars), including compensation, was paid directly from the Arnold Foundation grant with their knowledge and consent. According to Attia, donors other than the Arnold Foundation didn’t pay for salary or any other overhead; their funds went exclusively to research. Although Attia’s salary may seem high for a 501(c)(3) nonprofit, at its highest it was set following the recommendation of an external firm specializing in nonprofit executive compensation, at the request of the Board of Directors and with the approval of the Arnold Foundation. It seems that there was no self-dealing and my original post was incorrect in saying that NuSI executives “paid themselves enormous salaries.”

The collapse of NuSI

The reasons for the collapse of NuSI are more complex than what I portrayed in my article, and what appeared in the Wired article. A key reason NuSI collapsed is that it lost its funding, in part due to the fact that Attia was its most effective fundraiser and he left the organization at the end of 2015. Although Hall left NuSI’s Energy Balance Consortium (EBC) in 2016 due to scientific interference and personality conflicts, effectively shutting down the EBC at least temporarily, NuSI was larger than the EBC and the organization may have continued without it had funding not collapsed in 2016. The reasons why Attia left NuSI are complex, but they include undesired changes in his job requirements and personality conflicts.

Interference in the research process

NuSI did have its problems, some of which were described in my post and the Wired article, and some of which weren’t. I still believe NuSI tried to play a greater role in the research process than it should have, interfering with science and alienating certain researchers. Attia explained that from NuSI’s perspective, their intention was to create an adversarial collaboration that would be more effective at finding the truth than either camp alone. I actually think this is an admirable goal in many settings, and Daniel Kahneman, PhD describes a nice example of it in his book Thinking, Fast and Slow.

However, there are two reasons why I think it wasn’t a good idea in this particular case. First, this wasn’t the original agreement NuSI reached with EBC investigators. As Hall told me at the beginning of this saga in 2012, and reiterated in a recent conversation, the original agreement was that NuSI would fund studies on topics of interest and let the researchers take it from there. In other words, they would not be involved in the conduct, analysis, or reporting of the studies. Without this assurance, Hall wouldn’t have agreed to work with NuSI in the first place. However, after the EBC reported the results of its first study, which were unfavorable to the carbohydrate-insulin hypothesis, NuSI reneged on this agreement and began inserting itself into the research process in an increasingly assertive manner. The concept of adversarial collaboration emerged after these unfavorable findings and corresponded with NuSI’s desire to have greater control.

The second reason I think adversarial collaboration wasn’t a good idea in this case is that it was too dangerous due to the personalities, incentives, and strong beliefs of certain people on the NuSI side. This was a liability I flagged in 2012, which was apparent in the promotional materials NuSI published, and I think my concerns about this were realized. A key reason why I decided to support NuSI initially, despite this reservation, is that I was assured (as were Hall and his collaborators) it wouldn’t have control over the research process.

As I discussed in my initial article, NuSI did ask Energy Balance Consortium researchers to withdraw their abstract from the American Diabetes Association meeting, but this was because the researchers didn’t provide it to NuSI two weeks in advance, which their contract stipulated. While Attia argues that such agreements are common for this type of funded research, it still seems a little odd to me that NuSI actually enforced this rule and prevented the data (which challenge the carbohydrate-insulin hypothesis) from appearing at a major scientific meeting. Hall told me he wasn’t aware of this clause when he submitted the abstract. This incident would be less concerning if it weren’t part of a larger pattern of interference.

Conclusions

Although it still seems clear that NuSI interfered with the research process and alienated EBC researchers, I now believe the story of NuSI’s “collapse” is more complex than it initially appeared. I apologize to anyone who was harmed by inaccuracies in my initial article, particularly the salary issue. I’ll be annotating my original post to indicate the parts that I now think are wrong or misleading.