Discussion about this post

User's avatar
Phillip Muza's avatar

This is a very interesting article, thank you! But I believe it might be a little bit too reductionist. Sure, we can assume that random mutations introduced into a generation of mice might affect the mice phenotypically - but we know that even without the mutations, biological systems are inherently stochastic.

The replication crisis is a by-product or even a consequence of working with biological systems, and believing that all genetically identical animals should behave phenotypically the same is wrong. I'm not sure how much of a contribution of random mutations would have on the phenotypes of these animals - but you are right, they should probably be sequenced more often. Which is what major mouse houses do anyway. For smaller labs, re-running experiments is incredibly expensive - both financially and temporally... if only they had more money then maybe the reproducibility crisis wouldn't be so bad!

Expand full comment
halvorz's avatar

"Scientists are unable to reproduce experiments performed by their peers about 70 percent of the time. This is true across fields, from domains as disparate as cancer biology and psychology."

That is not what the cited article says, at all. The only mention of 70 percent in the article is here: "More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments."

That is to say, the 70% refers to a fraction of researchers, not a fraction of experiments/papers.

They do give numbers for what fraction of papers could be reproduced elsewhere in the article, but *only* for cancer biology and psychology, and only from two studies -that is to say it cannot be extrapolated across fields. The numbers they give are 10% for cancer biology and 40% for psychology, which would translate to 90% of cancer papers and 60% of psychology papers being irreproducible, respectively. If you average that out you get 75%, which is at least close to 70%, but, again, this number cannot be extrapolated across fields, and as the article states, there is no real consensus on what the true number is for any given field, let alone across research as a whole:

"The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland."At the current time there is no consensus on what reproducibility is or should be.” But just recognizing that is a step forward, he says. “The next step may be identifying what is the problem and to get a consensus.”"

Additionally, the cancer number is deeply flawed for multiple reasons. I find it highly unlikely that cancer research is *less* reproducible than psychology research, as we have a much, much firmer empirical and theoretical basis for cancer research than we do for psychology, and in practice cancer research has led to many highly successful medical therapies, in contrast to psychology.

More specifically, the 10% reproducibility number does not come from a systematic survey of the cancer literature, unlike the psychology study. Rather, the researchers chose a small number of papers "...that described something completely new, such as fresh approaches to targeting cancers or alternative clinical uses for existing therapeutics." That is to say, they chose the most surprising, groundbreaking, exciting papers that could lead directly to new cancer treatments. This will inevitably select for less reproducible papers.

It's definitely worth improving the reproducibility of scientific research but publishing extremely low and unsubstantiated reproducibility estimates around is both unhelpful to that specific goal and will likely lead some readers to completely distrust scientific research, even in fields with a strong track record -for example, vaccinology.

Expand full comment
2 more comments...

No posts