Discussion about this post

User's avatar
halvorz's avatar

"Scientists are unable to reproduce experiments performed by their peers about 70 percent of the time. This is true across fields, from domains as disparate as cancer biology and psychology."

That is not what the cited article says, at all. The only mention of 70 percent in the article is here: "More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments."

That is to say, the 70% refers to a fraction of researchers, not a fraction of experiments/papers.

They do give numbers for what fraction of papers could be reproduced elsewhere in the article, but *only* for cancer biology and psychology, and only from two studies -that is to say it cannot be extrapolated across fields. The numbers they give are 10% for cancer biology and 40% for psychology, which would translate to 90% of cancer papers and 60% of psychology papers being irreproducible, respectively. If you average that out you get 75%, which is at least close to 70%, but, again, this number cannot be extrapolated across fields, and as the article states, there is no real consensus on what the true number is for any given field, let alone across research as a whole:

"The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland."At the current time there is no consensus on what reproducibility is or should be.” But just recognizing that is a step forward, he says. “The next step may be identifying what is the problem and to get a consensus.”"

Additionally, the cancer number is deeply flawed for multiple reasons. I find it highly unlikely that cancer research is *less* reproducible than psychology research, as we have a much, much firmer empirical and theoretical basis for cancer research than we do for psychology, and in practice cancer research has led to many highly successful medical therapies, in contrast to psychology.

More specifically, the 10% reproducibility number does not come from a systematic survey of the cancer literature, unlike the psychology study. Rather, the researchers chose a small number of papers "...that described something completely new, such as fresh approaches to targeting cancers or alternative clinical uses for existing therapeutics." That is to say, they chose the most surprising, groundbreaking, exciting papers that could lead directly to new cancer treatments. This will inevitably select for less reproducible papers.

It's definitely worth improving the reproducibility of scientific research but publishing extremely low and unsubstantiated reproducibility estimates around is both unhelpful to that specific goal and will likely lead some readers to completely distrust scientific research, even in fields with a strong track record -for example, vaccinology.

Expand full comment
AG's avatar
May 16Edited

I wonder if you could put something like this in grant applications to get funding for reproductive tech, similar to how anti-aging scientists are sneaking in longevity research through pet longevity.

The pitch would be that you could unlock both standardization and customization for mouse studies, if instead of breeding lines you could create colonies from immortalized cell lines. Under this project, you could justify research into things like gene editing, oogenesis, and artificial wombs.

Expand full comment
3 more comments...

No posts