The Headlines vs. The Reality
A new report from the (fictional) Institute for Meta-Research has just dropped, and it’s a big one. Researchers used automated text analysis to scan 70,000 preprints—early, non-peer-reviewed versions of scientific studies—published over the last five
years on servers like bioRxiv and medRxiv. Their top-line claims are startling: that the strength of evidence in conclusions is weakening over time, and that certain fast-moving fields show signs of significant contradiction between initial findings and later work. The media has seized on this, with headlines suggesting a crisis of reliability in science. These claims are certainly attention-grabbing, and the scale of the analysis is impressive. It promises a bird's-eye view of how science is being done in the 21st century. But the story behind the data is more nuanced than the boldest claims suggest, and it all comes down to the nature of a preprint.
The Crucial Context: What's a Preprint?
To understand the 70,000-preprint analysis, you first need to understand what a preprint is—and what it isn't. Think of a preprint as a scientist's first draft, shared publicly before it has undergone formal peer review. For decades, researchers in fields like physics have used preprint servers to share findings quickly, get feedback from colleagues, and establish when a discovery was made. The COVID-19 pandemic supercharged their use in biology and medicine, as scientists raced to share data on the virus. The key benefits are speed and openness. Instead of waiting months for a journal to publish their work, researchers can get it out in days. However, there is a critical tradeoff. Preprints have not been vetted by independent experts in the same field. While they undergo basic screening to weed out non-scientific or dangerous content, they haven't been subjected to the rigorous critique that peer review provides. They are, by definition, a work in progress.
Pumping the Brakes on Strong Claims
This is why the findings of the 70,000-preprint analysis require careful handling. If the study suggests a decline in the reliability of preprint conclusions, it might not be pointing to a decline in science itself. Instead, it might simply be reflecting the raw, unpolished, and sometimes messy reality of the scientific process. Scientists use preprints to float early ideas and preliminary data. Some of those ideas will pan out; many will not. Findings often change significantly after peer review. One real-world study found that while most life-science preprints are similar to their final published versions, about 17% of COVID-19 preprints had major changes to their conclusions after peer review. Therefore, an analysis that only looks at preprints is analyzing a collection of drafts, not finished products. Any claims about a 'crisis' based solely on this data are premature. The public and journalists sometimes misinterpret these early findings as established fact, which can erode trust when the science evolves.
How to Be a Smarter Science Reader
So, how should you approach news about a study like this, or any study based on preprints? First, always check the source. Does the news report make it clear that the findings are from preprints and are preliminary? Reputable science journalism will always include this caveat. Second, treat preprints as a direction of travel, not a final destination. They show what scientists are currently working on and thinking about, but the conclusions are subject to change. Third, look for the response from the scientific community. Are other experts commenting on the work, pointing out flaws or offering alternative interpretations? Public commenting is a feature of many preprint servers, though it's not as thorough as formal review. Ultimately, the rise of preprints is a positive development for the speed and openness of science. But for those of us outside the lab, it means we need to become more sophisticated consumers of scientific information, embracing curiosity while maintaining a healthy dose of skepticism.
