One of the basic premises of scientific research is that repeating the same experiment under the same conditions should give the same result. However, anyone who’s ever spent any meaningful time in the lab will recognize — with no small amount of frustration — that no matter how good a scientist you are, this isn’t always true of even your own experiments, let alone trying to recreate somebody else’s. This systemic inability to replicate experimental results, even when following published methods, is the reproducibility crisis in a nutshell.
The term “reproducibility crisis” first appeared between 2010 and 2012, after researchers at Amgen and Bayer published papers that rocked the scientific establishment when they demonstrated that their in-house R&D teams could only replicate the results of landmark preclinical studies in 11% of cases (Amgen) and 20–25% of cases (Bayer).1,2 This, then, was real-world confirmation of Ioannidis’ 2005 mathematical simulation, which identified “that for most study designs and settings, it is more likely for a research claim to be false than true.”3
Why Good Scientists Get Different Results
While the reproducibility crisis is a real problem, this is absolutely not a result of any moral or capability failure on the part of scientists. Scientists are not failing science. Instead, the systemic architecture is failing scientists. What does that mean? Well, let’s consider where issues with reproducibility come from.
Biological Noise
Biology is an inherently “noisy” science. Cell lines from different sources have “drifted” genetically in different ways, the number of times a cell line has been passaged can change how it responds to certain stimuli, different seeding densities, environmental controls, batches of antibodies — even brand of reagents — can all make a difference to the result of an experiment.
Pragmatism vs. Perfection
All scientists would like to believe that they’ve tested their hypotheses ad infinitum before publishing their results, but the reality is much messier. Time is money, after all, and academic grant funding or a biotech company’s runway only stretches so far and for so long. At some point, perfection becomes the enemy of progress, and research has to move forward to secure the next tranche of funding.
The Three-/Four-Year Staff Cycle
In academia in particular, a tenured laboratory might have been there for decades, but its membership will — for the most part — be a revolving door of Ph.D. students and postdocs. Each time somebody leaves the lab, it’s inevitable that some of their “scientific magic” goes with them. If they’re lucky, the Ph.D. student whose project relies on following up on this research will be left with a meticulously completed lab-book and a freezer full of perfectly organized and carefully labelled samples. I would put money on the fact that few “follow on” students or postdocs are so lucky! While this is more obviously problematic in academia, the same issues of staff turnover and loss of institutional knowledge affect biotechs too.
Publish or Perish
Publish or perish is the unofficial motto of academic career plans. Academic researchers experience intense pressure to continuously publish high impact publications to secure funding, tenure, and professional longevity. But scientific journals prioritize novelty over rigor; a meticulous replication of somebody else’s data is unlikely to be published. And of course, the same race to patent exists in industry too. Actively disincentivizing reproduction of previous findings only adds fuel to the fire of the reproducibility crisis.
Missing Metadata vs. FAIR Data Principles
While every peer-reviewed publication includes a materials and methods section, this is really just the bare bones of the protocol. The true story behind the result lies in the experimental metadata; when, how, and under exactly what conditions were these data recorded? If these metadata exist, they are often lost in the pages of a lab book or a scribbled calculation on the back of a glove. It’s not searchable, shareable, or structured.
In 2016, a consortium of scientists and organizations introduced the concept of FAIR data principles to combat the reproducibility crisis.4 This is a set of guidelines designed to ensure that all scientific research is findable, accessible, interoperable, and reproducible. Demonstrating a plan to ensure that your data will meet this criteria is now a mandatory requirement for research funding from many major UK organizations, including The Wellcome Trust, The Medical Research Council, and Cancer Research UK.
However, despite the best intentions, without the “missing metadata” that rarely form part of a research publication, it’s nearly impossible for another scientist to faithfully recreate an experiment without costly and labor-intensive optimization to find the “secret formula” to success.
Could Lab Management Software be the Solution?
“Digital transformation” has been a biotech buzzword for years now, and almost all companies use some level of electronic lab notebook (ELN) to record data and timestamp results, or a LIMS (laboratory information management system) to track their samples and inventory. Adoption of first-generation lab management software is far less common in academia, but even within fully digitized biotechs with multiple software subscriptions and a clear eye on 21 CFR Part 11 compliance, experimental metadata can slip through the cracks of these siloed systems.
Perhaps the solution could lie in a unified digital thread; a single continuous datastream that links experimental design, at-the-bench actions, recorded results, and physical sample tracking?
This is the concept behind Lab Thread, a single lab management software solution designed to connect a laboratory’s entire workflow from DNA construct design, through in-lab dilution calculations and buffer recipes, to recording results and data analysis in the ELN, and then tracking the location of the final physical sample with the built-in LIMS. By connecting every step of the workflow (and combining scientific functionality with full project management orchestration), it becomes much harder for those all-important experimental metadata to slip through the cracks.
Once all these metadata are recorded, it’s only a small step further to take advantage of advances in AI or other analytical tools to conduct “meta-analysis at scale”. If five scientists do the same experiment, and it works three times out of the five; it’s no longer a question of whether this data are really valid, but rather just identifying that, in the two failed replications, the incubator temperature had dipped by two degrees.
Digital transformation doesn’t have to be expensive. It doesn’t have to involve robotics and whole system overhauls. Instead, it can mean quietly implementing the right lab management software to ensure that every discovery you make is based on a foundation of traceable, verifiable, and reproducible data.
References
1. Begley, C, and L Ellis. “Raise standards for preclinical cancer research.” Nature. 483: 531–533 (2012).
2. Prinz, F, T Schlange, and K Asadullah. “Believe it or not: how much can we rely on published data on potential drug targets?” Nat. Rev. Drug Discov. 10: 712 (2011).
3. Ioannidis, JPA. “Why most published research findings are false.” PLOS Medicine. 2: e124 (2005).
4. Wilkinson, M, M Dumontier, I Aalbersberg, et al. “The FAIR Guiding Principles for scientific data management and stewardship.” Sci. Data. 3: 160018 (2016).
