Major Publications plan on Raising the Standards in Biomedical Research to Improve Reliability and Reproducibility

In 2005 Stanford School of Medicine’s Dr.  John P.A. Ioannidis wrote in his PloS Medicine article “Why Most Published Research Findings Are False”, “There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims.”  Unreliable research and irreproducible data have been the status quo of scientific research since inception as Scientific American guest blogger Jared Horvath says in his blog post “The Replication Myth: Shedding Light on One of Science’s Dirty Little Secrets”, alluding to iconic scientists like Galileo, Dalton, Milikan, etc. But as time passes more attention is focused on the flaws in publication systems.  The problem starts in the laboratory but more rigorous journal scrutiny can help halt insufficient publications.  Recently, editors and publishers of more than thirty major journals, including Science and Nature, proposed new standards to improve reproducibility, focusing on biomedical research. 

In research, reproducibility is the gold standard but it’s unattainable because, as Ioannidis put it, “a major problem is that it is impossible to know with 100% certainty what the truth is in any research question.”  However, the unity of publishers is promising and the proposed new standards can improve the quality of the studies.  Rigorous statistical analysis, transparency in method and data reporting, consideration of refutation, and guidelines for image based data and more detailed biological material description can help the interpretation and reproducibility of published data. 

                Statistical analysis and accuracy in published articles have in part been faulty due to lacking statistical literacy in scientific fields.  Ioannidis criticizes the notion of single studies assessment by formal statistical significance, usually p-value less than 0.05, in medical research articles contributing to non-replicable data.   P-value help researcher determine the significance of their results in hypothesis-based testing, and when it is less than 0.05 the null hypothesis is rejected, concluding the existence of a relationship or phenomena.   In Nature’s 2013 “Announcement: Reducing out irreproducibility” they brought attention to the low number of “biologists receiv[ing] adequate training in statistics and other quantitative aspects of their subject.”  Therefore, their new standards will encourage researchers to closely examine statistics and transparency by providing checklists and by consulting statisticians on certain papers at editors’ discretion. 

                The checklist will also encourage authors to disclose technical information and experiment description in full.  For an animal experiment, for example, authors will be prompted to report source, species, strain, sex, etc.  Recently, I was reading up on the relationship of cholesterol level and statin (lipid-lowering agent) and Alzheimer disease for my biochemistry class, and the correlation is unclear because of inconsistent data; some studies found beneficial effects while others found no correlation at all.  The issue, according to a Neurological Review’s “Cholesterol Level and Statin Use in Alzheimer Disease” article, is that different studies used statins with non-identical brain-blood barrier permeability, analyzed the effects of different types of cholesterol, and studied patients with different Alzheimer disease severity.  Such inconsistencies due to vague specifications would lead to irreproducibility, as it has.

                Improving reproducibility of original papers is crucial to the scientific community because, according to Nature, in academia the pressures to publish and access funds “provide little incentive to pursue studies and publish results that contradict or confirm previous papers”.  In other words, scientists aren’t crosschecking other scientists work because it’s rarely “welcome[d] from journals and funders, even as money and effort are wasted on false assumptions.”  Raising the standards of publication is one step in ensuring public trust in science.

Sources:

http://blogs.scientificamerican.com/guest-blog/2013/12/04/the-replication-myth-shedding-light-on-one-of-sciences-dirty-little-secrets/

http://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852

http://www.nature.com/news/journals-unite-for-reproducibility-1.16259

Cholesterol Level and Statin Use in Alzheimer Disease. II. Review of Human Trials and Recommendations. Neurological Review. Nina E. Shepardson, MS; Ganesh M. Shankar, MD, PhD; Dennis J. Selkoe, M. 2011

Why Most Published Research FindingsAre False.  Plos Medicine.  John P. A. loannidis. 2005