WHAT'S INSIDE

Reading and Doing Neuroscience Research: Problem, Solution and an Insight From Augustina Otero

John Hartung, PhD
Associate Editor of the Journal of Neurosurgical Anesthesiology

John Hartung, PhD

After thirty years as the Associate Editor of the Journal of Neurosurgical Anesthesiology, I have come to agree with John Ioannidis’ conclusion that “False findings may be the majority or even the vast majority of published research claims ... research findings may often be simply accurate measures of the prevailing bias” (PMID: 16060722).  Within anesthesiology, approximately 75% of published research is inadequately substantiated (PMID: 4003779 & 12357186) and investigations by pharmaceutical companies have found the same rate of inadequately substantiated findings across other medical specialties. As put by Hiltzik, “Science has lost its way, at a big cost to humanity” (http://articles.latimes.com/2013/oct/27/business/la-fi-hiltzik-20131027).

Unlike Hiltzik, I am not convinced that science had greater integrity in the past. Current assessments may be the result of increased detection. Either way, at every meeting of journal editors that I have attended, we complain about the publication of inadequately substantiated findings and discuss ways to make improvements, but the bottom line remains that journal editors are not able to do the job that many readers assume they are able to do.

One purpose of this lecture will be to highlight the effect of bias on research, primarily by examining the history of publications on induced hypothermia after head injury. Another objective will be to outline a procedure that could reverse the ratio of good/bad science from 25/75 to 75/25 by forcing passionate scientists to become dispassionate scientists - a procedure that the NIH and National Library of Medicine (NLM) could arrange, perhaps with the cooperation of Thompson Reuters’ Institute for Scientific Information (ISI).

The way things work today, as you know, authors can submit a revision of a rejected manuscript to a different journal in a manner which implies that it is the first submission of a new manuscript. This gives authors several opportunities to finesse methodological issues that were raised by reviewers of prior versions, eliminate secondary results that invited questions about primary outcomes, develop new primary outcomes, and even change results.  None of that would be possible if every manuscript was forwarded to every journal through a central clearing house (NLM?) - a clearing house that would also forward every manuscript’s tracked history - including all prior versions, all reviews of all prior versions, and all prior editorial correspondence.

That could be accomplished if the clearing house retained a digital copy of every manuscript and used modified plagiarism software to compare each manuscript to every other manuscript that has been submitted through the clearing house – to determine whether a manuscript is what its authors purport it to be, i.e., a first submission of a new manuscript, a revision of a manuscript being sent to the same journal, a revision of a manuscript being sent to a different journal, or an attempt to submit a revision of a rejected manuscript to a different journal without inclusion of its history.

The key here would be encouraging journals to require authors to submit manuscripts through the clearing house, and agree to having all editorial correspondence and all reviews sent back to authors through the clearing house. As with the adoption of ClinicalTrials.gov, major journals would readily agree (BMJ, NEJM, etc.), which would increase pressure on second-tier journals, and so on down the line. A couple of years after initiation, an incentive could be added regarding the speed of indexing at PubMed, with articles published in clearing house journals receiving priority. After a substantive majority of journals joined the clearing house procedure, ISI could notify journals that bibliometrics like Impact Factors would only be calculated for clearing house journals. Eventually, NIH/NSF funding could require authors to submit manuscripts that report funded results exclusively to journals that require submission through the clearing house.  Ultimately, journals that operate outside of the clearing house would not be indexed in PubMed.

The immediate effect on investigators would be to cause them to invest more effort and thought into research design, because covering up design inadequacies in ‘doctored’ revisions would become nearly impossible. The effect on publishers would be shrinkage of their journal portfolios, because authors’ practice of starting near the top and working their way down the Impact Factor scale, such that some version of almost every manuscript eventually gets published, would be greatly curtailed. Indeed, instead of 5,500-plus indexed medical journals, we might end up with fewer than 2,000. This would be the big payoff for science, not because fewer journals would merely lead to more efficient literature searches, but because the literature searched would contain far less misinformation.

Science progresses at an impressive rate despite the adage “Odds Are It’s Wrong” (https://www.sciencenews.org/article/odds-are-its-wrong).  Imagine the rate of progress that would follow from reversing the ratio of good/bad science from 25/75 to 75/25!

For the connection to Augustina Otero, see “So Much Promise, So Little Power” (JNA 2001, pp 1-2) click here.