… much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. …
The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. …
Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.
Strong statements such as these about the parlous state of scientific research and publication are not uncommon. What is uncommon in this case is that these comments were made by Richard Horton. Richard who, you ask? It’s Dr Richard Horton, Editor-in-chief of The Lancet, who calmly lobbed in an incendiary ‘Comment’ piece, ‘Offline: What is medicine’s 5 sigma?‘
First, what’s the meaning of this seemingly innocuous title? It refers to acceptable significance in particle physics research. As Horton explains, “In particle physics, significance is set at 5 sigma—a p value of 3 × 10–7 or 1 in 3·5 million (if the result is not true, this is the probability that the data would have been as extreme as they are).” Clearly this makes the significance levels set in typical spheres of science (and humanities, of course), usually p<0.05 (that’s 1 in 20), pretty laughable.
What prompted him to write this piece? It was his attendance at a gathering of leaders in the field. The “symposium—on the reproducibility and reliability of biomedical research, held at the Wellcome Trust in London last week—touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.”
Interestingly and significantly, given the sensitivity of the topic, the symposium was held under the Chatham House Rule. And its conclusions?
The conclusion of the symposium was that something must be done. Indeed, all seemed to agree that it was within our power to do that something. But as to precisely what to do or how to do it, there were no firm answers.
Disappointing yes, but surprising no, as is apparent if you read the full text of the piece. That’s because each part of the large and complex system of scientific research publication require serious change, and each is replete with entrenched inertia. Universities, government agencies, publishers, and the commercial scientific community (I include the hugely powerful multinational pharmaceutical companies here) all need significant change.
You might be interested in how the Academy of Medical Sciences reported the symposium. And if you want to see how others are reacting to Horton’s piece, a typical one is found in the Phoenix Rising discussion board under the title of ‘Lancet editorial: “poor methods get results”, lack of action from research community‘. It gets a bit bogged down in the p value issue, but provides a sense of the frustrations in the medical research community.
There are of course worthy individuals working at identifying and fixing critical weaknesses in the system. All credit to them, as they must at times find it particularly frustrating to see how slowly they make inroads (if any!). Those of which I am aware include Dr Ben Goldacre, whose Bad Science website contains a wealth of his ever-accumulating evidence of “the misuse of science and statistics by journalists, politicians, quacks, drug companies, and more.” He’s thankfully becoming quite influential, judging by his ‘About‘ page (as far as I know, he wasn’t at the symposium, rather unfortunately). And of course he’s done a TED talk, so here it is so you can see him ‘in action’.
Who else? Well, Dr David Healy has done a good job on the pharmaceutical companies with Pharmageddon. You can also watch an excellent talk he gave recently at the Yale symposium “New Data and New Hopes Call for New Practices in Clinical Psychiatry“. There’s also Prof. Peter Gøtzsche from the Cochrane Centre, who was in Melbourne for a talk a few months ago which I attended. He’s the author of the subtly titled Deadly medicines and organised crime: How big pharma has corrupted health care (2013), and later this year will release Deadly psychiatry and organised denial (to appear in September 2015).
I’ll stop now. Let’s hope that some of the ‘firm answers’ that Richard Horton craves are found soon.