I like statistics and spent most of my intercalated degree ‘using’ medical stats (essentially, writing programs on an IBM 360 mainframe to handle a large dataset, that I could then interrogate using the GLIM package from the NAG). Yes, the days of batch processing and punchcards. I found — and still find — statistics remarkably hard.
I am always very wary of people who say they understand statistics. Let me rephrase that. I am very suspicious of non-professional statisticians who claim that they find statistics intuitive. I remember that it was said that even the great Paul Erdos got the Monty Hall problem wrong.
The following is from a recent article in Nature:
What will retiring statistical significance look like? We hope that methods sections and data tabulation will be more detailed and nuanced. Authors will emphasize their estimates and the uncertainty in them — for example, by explicitly discussing the lower and upper limits of their intervals. They will not rely on significance tests. When P values are reported, they will be given with sensible precision (for example, P = 0.021 or P = 0.13) — without adornments such as stars or letters to denote statistical significance and not as binary inequalities (P < 0.05 or P > 0.05). Decisions to interpret or to publish results will not be based on statistical thresholds. People will spend less time with statistical software, and more time thinking.
There is lots of blame to go around here. Bad teaching and bad supervision, are easy targets (too easy). I think there are (at least) three more fundamental problems.
- Mistaking a ‘statistical hypothesis’ for a scientific hypothesis, and falling into the trap of believing that statistical testing can operate as some sort of truth machine. This is the intellectual equivalent of imagining we can create a perpetual motion machine, or thinking of statistics as a branch of magic . The big offenders in medicine are those who like adding up other people’s ‘P’ values — the EBM merchants, keen to sell their NNT futures.
- The sociology of modern science and modern scientific careers. The Mertonian norms have been smashed. It is one of the counterintuitive aspects of science that whatever its precise domain of interest — from astronomy to botany — its success lies less with a set of formal rules than a set of institutional and social norms. Our hubris is to have imagined that whilst we cling to the fact that our faith in science relies on the ‘external test in reality’, we ignored how easy it is for the scientific enterprise to be subverted.
- This is really a component of the previous point (2). Although communication of results to others — with the goal of allowing them to build on your work — is key, the insolence of modern science policy has turned the ‘endgame’ of science into this communication measured as some ‘unit’ based on impact factor or ‘glossy’ journal brand. But there is more to it than this. The complexity of modern science often means that the those who produce the results of an experiment or observation are not in a position to build upon them. The publication is the end-unit of activity. So, some bench assay or result on animals might lead others to try and extend the work into the clinic. Or one trial might be repeated by others with little hard thought about what exactly any difference means.Contrast this with the foundational work performed by Brenner, Crick and others. Experiments were designed to test competing hypotheses, and were often short in duration — one or maybe two iterations might be performed in a day. Inaccuracy or mistakes were felt by the same investigator, with the goal being the creation of a large infrastructure of robust knowledge. Avoiding mistakes and being certain of your conclusions would allow you not to (subsequently) waste your own time. If you and your family are going to live in a house, you are careful where you lay the foundations. If you plan to build something, and then sell to make a fast buck, the incentives lie in a different place. Economists may be wrong about a lot of things — and should be silent on much more — but they are right about two important things: institutions and incentives matter. Period.
Science has been thought of as a form of ‘reliable knowledge’. This form of words always sounded almost too modest to me, especially when you think how powerful science has been shown to be. But in medicine we are increasingly aware that much modern science is not a basis for honest action at all. Blake’s words were to the effect that ‘every honest man is a prophet’. I once miswrote this in an article I wrote as ‘every honest man is for profit’. Many an error….