The peer review drugs don’t work

by reestheskin on 07/06/2015

Comments are disabled

I was going to ignore this, but I have just chanced upon some more comments about it, so writing therapy is required. The article with the above title was written by Richard Smith, and published in the THE a few weeks back. I  always worry if I agree with Richard Smith’ arguments, but on this topic he is largely correct:

Yet peer review persists because of vested interests. Absurdly, academic credit is measured by where people publish, holding back scientists from simply posting their studies online rather than publishing in journals. Publishers of science journals, both commercial and society, are making returns of up to 30 per cent and journals employ thousands of people. As John Maynard Keynes observed, it is impossible to convince somebody of the value of an innovation if his or her job depends on maintaining the status quo.

Scrapping peer review may sound radical, but actually by doing so we would be returning to the origins of science. Before journals existed, scientists gathered together, presented their studies and critiqued them. The web allows us to do that on a global scale.

But I think he gets a few things dangerously wrong.

First, journals have taken over ‘peer review’ forgetting that its original meaning was judgment by one’s peers. It was always silly to image that 2 -10 people judging a MS could provide robust scrutiny or recognition of excellence. Peer review is why we recognise Watson and Crick, or Wiesel and Hubel, a half century on. Journal editors use this intellectual sleight to make their organs more appear more powerful, when in point of fact many  journals work outside the real community of science.

Second, he states, ‘The most cited paper in Plos Medicine, which was written by Stanford University’s John Ioannidis, shows that most published research findings are false.’ This is imply untrue. Ioannidis’s study was limited to medicine and largely concerned papers that tried to define truth using ‘probability management’. One the reasons we are in this mess, is that we have confused what I would call academic A/B testing with science. The former is, at best, technology assessment, and however important it is for medicine , it is not science— or at least not science if we define science as the attempt to  produce broad and deep conceptual understanding of the natural world. You cannot do science without an attempt to produce theories about how the world works. That is why most RCT fail as science, because they are not obsessed with the structure of the world. They are frequently expensive forms of A/B testing.

Third, he states: ‘Cochrane reviews, which gather systematically all available evidence, are the highest form of scientific evidence.’ They are not the highest form of scientific evidence; and they do not systematically gather evidence (much of the evidence is in people’s heads). What they frequently are, are reasonable attempts to summarise domains of knowledge in which we lack critical insight, and have to rely on probability management (as if often the case in medicine). You may have to do this, but nothing in a Cochrane compares with say the theory of evolution, or the base structure of DNA, or how we define fitness, or laws of the conservation of energy etc etc. Probably the greatest series of scientific breakthrough over the last half century — the breakthroughs that allow the medium I write on to exist — owe nothing to Cochrane. The more you need Cochrane, the less you can be confident about what you are saying.

Rant over. I feel better already.