How do we educate students about how real discovery works?
I was musing over this article, party because it is a longstanding interest of mine—how do we acquire useful new knowledge— but also in the context of this blog on medical education, how do we get across to students how medical advance has occurred. Without getting into the ‘what can we learn from history’ subroutine, I think the topic important, and one that we cannot assume students will learn to think deeply about, without some guidance or prompting. To my mind, the role of education here is the classic one: as a detergent to propaganda.
The editorial describes changes at the mental health part of NIH (NIMH) in which the new director has made clear that to be funded, clinical trials have to include some test of the underlying biological mechanism. The line is that too many trials are black-box tests, in which if the results are negative, nothing is learned. ( It is suggested that 50% of the studies currently funded, would not be funded if submitted as new grant proposals at some future time). I think they are targeting the sort of pragmatic NHS style RCT which I find so depressing. The reason is simple: without a construct, or a genuine scientific hypothesis (and I do not mean a statistical one), we have no idea whether the conclusions of any study will apply at any future time, or in any other population. And a cognate fact is that we know trials are noisy and often unreliable guides to what is going on. Fisher warned of this nearly a century ago: we tend to try and rely on statistics when we know little about what is really going on, when what we need is more thinking, and much more repetition.
As it is, we often capture very little of the routine clinical encounter in many clinical trials. They are guides to what is going on, not rules to tell us how to behave. If the effects are very large, we can perhaps ignore much of this. But so often, we only conduct large studies, because the effect sizes are so small. If we think about it in simple terms, the R2 values are far too low: most of the variance is random, and unexplained. They are not good experiments and, it is no surprise, that we are finding out that so many papers published are wrong. Most RCT do not present the information this way, because it would be apparent we know little about what will happen to our patients. Not always, however. Systemic retinoids for acne have, I suspect, an NNT of close to 1. But even there, we had a clear demonstration of efficacy, before we had much insight into mechanism —but researchers went searching to complete the circle—not to the next RCT in a different domain.
However, what the editorial dismisses, is what I would most want to get across to students. The article (discussing psychiatric drugs) states that:
By the early 1990s, the pharmaceutical industry had discovered — mostly through luck — a handful of drug classes that today account for most mental-health prescriptions.
This is a real travesty, and again supports my adage that Nature doesn’t really understand medical research or medicine. Many of the leads were not luck, but the results of astute clinicians / pharmacologists interested in what happened to their patients. Not so much those immersed in the use of rating scales, or obsessed by assuming anything interesting must be due to chance, but acute observers that provided insights worth following by pharma. Calling this luck, is like saying Charles Darwin was just lucky (although I would accept Wallace was deeply unlucky). This for me is just another representation of the master clinician, one whose expertise is based around a knowledge of patients, and who thinks about what happens to them. This is a style of medicine we are in danger of losing. Students should know that the obsession of thinking about patients is what underpins and drives clinical advance. This is not at the expense of sensible clinical experiments, or wet-bench work, just an acknowledgement that medicine has its own intellectual heartlands, and we need to communicate this to the next generation, because it is in danger of being killed off by a pincer movement of ‘protocols’ on the one hand, and a confusion that biology and medicine are synonymous, on the other. As far as discovery in psychiatry is concerned, few can equal David Healy for explaining how we got where we did. For some other areas, see what I wrote in Science over 10 years ago. The problem for the undergraduate teacher is how to integrate real knowledge of statistics and experimental design, with a knowledge of how genuine clinical advance occurs.