Speaking at a Lib Dem conference in the Coalition’s earlier days Linda Jack, a Lib Dem activist, called for a commitment to “evidence-based policy”. She was interrupted by strong applause from the floor. Ms Jack is a feisty activist, but is not known for thought leadership. Her use of the idea, and the applause she got for it, shows that evidence-based policy has become a mainstream idea in some liberal circles. Not long ago it was a rather abstruse, fringe idea pushed by academics who wanted the extra public funding that it would require. Why is it now hurrah phrase used by political activists? Is this a good thing?
But first, what is it? No many people just think that it means that any public policy idea should first be based on some kind of evidence that it works, rather than just sounding like a good idea. But, to those that take the time and trouble to advocate it, it in fact refers to a particular type of evidence: statistical studies comparing the effects of the policy in action against some kind of control group. It takes its inspiration from medicine, and, indeed, some of its strongest advocates, like the writer and journalist Ben Goldacre, and Lib Dem former MP Evan Harris, are medical doctors.
Evidence-based therapies are all the rage in modern medicine. Statistical evidence techniques have long been used in drug trials, but their use is widening to other areas. It forms the core of policy advice put forward by Britain’s National Institute for Health and Clinical Excellence (known as NICE). What is interesting about these evidence-based therapies is their pragmatic element. Treatments are recommended because they are shown to have benefits, even if the explanation is unclear. I am taking medication to reduce ocular pressure, because ocular pressure seems to increase the risk of glaucoma. My consultant told me that why this was so was not understood – the treatment was prescribed purely on the basis of the evidence. Although the technique is often described as scientific, this pragmatism takes it away from classic scientific method – not in the rigour of the testing, but in the lack of a theoretical model to drive the hypothesis. No matter; a lot of useful therapies are being put into use, and some ineffective ones are being weeded out.
So it is quite natural for people to want to use the technique for non-medical areas. An early example of this was the testing of conditional cash transfer programmes in South America in a bid to raise levels of schooling and reduce poverty. A programme would be devised, and participating villages would be compared to ones outside, preferably with a random assignment between the groups. These studies helped make the case for these programmes, which are now a standard part of the anti-poverty tool set, and are credited with particular success in Brazil (the Bolsa Familia programme). A lively academic debate has been provoked as to how useful the technique is.
What are the problems? Most of the debate that I have read about focuses on two issues: the rather limited nature of the questions that you are able to gather evidence on, and the huge difficulties of gathering untainted evidence, especially if it is not possible to do large scale randomised trials, which it usually isn’t. It is disappointing that wider public debate is so limited, though, and evidence-based policy has simply become a warm, apple pie idea, without people asking searching questions as to what it is and what its limitations might be. There is a dark side to it.
This dark side is in fact evident in the medical model. Dr Goldacre has made his name in using evidence-based ideas to expose charlatan claims for fringe treatments that often get uncritical publicity in the press. This is good, but he*, and especially his disciples, swiftly move on to attacking alternative therapies in general. Homeopathy is a favourite target, since its use of extremely diluted solutions defy scientific common sense.
This is an interesting case. I don’t know much about homeopathy, but from what little I do know it places great reliance on three ideas: that you should look at the whole person; that mind and belief are a critical element of therapy; and that every person is an individual. These are three blind spots in statistical evidence techniques. They can only be used to test very simple propositions, so it is necessary to break down the whole person into a limited number of measurable symptoms. It is impossible to distinguish mind and belief effects from the so-called “placebo effect”; the placebo effect often works, but is excluded and ruled out of order by the evidence advocates since it is so difficult to test. And statistical evidence techniques depend entirely on using general rules, and do not attempt to find treatments that will work for everybody. So homeopathy is untestable using evidence-based techniques. That is a problem (how do you spot the charlatans?) but it does not make it rubbish. In fact what the evidence advocates are trying to do is to impose a particular belief system on what should and should not be included in health therapies. There is a world of propositions that are testable by statistical techniques, and a world that is beyond their scope. Both are big and important.
And what about evidence-based policy? The idea is bandied around very loosely by political activists, and most have very little understanding of the full implications of the technique or its limitations. Why are they so keen, then? At first I thought it came from the habit of politicians (including, and especially, the “non-political” sort) of using loose statistical associations to support their advocacy – to try and give themselves more credibility. This happens: I see much nonsense around the wisdom or otherwise of the government’s policy of converting schools into academies free of local authority supervision. But the cover was truly blown for me when I saw Dr Harris at a fringe meeting at the recent Lib Dem party conference. This was on the government’s “Free Schools” policy. Because of the difficulties of gathering evidence to test any policy proposal, he could knock any proposal down at will on the basis of lack of evidence. It is a powerful weapon with which to defend the status quo (which, of course, you do not need to test…).
Evidence-based policy, in the limited sense that its advocates use, is no doubt a useful tool, but of quite limited value in practice. We need to broaden our idea of what constitutes proper evidence, and develop an understanding of where good old-fashioned human judgement and instinct is more appropriate, given its speed, responsiveness and ability to handle both complexity and individual variations.
* Dr Goldacre is very careful in his use of words. His actual attack on alternative therapies may not be as direct as I am implying. He is, rightly, more interested in challenging false claims about evidence than in challenging therapies that make no such claims, but where conventional evidence is lacking. This not true of many of his fellow travellers – I have read much mockery of homeopathy online.