The dark side of evidence-based policy

Speaking at a Lib Dem conference in the Coalition’s earlier days Linda Jack, a Lib Dem activist, called for a commitment to “evidence-based policy”.  She was interrupted by strong applause from the floor.  Ms Jack is a feisty activist, but is not known for thought leadership.  Her use of the idea, and the applause she got for it, shows that evidence-based policy has become a mainstream idea in some liberal circles.  Not long ago it was a rather abstruse, fringe idea pushed by academics who wanted the extra public funding that it would require.  Why is it now hurrah phrase used by political activists?  Is this a good thing?

But first, what is it?  No many people just think that it means that any public policy idea should first be based on some kind of evidence that it works, rather than just sounding like a good idea.  But, to those that take the time and trouble to advocate it, it in fact refers to a particular type of evidence: statistical studies comparing the effects of the policy in action against some kind of control group.  It takes its inspiration from medicine, and, indeed, some of its strongest advocates, like the writer and journalist Ben Goldacre, and Lib Dem former MP Evan Harris, are medical doctors.

Evidence-based therapies are all the rage in modern medicine.  Statistical evidence techniques have long been used in drug trials, but their use is widening to other areas.  It forms the core of policy advice put forward by Britain’s National Institute for Health and Clinical Excellence (known as NICE).  What is interesting about these evidence-based therapies is their pragmatic element.  Treatments are recommended because they are shown to have benefits, even if the explanation is unclear.  I am taking medication to reduce ocular pressure, because ocular pressure seems to increase the risk of glaucoma.  My consultant told me that why this was so was not understood – the treatment was prescribed purely on the basis of the evidence.  Although the technique is often described as scientific, this pragmatism takes it away from classic scientific method – not in the rigour of the testing, but in the lack of a theoretical model to drive the hypothesis.  No matter; a lot of useful therapies are being put into use, and some ineffective ones are being weeded out.

So it is quite natural for people to want to use the technique for non-medical areas.  An early example of this was the testing of conditional cash transfer programmes in South America in a bid to raise levels of schooling and reduce poverty.  A programme would be devised, and participating villages would be compared to ones outside, preferably with a random assignment between the groups.  These studies helped make the case for these programmes, which are now a standard part of the anti-poverty tool set, and are credited with particular success in Brazil (the Bolsa Familia programme).  A lively academic debate has been provoked as to how useful the technique is.

What are the problems?  Most of the debate that I have read about focuses on two issues: the rather limited nature of the questions that you are able to gather evidence on, and the huge difficulties of gathering untainted evidence, especially if it is not possible to do large scale randomised trials, which it usually isn’t.  It is disappointing that wider public debate is so limited, though, and evidence-based policy has simply become a warm, apple pie idea, without people asking searching questions as to what it is and what its limitations might be.  There is a dark side to it.

This dark side is in fact evident in the medical model.  Dr Goldacre has made his name in using evidence-based ideas to expose charlatan claims for fringe treatments that often get uncritical publicity in the press.  This is good, but he*, and especially his disciples, swiftly move on to attacking alternative therapies in general.  Homeopathy is a favourite target, since its use of extremely diluted solutions defy scientific common sense.

This is an interesting case.  I don’t know much about homeopathy, but from what little I do know it places great reliance on three ideas: that you should look at the whole person; that mind and belief are a critical element of therapy; and that every person is an individual.  These are three blind spots in statistical evidence techniques.  They can only be used to test very simple propositions, so it is necessary to break down the whole person into a limited number of measurable symptoms.  It is impossible to distinguish mind and belief effects from the so-called “placebo effect”; the placebo effect often works, but is excluded and ruled out of order by the evidence advocates since it is so difficult to test.  And statistical evidence techniques depend entirely on using general rules, and do not attempt to find treatments that will work for everybody.  So homeopathy is untestable using evidence-based techniques.  That is a problem (how do you spot the charlatans?) but it does not make it rubbish.  In fact what the evidence advocates are trying to do is to impose a particular belief system on what should and should not be included in health therapies.  There is a world of propositions that are testable by statistical techniques, and a world that is beyond their scope.  Both are big and important.

And what about evidence-based policy?  The idea is bandied around very loosely by political activists, and most have very little understanding of the full implications of the technique or its limitations.  Why are they so keen, then?  At first I thought it came from the habit of politicians (including, and especially, the “non-political” sort) of using loose statistical associations to support their advocacy – to try and give themselves more credibility.  This happens: I see much nonsense around the wisdom or otherwise of the government’s policy of converting schools into academies free of local authority supervision.  But the cover was truly blown for me when I saw Dr Harris at a fringe meeting at the recent Lib Dem party conference.  This was on the government’s “Free Schools” policy.  Because of the difficulties of gathering evidence to test any policy proposal, he could knock any proposal down at will on the basis of lack of evidence.  It is a powerful weapon with which to defend the status quo (which, of course, you do not need to test…).

Evidence-based policy, in the limited sense that its advocates use, is no doubt a useful tool, but of quite limited value in practice.  We need to broaden our idea of what constitutes proper evidence, and develop an understanding of where good old-fashioned human judgement and instinct is more appropriate, given its speed, responsiveness and ability to handle both complexity and individual variations.

* Dr Goldacre is very careful in his use of words.  His actual attack on alternative therapies may not be as direct as I am implying.  He is, rightly, more interested in challenging false claims about evidence than in challenging therapies that make no such claims, but where conventional evidence is lacking.  This not true of many of his fellow travellers – I have read much mockery of homeopathy online.

10 thoughts on “The dark side of evidence-based policy”

  1. Your foray into homeopathy apologism contains a lot of straw men, but perhaps you meant it to. Anyway, that’s by the by – I can’t speak for anyone else, but when *I* talk about evidence-based policy I am using the word evidence in the legal sense. That is, strong statistical evidence is valuable, empirical evidence of what one person has seen is still valuable but slightly less so, what some bloke told you down the pub is still evidence, but it’s hearsay and not worth very much, and your idealogical assumptions are worth bugger all.

    Evidence is not an on/off switch, it’s a sliding scale.

    1. Thanks Jenny. I agree entirely that we need a more nuanced understanding about the quality of evidence. There are too many people out there who like to see things in black and white. When this is combined with what I can only call a quasi-religious attitude – “sciencism” – a belief that the whole of life can be summed up in a series of neat, verifiable laws and chemical reactions, we get what I call the dark side. It’s not just a question of the quality of the evidence – the degree of corroboration, etc – but whether it addresses the question being asked. That’s my point about homeopathy, about which I actually know very little (though about as much as most of its critics…). I don’t actually know whether it is an exercise in cynical charlatanism, apart from trusting some friends who are sympathetic to it – equivalent to your man at the pub. What I do know is that the criticism I have read clearly misses the point, especially when they bang on about evidence – they are looking at the wrong sort of evidence. They object to it because it conflicts with their belief system, not because they have evidence that it doesn’t work.

  2. There is a major flaw in ‘evidence-based policy’ that you don’t really touch on, which is that it is based on technocratic assumptions.

    Of course, one should not support propositions that fly in the face of the evidence, such as the idea that the MMR jab causes autism. But ultimately, most political decisions are moral choices.

    If that were not the case, there would be no need for democratic politics. All decisions could be left to an expert consensus – a technocracy.

    We are on dangerous ground when anyone making a clear moral choice can automatically be dismissed for making ‘value judgements’. We are on even more dangerous ground when someone making value judgements pretends that their view is dispassionate, evidence-based policy.

    1. Thank you Simon. That is a very good point. It puts me in mind of the last government’s policies, especially on the NHS. The Primary Care Trusts were technocrats sent out to make our lives better for us with negligible democratic exchange. I’m sure they were very keen on evidence-based policies. I wish I could believe that this government’s reforms are going to improve matters.

  3. Interesting read – I am a big fan of Goldacre’s work on randomised control trials – have you seen the Cabinet Office Paper he and a few others put together earlier this year?

    Regarding homeopathy, I don’t agree that it is immune to statistical analysis. What I gather is that in tests it performs no better than a placebo – i.e. a tablet (which is in reality just glucose) that a doctor might tell you will have a certain effect.

    Placebo effects can be powerful – I once read an interesting study looking at the effect of a placebo as its perceived strength increased. Placebos that patients administered via injection outperformed tablets taken twice daily, which in turn outperformed tablets taken once daily.

    1. I haven’t seem Ben Goldacre’s Cabinet Office paper. I am a bit nervous that too much weight can be placed on this type of technique – but it is surely very useful. I don’t think they are a huge amount of use when applied to institutional or management structures, for example. I’m not sure if Dr Goldacre does – but he is a vocal opponent of the NHS reforms, and one line of opposition is that they are not “evidence-based”. The government has made some rather spurious claims about evidential support for them, though. Randomised trials are unlikely to tell you much that is useful in such a complex area with so many variables. Other forms of evidence may be more appropriate.

      On homeopathy the power of the placebo effect is an interesting issue. The fact that there is a placebo effect does show that it may have potential – but it gets so tangled up with beliefs. Therapies like this always push at the boundaries of such testing techniques, and will be regarded as cheating by conventional practictioners. There may be broader statistical tests that can be used to get closer to how it works – but will lack precision.

  4. The cabinet paper is a great primer (http://www.cabinetoffice.gov.uk/resource-library/test-learn-adapt-developing-public-policy-randomised-controlled-trials).

    Though I agree, not all policy can be tested in that manner – but there are certainly some instances where it can be applied to get the best outcome for both recipients of government services and the gov’s coffers.

    On homeopathy, the fact that it does have a placebo effect doesn’t mean it has any potential. If, as a conventional doctor, told a large group of cancer patient that the glucose tablets I was giving them were the latest advance in medicine and have a sky-high effect rate, they’d likely outperform another group of patients I gave no such ‘miracle’ medication. Would the difference mean much? Not really – they difference would show the power of the human mind, and how it can respond to suggestion. Not sure how it can be said homeopathy may “push at the boundaries of such testing techniques” – rather it shows that there’s a belief that homeopathic treatments work that gives an effect comparable (at the most) to a placebo.

    A pretty good summary of why this is not a great basis for medicine can be found on an article written earlier this year (http://www.guardian.co.uk/science/blog/2012/apr/03/homeopathy-why-i-changed-my-mind.

    1. Thank you Daniel. I have flicked through the cabinet paper, and it looks very clear. But I can’t find any analysis of the types of problems that randomised trials would help with and those it wouldn’t. It implies that it is particularly useful for incremental changes to static procedures (changes in standard letter wording, etc) – and that makes a lot of sense. But the politcal heat around evidence and policy is always about much bigger changes, like GP led commissioning or school academy status. It is my contention that randomised trials are not very helpful for this time of policy (even if they produced a clear result, they wouldn’t provide enough information on why) – though other forms of evidence may be available.

      As for homeopathy I think you would have a point if it was setting itself up as an alternative version of conventional medicine, whereby they are about the design of potions which work to cure diseases in an of themselves (i.e. the chemicals in the potion are doing the work). The article’s author seems to think so – and he has much more experience of homeopathy than I do. My understanding, based on a few conversations with friends, is very different. They say that the potion is not of itself a particulalry important part of the treatment – what is important is the relationship between the patient and the practitioner. Randomised trials go to a great deal of trouble to exclude such relationship effects – they are “cheating”. That’s what I mean about pushing the boundaries. Perhaps there is good homeopathy and bad homeopathy, the latter pretending to work like conventional medicine.

      I would refer you back to my original post about three ways in which the boundaries get pushed. Whole person (i.e. looking beyond the immediate symptoms), mind over matter, and tailoring to the individual. All three break the rules of randomised trials.

Comments are closed.