The dark side of evidence-based policy

Speaking at a Lib Dem conference in the Coalition’s earlier days Linda Jack, a Lib Dem activist, called for a commitment to “evidence-based policy”.  She was interrupted by strong applause from the floor.  Ms Jack is a feisty activist, but is not known for thought leadership.  Her use of the idea, and the applause she got for it, shows that evidence-based policy has become a mainstream idea in some liberal circles.  Not long ago it was a rather abstruse, fringe idea pushed by academics who wanted the extra public funding that it would require.  Why is it now hurrah phrase used by political activists?  Is this a good thing?

But first, what is it?  No many people just think that it means that any public policy idea should first be based on some kind of evidence that it works, rather than just sounding like a good idea.  But, to those that take the time and trouble to advocate it, it in fact refers to a particular type of evidence: statistical studies comparing the effects of the policy in action against some kind of control group.  It takes its inspiration from medicine, and, indeed, some of its strongest advocates, like the writer and journalist Ben Goldacre, and Lib Dem former MP Evan Harris, are medical doctors.

Evidence-based therapies are all the rage in modern medicine.  Statistical evidence techniques have long been used in drug trials, but their use is widening to other areas.  It forms the core of policy advice put forward by Britain’s National Institute for Health and Clinical Excellence (known as NICE).  What is interesting about these evidence-based therapies is their pragmatic element.  Treatments are recommended because they are shown to have benefits, even if the explanation is unclear.  I am taking medication to reduce ocular pressure, because ocular pressure seems to increase the risk of glaucoma.  My consultant told me that why this was so was not understood – the treatment was prescribed purely on the basis of the evidence.  Although the technique is often described as scientific, this pragmatism takes it away from classic scientific method – not in the rigour of the testing, but in the lack of a theoretical model to drive the hypothesis.  No matter; a lot of useful therapies are being put into use, and some ineffective ones are being weeded out.

So it is quite natural for people to want to use the technique for non-medical areas.  An early example of this was the testing of conditional cash transfer programmes in South America in a bid to raise levels of schooling and reduce poverty.  A programme would be devised, and participating villages would be compared to ones outside, preferably with a random assignment between the groups.  These studies helped make the case for these programmes, which are now a standard part of the anti-poverty tool set, and are credited with particular success in Brazil (the Bolsa Familia programme).  A lively academic debate has been provoked as to how useful the technique is.

What are the problems?  Most of the debate that I have read about focuses on two issues: the rather limited nature of the questions that you are able to gather evidence on, and the huge difficulties of gathering untainted evidence, especially if it is not possible to do large scale randomised trials, which it usually isn’t.  It is disappointing that wider public debate is so limited, though, and evidence-based policy has simply become a warm, apple pie idea, without people asking searching questions as to what it is and what its limitations might be.  There is a dark side to it.

This dark side is in fact evident in the medical model.  Dr Goldacre has made his name in using evidence-based ideas to expose charlatan claims for fringe treatments that often get uncritical publicity in the press.  This is good, but he*, and especially his disciples, swiftly move on to attacking alternative therapies in general.  Homeopathy is a favourite target, since its use of extremely diluted solutions defy scientific common sense.

This is an interesting case.  I don’t know much about homeopathy, but from what little I do know it places great reliance on three ideas: that you should look at the whole person; that mind and belief are a critical element of therapy; and that every person is an individual.  These are three blind spots in statistical evidence techniques.  They can only be used to test very simple propositions, so it is necessary to break down the whole person into a limited number of measurable symptoms.  It is impossible to distinguish mind and belief effects from the so-called “placebo effect”; the placebo effect often works, but is excluded and ruled out of order by the evidence advocates since it is so difficult to test.  And statistical evidence techniques depend entirely on using general rules, and do not attempt to find treatments that will work for everybody.  So homeopathy is untestable using evidence-based techniques.  That is a problem (how do you spot the charlatans?) but it does not make it rubbish.  In fact what the evidence advocates are trying to do is to impose a particular belief system on what should and should not be included in health therapies.  There is a world of propositions that are testable by statistical techniques, and a world that is beyond their scope.  Both are big and important.

And what about evidence-based policy?  The idea is bandied around very loosely by political activists, and most have very little understanding of the full implications of the technique or its limitations.  Why are they so keen, then?  At first I thought it came from the habit of politicians (including, and especially, the “non-political” sort) of using loose statistical associations to support their advocacy – to try and give themselves more credibility.  This happens: I see much nonsense around the wisdom or otherwise of the government’s policy of converting schools into academies free of local authority supervision.  But the cover was truly blown for me when I saw Dr Harris at a fringe meeting at the recent Lib Dem party conference.  This was on the government’s “Free Schools” policy.  Because of the difficulties of gathering evidence to test any policy proposal, he could knock any proposal down at will on the basis of lack of evidence.  It is a powerful weapon with which to defend the status quo (which, of course, you do not need to test…).

Evidence-based policy, in the limited sense that its advocates use, is no doubt a useful tool, but of quite limited value in practice.  We need to broaden our idea of what constitutes proper evidence, and develop an understanding of where good old-fashioned human judgement and instinct is more appropriate, given its speed, responsiveness and ability to handle both complexity and individual variations.

* Dr Goldacre is very careful in his use of words.  His actual attack on alternative therapies may not be as direct as I am implying.  He is, rightly, more interested in challenging false claims about evidence than in challenging therapies that make no such claims, but where conventional evidence is lacking.  This not true of many of his fellow travellers – I have read much mockery of homeopathy online.

Cardinal O’Brien: is this all that’s left of Christianity?

After a wonderful day out in the sun, I return home to a Twitter feed bulging with reaction to Cardinal Keith O’Brien’s Easter sermon in Edinburgh.  It was near top of the BBC news this morning, but it was surprising difficult to locate their coverage on the BBC website (here) this evening.  Archbishop Rowan Williams’s sermon (something about happiness) got much more coverage.  From this coverage I find it rather unclear what Cardinal O’Brien actually said, beyond an attack on “aggressive secularism”, and calling for all Christian denominations to unite against it in defence of traditional Christian values.  This was enough to get Evan Harris wound up and twittering.

I do support state secularism, but my deeper reaction to Cardinal O’Brien’s sentiments are about what it means to Christians.  There seems to be a large body of people for whom a (the?) fundamental purpose of Christianity is to defend its traditional values.  I have heard people on the radio suggesting that the Church got these values right at the start, and that to change them to suit modern fashion is a betrayal.  There is a massive reaction against accepting gays – quite disproportionate to the significance of the issue.  Likewise, many react badly to the idea of women priests and bishops, to the extent of switching from Anglican to Catholic denomination.

I find this very strange because it seems so at odds with the teachings of Jesus in the gospels, to say nothing of how the church has evolved since.  The central message to me of the Sermon on the Mount was that Christians should not be deceived by the letter of the law, but always go to the underlying purpose, and to do so with humility and love.  This is a dynamic message, allowing practices to be continuously re-interpreted.  One of the examples in the sermon was the observance of the Sabbath – where Jesus said that it was nonsense to be totally strict.  And so the church did adapt, notably by extending membership to Gentiles. And this adaptation has been dynamic.  Take its attitude to women.  It is quite clear that in the very early days, the time of Paul, women had a leadership role in the church.  But as Christianity became closer to the establishment this was less acceptable, and so doctrine changed (including some rather dubious epistles making it into the Bible, supposedly from Paul himself).  More recently, women have been returning to prominence, although the Catholic hierarchy are still determined to hold their line in the sand.

Defence of tradition actually undermines the core Christian message.  It is a doctrine to hide behind rather than face up to the challenges that real faith should lead you to.  It is a message of despair.  Is that really all that is left of this once great faith?  At least Archbishop Williams is trying to use his Easter pulpit to promote a message of hope.