The information gap – the big issue of our age

I felt a brief pang of sympathy for former Prime Minister David Cameron last week, as he endured grandstanding “questions” from MPs over his lobbying on behalf of Greensill Capital. He probably really did think that what he was suggesting on that company’s behalf would be helpful to the government. But nobody cares about that; they just wanted the masochistic pleasure of giving him a beating. Such is politics. There is a deeper issue, though.

That issue is what I call the “information gap”. Governments do not have the information they need to use their resources efficiently on our behalf. We don’t get the right sort of help when we need it; lots of people get help they do not need. The government’s aid to businesses to help survive the covid lockdown was a case in point. Many needy businesses were unable to comply with the conditions and received no help. But many fraudulent companies were also set up to make claims, costing the state many millions, so it is thought. Greensill’s business was to lend money to businesses against outstanding invoices. Could this have helped close this information gap? Probably not, but it surely merited a closer look. It is not clear that the Treasury ever gave it that look. Mr Cameron does not seem to have been given any clear reason why they considered his suggestion unsuitable. There turned out to be good reasons to turn Greensill down, but Mr Cameron wasn’t wrong to put the question.

The information gap is nothing new. It arises whenever a large, centrally-managed organisation has to interact with the world beyond the direct experience of the core group of 100 or so people that run it. We might call it “Empire Syndrome”. The classic way to deal with it is for the elite to create an arbitrary set of rules for outlying officials to follow, and simply to shrug when this leads to bad outcomes – call this the “Imperial Method”. Conscientious officials try to optimise these rules to minimise bad outcomes and get some sort of positive return. Other officials don’t care – they just derive pleasure from the exercise of arbitrary authority and the status that this confers. The alternative solution is to allow local officials broad discretion to exercise their judgement based on the achievement of broad objectives. This brings problems of its own. Discretion is a opportunity for corruption; also it allows power centres to build up and undermine the authority of the central elite.

The imperial method is deeply ingrained into our psyche. Most people assume this is the way things should be done. You can see this from how annoyed people are getting with the government’s regulations on foreign travel. Nobody is asking whether it is wrong to expect anybody to offer advice that is going to be applicable in all circumstances. As a school governor there was an expectation that you would create files of detailed policies approve them, even to run a primary school of 200 pupils. When Chair of Governors I learned not to spend too much time on these, although developing a policy can be a good way of thinking through problems in advance. As an operations manager for a department of over 100 people I remember being advised by a lawyer that management-by-policy was what you had to do: you create rules and people follow them, and that way you could manage risks of legal liability. I tried to do this but soon gave up. The rules never could keep up with the world as you encountered it, and they soon became so complicated that nobody could master them, especially if you kept updating them. In the face of the real world you had to be more flexible – discretion and honest two-way communication between the levels of management were the key.

Interestingly, one part of our state services complex has actually thought about this problem and developed a solution: the military. The Prussian army and then Germans were the pioneers in the 19th Century. They developed a system of discretion and honest communication to govern their operations. In many ways this operated against their authoritarian culture, but the system improved over time. By the time of the Second World War it had developed into a highly effective system which meant that man for man the German Army was most effective in the world, alas for humanity, until Hitler undermined it toward the end. It has been widely copied since, notably by the British and US armed forces. Which is one of the reasons why these institutions are now held in such high regard by the public. But the system requires immense discipline and a very strong culture of working towards common objectives.

Back in the 1990s it became fashionable for large businesses to develop analogous systems to run their businesses. This included reducing the number of layers of management and empowering junior and middle managers. It never caught on in the public sector, and by the 2000s central authority and management pyramids became the norm again everywhere. I’m not quite sure why this was, but I suspect that senior officials feared that they would be blamed for bad decisions made in junior ranks, as the public and politicians are only too happy to do.

New technology may also have played a role. Developers of information systems offered their own automated way of closing the information gap, by baking the rules into control systems. This started with workflow technology, which can be used to empower workers by making more information available to them, but instead became used for tying them down, including the measurement of toilet breaks. The process of “de-skilling” began. This had a dire effect on the public sector, as de-skilling became the basis of many an outsourcing contract, causing catastrophic loss of effectiveness. But this was but the start of a journey; the technologists started to talk about the Holy Grail – the use of Artificial Intelligence to intermediate between organisations and the public, so that you wouldn’t need people at all. AI is powerful technology, with many important uses, but it cannot possibly as effective as a skilled professional in helping people solve complex personal problems. Many, if not most, AI applications are pre-destined to fail.

Unfortunately too many people believe that technology can solve the information gap by itself. The Chinese Communist Party seems to have embarked on this road. Providing the data that the state needs to make its systems work means a steady erosion of privacy; ensuring its consistency means standardisation; maintaining confidence in the system means the suppression of honest communications. And so on. Ultimately the Imperial Method means that people have to conform to the needs of the state, rather than the other way around. This is truly the road to serfdom.

But the answers are not too difficult to find. Free markets will find appropriate solutions for most of our needs if we are alert to excessive market power by large businesses. State services need to be mediated by empowered professionals held accountable by people who understand the challenges they face. Countries like Britain, with a strong public service culture, can do this. There are examples of excellent public service – the two primary schools where I had the privilege to be governor are undoubtedly among them. Technology can empower other than enslave.

Too few people appreciate this in government, or amongst those holding them to account. But as public service systems fail, perhaps understanding will grow. There is now a growing consensus that the Test and Trace system for coronavirus was approached in completely the wrong way, and should have been localised from the start. If more aid to businesses had been intermediated at local level then perhaps more businesses could have survived and more frauds stopped.

Organisations like Greensill were never going to be the answer. But the more important question is whether the Treasury ever really cared about the problem.