How decisions on data and analysis have biased the referendum debate

oFZ3nTdrBy Jim and Margaret Cuthbert

Decisions on what data to collect, and how it should be analysed, are not on the face of it the most exciting topics in the world: and yet, some of these decisions have seriously biased the referendum debate. Here we explain how, and suggest what should be done about it.

This note develops on a paper we published in the last issue of Significance, a journal of the Royal Statistical Society. In that paper we gave two examples.

One example was the annual report on public expenditure and revenues by the Scottish government (GERS). Until very recently, all that has been produced by way of official figures on flows into and out of Scotland has been the GERS analysis of flows associated with government – there has not been a proper analysis of the flows associated with trade and capital movements. And yet this broader picture really matters: the lack of it has had a profound effect on channeling the independence debate into concentrating on Scotland’s government deficit/surplus. If a fuller set of figures had been available, then the debate would naturally have turned to consider other areas, which have been almost entirely neglected – like, what has Scotland’s current account balance been with the rest of the world: what has been the non-governmental financial outflow, (and it would, in most years, have been a substantial outflow): what steps would be open to a Scottish government to maximise the benefit from such an outflow – and so on.

Our second example was the Office for Budget Responsibility’s approach to economic forecasting. The OBR has adopted the detailed Treasury macroeconomic model as one of the main tools it uses in producing its forecasts. However, they make the key assumption that, by around the end of the 5 year forecast period, inflation will be stable, and the economy will be operating on its trend line of potential output. In a critical respect, therefore, the OBR assumes the success of economic policy. As a result, OBR significantly understate the risks attaching to the UK economy. This matters for the referendum debate, because a whole swathe of analysts/commentators, (like the Institute for Fiscal Studies, the National Institute of Economic and Social Research, and our own dear Treasury), who have produced forecasts for Scotland in relation to the referendum, simply took the OBR’s UK forecast as a given, disaggregated to Scotland, and then added in a variety of risks for Scotland. This approach had the effect of almost completely ignoring UK risk, hence fundamentally biasing the debate.

Those who want more detail on these examples can refer to our Significance article, (or a pre-peer reviewed version on under Theme 6). Here, we will be looking in more detail than in Significance at how things went wrong and what can be done.

It is actually quite rare, while not unknown, for governments to lie with statistics: they don’t need to. Instead, they have powerful tools which they use to set the statistical agenda in their favour. The above examples illustrate two of these tools.

The obvious one is control of resources. In the case of GERS, Ian Lang knew exactly what he was doing when he allocated resources to the production of the first GERS in 1992. As he wrote to John Major, “I judge that it is just what is needed at present in our campaign to maintain the initiative and undermine the other parties. This initiative could score against all of them.” And likewise, successive unionist governments knew exactly the effects when they failed to resource extension of GERS to a proper set of accounts for Scotland.

The OBR example illustrates another mechanism of control. OBR is independent: but its terms of reference were set by George Osborne. So as soon as OBR accepted what is basically a remit to produce forecasts, a satisfactory outcome from George Osborne’s viewpoint was virtually guaranteed. This is because a rational forecaster in a policy influenced environment will usually assume the success of policy: if it is obvious to the independent forecaster that the process is currently heading for, say, an undershoot – then this will be equally obvious to the controlling agent. So the forecaster has to assume that the agent will take corrective action.

In addition, there are other powerful factors which ensure that the agenda for statistics and analysis usually implicitly favours central government and the general status quo.

One factor is the power of patronage: you get the Governor of the Bank of England whom you appoint, and he/she will then be looking for re-appointment in the first instance, and a suitable honour towards the end of their “successful” term. And there is also the powerful influence of what one might call Establishment bias: appointees may be strictly neutral in Party political terms, but, having emerged from Establishment grooming, are very unlikely to set in train the production of analyses which might raise awkward questions about the stability and justice of the status quo.

There is also the sheer effect of centralisation. GERS provides another very good illustration of this. In its early years, the production of GERS relied upon data already refined and packaged by central departments, namely, the Office for National Statistics and the Treasury. Government secrecy meant that the detail of the analysis, and in particular, the basic data set, was not released, either to individual government departments or to the public. This meant that GERS was effectively beyond scrutiny: and a number of major errors went undetected. The situation only changed when a freedom of information request, (in fact, by us), secured general release of the relevant data set in 2005.

Given that the control of resources, and of the power of patronage, still rests overwhelmingly with London, it is not surprising that the analysis agenda has favoured the unionist side in the independence debate. Unfortunately, however, the situation has been even worse than it needed to be. While the SNP has been in power in Scotland since 2007, and could have taken significant steps during that time to redress the information imbalance, it was actually very slow to learn this important lesson of power. For example, despite repeated requests, it was only in November 2013 that the Scottish government produced initial estimates of Gross National Income. It was only at this stage that we had provisional estimates fleshing out GERS into a fuller set of international accounts for Scotland: too late, too late to rebase the agenda of the debate.

Another example is that the Scottish government could have used its control of land registration to open up information on who owns Scotland: and similarly, could have published much more data on who receives payments under the Common Agricultural Policy. Better information in these areas could have focused an intense debate on the need for reform of land tenure.

So the SNP needs to learn important lessons of power. But it is actually a counsel of imperfection just to argue that one side of the independence debate should descend to the level of the other. In a better world, much more would be done generally to loosen the grip of government on the statistical analysis agenda. For one thing, government should lose much of its power to control analytical resources. It should not be the government of the day, but Parliament, which sets the main priorities, and the budget, for data collection and analysis. And it should not be the Chancellor of the Exchequer who appoints the Governor of the Bank of England, or sets the terms of reference for the OBR. But in addition, the government’s overall powers of patronage should be much reduced: and a good starting point would be the complete abolition of the honours system.







Categories: Commentary

5 replies

  1. Important article. Lot’s of food for thought. Didn’t know about Ian Lang’s part in all of this – neither, I would guess, does the vast majority of people in Scotland. The SNP government at Holyrood could only do so much. Previous Labour governments at Holyrood, of course, didn’t give a damn about Westminster’s manipulation. We will only find out the true scale of Westminster duplicity after a Yes vote – there’s the rub.

  2. Many thanks for this article, let’s get all of it into some kind of brutal perspective.

    When Ian Lang wrote to John Major, “I judge that it is just what is needed at present in our campaign to maintain the initiative and undermine the other parties. This initiative could score against all of them.”

    This was nothing less than a constructive and deliberate attempt to misinform the Scottish people who would then directly suffer the consequences, in whatever form that would take.

    We talk of voter apathy and the ‘missing millions’ who are now being recovered by YES and it’s just so right that he tables have turned in such a way; social manipulation should be labelled a crime and politicians, the media and the press barons should be well and truly shafted for setting out to do so.

    We truly need a solid YES for democracy’s future and the burst up of this corrupt political reward system.

  3. I would concur with virtually everything the authors have to say. The selective bias that affects data collection and analysis operates not only at a national level but just as effectively much lower ‘down the line’ as my own research career indicates.

    I am now retired but at one point prior to becoming a University lecturer I was asked to give at very short notice (about ten minutes) to give a talk on my career to date to some undergraduate students in the final week of their studies. Unable to prepare any formal presentation I simply ran through my career in a conversational manner. Only then did it become clear to me how much the information available to the public had been influenced by factors that would not be disclosed.

    A few examples will make this clear, and also demonstrate that whilst the facts that are reported can be accurate, lying behind the facts there is a selection process that needs to be examined.

    This selective bias was most obiously made explicit when I carried out my first project in order to acquire an MSc degree. The funding for an investigation into accidents at work came from a trade union who stipulated that I could reach any conclusion I liked so long as they were not detrimental to the interests of the union. Prior to reaching this stage, needing access to various business to carry out the research I had been turned down a number of times either by management who feared the conclusion would not be in their interests or the workforce ditto. The most serious interference however led to my resigning rather than report false conclusions after a year long research project for a major UK company. The study was completed and duly presented with the false conclusions by my ex-colleagues who, I trust, had difficulty explaining my absence.

    The absence of research information also can have a major impact on policies. One three year project I was involved in received virtually no funding from official sources and was carried out by numerous iinterested parties to overturn conventional wisdom regarding an issue related to alcohol problems. No research body would fund the research, preferring to maintain the status quo. There was probably no intentional bias here; the project was probably not deemed important enough to justify the expenditure had it been fully funded.

    Another project was funded for a year by a Government Department but six months into the work my colleague and I were told we either had to change markedly what we were doing or lose our funding. Finally, a comment on research that was carried out, but not by my colleagues and I. Proposals had been presented to the Ministry of Defence for some research regarding the retention of personnel. The proposal was turned down. However, some time later, my colleagues and I found that virtually identical work was being carried out elsewhere. The similarity in detail to our proposals was such it seemed that our proposals had been passed on to others, either because we were deemed inappopriate or those who received the funding were known to whoever was responsible for funding the work.

    I present these examples to give confidence to anyone who wishes to challenge official statements and data.
    Behind any data or conclusion, however accurate, is always a process of selection. It is therefore appropriate to question the reasoning and rationale of whoever is presenting data or analysis.

    In an ideal world everyone would honestly disclose their affiliations and interests, as academics presenting their work on The Conversation do. This is rarely the case in main stream media. Readers should therefore seek as much information about the background of presenters using social media sites, etc. It is often very revealing.

    I cannot claim my presentation here is unbiased. Although unable to vote in the referendum I would like to see a YES vote in the belief, not only would it benefit citizens of Scotland, it would, indirectly benefit myself and my family, by providing a stimulus for changes needed in other parts of the UK.

  4. It is clear that the opinion polls do not touch the schemes. That is a fact. I have out and about in West Dunbartonshire and the response to the yes campaign in places like Faifley and Drumchapel has been absolutely awesome. Literally hundreds of Yes posters with flags hanging out of people’s windows. It has been an incredible sight to see. And these polls that seem to put No in the lead,? As Rab C Nesbitt would say ‘ a load of Sh***! They are based in middle class areas in the suburbs and the borders. And they fool no-one.

  5. valuable article! I use these type of articles when putting my point to undecideds. This one however is particularly analytical.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: