91Èȱ¬

As Britain remains cold and snowy, an interesting little dispute has boiled up between the Global Warming Policy Foundation (GWPF) and the Met Office over the quality of longer-range weather forecasting.

And this is illuminated by documents obtained by the 91Èȱ¬ under freedom of information from the Met Office. These shed new light on the problems faced by the Met Office in its public communications and the strategies it has adopted for tackling them.

The Met Office is from the GWPF, for its "poor advice" on the likelihood of a harsh and cold winter.

The GWPF is to a map published on the Met Office website in October which indicated that the UK was likely to experience above-normal temperatures in the ensuing three-month period.

For the GPWF, which is sceptical of the Met Office and other mainstream analysis of global warming, this is evidence of a to under-predict cold weather and over-predict mild winters.

The Met Office replies that , which feature in the scientific research section of its website, are probabilistic estimates of the chances of a range of outcomes and as simple weather forecasts that can be right or wrong. It to squash news stories in October that it was predicting a mild winter.

It should also be noted that, according to the on transport and winter resilience earlier this week, the Met Office did give "early indications of the onset of a cold spell from late November at the end of October".

This argument is linked to , but part of the background is the major difficulty the Met Office has faced for some time over forecasting seasonal weather and conveying its views to the public.

It goes back to the well-publicised (and in due course much ridiculed) of "a barbecue summer" for summer 2009, which turned out to be true if you use your barbecue for collecting rainwater. It became one of the wettest summers in the past century. The widespread derision that resulted left the Met Office feeling badly burnt (while the nation's sausages were not).

The documents we requested show that scientists within the Met Office were uneasy about the language of this prediction. One internal report states:

"The strapline 'odds on for a barbeque summer' was created by the operations and communications teams to reflect the probability of a good summer. Concern over the use of the strapline and its relationship to the scientific information available was expressed by the scientific community, who were not consulted prior to the media release."

The Met Office then resolved to use "more conservative terminology" in future. But its seasonal prediction for last winter was also awry, failing to signal sufficiently the long and severe cold spell.

An internal executive paper noted the impact as follows:

"Unfortunately, less 'intelligent' (and potentially hostile) sections of the press, competitors and politicos have been able to maintain a sustained attack on the Met Office ... The opprobrium is leaking across to areas where we have much higher skill such as in short range forecasting and climate change - our brand is coming under pressure and there is some evidence we are losing the respect of the public."

This report argued that one downside of the seasonal forecasts was that they remained on the website and could easily be later compared to reality. It said:

"One of the weaknesses of the presentation of seasonal forecasts is that they were issued with much media involvement and then remain, unchanged, on our website for extended lengths of time - making us a hostage to fortune if the public perception is that the forecast is wrong for a long time before it is updated."

In contrast it noted that the "medium range forecast (out to 15 days ahead) is updated daily on the website which means that no single forecast is ever seen as 'wrong' because long before the weather happens, the forecast has been updated many times."

The intense embarrassment over the seasonal calculations prompted the Met Office to rethink its approach to predictions for several months ahead. It stopped publishing a seasonal forecast for the UK for public consumption (although it added a rolling 30-day view to its ). Instead it decided to put probabilistic seasonal data on the scientific pages of its website where, in the words of a board paper, such figures can be "more targeted towards users who appreciate their value and limitations".

As another document put it, "'Intelligent' customers (such as the Cabinet Office) find probabilistic forecasts helpful in planning their resource deployment."

A communications plan in February 2010 instructed staff that "interested customers" should be told the three-month outlook will be available on the research pages of the website but that "this message should not be used with our mainstream audiences".

Met Office staff clearly feel the general British public find it difficult to cope with probabilistic statements. A board paper from September 2009 states: "Feedback from Met Office surveys suggests that users would rather receive a deterministic forecast."

It adds: "It is considered that the task of educating the UK public in interpreting probabilistic information will be neither a short-term, nor simple task." It compares this unfavourably with the apparently greater ability of the US public to grasp such material.

Its location and temperate climate mean that predicting the British weather is a tricky task - especially when your audience is members of the general British public, who don't like probabilities and who may not be the most "intelligent customers" or able to cope with understanding the uncertainty of the longer-range predictions. Thus the problem for the Met Office is not only the variability of the British weather, it's also coping with the intellectual capacity of the British public. That, anyway, seems to be the view within its staff according to these FOI disclosures.

Are they right? Your answers can either be expressed deterministically or probabilistically, according to your own taste.

• Bookmark with:
• |
• |
• - What's this?

I think we're at an intriguing stage in the development of freedom of information in the UK. A year ago the context surrounding FOI was dominated by the role it had played in the MPs' expenses saga and the repercussions. Now that we're moving on from that particular while important matter, we can discern some interesting general themes - so here are some of my end-of-year reflections.

Data, data, data. That's what we've been getting - and that's what we're going to get more of.

This year has seen a rapid increase in the flow of government data being released to the public. It during the previous Labour government and has accelerated under the coalition, particularly so far with financial data. And there's .

For the coalition this is clearly partly driven by the notion that transparency cuts waste and saves money (a point which the Information Commissioner Chris Graham is also keen to make when he gets the chance). But there's more to it than that - partly due to the work of some devoted campaigners, the zeitgeist is now for opening up government data and giving outsiders the chance to work with it.

For me, the change in attitudes is encapsulated in one small example. You can now see the location of London Underground trains in real-time . The private individuals who created this map exploit . In contrast a few years ago London Underground refused to send me a District Line timetable.

As a journalist, one of the consequences I reflect on most often is naturally how the data deluge will affect journalism. One of my colleagues, Mark Easton, wrote about some of the difficulties when large quantities of spending data were released last month.

Data journalism is still very much in its infancy. But it's bound to develop and become increasingly important. And I think we will see journalists competing for what I call "scoops of analysis" - getting the story first in data journalism will mean doing the calculations and finding the insights in the data which no one else has yet spotted and analysed.

The phrase "scoops of analysis" was actually used, as I recall, previously in the 1980s and 1990s ("popularised" would be rather too strong) by the former 91Èȱ¬ Director General John Birt, although at the time of my writing this Googling it surprisingly produces no results. The Birtist doctrine was that journalism should have a "mission to explain", and he liked the analytical approach. This view was far from universally popular within 91Èȱ¬ News, where many staff preferred to think of journalism as story-getting.

But the context is now very different. I envisage that from now on breaking news (and that means story-getting) in the field of data journalism will increasingly be based on scoops of analysis.

More generally than the issue of data release, the impact of the freedom of information system is both about policy and delivery.

We discovered this year that the prime minister who introduced FOI thinks himself a "nincompoop" for so doing. Tony Blair now worries about its effect on frank exchange of information within government. This is much talked about anecdotally, although an important academic study issued this year did not find FOI had such a chilling effect.

Tony Blair with his former chief of staff Jonathan Powell

I was particularly intrigued by the mixed messages in the discussion of this topic in The New Machiavelli, the (incidentally very entertaining) book published this year by Jonathan Powell, Mr Blair's former chief of staff.

He says FOI has resulted in a "reduction in the amount of confidential work conducted on paper", but he also writes "I have often been told that I should simply stop sending emails, but you cannot do that if you are running a place like Number 10." There is a practical conflict between caution and effectiveness, a tension which will doubtless remain.

But whatever the merits of the policy itself, for its first few years delivery has been a problem. Until recently, if you'd asked me for one word to characterise the UK's freedom of information system, I would have answered "delay, delay, delay." Now I might merely say "delay".

In his first 18 months in office the Information Commissioner Chris Graham has managed to speed up the rate at which his office assesses complaints. One measure of this is that, , his office now has no complaint under consideration for over two years. The fact that this statement counts as good news indicates how bad the situation had become. [180KB PDF] there were over 30 such cases.

This is a crucially important change to how FOI works in practice. My impression anecdotally is that the ICO's progress in tackling the backlog it was struggling with is having a knock-on effect on the efficiency of the rest of the FOI system - it has made public authorities somewhat less complacent about their own delays. And Mr Graham now clearly feels more confident [152KB PDF].

Ìý

I expect that, for many, the current controversy they most associate with the term freedom of information is that of Wikileaks and the US diplomatic cables - although, strictly speaking, as a leak it's an entirely separate matter from FOI. But one thing this does illustrate (as indeed did the MPs' expenses controversy) is that leaks can go much further than material the authorities officially if reluctantly decide to release in reply to FOI requests. Although the 91Èȱ¬ has obtained under FOI some rather interesting reports sent back home by British ambassadors, we have been denied other more sensitive ones.

Much has been written about Wikileaks and how much the public should know about diplomatic communications, but there is one point I think has been largely missing - and it connects to large releases of data both under FOI and of the government's own volition.

I've heard civil servants say that the public disclosure of spending data and its reporting in the media has actually given them much easier access to information about state expenditure, in some cases to material which they felt was useful to their job and previously hard to obtain.

Similarly, of the three million or so Americans who apparently had access to Siprnet, the network which contained the cables leaked to Wikileaks, how many I wonder had actually become aware that (to take for example the ) Arab leaders had been pressing the US to attack Iran?

Widespread revelations of state information may or may not be in the public interest; but they're not only to do with what the citizen knows about the state, they are also to do with what the state knows about itself (and yes, sometimes some of us in the 91Èȱ¬ learn about other parts of the 91Èȱ¬ due to the 91Èȱ¬'s own FOI disclosures).

Another current point of interest is the position of private companies in numerous different fields in relation to information access. The coalition government has extended FOI to academy schools in England (in legal terms they are private not state institutions). The Scottish Information Commissioner Kevin Dunion for FOI to be extended in Scotland to various private bodies such as private prisons and Glasgow Housing Association.

But a recent [140KB PDF] ruled that the private water companies of England and Wales did not have to abide by the Environmental Information Regulations. I suspect we may see further developments in this area.

And finally, we're waiting for the coalition government to announce its plans for developing FOI.

[476KB PDF] that it will extend the scope of the Freedom of Information Act but we have yet to see the detail of what it will do. However of ensuring the process is not "swamped" by "people who overload the system". They say they're not sure the balance is right at the moment.

We now know where Tony Blair felt the balance should lie on freedom of information. We may soon find out more about the thinking of the current government.

Ìý

• Bookmark with:
• |
• |
• - What's this?

If you want to know how the UK is run, the Cabinet Office has issued just the thing for you - its draft , which sets out the main laws and conventions affecting the conduct of government.

It is mainly intended as a guide for ministers and civil servants, but the government says it is publishing it generally as part of its agenda for promoting transparency about the workings of the state.

I confess that I haven't yet read the entire 149-page manual since it was released yesterday, but I have looked at the section on freedom of information at the end. And I was surprised to discover that a document of this status contains errors in what it says about the law.

For example, paragraph 389 states that information may be exempt from FOI if it "relates" to defence or international relations or numerous other matters it lists. In fact, for most of these areas, what the actually says is that the exemption can only apply if disclosing the information "would, or would be likely to, prejudice" defence, et cetera.

The same paragraph also asserts that another exemption is "if the cost of complying with the request would exceed £600". Strictly speaking this isn't an "exemption", but putting that more pedantic point to one side, the statement is still extremely misleading.

Under the , the relevant cost to take into account is not the overall cost of complying with the request; it is only the cost of determining whether the information is held, and locating, retrieving and extracting it (which may well be significantly less than the total cost of responding).

It does seem rather strange for a document of this importance to contain legal errors. However, the Cabinet Office . Since I do not know if the Cabinet Secretary Sir Gus O'Donnell is an avid reader of this blog, I have decided to perform my civic duty and will be e-mailing the address provided for sending in comments.

• Bookmark with:
• |
• |
• - What's this?