Accuracy of forecasts

[franksingleton]

I think they would only need to say "our confidence is high"or "low" ... or even "very poor" for which read non existent..

............
.

When it comes to the next few days you will sometimes hear/see the BBC weather people be quite positive for the next 5 or 6 days. At other times, they are far more cautious and will say something like “keep tuned in,” or “keep watching the forecasts.” Those statements are based on the model ensembles.

When it come to today or tomorrow they are usually far more certain although, of course, there are always problems with showers. These are pretty ransom events. The forecast may well be correct but anyone not having a shower may well complain. Of course, there are often timing problems, especially when the forecaster is covering a large area.

Those who really feel strongly about it should write/email the Met Office. Given enough weight of opinion they might change their collective mind. But, whatever they do, somebody will find a reason to jeer. I can hear it now, “can’t they make up their f***9ng minds?”

 

[franksingleton]

Interesting that once-time Met Office Senior Manager Frank S. seems now to be coming round to
..............
/QUOTE]

There is so much here that I do not have the time to go through and deal with point by point. I do seem to recollect an article in PBO about forecast accuracy which began with an example of a totally incorrect forecast. When I asked the author to give me dates and times, he could not. I see the same rather one-sided screed here. Do you know the name of the author?

I am long retired. But I have no reason to believe that the library and archives are not open to anyone. See http://www.metoffice.gov.uk/learning/library. If you cannot gain access, then try your MP rather than moan here.

I can assure you as a former insider that the Met Office was always and still is self critical see http://www.metoffice.gov.uk/about-us. You might not like their assessments but do you have any better objective suggestions? By that I mean other than journalistic hype and misrepresentation.

I am well aware of the Brettle paper. I am also all too well aware of the problems of making meaningful assessments of forecasts, in particular of wind. I was not happy with the letter that I quoted at http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Assessment, not least because the results seemed to good to be true. But, do you have a better way of assessing Inshore waters Forecasts?

 

[BruceDanforth]

I don't mind about the (in)accuracy just so long as there aren't any gales.

 

[TimBennet]

The Met Office doesn't have a culture of 'secrecy' per se, but rather one it shares with the Hydrographic Office and Ordinance Survey for example, of seeing all its data as having a commercial value.

This came as a result of them becoming semi self financing operations under the various political reforms that have been foisted on them over the past 25 years. You can't have the CG giving out information for free on demand if you're trying the sell the same information on a pay to listen phone line.

In the States, it's accepted that any data that was compiled using tax payers dollars should be in the public domain. The only charges that can be levied for it are to cover the costs of disseminating the information, not compiling it. So a paper chart will cost a nominal amount to cover the paper and printing, but the data is free, so citizens are welcome to photo copy it and use it for their own personal use.

On the issue of forecast accuracy as research or even when just listening to them, you need to consider them broadly of two types; depending on whether they were made by predicting the behaviour of low pressure systems or by predicting the behaviour of a high pressure. The accuracy is much higher for the first category, where 24 to 48hr predictions are consistently good. But be very wary of anything that tries to predict the behaviour of an anticyclone. Thirty six hours out and you of into the world of Hans Christian Anderson!

 

[simonjk]

Wow!

Loads to go on there ( I wish the board would notify me when someone comments on a watched thread, very strange! Apologies to all for not coming back earlier but it's been a rather busy weekend with hurricanes and gales!)

We probably firstly need to recognise that the Met Office is not the monolith it once was (whether by accident or design). It seems to have reigned itself in over recent years and seems to be a changed beast. This could be because their focus is more on research; much government funding has been in this direction of recent years, although I stress I have no evidence to back up this supposition. As sailors we probably shouldn't get quite so hung-up about them as their relevance is diminishing year on year, although that is not to detract from the excellence in certain areas of weather and climate prediction that they display.

Conversely the UK private weather sector seems to have taken on the role of the monolith. There are a few large, global companies who are now servicing a plethora of contracts and look very similar to the Met Office.

That leaves the smaller private companies (of which I run one) and crack-pots (of which I hope I m not!).

The respectable smaller companies would never make outlandish accuracy claims. It will always be that those companies simply state the customers that use their services already, the qualifications and/or experience of their staff and their longevity in the commercial market place.

Then there are those who emphasise the accuracy of there forecasts and the resolution of their models. No, there is nothing wrong with this per-se so long as the companies state to what parameter(s) their accuracy figures pertain and for which location. Such accuracy statements should be independently verifiable.

Likewise if a very high resolution model is claimed then the company should state which model is used as its starting point' and how they manage to get such fantastic resolution when the billions of dollars put into sate backed computer models over many years has not managed to achieve such astounding resolution. Again, and accuracy claims should be backed up with independently verifiable data.

Now, what about forecasting? Well, as you know I am bench forecaster at heart. Despite the qualifications I still describe myself as an 'amateur weather enthusiast'. My forecasting knowledge has been built over many years experience and I like to think I have a 'fel' for the weather (oh no, crackpot!).

But when I am forecasting, lets say for the Channel I have in my mind the average user of my service; that is usually a sailor, taking his yacht and family on a short passage, or perhaps even an overnighter to France or back to the UK.They don't want to be in uncomfortable seas, or strong winds.

I therefore take this into account when issuing a forecast. There is no way that I can predict the winds around every headland otherwise the forecast would be so long s to be unreadable.

What I can do though is to forecast for where most sailors might be; open water, close to the coast etc... I have that 'typical' sailor in mind and, yes if there is a borderline F5/F6 I'l edge into the F6 category.

Now, it's impossible to verify whether the forecast of an F6 was the correct decision, but based on my 'sailor' I would argue that it was the right call.

There is certainly nothing wrong with stating confidence levels, but what are we actually stating? Confidence in the accuracy of the model, the likelyhood of the speed actually occurring, or he competence of the skipper?

And we also need to be aware of perceptions. I gaurantee that if you have a rough passage with a F4 wind, you will summise it was more like a F6!

Okay, I'll leave it there and hope I have not waffled too much. This is suject very close to my heart, so I watch with interest!

Simon

 

[franksingleton]

In the good or bad old days, the Met Office was able to give just about everything away free. Hence the old weather centres that some may remember. They were a “Good idea,” but became untenable as more and more people wanted weather information for a myriad reasons. Just imagine the staff that would be needed to deal with telephone calls from 10% of Solent sailors asked for “latest weather, please.” I well remember one of my staff spending hours on a request from a primary school class for “Some information ....” All very laudable but I had to account how my staff were spending taxpayers’ money – our taxes.

Then came Mrs T and her guru Rayner. The eventual outcome was that they now work under Trading Fund rules. Don’t argue with me about that. Go and read what it means. I know, because I was involved in implementation. We did not all like it but we all had to admit that it was necessary in the interests of the taxpayer. OK. You may doubt that but I can make good arguments for both ways of working. Unfortunately, it was not for me to decide. It was HMG.

How relevant is the Met Office? Far more than Simon says. They are in the front rank for numerical weather prediction and a leader in meso-scale prediction. That feeds into the national good through warnings of severe weather in a way that nobody else could do without that information. There are many other functions that few of us see. Rightly, they leave much medium range forecast development to ECMWF although there are close ties and co-operation as there are with all other national weather services.

I have suggested before, but I do not think that Simon would want to do it, that he should offer to produce GMDSS forecasts under contract – perhaps for access to Met Office model output. Although I have a high regard for Simon and his approach, I think that he would do better job because of the constraints in meeting what the customer ie the MCA wants.

I think that there is a very real place for the kind of human service that Simon can provide. On several occasions on these forums, I have recommended him. Personally, for sailing use, I would not pay any of the commercial weather modellers referred to in the first post of this thread... First, their claims seem highly unlikely. Second they claim to run models in ways that the Met Office and ECMWF cannot and I have to wonder why. Bor would I pay for any totally automated service such as Météo Consult. It is worth paying for intelligent, experienced human input.

Simon, do you know how these various commercial modelling firms initialise their models? When I last asked, none of those that replied started from a detailed analysis. That may have change but, if it has, nobody has said so. They still claim to manage what world leaders in the field cannot.

I have to go back to what I have said before. A computer model can produce different results from the same data. That is because the data cannot define the atmosphere precisely enough and the models are not precise enough.

Enough for now.

 

[AndrewB]

As far as I can see, MAE is only valid for a scalar quantity eg temperature. I THINK that they are applying it to wind speed and forgetting the direction. My example of a zero error in speed but a large vector error is what is really behinf the measure used. Any competent statistician would use RMSVE. Your guess is as good as mine why they do not use a meaningful measure. Answers on a postcard.

I'd like to understand better exactly what RMSVE is. It seems to me that velocity errors can be measured quite easily, e.g. a forecast of wind 5kts N and an actual wind of 10kts S is an error of 15kts (not 5kts!). Its a fairly easy piece of trig to measure the velocity error when the forecast and actual are in other, different directions, and the MAE would then be the simple average of these errors, either through time or over space. IS RMSVE calculated similarly, except that you use the square root of the average of the sum of the squares of these errors?

Statisticians don't always prefer sums-of-squares errors over mean absolute errors. The former became more popular because the distribution theory is more tractable and the 'least-squares' method results in relatively simple formulae for use with forecasting models: not because its an inherently better approach.

P.S. Of course I share your suspicion that the accuracy claims of some forecasters are fudged. Lies, damned lies, and statistics!

 

[simonjk]

I probably did the Met Office a bit of a dis-service by describing them as irrelevant, you are correct that they are at the forefront of modelling research and climate prediction, but it's worth remembering they are neck and neck with other taxpayer funded organisations in the US, Europe and Asia.

Perhaps the best similarity is British Airways. Remember the 'fights' in the 1980's and how now BA are just another, albeit major carrier? That is how I see the Met Office these days.

Yes Frank, like the idea of the GMDSS being contracted out, although I suspect the Met Office would not be happy about it. I'd certainly be at the forefront of that race and would produce independently verifiable accuracy statistics on the web each day.

However, if the reaction is similar to that as it was to my approach to the Met Office to collaborate with Weather School, then I don't think the door would open! I offered t use Met Office data in my Weather Schools and operate their training classes for them under contract with key performance measures and value for money tests built in...it initially received a favourable response, but then the door was well and truly slammed shut! That's why Weather School goes it alone, makes money, gets great feedback and is now the pace that people come to from all across the world to learn about the weather!

You mentioned initialisation of the 'models'. I suspect most of these are initialised off the freely available GFS or perhaps even a version of the HIRLAM. BY sticking the data through an enhanced topographical model, one could increase the resolution significantly, although the errors which are features of the initialised model parameters are still present, but may be more difficult to identify.

Take a look at Hurricane Isaac which is about to hit the southern USA. Despite the billions of dollars of computer modelling from some of the worlds most brilliant minds, we still are not sure where landfall will take place in just 30 hours time. The actual decision about landfall is down to a human forecaster using his skill and experience.

Of course, that' not to say that GRIB forecasts are not useful to sailor's, they certainly are and can certainly give god guides. However, as I kept saying (sorry to be boring) speeds are often underplayed in the GRIB files due to the smoothing effect of the output so one should always use with caution.

Simon

 

[wotayottie]

As far as I can see, MAE is only valid for a scalar quantity eg temperature. I THINK that they are applying it to wind speed and forgetting the direction. My example of a zero error in speed but a large vector error is what is really behinf the measure used. Any competent statistician would use RMSVE. Your guess is as good as mine why they do not use a meaningful measure. Answers on a postcard.

There are three issues with any wind forecast ( speed, direction, and timing) and any measure of forecast accuracy that combined all three or even two of them would in my view be misleading. The three issues have different importance in planning a journey.

Direction is the one with least sensitivity to error - it rarely if ever makes a big difference if the wind direction is 10 degrees off. Wind speed is the next most important, at least in my view but a 25% error between a 10 knot forecast and a 12.5 knot reality is much less important that the same percentage between 20 and 25 knot.

For me the most critical often is the timing. In a country plagued by unstable weather patterns, I am often timing a passage to fit between two unacceptable weather forecasts. Sadly this is also the area where the Met Office accuracy seems worst - they are not bad at forecasting what is going to happen but often poor at saying when it will arrive.

So personally I would want to see accuracy statistics separately stated for all three factors. That said I would much prefer more accurate forecasts.

P.S. Right the way through the last week the inshore waters for the Bristol channel has been tinged in red for a force 6 when none of the usual internet forecasts have shown the same wind level. Most of the time the internet ones have been nearer the mark. Why? Is it the Fish / hurricane factor?

 

[oldbilbo]

In the good or bad old days, the Met Office was able to give just about everything away free - F. Singleton

In the good old days, as a daily user of military aviation forecast services, I had a huge respect for the Met Office's forecasters. I still have. Their commercial managers are in a different league.

The first time I visited the Met Office/Bracknell in 1995, their library staff and forecaster colleagues couldn't have been more helpful. The second time, at Exeter in 2010, I was treated to coffee in the foyer by two smooth Public Relations types, and my request for library access to historical data similar to that gathered in 1995 was politely refused. The third time I sought to visit the library, later in 2010, I was bluntly refused access, not permitted past the security gate. No reason given.

Here's what a few others have had to say about the Met Office and the way it has been meeting its obligations....

"First it was a national joke. Then its professional failings became a national disaster. Now, the dishonesty of its attempts to fight off a barrage of criticism has become a real national scandal. I am talking yet again of that sad organisation the UK Met Office." Christopher Booker - Telegraph

"Regular readers will remember the intense period of blogging activity during the 2010/11 winter about the Met Office’s weather forecast failures and our work in exposing their fraudulent attempt to conceal the reality of their seasonal forecasting activity." AutonomousMind2011

"Met Office boss lie exposed by own official...

The Chief Executive of the UK Met Office, John Hirst, appeared on the Daily Politics show earlier this week. Hirst was grilled in a very un-BBC manner by Andrew Neill about the department’s poor forecasting performance and Hirst’s extraordinary 25% pay increase. The hike in performance related pay saw Hirst receive between £195,000 and £200,000 in pay and performance bonuses in 2008/9. But John Hirst’s claims have been shown to be lies, by none other than one of his own senior Met Office officials ….Keith Groves." AutonomousMind2009

It's not the first time a Met Office Chief Executive has been hauled over the coals for 'creative accountancy'. In the past decade yet another of them has been required to hand back a large portion of his performance bonus after the National Audit Office found he had 'massaged' the organisation's annual results very much to his personal benefit. In any other developed jurisdiction he would have had his collar felt....

But what can we do to improve things at this hugely-expensive but under-performing public body? There is a group charged with protecting our interests and the service we get from the Met Office - the Public Weather Service Customer Group.

"The PWS provides a coherent range of weather information, services and advice and weather-related warnings that enable the UK public to make informed decisions in their day-to-day activities, to optimise or mitigate against the impact of the weather, and to contribute to the protection of life, property and basic infrastructure. The PWS also provides relevant and timely information to Public Sector users to enable them to fulfil their duties."

"It acts as the customer on behalf of the public for free-at-point-of-use weather services and advice... It also acts as guardian on behalf of the public..." ​

Membership may include representatives from: Cabinet Office, Civil Contingencies Secretariat, ACPO – Association of Chief Police Officers, CFOA - Chief Fire Officers Association, Local Government Association, Scottish Government, Welsh Government, Northern Ireland Government, Environment Agency, Highways Agency, Maritime and Coastguard Agency, Civil Aviation Authority...

All senior civil servants, paid and pensioned by HMG. Not a representative of the public among them. No RYA. No PBO. No incentive to initiate change, challenge the complacency, or otherwise rock the boat.

The Met Office formally supports this government's Open Data User Group, through the Public Data Group. Ever heard of them or what they do? No? There's a surprise....

The question I ask is "Are we being served - or just shafted?"

 

[franksingleton]

There are three issues with any wind forecast ( speed, direction, and timing) and any measure of forecast accuracy that combined all three or even two of them would in my view be misleading. The three issues have different importance in planning a journey.
.....

Several points.

I cannot see how to derive a confidence factor that would be a sensible and meaningful measure.

Direction can be very important, more so than force. I can cope with a 30 knot wind 60 degrees of the bow even if it is 10 knots stronger than forecast. I do not like a 15 knot wind 30 degrees off the bow. I have to start beating and that cocks up my timing for crossing the bar when I was expecting a 20 knot wind 60 degrees off.

NO. There is no such thing as the Fish effect. You have to remember that the Met Office MUST issue a gale warning if a gale MAY occur in an area. They can only cancel a warning if they are sure that there is no gale. Inevitably, that leads to an impression of over warning..

 

[franksingleton]

I'd like to understand better exactly what RMSVE is. It seems to me that velocity errors can be measured quite easily, e.g. a forecast of wind 5kts N and an actual wind of 10kts S is an error of 15kts (not 5kts!). Its a fairly easy piece of trig to measure the velocity error when the forecast and actual are in other, different directions, and the MAE would then be the simple average of these errors, either through time or over space. IS RMSVE calculated similarly, except that you use the square root of the average of the sum of the squares of these errors?
..........

RMSVE is the equivalent of RMSE for a scalar quantity. It has the merit of weighting larger errors more than small ones. In a normal distrubution, 2/3 rds of all errors should be within one RMSE/RMSVE.

It is not high powered statistics, it is a standard tool used by many experiamental physicists and others.

 

[franksingleton]

I probably did the Met Office a bit of a dis-service by describing them as irrelevant, you are correct that they are at the forefront of modelling research and climate prediction, but it's worth remembering they are neck and neck with other taxpayer funded organisations in the US, Europe and Asia.

..........
/QUOTE]


Simon

There are many issues here, not least the role of the private sector that have never been satisfactorily resolved. I wish they had because there is too much work out there for the met Office to handle properly. One problem that is still there is the financing. To put it crudely should a firm using raw material from the Met Office be able to benefit at taxpayers’ expense?

Initialisation of the commercial models, from what you say, totally unsatisfactory. The big players put as much effort into data assimilation as into the forecast. The minnows take short cuts. To me, it is axiomatic that unless you have a good start, the outcome will be in more doubt than if you did have a good start.

The problem with any objective forecast, GRIBs, HIRLAM, or any of the commercial modellers is that they are deterministic and not probabilistic. They are one solution of many and may not be the best. That is why I keep saying that ensembles are vital.

 

[franksingleton]

In the good old days, as a daily user of military aviation forecast services, I had a huge respect for the Met Office's forecasters. I still have. Their commercial managers are in a different league.
............

There are so many issues here and you clearly have some big bees in your bonnet.

Oh, yes, I remember now. It was a reporter called Bill Bailey who wrote the PBO article. The last section was good and gave the message that I give about using forecasts.

The first two-thirds were journalistic clap-trap. He began by quoting a young couple struggling into St Peter Port after a pounding in an un-forecast storm. When I asked for dates and times, he had to admit it was a fiction. If forecasts were so awful why did he not find a good example? Why demean himself with an unnecessary fabrication?

 

[AntarcticPilot]

Can I draw everyone's attention to this excellent link on Frank's web pages?

It is worth bearing in mind that chaos studies has its roots in weather prediction; this is where computational chaos was first observed (i.e. small changes in initial conditions resulting in vastly different end results). Initialization of models isn't just a theoretical nuisance; it is utterly fundamental to the whole business.

The point is not that weather forecasts are inaccurate, but that they are INEVITABLY subject to inaccuracies. That is why ensemble forecasting is a Good Thing - it shows the degree of unpredictability at a particular time. Weather systems are chaotic, and basically unpredictable at some scale or another. Fortunately for us, the horizon for predictability tends to be at something like 3-7 days out, depending on the particular circumstances. Forecasts beyond 7-10 days are really not much better than educated guesses, and long range forecasts are necessarily VERY general, and not based on the same methodology as the usual forecast. The horizon for predictability varies with the distances concerned as well.

Frank is quite right to ask how the inevitable inaccuracy in forecasts is measured, and equally right to suggest that a particular measure is misleading at best and a "terminological inexactitude" at worst!

 

[franksingleton]

Can I draw everyone's attention to this excellent link on Frank's web pages?
..............

Many thanks. Extremely clearly put. Better than some of my posts.

 

[wotayottie]

To put it crudely should a firm using raw material from the Met Office be able to benefit at taxpayers’ expense?

Yes. They are taxpayers too. Its outrageous that government agencies not only use taxpayers money but then charge the same taxpayers for what they have already bought. Contrast the free charts available in the US with UKHO charging time and again often for info from Nelson's day

 

[Cantata]

I can put up with the actual wind being not quite what is forecast (and I'd cope especially well if they attached a percentage likelihood to it) but so many times the Inshore waters forecast in my experience is laughably wrong even with the actual weather at the time of the forecast. See my earlier post in this thread - "Early rain clearing" - we hadn't seen any rain for days, it was a cloudless day; "F5-6, decreasing 3-4 later" - two hours later, there was no wind.
Perhaps the 'Thames' area is worse than others. Everyone I sail with is well used to staring in disbelief at the VHF as the forecast is uttered, and shouting "why don't you look out of the window!".

 

[wotayottie]

.

I cannot see how to derive a confidence factor that would be a sensible and meaningful measure.
.

Why bother? Does anyone other than a met office type even look at confidence data? For that matter, how many members of the public would really understand what it meant anyway?

It's an academic issue maybe of interest to you and Simon but mostly not to the rest of us.

 

[oldbilbo]

He began by quoting a young couple struggling into St Peter Port after a pounding in an un-forecast storm. When I asked for dates and times, he had to admit it was a fiction.

Bolleaux, Mister Singleton. Not a fiction at all, but my very own real-life encounter - with the lass who is now my wife. I am well aware you have done not much else with your life other than be a middle-aged, middle-ranking civil servant piloting an ever-increasing desk around the corridors, but those who take an interest in other's trade skills will recognise the common-or-garden writers' 'hook' - a described incident intended to engage the readers' interest in what follows.

In the context of a magazine article, it is a well-respected, even expected, wordsmithery device and that was firmly based on personal experience. If pressed, there were minor aspects one could describe as an 'ensemble' - but you're familiar with that term, I believe.

The PBO article, written by a well-respected Chartered Meteorologist and, er, myself, did engender considerable interest. It was attacked vehemently by a couple of Met Office 'apparatchiks' who sought to divert attention from the failings so accurately illustrated. I am advised that the Chief Scientific Officer of the Met Office apologised to the 'well-respected Chartered Meteorologist' some years later for 'an undeserved and scurrilous defamation made by an unauthorised junior official'. Not you, by any chance, was it?

Be that as it may, essentially the same article, stripped of the bits that made the story readable in PBO magazine, was re-published in the February 1998 volume of 'Weather', the Royal Met Society's journal, and sparked off a genteel debate among pro met-men ( and women ) about the issues raised. That continues today. So in a very real sense, the article achieved much of its purpose.

I find myself wondering why you expend so much of your energy resisting change instead of, with your considerable knowledge of the cogs and wheels in the Met Office, facilitating it.