Accuracy of forecasts

[franksingleton]

As a meteorologist I am all too aware that there is always some uncertainty in forecasts, See http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Limitations. That is why the Met Office and the other major payers run ensembles of forecasts. Whilst I expect commercial firms to make exaggerated claims about their products, I never fail to be amazed at claims about forecasts such as

…………detailed marine forecast
Outstanding accuracy
Worldwide forecasts using a 1km resolution
……….showing detail every hour!
Superior forecasting …………
Detailed ……. information for your local area
……highest quality weather forecast data
precise …..… accuracy of one nautical mile

No names no pack drill. Of course, much depends on what is meant by words such as “accuracy” and “precise.” However, I have now seen a statement that says

“.............., our data is exceptionally accurate. We produce forecast data with market-leading Mean Absolute Error (MAE) figures: within 1m/s on a 36 hour forecast against observed data over a year.”

I have no reason to doubt the accuracy of the statement but, an error of 1 m/s ie about 2 knots sounds impressive for a 36 hour forecast. However, what does MAE mean? I can see how it works for a scalar quantity such as temperature although the usual measure of accuracy would be RMSE or SD (root mean square error or standard deviation.)

It is not at all clear how MAE works for a vector quantity, ie wind. If it refers just to the wind speed then a forecast of 240 ° 10 m/s and an outcome of 300° 10 m/s would be a zero error of wind speed but a vector error of 10 m/s. In other words, instead of a fast one reef reach from Sables d’Olonne to Ile d’Yeu, it would be a two reef hard slog.

Have I got it wrong? Is there some other interpretation of MAE as applied to wind forecasts? The usual statistic is RMSVE – root mean square vector error.

I note that the Met Office quotes accuracy figures for temperatures, rainfall, sunshine, weather but not for surface wind. I assume that this might be because of the high variability of wind in space and time.

Any views on MAE and wind?

 

[wotayottie]

I note that the Met Office quotes accuracy figures for temperatures, rainfall, sunshine, weather but not for surface wind. I assume that this might be because of the high variability of wind in space and time.

Any views on MAE and wind?

Or because the wind forecasts are inaccurate and unreliable ? It's human nature to trumpet your successes and not to mention your failures.

P.S. Wonder if you are posting ( like me) because you are stuck somewhere waiting a weather window and fed up of forecasts that turn out to be wrong?

 

[savageseadog]

Meteorologists are predicting global warming very accurately too

 

[Heckler]

Or because the wind forecasts are inaccurate and unreliable ? It's human nature to trumpet your successes and not to mention your failures.

P.S. Wonder if you are posting ( like me) because you are stuck somewhere waiting a weather window and fed up of forecasts that turn out to be wrong?

The forecast for the run from MH to HH was anything from 4-5 to 7 and I saw an 8 at one point, in fact it was piggin lumpy getting to skokholm and turning right with 26kts, then settling at 19 kts for most of the night then dropped to 14kts then picked up to 19 kts at HH.
Stu

 

[franksingleton]

Or because the wind forecasts are inaccurate and unreliable ? It's human nature to trumpet your successes and not to mention your failures.

P.S. Wonder if you are posting ( like me) because you are stuck somewhere waiting a weather window and fed up of forecasts that turn out to be wrong?

Partly, yes, on Ile d’Yeu, but quite happy. No rush to get anywhere.

But my real reason is that the various firms all make extravagant claims that I do not think can be justified. When verifying wind forecasts, the Met Office does so for upper level winds because the UK and the US, jointly, run by mutual back-up, the World Aviation Forecasts Centre providing planning winds for all civil aircraft. With surface winds, it is far more difficult. Just think how much the wind varies over short periods. Do you verify in terms of gusts? If so, what length of gust? Or the 10 minute “synoptic” wind? Or an hourly average? The figure quoted in my first post seems to me to be nonsense but I may have missed something. I am not aware of any national weather service making such claims.

One of the few assessments of wind forecasts was in a letter to PBO. See http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Assessment. I wrote a letter to PBO and put up this page rather hoping to start a debate but nobody rose to the bait. As a former forecaster who has some experience of forecast assessment and as a yachtsman user I do see several facets of this issue but do not have an answer.

As someone once said, a weather forecast is rather like an oil painting. It can look great from a distance in that all the shapes look good. Get in close and it all look very different.

 

[simonjk]

Good post Frank!

I don't think it is just commercial organisations that make extraordinary claims, government organisations are capable of doing that too, although admittedly less so in 's a dilemmthe last 12-months or so.

I do find it laughable when I see claims of 0.01 deg resolution and accuracy of 99%+. These are figures that it is just not possible to achieve; although to be more precise a resolution of 0.01 deg is possible to achieve but should not be used to infer superior accuracy because of the enhanced resolution.

It's a dilemma I sometimes face. At weatherweb I sometimes get comments such as "you forecast a F7 but the most I saw was a high F6". It seems that for some the expectation from the forecast is too great.

In the 'heads-up' I put out last night I stated that I may have been too pessimistic, but looking at the obs today I think I was pretty close and that certainly somewhere would have received the winds I forecast. This is a problem of assessing forecasts based on spacial locations which are far apart.

Ultimately I suppose what we all need to remember is that too high expectations of a forecast, combined with too high claims of accuracy equals a recipe for, at best, dissatisfaction and, at worst, disaster.

My response to the "how accurate are your forecasts?" is that customers pays their money and takes their choice, if we weren't accurate, they wouldn't use us...seems a reasonable response to me?

Okay, rabbited on too long!

Simon

 

[little_roundtop]

 

[AndrewB]

I know nothing about weather forecasting but would assume MAE is the equivalent of Mean Deviation, just as RMSE is the equivalent of Standard Deviation. MAE has some advantages over RMSE - (i) it's a little easier for non-specialists to grasp and (ii) it isn't so affected by outliers, exceptional errors (though as Little Roundtop illustrates, we weather forecast users tend to be more impressed by the outliers).

Maybe they are not using MAE in a vector sense, just in terms of speed. That would favour their accuracy claims if they get the direction totally wrong. However, there is no reason in principle why MAE could not be calculated in terms of vector differences.

Their accurancy claims appear to be specific enough to be verifiable. If it is valid, maybe the Met Office had best poach their model? Otherwise, call foul play and march them down to the Trading Standards Office!

 

[gerry99]

When I was doing my YM I was always taught to remember its is a FORECAST not a guarantee. The physics of weather are extremely complex, hence the growth in the use of supercomputers in the process, but fundamentally a weather forecast is just that! Whilst the accurancy of forecasts has improved considerably over the past decade there are many folks who stoll seem to think it is a guarantee

 

[franksingleton]

I know nothing about weather forecasting but would assume MAE is the equivalent of Mean Deviation, just as RMSE is the equivalent of Standard Deviation. MAE has some advantages over RMSE - (i) it's a little easier for non-specialists to grasp and (ii) it isn't so affected by outliers, exceptional errors (though as Little Roundtop illustrates, we weather forecast users tend to be more impressed by the outliers).

Maybe they are not using MAE in a vector sense, just in terms of speed. That would favour their accuracy claims if they get the direction totally wrong. However, there is no reason in principle why MAE could not be calculated in terms of vector differences.

Their accurancy claims appear to be specific enough to be verifiable. If it is valid, maybe the Met Office had best poach their model? Otherwise, call foul play and march them down to the Trading Standards Office!

As far as I can see, MAE is only valid for a scalar quantity eg temperature. I THINK that they are applying it to wind speed and forgetting the direction. My example of a zero error in speed but a large vector error is what is really behinf the measure used. Any competent statistician would use RMSVE. Your guess is as good as mine why they do not use a meaningful measure. Answers on a postcard.

 

[Cantata]

How do these people measure how accurate they were?
I recall a debate on here and elsewhere a couple of years back where I and others bemoaned the consistent lack of accuracy (as perceived by us) of the Inshore Waters forecast. Someone sprang to their defence and said the Inshore forecasts were such and such accurate, may have been 90%. But no explanation was forthcoming on how the actuals were measured. For instance where would actual data for the Thames Estuary (my patch) come from?
In my sailing this year I'd still say these forecasts are poor. For instance last week, heading north across the shipping lanes NE of Sheppey, I heard the 1200 forcast on the 1310 CG MSI b'cast. "SW 5-6", it said, "backing W and decreasing to 3-4 later." "Early rain clearing" it said.
We hadn't seen rain for several days and at the time it was a beautiful cloudless day. The wind was SW 4-5, but by 1430 had dropped completely. Can't win 'em all, I suppose.
The Gib Point to N.Foreland area is admittedly very large.

 

[franksingleton]

Good post Frank!

Simon

This post is really telling my granny how to suck eggs but I hope that it might help others.

First, many people do not understand about predictability and its limitations. That seems to include the commercial firms that run meso-scale models. ECMWF and the major national weather services do recognise the limitations in weaher prediction. That is why they use model ensembles.

As you are well aware, a numerical model can produce different forecastsm sometimes quite different, starting from the same data. That is because the availanle data are simply not good enough to define the atmosphere precisely enough. There are always uncertainties in the data and there is no analysis scheme that will produce a definitive analysis. That is what ensembles are for 0 to use the same data, putting in small ransom variations in the analysis and seeing how similar or not are the resukts.

The second point is that wind is extremely different to define and forecast. The picture here of a wind record (direction st the tope and speed lower down_shows just how variable wind is even on a fairly quiet day and a quieter evening.

Differential heating causes pressure gradients. These lead to air moving as the atmosphere tries to settle down to a state of equilibrium. But, as the air moves, it redistributes air and changes air pressure. At the same time, all the heating anf cooling effects are changing. So, the atmosphere is always changing.

All sailors wwill be well aware of the outcome if not the reasons for it happening. That is why wind is incredibly difficult to get right. One of my examples is a W F4 up the Channel. Pressure at Soyjamptone, say, 1004 pressure at St Malo will be about 1008. If the weight of air over Southapton is less by 0,1% then the pressure will be 1003 and the winds F5.

Our passage today has been a good example of what forecasts can and cannot do.

Last Wednesday we were leaving the river Vilaine. GRIB forecasts suggested that Friday and Saturday would be windy. Going south, Ile d’Yeu would be a good plave to be. The same forecasts also showed that we would possibly be able to move on to Sables d’Olonne on the Sunday but that Monday would have southerlies.

That all went pretty well. Friday and Saturday were windy, moreso than indicated. Sunday was a good day to move on. Forecasts for Monday still showed southerlies.

The forecast from Météo France on VHF, Météo Consult and the GRIBs all showed light NW becoming W to SW F2-3. A gentle sail with a favourable wind was the agree story. In the event, the wind started non-existent, came in a nice SW, up toF4. It then dropped to a F2 and backed to SSE. Approaching Sables d’Olonne, it went more SSW and increased a little.

 

[Tomahawk]

I would like to see forcasts give an indication of their confidence in the accuracy of the forcast itself..

The are some days when you can look at the analysis and say tomorrow it will rain at about 10 OClock and it does.. other days there are a lot of compeeting influences and any of them could dominate so it is impossible to be accurate..

 

[franksingleton]

How do these people measure how accurate they were?..............

.

It may seem strange to you-all but assessment of forecasts is remarkably difficult. The MCA did mount an exercise- see http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Assessment. In a letter to PBO, I did express some doubt about how the assessment was made and whether it was would really reflect what you and I see when we go to sea. How do you verify a forecast that covers a 24 hour period against something that is continuously variable in space and time?

It is when I see statements that are not supported by verifiable facts claiming great accuracy that I get worried. That worry is increased when I see claims using MAE for wind speed without reference to direction. To anyone who is out there sailing. It must be obvious that claims for precision are nonsensical. Weather is simply not precise.

Models can be assessed objectively and this is done primarily to see whether different formulations give improvements. There are various checks. RMSE of surface pressure against analysed pressures is a good check. Because winds at higher levels are more steady (not having small topographical eggects0 and because the UK is one of two World Aviation Forecast Centres, assessments are made at upper levels – 500, 200 hPa for example.

When I give talks, I usually show an 8 day sequence of wind forecasts using the US GFS GRIBs by comparing each day’s forecast to the analysis for that day. Do it yourself or take a look at http://weather.mailasail.com/Franks-Weather/Grib-Forecast-Examples.

 

[franksingleton]

I would like to see forcasts give an indication of their confidence in the accuracy of the forcast itself..

The are some days when you can look at the analysis and say tomorrow it will rain at about 10 OClock and it does.. other days there are a lot of compeeting influences and any of them could dominate so it is impossible to be accurate..

The Met Office does (or did) review this idea from time to time. The problem always will be that many will not understand or know how to use that information. I have only to recall all the fuss over seasonal forecasts which are always expressed in probabilistic terms. The press does not or does not want to understand – eg the “barbecue summer” was a probability not a definite statement. In hindsight it was still a strong probability.

Not long ago the duty chief forecaster was in a dilemma. His ensemble models said 20% chance of show in London. What would the councils have done with that information? More importantly, perhaps, what would the press have done? They would have trumpeted “Snow expected in London,” and then lambasted the Met Office when it did not happen. In the event the forecaster made a decision to say “No snow in London.”

But, yes, because ensembles are now used as routine, confidence factors could be given.

 

[Cantata]

The Met Office does (or did) review this idea from time to time. The problem always will be that many will not understand or know how to use that information. ......

I seem to recall from the times when I made frequent visits to the USA that percentage probabilities were used in every forecast seen on the TV. Are the Brits perceived as being less able to understand probabilities than Americans?

 

[franksingleton]

I seem to recall from the times when I made frequent visits to the USA that percentage probabilities were used in every forecast seen on the TV. Are the Brits perceived as being less able to understand probabilities than Americans?

I can only refer you to all the hou-hah about seasonal forecasts. You may well be able to use such information sensibly. It is clear that many cannot. Or, maybe it is our wretched press that makes a song and dance about it.

 

[maxi77]

I seem to recall from the times when I made frequent visits to the USA that percentage probabilities were used in every forecast seen on the TV. Are the Brits perceived as being less able to understand probabilities than Americans?

Are you so sure the Americans do understand probabilities, are perhaps the probabilities added as protection from litigation rather than a desire to inform the public.

 

[Tomahawk]

The Met Office does (or did) review this idea from time to time. The problem always will be that many will not understand or know how to use that information. I have only to recall all the fuss over seasonal forecasts which are always expressed in probabilistic terms. The press does not or does not want to understand – eg the “barbecue summer” was a probability not a definite statement. In hindsight it was still a strong probability.

Not long ago the duty chief forecaster was in a dilemma. His ensemble models said 20% chance of show in London. What would the councils have done with that information? More importantly, perhaps, what would the press have done? They would have trumpeted “Snow expected in London,” and then lambasted the Met Office when it did not happen. In the event the forecaster made a decision to say “No snow in London.”

But, yes, because ensembles are now used as routine, confidence factors could be given.

I think they would only need to say "our confidence is high"or "low" ... or even "very poor" for which read non existent.. A simple explanation on the front page of the weather page should be good enough..

At least if they say rain together with "our confidence is low" , it is just as likely to be hot and dry, whereas if they say rain and the confidence is high... you know will get wet ... And yes it is difficult when a front is sweeping over Scotland bringing lashings of rain.. you can be confident it will rain up north.. however further south the tail of the system is weakening and it may or may not rain depending on the building strength of a nearby high.. ven so with ensembles it should be possible to say something about the confidence of various parts or the model..

 

[oldbilbo]

When I give talks, I usually show an 8 day sequence of wind forecasts using the US GFS GRIBs by comparing each day’s forecast to the analysis for that day. Do it yourself.....

Interesting that once-time Met Office Senior Manager Frank S. seems now to be coming round to our way of thinking. The Met Office, for decades a branch of the MoD, has always had a culture of 'official secrecy' and a horror of anyone other than its own approved staff taking any interest in its in-house activities and, of course, its record of successes and failures.

Back in 1996 'Practical Boat Owner' published 'A Short Survey Of Forecast Performance', where the forecasts for Sea Area Portland, for a full year, were compared with the actual conditions reported by the Channel Light Vessel Automatic. This 'population' was analysed by independent Chartered Meteorologist Mike Brettle and the results published by PBO's fiery editor George Taylor. The briefed 'research question' was 'Compared with the 24-hour forecasts on a Friday and a Saturday night, on which yachtsmen would base their decisions, what actually were the outcomes at sea twelve hours later? How reliable and useful to us, then, were the Shipping Area forecasts and what can we infer from this'.

There was then no publication of any measures of forecast success rates. Then as now, we could not look at any analysis in the public domain which showed clearly whether the Met Office were getting it about right, perhaps improving as time went by, or simply marking time. The primary purpose of the article was to encourage a little fresh breeze to blow through the stuffy and patronising Met Office engagement with its many thousands of small-craft users, in the hope that some better understanding and perhaps some better forecasting would emerge.

Predictably, the Met Office 'panjandrums' were outraged, and rushed to attack the impudence of a sailing magazine publishing stuff that was officially their prerogative. Predictably, nothing changed. Some years later, essentially the same PBO article was published within the Royal Met Society's magazine 'Weather', re-written for a different audience by Mike Brettle, and ever so slowly the Met Scientists' Mafia began to accept that public interest in the outcomes of their work was not necessarily 'a bad thing'.

Today, we have numerous meteorologists' sites - Chris Tibbs', Frank Singleton's , Simon Keeling's , and Passage Weather prominent among them - offering perspectives, guidance and met services. Crucially, we can now access them all readily and compare for ourselves.

The Met Office is no longer the 'keeper of the keys' to the results of this science. The truth is out there....

However, readers might care to muse on the uncomfortable fact that, in 1996, a researcher could and did visit the Met Office Library ( a public library in Bracknell ) and simply consult the records of Forecasts and Actuals for the relevant places and times. We had a right of access. Just over a year ago, when PBO sought to repeat the same exercise to identify what if any differences or improvements had occurred in the intervening 17-odd years, we were denied access to the records. The unhealthy cult of official secrecy is still strong down at Met Office Exeter.

I have crossed swords with Frank Singleton before - not on how best to interpret forecast charts, but on the Met Office's stifling and patronising culture of 'this is our information; you will be granted only such access as we deem fit - and you most certainly will not get anywhere near any data that might possibly suggest we managers are under-performing in any way'.

Official Secrets Act versus Freedom Of Information Act?

It wouldn't happen in the USA, where they have much-enhanced public weather services, with the Inshore Weather Forecasts and Actuals we, too, really want made available via continuous VHF broadcast. Over there - and elsewhere in the developed world - professional met managers and coastguard organisations have seen the obvious sense and simply got on with it. Here, the 'dinosaurs' spend their time rehearsing reasons why they
can't....