Weather forecasts - anyone keep tabs on accuracy?

[Conachair]

Geeting a bit obsessed these days with the weather. This arrived on the doorstep this morning. http://www.amazon.co.uk/Heavy-Weath...1?s=books&ie=UTF8&qid=1326744924&sr=1-1-spell
Only opened the cover but looks like a very fine book to have onboard.

So how to check if you are getting it anywhere near right? No point in trying to remember, everyone thinks they're brighter than they are (just look in the lounge )


Current plan is for a spreadsheet, each day do a 5 day forecast for a specific area from a look at the surface charts and 500mB charts , then try and put some kind of accuracy on them whenthe tinme comes, maybe a cell colour or something.

Anyone else ever try anything like this?

And how to the professions do it, they must have some kind of empirical scorecard to keep track on things.

Frank?

 

[Grumpybear]

the Met Office! the bonuses of the head honchos depend on it. a few years back the chief exec was caught manipulating the results to his own advantage. needless to say he just got a limp slap onn the wrist, as I recall.

on second thoughts, that isn't what you meant.....

 

[Conachair]

Nearly went into one about poeple being down on the met office without the faintest idea about real/ vs forecast data. Please no!!!!!!!!!! Go rant on your own thread!

Anyway, lets see how we can do from here..

http://www.weathercharts.org/ukmomslp.htm

Anyone care to join in?

So far shooting from the hip...

16/01/11 17/01/11 18/01/11 19/01/11 20/01/11 21/01/11
light wind, sunny , good. ssw 2/3. sunny good sw4 rain morning more rain veering wind afternoon WNW2/3 rain clear o/n WNW2/3 scat/ cloudy. chilly NW2/3 scat cloud chilly


Blimmin formatting

 

[estarzinger]

http://www.amazon.co.uk/Heavy-Weath...1?s=books&ie=UTF8&qid=1326744924&sr=1-1-spell
Only opened the cover but looks like a very fine book to have onboard.

Sorry to tell you, but from a practical perspective the 500mb charts have limited practicality for a cruising sailor. That information is already built into the grib (and met office chart) outputs to the best of the supercomputers ability and you are very unlikely to do any better.

do a 5 day forecast for a specific area

It depends on what you mean by accuracy. The accuracy of a specific forecast of wind direction and strength is about nil 5 days out (eg no better than just predicting the prevailing/pilot chart wind). There is some very small amount of accuracy to the 5 day macro feature forecast, but it's still quite small. You can see this from the hurricane forecasts where they do track and measure accuracy.

One thing that is very valuable for a cruising sailor is to be able to see and understand when the forecast has decent stability/probability vs when it does not. If it has decent probability you can do clever routing and cut the margin or error, while when it has low probability you need to set up so you are in good shape if other things than the forecast pattern happens.

But to directly answer your question . . . yes, folks do measure forecast accuracy - in various different ways. Just for example below is the 24hr precip forecast accuracy for my current location:

The Weather Channel 74.15%
Intellicast 74.06%
AccuWeather 71.88%
National Weather Service 71.75%
CustomWeather 68.93%
NWS Digital Forecast 67.58%
Persistence 48.97%

Note that this accuracy decays roughly with the power of time, so if we take .7 as the 24hr accuracy then the 5 day precip accuracy might be .7^5=17%

 

[Allan]

I heard recently that the BBC are planning to run long term test of weather report accuracy. I think they said that the Met office had refused to join in.
Allan

 

[Conachair]

Originally Posted by Conachair View Post
http://www.amazon.co.uk/Heavy-Weathe...4&sr=1-1-spell
Only opened the cover but looks like a very fine book to have onboard.

Sorry to tell you, but from a practical perspective the 500mb charts have limited practicality for a cruising sailor. That information is already built into the grib (and met office chart) outputs to the best of the supercomputers ability and you are very unlikely to do any better.

Don't care Still fascinating to learn more about it Like *why* that low the other day which would normally have spanked into Ireland ended up off Morocco.

Still, going well so far, forecast of about an hour ago is holding up nicely

 

[Roberto]

looks like a very fine book to have onboard.

A fine book it is, imho

 

[webcraft]

I heard recently that the BBC are planning to run long term test of weather report accuracy. I think they said that the Met office had refused to join in.
Allan

No, the Met Office are going to take part - and so is Piers Corbyn. Should be interesting.


BBC to carry out weather forecasting test



- W

 

[Koeketiene]

Weather forecasts - anyone keep tabs on accuracy?

When sailing in the Channel, it's been our experience that the Meteo France 24hr forecast is pretty much always spot on.

 

[Conachair]

Changed to source to here:-
http://www.weathercharts.net/northwood-analysis-prognosis.htm

Pretty sure this what northwood transmits via weatherfax so makes sense to use it.

 

[ukmctc]

I use USGRIB and for the last 2 years its been 99% bang on.

 

[binch]

Med weather comparisons

I once spent a year sailing west of Italy. I monitored Spanish, French and Italian shipping forecsts. The Italian were easily the most reliable.

 

[franksingleton]

Geeting a bit obsessed these days with the weather.

............

And how to the professions do it, they must have some kind of empirical scorecard to keep track on things.

Frank?

Assessment of weather forecasts is a jungle full of booby traps. I have a number of apochryal stories up my sleeve.

The Met Office has some objective verifications of parameters like temperature at specific locations. The results are meaningful for those locations. Verification of an area forecast such as a sea area or a stretch of coast is more problematic because of the spatial and temporal variability of weather.

See http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Assessment. I believe that this is an ongoing exercise but have heard nothing recently. Perhaps a forumite might quote my page and write to MCA and the Met Office to ask.

All Met services running Numerical Weather Prediction models run continual checks on performance. This is partly to monitor the output and, partly, to check that changes to the model, which are frequent, do result in improvement.

These checks are many and various. One is the RMSE of surface pressure over a specific area, such as the North Atlantic, over the whole N Hemisphere or even over the whole globe. These data will not mean much to us in terms of forecast accuracy but do give a pecking order of the models. Centres will check their own model output and that from other centres. A short while ago, ECMWF was doing best of all – but remember that they start their run later than the others as they are really looking 3+ days ahead. The UK came next followed by the rest in a bunch. However there was little difference between them all.

In my talks, I usually show a sequence of forecasts from the GFS (zyGrib, Ugrib, Saildocs, Passageweather, XCWeather and the various iPad apps etc all use the GFS) and compare forecasts from one to 8 days ahead with what actually happened. Some of the examples I used are at http://weather.mailasail.com/Franks-Weather/Grib-Forecast-Examples. At some stage, I will add results from examples used this year.

In general terms, the GFS GRIBs give good guidance to 5 or 6 days ahead, not usually misleading at 7 days and variable to the extent of not being useful at 8 days.

You can, of course, see http://www.metoffice.gov.uk/about-us/who/accuracy. I know that the more cynical ones will sneer at this page. However, there is no point in the Met Office or any other forecasting organisation fooling itself.

When anyone says that this or that forecast is accurate, or someone claims accuracy for their service, I always want to ask how they define accuracy and how they measure it. For organisations other than the National Met services, the answer would no doubt be a stony silence.

Sorry, yet again for the length of this reply. Questions on weather can rarely be answered satisfactorily in a few words.

 

[franksingleton]

I once spent a year sailing west of Italy. I monitored Spanish, French and Italian shipping forecsts. The Italian were easily the most reliable.

Interesting. Having spent 10 years in the Med, I gave the Italians the lowest marks of all on all counts. Poor presentation, poor organisation of their forecasts, unhelpful forecasts. On the west of Sardinia, I preferred to use Météo France.

My trenchant views are at http://weather.mailasail.com/Franks-Weather/Marine-Weather-Forecasts-Mediterranean-Adriatic#italy if anyone is interested.

 

[Liz_I]

Unfortunately most folks want a weather forecast to be 100% accurate. In reality that isn't going to happen.
Get a forecast for the middle of the Channel, North Sea, Biscay etc and it will be fairly accurate. Go inshore and although the forecast will vary from offshore it cannot be totally accurate. Hills, mountains, valleys, plains + dozens of other obstacles will effect the speed and direction of the wind.
Best bet is to get 2-3 forecasts from different sources plus local ones from the tv/newspapers/Navtex. Compare these over a few days along with your own observations. It's interesting to see what you come up with.
Failing that it is usually 'just pick the forecast that suits you best'!

 

[KellysEye]

>Meteo France 24hr forecast is pretty much always spot on

Agree. From the UK to the Caribbean we used SatC to get Met Office, Meteo France and NOAA forecasts. MF were the most accurate. From experience I wouldn't trust a forecast beyond three days but it does depend where you are. In some places the weather can stay the same for months.

 

[estarzinger]

In general terms, the GFS GRIBs give good guidance to 5 or 6 days ahead, not usually misleading at 7 days and variable to the extent of not being useful at 8 days.

My experience (offshore) would suggest you are being a bit optimistic. I would have said - generally good guidance 3 days ahead with a vague hint at the possibility 5 days out. (and that's been true for both raw gfs gribs and "human adjusted" weather routers advice).

There are of course differences depending on location. Just for instance - sailing east from NZ (in the 40's) the models often have a difficult time determining whether lows will come over the top of NZ, over the N island, or between the N & S islands and that dramatically effects the subsequent developments. There are also vast differences between the tropics (where the current wind strength and direction is surprisingly often wrong - case in point this last fall the last 24 hours approaching the BVI when all the models were showing 10-15 from the E and in fact there was 0-20 from the North) and the higher latitudes (where the systems are more compact and intense).

Our experience is that most sailors leave heading offshore thinking that the forecasts are going to be more accurate than they in fact are. As I commented in a post above, I think one of the most useful and practical a cruising sailor can understand/learn is to see/know when a forecast development is 'high probability' vs when it is 'low probability'. When its high probability you can do clever routing and squeeze thru short windows while when its low probability you want to prepare contingency plans so you are well positioned no matter what occurs. I have always been a bit surprised and disappointed the met profession does not assign a 'certainty rating' to a forecast - but the cruising sailor with some study can in fact determine quite a bit about the likelihood of a particular forecast occurring. (by looking at different models and one model's stability over time and looking for particular features - like slack zones and secondary lows etc)

 

[Roberto]

I have always been a bit surprised and disappointed the met profession does not assign a 'certainty rating' to a forecast

Just for info, Meteo France assigns "likelihood factors" (facteur de confiance) to some of their forecasts, daily for the general forecasts on tv, but also for some of their marine forecasts: sorry can't remember which one, by memory offshore ones going beyond GMDSS horizon, which probably means "forecasts you have to pay for"
btw 1 is lowest, 5 is highest likelyhood

 

[franksingleton]

Unfortunately most folks want a weather forecast to be 100% accurate.

Ttrivial forecasts apart, no forecast is ever 100% accurate although, I suppose, it all depends on what you mean by accuracy.

Hills, mountains, valleys, plains + dozens of other obstacles will effect the speed and direction of the wind.

Ditto round the coast. No inshore waters forecast will please everyone along that coast.

Best bet is to get 2-3 forecasts from different sources ...........

The problem is to get the truly independent. Met services see each others forecasts. Most of those automatic forecasts will be interpolations from the GFS.

 

[franksingleton]

My experience (offshore) would suggest you are being a bit optimistic. I would have said - generally good guidance 3 days ahead with a vague hint at the possibility 5 days out. (and that's been true for both raw gfs gribs and "human adjusted" weather routers advice).
.......

I have not done the exercise in the tropics as my talks are usually to UK based sailors interested in NW Europe, Baltic and Med. However, I suggest that you try it yourselves.

Get an 8 day GRIB forecast - zyGrib or UGrib are the most convenient. Then for the next 8 days, compare the T=0 for each day with the T+1, T+2, T+3 days etc up to 8 days. Doing this, I have found that days 1 to 5 are usually pretty good. That is used as a planning tool you would have no surprises. In my last one, the T+% did not look very good but T+51/2 did ie a 12 hour timing error. T+6 and T+7 are usually not wildly wrong ie not seriously misleading. T+8 is hit and miss.

It is many years since I last sailed in the caribbean and my tru tropical sailing was all in dinghies although I had two years as a forecaster in the days of steam.

Tropical weather is often very local and convection often dominates. Models cannot deal well with detail on that scale although models can handle tropical storms. From time to time Miami Forecast Center has thanked the UK for its model output.

Southern hemisphere forecasting is still a problem partly, perhaps mainly, because of the lack of in situ data. Marvellous though satellite sensing is, the measurements are very difficult to handle.