Weather forecasts - anyone keep tabs on accuracy?

[estarzinger]

However, I suggest that you try it yourselves. Get an 8 day GRIB forecast. Then for the next 8 days, compare the T=0 for each day with the T+1, T+2, T+3 days etc up to 8 days.

I have. I am a pretty intensive user of weather product. We have sailed twice round the world (over 15 years), looking closely at the weather forecasts the whole way - fax charts on the first trip and gribs and shore routers on the 2nd. I do exactly what you are suggesting whenever we are looking for a window to leave, because it helps me understand how stable/consistent/probable the forecasts are at that moment.

Some years ago, in between the two trips, I had a somewhat boring corporate job, and every day for a year compared the past 5 days of 5 day forecasts with what what was actually being reported on a buoy off cape hatteras. I then gave a 'pass/fail' accuracy rating to the wind (direction and strength) forecasts - esentially say pass if the forecast was accurate enough to select the proper sail combination (within 5kts and 20 degrees). Using that criteria, the 24 hr forecasts were only 40% accurate, and it decended in a power curve from there. That was a decade ago now, and things have improved some, but at least in our direct experience, still less accurate than many would think.

My wife gives 'practical' weather seminars occasionally and she is often in a venue where another met professional also gives seminars which are more theoretical. Its a bit funny, as he spends a lot of time on 500mb charts saying they are essential, and she says a cruiser should generally understand the vertical nature of the atmosphere but practically speaking should not bother with the 500mb product. And he says that the accuracy is terrific and if you know what you are doing you can pretty much avoid the nasty stuff, and she starts out showing three recent examples (which are always easy to find from our last 3 months gribs) where the models fell apart quite quickly. And then goes on to my point about assessing the forecast stability/probability.

 

[Conachair]

Thanks for all the pointers guys. As with so much of sailing it's another never ending learning curve.

every day for a year compared the past 5 days of 5 day forecasts with what what was actually being reported on a buoy off cape hatteras.

Good Idea . My spreadsheet gains another tab. Sandetti light vessel and XC seems as good a place as any.

When anyone says that this or that forecast is accurate, or someone claims accuracy for their service, I always want to ask how they define accuracy and how they measure it.

Sorry guys but I'm with Frank here, I just don't buy all the "XYZ is better than zyx", humans just don't do objective, we see what we want to see and once our minds are made up thats it. Unless, like Evans you write it all down and check it with the real world afterwards.

Weatheronline expert charts seems a great way to do some research, http://www.weatheronline.co.uk/cgi-bin/expertcharts?LANG=en&CONT=euro&MODELL=gfs&VAR=prec

Frank, am I right in thinking that the "base" alters when the model was run? So you can see what last weeks model said about today?

A few thoughts re gribs: I sort of felt offshore that I should be using the opportunity to practice forecasting from synoptics more rather than just staring at the wind arrows. though sat phone call costs might have had a bearing on this Same with Herb, why let someone else tell you what to do instead of using a rare ocean crossing to practice standing on your own feet?

 

[franksingleton]

Yes, the “base” is the data analysis time for that computer run. ECMWF do slightly better than the rest because they do not have an immediate imperative. Their sights are on 3+ days. The UK, NOAA etc have immediate needs. ECMWF put more effort into data analysis and get more data in. Having said that they are slightly better than the UK and the UK slightly better than the rest in terms of RMSE of surface pressure.

Generally, of course, later data times will give better results. But, not always. ECMWF were pleased that their 5 day forecast had predicted the January 2010 la Rochell storm. They were less pleased that their 3 day forecast was not as good!

Some of the comments in other posts refer to previous years. NWP has been and is still improving. In 2001, the GRIBs and the DWD RTTY service were predicting Mistrals about 4 days ahead. In 2008, when we were last there, they were doing well 5 days ahead. I would now be looking 6 days ahead.

Please see my lecture examples at http://weather.mailasail.com/Franks-Weather/Grib-Forecast-Examples and, in particular, my latest one at http://weather.mailasail.com/w/uploads/Franks-Weather/Gribex-Nov2011.pdf.

The idea of comparing 5 day forecast with what happens at a point is reasonable, but there will be noise in the signal due to local effects and results will not look as good as looking at the whole patter. Put crudely, the atmosphere does not know itself to within one Beaufort force If anyone could write a bit of software to do that kind of exercise automatically it would be great. I have done some manual checks but it was very laborious.

I certainly do not buy the idea of using 500 hPa charts. We used to do that when I was a lot younger. The results simply do not compare with NWP. Remember that if you start using upper air charts, they will come from a computer model. By definition, NWP models will produce an internally consistent set of charts. They will interpret their own data far better than we can.

The user has to use experience, nous and commonsense to add local value to NWP/GRIB or synoptic charts. That will work for the first 24 to 36 hours. After that, and certainly at 5 or more days, you will not beat the computer models.

I could say a lot more but will leave it there.

 

[franksingleton]

One day weather course/seminar at CA House

In case anyone is interested, there are some places on a one day course at CA House, Limehouse Basin, on Sunday 29 January. This is in the list of courses run by the CA - http://www.cruising.org.uk/events/cahcourses.

On the 29th, I am giving 4 talks.

1. Back to Basics in which I give better explanations than you see in standard Yachtie textbooks of Coriolis, how pressure gradients form, how sea breezes occur. I have long thought that if you understand the basics, then the rest should follow.

2. The General Circulation. How the atmosphere works on the large scale. Trade winds, Easterly waves and hurricanes. Jet streams and extra-tropical depressions.

3. Numerical weather prediction. GRIB files, what they are, what they can and cannot do. Getting and using them with other tools.

4. Fronts and local effects – essentially, how the user can add detail to forecasts.

It is intended to be a fairly stretching type of day, not least for me, and numbers are being kept fairly small so as to encourage discussion after each talk.

 

[mcframe]

In case anyone is interested, there are some places on a one day course at CA House, Limehouse Basin, on Sunday 29 January.

Interested, but I don't think I can make the CA then, unfortunately.

Quick question 'tho:
Is it fair to say that UK 3-5 day forecasts are generally pretty accurate, but might happen sooner or later and/or further north or south than forecast; and that the average UK yachtsman can plan and apply some adjustment on that basis?

I'm thinking of the usual SW Atlantic system coming in, might speed up, might slow down, might get pushed north, might get pulled south situation rather than detailed specifics.

 

[Bosun Higgs]

Geeting a bit obsessed these days with the weather. This arrived on the doorstep this morning. http://www.amazon.co.uk/Heavy-Weath...1?s=books&ie=UTF8&qid=1326744924&sr=1-1-spell
Only opened the cover but looks like a very fine book to have onboard.

So how to check if you are getting it anywhere near right? No point in trying to remember, everyone thinks they're brighter than they are (just look in the lounge )


Current plan is for a spreadsheet, each day do a 5 day forecast for a specific area from a look at the surface charts and 500mB charts , then try and put some kind of accuracy on them whenthe tinme comes, maybe a cell colour or something.

Anyone else ever try anything like this?

And how to the professions do it, they must have some kind of empirical scorecard to keep track on things.

Frank?

Did it one summer a few years back. Conclusion - it was marginally more reliable to assume that today would be the same weather as yesterday than to believe the inshore waters forecast.

Interesting exercise but where does it leave you?

 

[franksingleton]

Interested, but I don't think I can make the CA then, unfortunately.

Quick question 'tho:
Is it fair to say that UK 3-5 day forecasts are generally pretty accurate, but might happen sooner or later and/or further north or south than forecast; and that the average UK yachtsman can plan and apply some adjustment on that basis?
....
.

Perhaps another time, if there is one.


Numerical weather prediction by all the major players (ECMWF, UK, NOAA, etc) are usually good at getting the general patterns right up to about 5 or 6 days. Please take a look at my lecture examples at http://weather.mailasail.com/Franks-Weather/Grib-Forecast-Examples. None is perfect. Some are better than others but they do give an idea of what NWP can and cannot do. Those are, of course the US GFS but the UK will be at least as good and ECMWF, slightly better on average.

In particular over the past 20 years or so I, as an observer/user of forecasts, have seen a marked improvement in prediction of change of weather type ie from a mobile westerly to a blocking high and settled weather. In my long past days as a senior forecaster we would never have been able to predict strong winds and snow, especially, as well as is done today and with such confidence. I used to see experienced forecaster get snow wrong 12 hours ahead.

However, detail within that general pattern may well be considerably uncertain even a day or so ahead. Whether those strong winds will occur over the Channel or be just that little further north may be in doubt in an otherwise good general pattern. When there is such uncertainty, a human forecaster will often express doubt. From time to time the BBC Weather Centre will either not give an outlook beyond a day or so or may well say that it is uncertain. Chaos rules, OK?

My personal approach, and what I always suugest, is that you should be looking to 5 or 6 days ahead, making plans, and updating on a daily basis, sometimes more often when it is really critical.

I could go on but that would mean giving you all my lecture notes and more!

 

[franksingleton]

Did it one summer a few years back. Conclusion - it was marginally more reliable to assume that today would be the same weather as yesterday than to believe the inshore waters forecast.

Interesting exercise but where does it leave you?

Would you cross the Channel on a persistence forecast? Would that tell you about the norther gales that kept you in Cherbourg when you should have been back at work?

 

[estarzinger]

The idea of comparing 5 day forecast with what happens at a point is reasonable, but there will be noise in the signal due to local effects and results will not look as good as looking at the whole patter.

That is in fact how a sailor (as opposed to a met pro) looks at and assesses forecasts accuracy. The sailor is at a point (or on a route) and wants to know what his weather is going to be. So 'local noise' is an integral part of the sailors experience of forecast accuracy.

I have to say I am puzzled by our difference of opinion on forecast accuracy. We just have not gotten the sort of accuracy you seem to be implying. I don't have to look very far for support - just look back to my last passage (Annapolis to bvi) and the models certainly did not have a good 5 day outlook on TS sean's development (that's just one most recent exmple, I could give you about one a month for our whole cruising experience).

Of course sometimes the 5 day forecast does sort of materialize, and you can display an example sequence of charts that shows that - I am surely not denying that. But you certainly can not deny that equally, there are (not infrequent ) times when the development even at 3 days is pretty completely different (in sailors terms) than the 3 day forecast. So, practically speaking, for the sailor, the need is to be able to assess the stability/confidence in the forecast, and be able to plan and position for what might happen. The sailor can use clever routing and tight windows if the forecast looks high confidence, but when it looks low confidence he/she needs to use bigger windows and position with a greater latitude for error in the systems development. I again don't have to look far - this was all demonstrated in heartbreaking human detail in Sean.

 

[franksingleton]

That is in fact how a sailor (as opposed to a met pro) looks at
........
.

First, I must point out that I am an active sailor. I was a professional meteorologist and, therefore, I have a realistic view of what forecasts can do and what they cannot do. When I give talks, I usually show a 7 day (now 8 day) GRIB forecast and compare with the actual analysed winds. See my page http://weather.mailasail.com/Franks-Weather/Grib-Forecast-Examples. As I say there, my only criterion was that the forecast should show some significant changes.

From those examples, but more from my annual cruising, 5 months a year over the past 11 years when I have been using GRIBs, I would claim that the GFS, a pretty average NWP model but the one we can all get easily, does well in terms of general patterns nowadays to about 6 days ahead.

However, and it is a big “however” for a sailor, weather is not precise. The atmosphere does not know itself to within one Beaufort force. That is the reality. That is why I said that comparing a forecast, whether GRIB or sea area, against an actual observation would introduce a lot of noise into the signal. It is a fair comparison but that fact has to be recognised.

Looking ahead and planning over about 5 or 6 days works pretty well. Back in 2001, I would have said 4 days. But, small detail, particularly when there are local topographical effects or in areas of convection can catch you out. This last year, 27 June, on passage from Ria Cedeira round the Galician coast to Viveiro, Spanish Met were saying up to F 4. Looking at the GRIBs and using a bit of commonsense, I thought that we might just get a F 6 around Capos Ortegal and Bares. In fact near Bares, the wind got up to NW F 8 and held that until down into Ria Viveiro.

I could have no criticism of the GRIBs. On the forecast, we would not have moved on from Cedeira for at least another 6 days. We knew that we would have to be in some safe haven for that time. We had left A Coruña on the 26th knowing all that. In the event, we and several others were weatherbound in Viveiro for that time. We moved on the 3 July.

I could quote many other examples of how watching the GRIBs carefully has meant that we were in the right places at the right times.

 

[theguerns]

keeping weather facts

We have been using weatheronline.co.uk for about 8 years now and have keept a check on the accuracy of its forcasts. In all this time we have found it to be 82.4% . In 2005 when the azores high collapsed it was the only weather site to give near accurate weather forcasts for the med and atlantic coast of portugal and spain. In fact the best we had was on crossing from Vivaero to Camaret it predicted a F5-6 NE for 3 days then going to NW F6-7.
We crossed in 60 hrs and arrived in Camaret having sailed all the way in a F5.

 

[franksingleton]

We have been using weatheronline.co.uk ..............

. In all this time we have found it to be 82.4% . .
.

Weatheronline now has output from several models. However, I am pretty sure that you were using and are referring to the Weatheronline GFS output. In other words anyone using Saildocs, Ugrib, zyGrib or global ie 50 km spaced data from almost any source would, in fact, have the same information.

Your figures are interesting. Without knowing how you define accuracy or measure it, I can only say that what you say ties in with my own experience of using GRIBs. Of course, there have been improvements since 2005. My comments refer to what happens now.

As a matter of interest, what other forecasts were you comparing Weatheronline with?

 

[snooks]

For work I check the weather a lot for shoots. I've found windfinder.co.uk reliable 9 times out of 10 for wind strength and direction 3 days or less away, and the day before I'll use weather.co.uk to check the cloud cover which I've found to be very accurate.

Don't use windfinder for cloud cover, or weather.co.uk for wind direction and strength

 

[franksingleton]

For work I check the weather a lot for shoots. I've found windfinder.co.uk reliable 9 times out of 10 .

I assume that you are using the Windfinder superforecast based on the GFS. They run a meso-scale model starting with the GFS as input. I do not think that they use any more detailed weather data so will not know about any pre-existing small weather features. If I am wrong, I hope someone will tell me. Nobody has done so far. I suspect that the problems of handling the amount of data to run a meso-scale model is beyond their capability.

They do use better topography. Consequently they are likely to improve on wind speeds and, maybe directions, near the coast but probably not over the open sea. I do not know the size of the area for which they run their meso-scale model but the forecasts are unlikely to be much good beyond 48 hours at most. The UK Met Office runs its NAE on the same size grid. However, since they reduced their global model grid to 25 km, the NAE is no longer regarded as operational. In other words it does no better than the global model. The GFS is not quite as good as the UK Global model and, it is possible that the Windfinder superforecast improves slightly. But, if the GFS gets the large scale pattern wrong then so will the meso-scale.

The Windfinder large scale model is the GFS ie the same as you get from zyGrib, Ugrib, Saildocs, XCWeather etc.

 

[snooks]

The Windfinder large scale model is the GFS ie the same as you get from zyGrib, Ugrib, Saildocs, XCWeather etc.

I find the presentation of the data better (for my use) so I have tended to stick with windfinder. But I find seeing what's happening on a national level handy too.

Sometimes I use the super forecast, but not always.

 

[franksingleton]

I find the presentation of the data better (for my use) so I have tended to stick with windfinder. But I find seeing what's happening on a national level handy too.

Sometimes I use the super forecast, but not always.

I think that this is much the case for many users of GRIbs or GRIB based information. They use whichever gives the most useful presentation for them.

I doubt that you would get much benefit from the superforecst. Beyond a couple of days it can, in fact be misleading.

 

[Blue5]

This last year, 27 June, on passage from Ria Cedeira round the Galician coast to Viveiro, Spanish Met were saying up to F 4. Looking at the GRIBs and using a bit of commonsense, I thought that we might just get a F 6 around Capos Ortegal and Bares. In fact near Bares, the wind got up to NW F 8 and held that until down into Ria Viveiro.

I could have no criticism of the GRIBs. On the forecast, we would not have moved on from Cedeira for at least another 6 days. We knew that we would have to be in some safe haven for that time. We had left A Coruña on the 26th knowing all that. In the event, we and several others were weatherbound in Viveiro for that time. We moved on the 3 July.

I could quote many other examples of how watching the GRIBs carefully has meant that we were in the right places at the right times.

Now that is interesting, I have just checked our log, we were on passage on 28th May from Viviero to Cedeira and the forecast was NE 4-5, we had winds of 38kts that persisted until we were into the shelter of Cedeira, initially we had a steady 20kts and it piped up instantly without warning, fortunately we were just using genoa.
Given your experience as well it is probably not an unusual occurrence

 

[Blue5]

Numerical weather prediction by all the major players (ECMWF, UK, NOAA, etc) are usually good at getting the general patterns right up to about 5 or 6 days. Please take a look at my lecture examples at http://weather.mailasail.com/Franks-Weather/Grib-Forecast-Examples. None is perfect. Some are better than others but they do give an idea of what NWP can and cannot do. Those are, of course the US GFS but the UK will be at least as good and ECMWF, slightly better on average.

I like the idea of getting an actual (grib) to compare against a forecast. Is this facility available on Zygrib or any other viewer, I seem to be struggling trying to find actuals, perhaps I am looking in the wrong place.

 

[franksingleton]

Now that is interesting, I have just checked our log, we were on passage on 28th May from Viviero to Cedeira and the forecast was NE 4-5, we had winds of 38kts that persisted until we were into the shelter of Cedeira, initially we had a steady 20kts and it piped up instantly without warning, fortunately we were just using genoa.
Given your experience as well it is probably not an unusual


occurrence

This is a case of when I say that the forecast was useful and gave good guidance.

The image below shows the 36 hour forecast from 27 May. The 12 hour on 28 May was similar.


First, even over the open sea, GRIBs will under forecast F3/4 and above by about 20% or one Force. In addition, the two Headlands, Bares and Ortegal are notably fierce and frequently cause stronger winds.

Quite often the Spanish Met service VHF makes particular mention of these two headlands. They did not on the day that we were there. What did AEMet say when you were there?

Yes, obviously I have over 10 years experience of using GRIBs and I know a fair amount about weather prediction. But, this is the kind of experience that teaches a good lesson. It illustrates what I have often said, that is that a Numerical Weather Prediction model can, at best, represent weather and topography on a scale of about 4 or 5 grid lengths. The GFS uses, in effect, a grid length of about 25 km but only gives output on a 50 km grid. It cannot deal with such local important detail. That is wThhere the human comes in. AEMet let you down but, I would argue that with some experience of using GRIBs and knowing what headlands can do, such events should not catch you by surprise - next time, at least!

There is some fairly heavy reading at http://weather.mailasail.com/Franks-Weather/Background-To-Forecasts where I try to deal with some issues regarding "accuracy."

 

[franksingleton]

I like the idea of getting an actual (grib) to compare against a forecast. Is this facility available on Zygrib or any other viewer, I seem to be struggling trying to find actuals, perhaps I am looking in the wrong place.

The best that you can do is the use the T=0. This is an analysis using the available data plus what is known as the “First guess field.”

All NWP models start with a 6 hour forecast from the last computer run with all the data used at that time. This is combined with all the data received between 6 hours previously and the new initial data time.

There may seem some circularity here but it really is very similar to what we used to do pre-computer forecasts. We used to place our new plotted chart over the previous one using a light table. The forecaster would then be looking at the latest data whilst doing a mental 6 hour forecast to get positions of fronts and centres in the right places.

The computer does the same but far more rigorously as it does a 4 dimensional analysis in that it combine the predictions from 6 hours previously at all 70 model levels and times (about 10 or 15 minute intervals) with all the new data that can be handled, also at all times and locations, globally and up to 80 km above the earth.

I hope that is clear!