Just how accurate is your favourite weather forecast model?

[GHA]

I strongly suspect that no one has much of of a clue

The images below are from the past little while, top one is GFS from wind guru, middle is recorded data from an unknown site nearby from windfinder and bottom from predictwind.

But how accurate is the recorded data? It's a few miles down the coast other side of Portimao and looks like it might be on a lighthouse on a cliff which would most likely read a lot higher than in an anchorage over in Alvor.

Interestingly predictwinds own model PWG based on GFS was probably the closest to the recorded data.

That said I've found their 2 models to be very sensitive and seem to sometimes go erratic quickly so tend to go with GFS as the most likely.

That said again.. I've just signed up for another year on predict wind as the display is so good, easy to see if the 4 models they offer agree which is probably a good sign.

And in general, they were all accurate enough - it was really flippin windy at 6am this morning on the hook trying to tie of the wind gen which somehow wasn't connected to the batteries anymore and had got up to a few thousand RPM


SO, ... massive amount of work involved recording data and somehow matching it to reliable live data - has anyone had a go at this?
Well past the capabilities of us mere mortals I reckon. To anyone who says "I use blahdeblah website cos it's really accurate" - you ain't got a clue . IMHO.

SO I'll stick to trusting GFS most initially but still with a fair degree of caution. And cross referenced with the synoptics.

Thoughts??

 

[lw395]

I think the Met Office has got wore since I've moved closer to Exeter?
For a 'tomorrow' forecast, XCWeather is better than the BBC or Metoffice.

For say 5 days ahead, I used to look at Wetterzentrale.de and if all the main players agreed, they were probably going to be about right.
The one which IMHO carried the most weight was NOGAPS.
These days I don't usually need to plan more than a day or two ahead, so I don't.

 

[GHA]

I think the Met Office has got wore since I've moved closer to Exeter?
For a 'tomorrow' forecast, XCWeather is better than the BBC or Metoffice..

But without doing a substantial load of logging and research - how do you know? Or if you did could you upload it for the rest of us to have a look at?


(PS - on a sailing forum we're talking about wind here of course)

 

[alahol2]

The forecast I have most grown to trust is Windfinder Superforecast. Like most people I only go by impressions and feelings of it being correct most of the time. I can't ever remember being nastily surprised by wind/weather that it hasn't forecast which is more than can be said for most.

 

[KellysEye]

What I have found that every different forecast, such as the Met office, Windfinder and Windguru are diffferent. Accuracy of any them I estimate to be 60%.

 

[GHA]

What I have found that every different forecast, such as the Met office, Windfinder and Windguru are diffferent. Accuracy of any them I estimate to be 60%.

Unless you pay then Windguru and Windfinder default at looking exactly at the same gfs model data. Good example of how bad we are at any kind of objective analysis when 2 different ways of presenting the same numbers seem different.

 

[alahol2]

Unless you pay then Windguru and Windfinder default at looking exactly at the same gfs model data. Good example of how bad we are at any kind of objective analysis when 2 different ways of presenting the same numbers seem different.

I don't think that is true. Windfinder seems to use current data (not delayed). The time difference in base data may well provide different forecasts.
Windfinder also state...

Forecast and Superforecast are based on different physical models and therefore may cause divergent predictions. Due to its higher horizontal resolution the Superforecast should be more accurate, especially for locations with a complex topography and local thermal effects

 

[lw395]

But without doing a substantial load of logging and research - how do you know? Or if you did could you upload it for the rest of us to have a look at?


(PS - on a sailing forum we're talking about wind here of course)

I haven't kept a rigorous log, but have compared the accuracy of BBC and XC over the last year.
XC has been consistently more accurate on the big picture of tomorrow's weather.
Wind speed and direction and whether it's raining or not.
The BBC often says roughly the right thing, but the timing is more likely to be wrong. E.g. a change of wind and rain or shine not in the right 3 hour window.

A few years ago I needed to plan the weekend from about Monday evening, I spent a few months writing down the forecasts as given Monday 6pm and as forecast Friday morning. Compare with what actually happened Sat an Sun. Looking for the big picture, e.g. is it worth entering a dinghy race weekend, don't want a flat calm or a hoolie.
Nogaps/Navgem seemed most reliable.

My feeling is that if several sources agree, you can have more confidence, but you want actual different weather models, not just xc and windguru processing GFS with different colouring-in.

 

[BigJoe]

Like most people we use several weather forecasting models, some mentioned above, simultaneously to get the best overall picture, however we now use Weather Track as our "go to" service.

Its uses GSF model, however, the BIG plus for us is you actually download the data, its takes only a second or two, then it can be viewed offline, or if you loose 3g or Wi-Fi signal.

The only downside is, as far as I'm aware it can only be used on smart phone of tablet.

 

[Tomahawk]

That is like asking which is the best anchor? They are different and some work better in some places than others in others.... me I prefer a Rocna

And I never trust the BeeB

 

[Poignard]

That is like asking which is the best anchor? They are different and some work better in some places than others in others.... me I prefer a Rocna

And I never trust the BeeB

I haven't seen any test reports on the BeeB. How does it perform when dragged across a beach by a LandRover?

 

[Moonbeam]

Like most people we use several weather forecasting models, some mentioned above, simultaneously to get the best overall picture, however we now use Weather Track as our "go to" service.

Its uses GSF model, however, the BIG plus for us is you actually download the data, its takes only a second or two, then it can be viewed offline, or if you loose 3g or Wi-Fi signal.

The only downside is, as far as I'm aware it can only be used on smart phone of tablet.

Weather Track looks like another GRIB file viewer (of which there are many), but only for apple?

For local sailing around Devon, I've also found Windfinder to be a pretty good interpretation of the GFS data. I think the trick is to learn the quirks of any given app and you start to get a good feel for what the real weather will be in comparison to the forecast. I find Windfinder to be good on both sea state and wind (gives good 'gust' data), but always overly optimistic on sunshine. Conversely, Metoffice is terrible for wind, but pretty accurate on sunshine and rain!

 

[Lucky Duck]

I find Windfinder to be good on both sea state and wind (gives good 'gust' data), but always overly optimistic on sunshine. Conversely, Metoffice is terrible for wind, but pretty accurate on sunshine and rain!

This

I have generally looked to sources 'land based' such as the BBC website for whether its going to rain or be sunny and the likes of Windfinder for details on the wind.

 

[Sandy]

As I live near Exeter and saw that the Royal Institute of Navigation and Royal Meteorological Society were having a day conference on "Sailing and Weather" last October. I popped along and attended the day. Outside of the main speakers an interesting discussion developed about how weather was presented to everybody. The conclusions that were drawn was that you need to have several sources and be able to did into the depths of web sites to get the detail that us, as specialist users, need. Personally, I find MeteoFrance provides me with information that is easy to read and needs less processing to understand.

 

[Moonbeam]

The conclusions that were drawn was that you need to have several sources and be able to did into the depths of web sites to get the detail that us, as specialist users, need.

Very true. Most of the apps/web forecast presentations are considerably dumbed down to make them accessible to all. As you say, we, as 'specialist users,' often require more depth. Take XC Weather for instance... this app was designed for pilots of sail planes, paragliders and hang gliders. However, the truth is that anyone of those users that seriously intends to go cross country flying will have to find and study much more detail such as lapse rates, convergence lines etc etc than is presented on XC weather.

I find the experts charts (look down the left hand side under weather charts) here http://www.weatheronline.co.uk/ to be very useful for getting the bigger picture.

 

[Neil_Y]

I've always compared a few sources, but my favourite for long term or past 2 days forecast is Yr.no

They don't say which model they use but the data comes from...
Weather forecasts on yr.no are based on data provided by Norwegian Meteorological Institute and its international partner institutions such as European Centre for Medium-Range Weather Forecasts (ECMWF) and European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT).

 

[GHA]

Has anyone actually tried logging and comparing with live data to get a grip on accuracy then?

Without a load of work, your favourite may well just be wishful thinking....

 

[lpdsn]

Has anyone actually tried logging and comparing with live data to get a grip on accuracy then?

Without a load of work, your favourite may well just be wishful thinking....

There's a lot to be said for that. Maybe the forecasts could be downloaded every 6 hours then compared to somewhere that will give you the actuals. I think you would need to be able to use an offshore met buoy or lightship to be able to compare it with the forecast for a particular position interpolated from forecast data. Anything too close to land and the variations between actuals and forecast would be masked by variations due to topology. Maybe use something like M2 or the Channel Light vessel, depending upon whether it is easier to download UK or Irish data. XCWeather certainly manages to intepolate the cell positions of the GFS to get a forecast at these locations - I presume it doesn't use a particulrly sophisticated alogorithm.

Intuitively, so without systematically collected data, my view is that the GFS 0.25 is pretty accurate these days for a good few days out. Obvious proviso that if it isn't settling down between successive runs of the model then it can't be trusted that much for that particular period.

I back that up by comparing it with the ECMWF and HIRLAM models on the Met Eireann website. I'm not convinced by the ECMWF. It does seem to under forecast winds (again gut feel based on my experience not systematically collected data) more often than the GFS 0.25 does.

 

[Mataji]

Has anyone actually tried logging and comparing with live data to get a grip on accuracy then?

Without a load of work, your favourite may well just be wishful thinking....

Would there be any point in doing this. The Met Office inputs live data into a massive computer which then runs this data against lots of past scenarios and comes up with predictions of probable outcomes. None of us could hope to log the amount of data they use. As anyone who sets sail knows, you can never really trust any forecast.

 

[jimbaerselman]

I think the question slightly misses the point.

Forecasts are estimates of probable outcomes. So the question is not really about accuracy . . . it's about the probability of being within a predicted range of outcomes.

The Beeb are actually quite good at this. They're clear about the level of reliablity of their forecasts. This level of reliablity depends on whether you're in a band of unsettled weather with rapidly moving fronts and depressions. Or sitting in a big, slow moving jet stream loop - with little likely change for several days.

All forecasts are inaccurate. The degree of inaccuracy depends how far ahead you're looking, and what climatic band you're in; trade winds, tropic thunderstorms, arctic high, or frontal zones with their ever changing streams of depressions and highs.

One good way to test the probability of certain outcomes is to look at sites which allow you to compare a range of models, looking (say) 3 or 5days ahead. One such site is windytyv.com.

Go to the bottom right of the screen, and your choice lies between ECMWF, GFS and NEMS (rest your cursor on whichever is showing, and you see the choice). If all the models for a particular time and date look the same, your forecast will be reliable. As the differences increase, your forecast will be less reliable - thus, less likely to be accurate.

Overlaying this observation are local effects. High terrain or diurnal heating or freezing will create strong local wind variations from forecasts.

So I take three views. If they agree, I'm confident that when I allow for local effects (if I know them!) I know what's coming.

If they don't agree, I know to allow for a range of outcomes - sometimes sufficient to render local knowledge irrelevant.

JimB