Accuracy of forecasts

[franksingleton]

Yes. They are taxpayers too. Its outrageous that government agencies not only use taxpayers money but then charge the same taxpayers for what they have already bought. Contrast the free charts available in the US with UKHO charging time and again often for info from Nelson's day

There is a good argument for what you say. Unfortunately, it is not HMT's view. In fact, it goes wider than that. Our guidance used to be that sectional interests should not benefit at the taxpayer's expense. For example, the needs of the GMDSS are met in the UK, as in many other countries (often not as well) by the inshore waters and shipping forecasts. It follows that if we yachties want a more detailed, more localised service then we would have to pay. Arguments about it being in the public interest to reduce number of rescue call oiuts, the better. The user pays is the dictum. No good arguing with me about it. Go to your MP.

 

[franksingleton]

Why bother? Does anyone other than a met office type even look at confidence data? For that matter, how many members of the public would really understand what it meant anyway?

It's an academic issue maybe of interest to you and Simon but mostly not to the rest of us.

It is a practical issue. I sail less than some and more than many. I do not know hat I would do if the forecast sau 65% (or any other figure.) Would that refer to force, direction timing?

I just would know how to start devising a useful and meaningful figure. But, I retired a long time ago. Start up a pressure group and apply pressure. despite what some say, the met office does listen - sometimes.

 

[franksingleton]

Bolleaux, Mister Singleton.
..............

I will ignore the insults but hope that the Forum manager takes note of your intemperate tone...

Two points only.

I asked Bill Bailey on at least two occasions for times and dates. He could not give them although he had ample opportunity. I was genuinely interested in seeing why there was such a bad forecast. I would have looked into the facts... I have heard a number of such stories where people either did not get a forecast, got the wrong forecast or did not understand the forecast. A lesson that I learned long ago was to check facts before trying to deal with such cases. There may well not be a case to deal with, as Paul Ferris found when he interviewed me for an in-depth Observer article...

Secondly, I do not set out to defend the Met office, Météo France, AEMet or any other National Met Service. My prime objective is to inculcate some better understanding about weather and weather forecasting. I see and hear about sailors getting into trouble needlessly. If I can help to give a better understanding of weather, forecasting – especially the limitations then I will feel that I, at least, will have done something positive.

 

[simonjk]

An interesting exercise here chaps would be to be given actual dates, times and locations of when a forecast is found to be inaccurate.

Then a proper investigation can be undertaken to ascertain if the forecast was incorrect or whether it could have been the interpretation that was incorrect. That is not a defence of inaccurate forecasts by the way, but it would be interesting to see.

On the issue of confidence you are correct that it may seem irrelevant for many, although if one could be educated to understand what confidence levels are trying to sat (i.e. a 90% gale risk means that given a similar synoptic situation on 9 out of 10 occasions a gale could be expected to occur) they may prove useful?

Simon

 

[franksingleton]

An interesting exercise here chaps would be to be given actual dates, times and locations of when a forecast is found to be inaccurate.

Then a proper investigation can be undertaken to ascertain if the forecast was incorrect or whether it could have been the interpretation that was incorrect. That is not a defence of inaccurate forecasts by the way, but it would be interesting to see.
...........QUOTE]

Simon

Forecasts go wrong. You know that as well as I do. Short period forecasts are better than 30 years ago but not, maybe, too obviously because of the sheer variability of weather in space and time. Forecasts up to about 5 to 7 days ahead are distinctly better although this year in the Charente, Vendee, Brittany area they seem not to been as good as last year. Météo France inshore forecasts have been less good this year – a subjective impression, I freely admit.

I am always chary when somebody has a horrendous bad forecast story. I am not saying that they are never correct. However, I do know that, on about 10 such occasions, when I was able to follow up the story, I have found that they were simply incorrect.

One in particular sticks in my mind was a RNLI rescue off the Lizard in a F10. The yacht was on passage from the Channel Isles to Cork. I asked for transcripts of the forecasts; 24 hours before they left there was a “Perhaps gale 8 later” in the forecast. At about the time they left it was “Perhaps severe gale 9. The skipper had the gall to say that he was surprised by the wind strength. OK, it was F10 and not F9!

It is always good to look at serious errors but if you follow every hare you will get involved in much nugatory effort. I would make a plea for the Yachtie mags to give full details, dates, times of forecasts used and times of issue. Of course, like the Mike Fish hurricane, if they gave all the facts, there would not be a story.

 

[oldbilbo]

I asked Bill Bailey on at least two occasions for times and dates. He could not give them although he had ample opportunity. I was genuinely interested in seeing why there was such a bad forecast. I would have looked into the facts...

Readers will be getting bored by this continued 'ad hominem' challenging of integrity as a poor substitute for intelligent exchange. It is boorish and aggressive, may have served him well in otherwise-courteous staff meetings in Bracknell among otherwise-courteous colleagues, but it doesn't cut any ice with me.

My first encounter with the 'defender of the indefensible' was at a joint meeting of the Royal Institute of Navigation and the Royal Met Society. That very successful series of joint-symposia meetings was an idea of mine as a committee member, brought to later fruition by the RIN's Small Craft Committee and their counterparts in the RMS. I was most pleased at how the concept turned out, but rather less pleased by the aggressive, finger-poking-in-the-chest behaviour of the abrasive Mr. Singleton at one of those symposia. I decided that I had no interest in the man and his style, and had no wish to subject myself to his uninvited interrogation. There were more interesting and productive conversations - and conversationalists - to follow.

Had the man expressed interest in the aims and means of the joint article, and engaged in an ordinarily-courteous exchange - as did others - then perhaps something productive might have emerged. But as it developed, I had no interest in being poked at and 'interrogated with hostility'.

Most of us here have encountered roughy-toughy weather at sea we didn't expect. Sometimes that's due to our failures to acquire a forecast, sometimes our failures to understand a forecast. Sometimes it's just due to the limitations of the forecast. However, I learned - was taught - from wrinkled, grey-bearded old QFIs always to get a good forecast, always to consult it, and always to interpret it around my judgement of what I/we, the aircraft, and the mission requires. I still do that, with boats and the sea and, like many others here, have long considered the Shipping Forecasts and the Inshore Waters Forecasts to be less than they could be.

I'm one of those who seeks better met information, the better to interpret it. There are others like me in that respect. One of the better ways of providing this is via 'continuous VHF broadcasting' - or VOLMET. Aviators have had this for decades. Other maritime nations have had this for decades. It's really not difficult. All I hear from the Frank Singletons of this world is why they/'we' cannot.

"If you're not part of the solution, you're part of the problem."​

 

[franksingleton]

Readers will be getting bored .............
.................]

For that very reason, I will not be replying to that post.

 

[LittleSister]

I would be grateful if Mr. Bilbo would put his bees back in his bonnet.

I was very interested in the discussion about how accurate forecasts could be, and how that could be assessed. I am not interested in hearing him rehearse his gripes about something that happened long ago. Normally I wouldn't mind just skipping the bits like that that I'm not intersted in, but here it seems to be dragging a discuission that was of intereest to many, and included input from a variety of useful perspectives, to an ill-tempered halt.

On a lighter note, and following on from the comments about the value some place on an indication of how confident the forecaster is of the forecast (one of the reasons I like Simon's forecast videos), a couple of years ago the glamorous presenter of the ITV forecast claimed that the rain should have arrived in Scotland earlier but was running late!

 

[Tomahawk]

Today we are not sailing on account of yestdays forcast which was giving rain and howling winds..

Yet right now it is a mill pond ... The forecast has about three bours to come good..

 

[franksingleton]

Accuracy assessment of forecasts

I began this thread by querying MAE as a meaningful indicator of accuracy of a wind forecast in meso-scale (or any) numerical weather predictions. I can well see that it has value for wind generation where directionality is not important. I can see no value for assessment of accuracy for sailing purposes where direction can often be more important than strength. I can always shorten sail but gentlemen do not beat!

Bill Bailey referred to an article by him and Mike Brettle some years ago which compared shipping forecasts with reports from Channel LV. Like many papers about weather, it was a best endeavours at the time. However, comparisons of values at a point with an area forecast are always fraught.

In April2010, PBO published a letter from the MCA giving some figures for “accuracy” of forecasts. See http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Assessment.

They state that over a period of one year –

• The “hit rate” for gale warnings (for sea areas) was 92%
• The “hit rate” for coastal strong wind warnings 96%.
• Within a typical 24 hour period forecast, 94% of winds were within one Beaufort force of prediction.

I am not entirely sure how these data were obtained but think that they were comparisons between worded forecasts and computer analyses at a dense (4 km ?) grid. Therefore they were areal comparisons that were not then available to Bailey and Brettle. In a follow up letter to PBO, I said that these data, honestly obtained, would not accord with experience out on the water – for many of the reasons discussed in this thread.

I had hoped that the two letters would spark interest and further discussion. To my great surprise, and dismay, the PBO editor did not pick this up as worthy of discussion; neither any Yachtie journalists nor the readership.

Surely there is enough interest for someone to go back to the April 2010 letter, do some questioning and try to produce a balanced and informative article that would enlighten us all.

 

[Tomahawk]

Several points...

NO. There is no such thing as the Fish effect. You have to remember that the Met Office MUST issue a gale warning if a gale MAY occur in an area. They can only cancel a warning if they are sure that there is no gale. Inevitably, that leads to an impression of over warning..

OK I see a potential problem here..
Forecasting is done by numerical modelling on very very big computers. However due to the complexity of the weather these models are never going to be totally accurate in one go, (see above). Thus the weathermen run the model a few times using the same input data and take the average of the outputs as the published forecast..

This means that sometimes there will be a gale warning in one of the computer models, but not necessarily in the other three of four which are used to produce the final forecast.. Does the MET office always broadcast a gale warning because that one computer model includes a possible gale? If so, they will by simple averages, broadcast a lot more gale warnings than actually come to fruition..

The Fish effect?

 

[AntarcticPilot]

I began this thread by querying MAE as a meaningful indicator of accuracy of a wind forecast in meso-scale (or any) numerical weather predictions. I can well see that it has value for wind generation where directionality is not important. I can see no value for assessment of accuracy for sailing purposes where direction can often be more important than strength. I can always shorten sail but gentlemen do not beat!

Bill Bailey referred to an article by him and Mike Brettle some years ago which compared shipping forecasts with reports from Channel LV. Like many papers about weather, it was a best endeavours at the time. However, comparisons of values at a point with an area forecast are always fraught.

In April2010, PBO published a letter from the MCA giving some figures for “accuracy” of forecasts. See http://weather.mailasail.com/Franks-Weather/Forecast-Accuracy-Assessment.

They state that over a period of one year –

• The “hit rate” for gale warnings (for sea areas) was 92%
• The “hit rate” for coastal strong wind warnings 96%.
• Within a typical 24 hour period forecast, 94% of winds were within one Beaufort force of prediction.

I am not entirely sure how these data were obtained but think that they were comparisons between worded forecasts and computer analyses at a dense (4 km ?) grid. Therefore they were areal comparisons that were not then available to Bailey and Brettle. In a follow up letter to PBO, I said that these data, honestly obtained, would not accord with experience out on the water – for many of the reasons discussed in this thread.

I had hoped that the two letters would spark interest and further discussion. To my great surprise, and dismay, the PBO editor did not pick this up as worthy of discussion; neither any Yachtie journalists nor the readership.

Surely there is enough interest for someone to go back to the April 2010 letter, do some questioning and try to produce a balanced and informative article that would enlighten us all.

Dear Frank,

You raise a point that falls within my own area of expertise, and that is the validity and value of comparing phenomena that have different (but overlapping) spatial extents. It is the same problem as using census data (collected on electoral ward boundaries) with data collected on post-code boundaries. The problem is known as the Modifiable Area Unit Problem, and the problem within weather forecasting comes about because you aggregate the point results from a weather model into statements about fixed areas. Statistically, my understanding is that this is only valid if the areas into which you are aggregating the results have larger spatial extents than the systems you are characterizing.

To give an example of the sort of thing I mean, I live in a village on the edge of the Fens. The parish has an area of about 100 square miles, the population of the village is about 10,000. So, the population density is about 100 per square mile. Easy, isn't it? But if you land in the majority of that area, you'll find vanishingly low population densities - most of it is East Anglian prairie! And where it isn't extremely low, it will be 1000 or more (much more) per square mile. So, my prediction of 100 people per square mile is wrong almost everywhere, because the area I am using to predict the population density doesn't match the distribution of the population. But that figure of 100 people per square mile is correct, even if it is totally misleading!

In GIS, this is regarded as a situation to be avoided if at all possible, as the results are not always useful.

What it means in practise is that a forecast (such as the shipping Forecast) where results are aggregated over large areas will almost certainly be wrong in at least part of the area concerned, possibly quite seriously wrong. So, for example, we have a gale warning for Forties, Dogger, German Bight and Fisher. We cannot tell whether that means that the whole of the North Sea is subject to a storm, or whether is a much smaller storm affecting the point where these 4 meet! Of course, the forecast MUST say "Gale Warning for .....", even if only a small part of the areas named is affected. But the yachtsman on passage from Inverness to Norway, crossing one of the areas concerned, might well only see clear skies and fair winds!

Of course, there are other confounding issues such as the effect of local topography; this is particularly important in places like the West Coast of Scotland where the wind speed and direction can change over very short distances.

Estimating the accuracy of forecasts must take this issue into account, I think. It must often be the case that a forecaster issuing area based forecasts will knowingly give a forecast that is incorrect for part of the area concerned, because it is important that warnings are given of violent weather. And equally, a forecast without a hint of bad weather may fall down if a local topographic feature causes funnelling or other gust-inducing effects. And, "local topographic feature" in this case means anything that falls below the resolution of the model - that is, things 10s of kilometres in extent!

I'd imagine that the only way round this would be to do away with fixed sea areas, and instead issue forecasts using areas based on the current meteorological situation - but how to send out the description of the areas in media that don't allow images? Of course, the general synopsis of the Shipping Forecast attempts to do this, and if used as intended, it should at least give a clue as to the extent of high winds etc.

However, to return to my main point, it may not be easy to create a proper statistical assessment of the accuracy of area based forecasts. In the worst case, it may not be possible, except in VERY coarse terms! You have the following possibilities:

1. Forecast correct for all parts of the named area
2. Forecast correct for part of the named area
3. Forecast correct for a small part of the named area, but showing extreme conditions (knowingly incorrect)
4. Forecast fails to predict extreme conditions in a small part of the area, but otherwise correct
5. Forecast correct but timing wrong
6. Forecast completely incorrect

I'd guess that the first and last are the least likely! And, given that most yachts are making coastal passages, I'd guess that most reported cases of incorrect forecasts are related to point 4.

Just one personal observation from the Clyde, and that is that it seems to me that the forecast is much more likely to be incorrect when conditions are atypical. For quite a lot of this summer, we've had easterly winds up there (which are normally rare), and the forecast has often been off in its timing and wind strength.

 

[franksingleton]

OK I see a potential problem here..
............

If you use the GFS GRIB forecasts you will know that the models will often underestimate the wind. The forecasters have to use judgement and experience both of the models and the weather. The time has not yet come when we can rely on output from a computer model. We are in the Vendée just now and look every day at the Météo France computer generatd forecasts at http://france.meteofrance.com/france/mer?MER_PORTLET.path=merprevisionville/8519451. They are not very good.

When it comes to putting fingers to keyboards, forecasters take the model output as guidance. The 24 model ensembles come up with results that may be significantly different or rather similar. The decision on what actually goes into the forecast and the issuing of gale warnings are human ones. In my ex[eminence as a user, I would always back the relevant National Met Service for the GMDSS type of forecast and a human, Simon K or a NMS forecaster if I wanted local detail; probably Simon for preference as he will be more focused. At present, d not recommend a computer product.

As regards the Dish effect, on a number of occasions, I have asked the Met Office person responsible for liaison with the MCA for reassurance that there is no policy or intent by individuals to over forecast strong winds. To do so would be counter-productive. In my time as a Senior Forecaster, I would not have allowed it. One of the Met office performance measures was, and probably still, relates to issue of gale warnings. It is probably published somewhere.

Dear old Mike Fish was desperately unlucky. As the BBC said, if they, showed repeats of the whole broadcast there would not be a story. To hell with the facts! It is unlikely that such a forecast error will occur again, but nobody would give a guarantee.

 

[franksingleton]

Dear Frank,

You raise a point that falls within my own area of expertise, ..................

Thank you. All very well put – although I guess that I am replying to someone with much experience of explain difficult issues to even more difficult students?

You are quite correct that fixed sea areas or stretches of coast are straitjackets. Opinion varies about the best way to deal with large areas with what is so often a moving and changing weather pattern. Looking at High Seas and offshore forecasts from the US, I find them difficult and complex to understand. One of my posts was at HQ Bomber Command where we were forecasting high level winds for V-Bombers. We had to divide an area up into areas such that each aircraft in each area would be putting the same wind data into their navigation systems. It was not easy to finish up with a sensible number of areas in which winds would be reasonably homogeneous. And that was for levels where there as little effect of topography.

The fixed areas system used throughout Europe and elsewhere has the merit of simplicity but at a cost. Anyone with half a mind can always find reasons to criticise.

For optimum use, the sailor has to take a forecast as a broad and general description. I often us the term broad brush. Interpretation will be helped by

1. Use of synoptic charts – although these, too, are “broad brush.”

2. Use of GRIBs. We really only have the IS GFS easily available. At some stage, I think that European countries will have to follow suit.

3. Experience and commonsense. If you sail in an area, you will learn the various types of local effects. You are unlikely to get them right, at least not all the time, but that will help you to have an idea about possible differences from the broad overall picture.

National Met Services take verification and monitoring of both their models and their forecast very seriously. They have to do so, partly to ensure that forecasts really are improving; partly to check what effects changes make to the models; partly because many forecasts, especially for marine, aviation and mountaineering use have safety connotations.

 

[simonjk]

This thread is excellent and shows the quality of the expertise and willingness to feedback at YBW.com (thought that needed saying).

Fascinating stuff AntraticPilot and something I need to read again and go through. I did undertake some research as part of my PhD along these lines, but you described the problem much better than I have!

As Frank says, computer models can't yet take over from humans, although I must admit they are very good and improving all the time. But I do still believe the human forecaster has 'the edge' (I would say that wouldn't I)?

I'd urge you all to take a look at, and use GRIB data (although many of you probably already do, even unknowlingly, especially if you use a website such as Weatheronline, Passage weather etc... as these derived their data from the GFS GRIB output).

Then by using human sourced forecasts such as the Shipping and Inshore waters forecasts, perhaps even the text of my Sailing Weather Information Service, you can contrast and compare and will soon get used to spotting situations when GRIB will underperform.

When I was writing my GRIB for Sailors book I prepared a presentation about GRIB and how to use it with the intention of extending it to a full Weather School. I could condense it into an hours webinar if anyone thinks it could be of use?

Simon

 

[franksingleton]

.........

...........But I do still believe the human forecaster has 'the edge' (I would say that wouldn't I)?.........

As someone who started in the business pre NWP, I cannot agree that the human has the edge. Although we were issuing forecast charts for up to Day 3, they were not very good. We had no obligation to provide worded outlooks beyond Day 2. To talk about a week ahead was unknown except in the most general terms in a monthly outlook and they were hit and miss. For short period forecasts one has only to look at snow forecasts in the last three winters. Never perfect but far better than we ever achieved 30 years ago.

The human can often improve or build on the computer models. He would be lost without them.

 

[dt4134]

"Accuracy implies precision but precision does not imply accuracy". In other words you need precision to be accurate but you can be as precise as you like and still wildly inaccurate.

The forecasters linked in the OP are making definite claims to accuracy and precision and it should not be too onerous to measure those precise forecasts against their claims. The original poster expects them to be inaccurate. He's undoubtedly right, IMHO, and should be able to demonstrate that if he feels so inclined.

(From memory as I don't have the original source to hand) FitzRoy's original met service was closed down because Parliament didn't consider his port by port forecasts to be sufficiently accurate. Apparently if he'd used modern sea areas his 'accuracy' would have improved substantially.

The Met Office's forecast doesn't even attempt a reasonable level of precision and therefore I'd argue that they don't even attempt a reasonable level of accuracy.

The most irritating thing is the vagueness in the timing. It makes a great deal of difference whether a forecasted gale arrives about 6pm or 3am the next morning, but the term 'later' is so imprecisely defined as in many cases it could mean either. In quite a few circumstances it makes the forecast as much use as a chocolate fireguard.

At least UGRIB gives a more precise forecast both in position, timing and wind strength. An experienced yachtsman should be able to add their own correcting factors. It also gives a clear picture on which to base an evaluation of the observed conditions against the forecast. So they can at least guesstimate whether that "perhaps gale 8" is really going to be a 10, or whether the perhaps 5 (usually meaning probably no more than F4) is really going to be a 7.

 

[dt4134]

An interesting exercise here chaps would be to be given actual dates, times and locations of when a forecast is found to be inaccurate.

I'va fair few memories of inaccurate forecasts but only one where I can give enough detail now for it to be checked.

25th June 2011, northern Cardigan Bay (Barmouth to Bardsey Sd) from about 3-4pm through till about 7-8pm.

The forecast we got was read out by a member of HMCoastguard in person and we were expecting F5 (I have a recollection 17- 18 knots being mentioned) and we actually got about 30-35 knots and gusts of much more. It did drop substantially by midnight.

 

[Tomahawk]

Today we are not sailing on account of yestdays forcast which was giving rain and howling winds..

Yet right now it is a mill pond ... The forecast has about three bours to come good..

So to put forecasts into the context of usability..
I didn't go racing today because yesterday evening the weather forecast was all howling winds and lashing rain.. and as usual the presenters went heavy on the rain and unpleasant parts..

Today in E England.. the morning was really quite pleasant. It began to cloud over about lunchtime and we did get a bit of a lashing early afternoon.. By which time we would have been supping in the bar.. it is now overcast and spotting with rain.. I could have enjoyed a very good sail and stayed dry..

True it did rain somewhere else like Wales where it always rains Etc.. but that is no use to people where we are..

Should I believe the next forecast or will it be wrong again..? then what happens if one day they will cry wolf and yes it will actually be a storm..

 

[Blue5]

I'va fair few memories of inaccurate forecasts but only one where I can give enough detail now for it to be checked.

25th June 2011, northern Cardigan Bay (Barmouth to Bardsey Sd) from about 3-4pm through till about 7-8pm.

The forecast we got was read out by a member of HMCoastguard in person and we were expecting F5 (I have a recollection 17- 18 knots being mentioned) and we actually got about 30-35 knots and gusts of much more. It did drop substantially by midnight.

Without wishing to sound sad or smug we tend to write the forecast and source in the boat log although I can't claim to include the issue time...............