Why is the Met Office weather forecast wrong?

[franksingleton]

I guess it would depend on the model. As you said, AI models don’t “understand” anything but yes perhaps they could extrapolate enough from what’s happened if given all of the data.
I think my reply was a bit of a knee jerk “but climate change is new” but I guess in the scheme of things what I said about the physics also applies and the AI would have data with rising CO2 and patterns changing. Whether it could extrapolate out to uninhabitable tropics and massive storms elsewhere I don’t know as there are no stats for it to lean on but in theory those are all just rules playing out if you zoom out enough.

I was really just reflecting thst it is an odd world. There is a strong correlation between CO2 and global temperature. The scientific approach is to say that that suggests a physical connection. They want to know why. Therefore, they study the physical processes, develop a model, show why the statistics produce the observed result. From that, given CO2 emission scenarios, they predict outcomes at various future times. Some people refuse to accept the results.

AI can take the same raw data. It could work just with observed temperatures in the observation record with CO2 emissions for the same period. The correlation is so strong that AI will recognise the link between CO2 and temperature. AI, if asked to do so, could produce forecasts at future times. Given the strong correlation there can be no reason why AI will not come up with a result showing continued warming. Of course, I go not know how close those will be to climate models, but it is bound to show continued warming. Because it is an AI result many people will accept the result. I am very happy that they should do so but cannot help reflecting that many people will more readily accept numbers coming out of a black box than from a scientific study that is based on understanding of the data.

It would be interesting to let AI loose on all the data available back to several million years ago. It could go back to when CO2 was higher than current values but decreasing to the values that we had several 100,000 years ago. I suppose that a weakness of AI for climate prediction is that it can only work on actual CO2, temperatures and emission data. I suppose that it could be fed withm future emission and mitigation strategies.

 

[AntarcticPilot]

I have no doubt that AI will produce some very clever results. Language is certainly a great challenge. Weather prediction is a very different problem. The number of variables and the complexity of the atmosphere are, I believe, beyond the capability of AI. For example, AI might be able to identify two days with exactly the same weather conditions. But, if they are not on precisely the same day of the year, then the input from the sun will be different and the two situations will have different outcomes. The complexities of language are enormous but lack the fluid physical interactions that drive the atmosphere.

There is also the issue of mathematical chaos, first observed in weather forecasting but found to be common in many environmental problems. Tiny changes below the limits of observation can cause vast changes in the outcome (Lorenz first noted this - see https://en.wikipedia.org/wiki/Chaos_theory#History). Very similar historical data can lead to vastly different outcomes, even if every possible observation is taken into account. @franksingleton and @JumbleDuck and I have had fun discussions of this!

 

[boomerangben]

On another part of the system, how much of the “inaccurate weather forecasts” is actually the user’s perception of the forecast being misaligned from the forecasters definitIon. Take for example in aviation, the maximum visibility that is forecast is 10km or more. Which sounds epic, but might only be 5nm which is pretty pants really

 

[lustyd]

On another part of the system, how much of the “inaccurate weather forecasts” is actually the user’s perception of the forecast being misaligned from the forecasters definitIon. Take for example in aviation, the maximum visibility that is forecast is 10km or more. Which sounds epic, but might only be 5nm which is pretty pants really

With sailing I find a lot of people forget that forecast wind might be 1 Beaufort force different to apparent wind even if it’s correct due to boat speed and tide. Most of us remember the gusts rather than wind too

 

[franksingleton]

On another part of the system, how much of the “inaccurate weather forecasts” is actually the user’s perception of the forecast being misaligned from the forecasters definitIon. Take for example in aviation, the maximum visibility that is forecast is 10km or more. Which sounds epic, but might only be 5nm which is pretty pants really

If you are talking about sea area or coastal forecasts broadcast by the BBC, HMCG, NAVTEX bear in mind the strict word limits. The BBC Shipping forecast has io be capable of being read in 3 minutes, 330 words for the whole broadcast. It has to be vey broad brush. It has to mention any severe weather possible somewhere in each sea area. Inevitably, it will be read as over-forecasting by some. NAVTEX slots impose similar restrictions. HMCG might not put word limits on texts but do want the same levels of brevity.
A gale warning MUST be issued if gale F8 MIGHT occur somewhere in a sea area. A gale warning can only be cancelled or allowed to lapse if the forecaster is sure that there are no gales existing or expected within 12 hours. A gale warning is a forecast. A cancellation is a statement of fact.

 

[franksingleton]

With sailing I find a lot of people forget that forecast wind might be 1 Beaufort force different to apparent wind even if it’s correct due to boat speed and tide. Most of us remember the gusts rather than wind too

A forecast saying F5 or 6, locally 7, perhaps F8 at times is perfectly sensible. Looking for ensemble examples for my Reeds WHB, 3rd edition, I looked at an AROME 16 member ensemble. For one point at 18 hours, the deterministic forecast was F6. The ensemble had speeds up to middle of F8 and down to bottom of F5. I am sure that any serious sailor will have experienced such events. Yet, I see (armchair?) sailors complaining.

 

[alan_d]

I blame that pesky butterfly always stirring storms, the over dressed moth.

Yes, the problem may be attaching sensors to the wings of a sufficient number of butterflies to feed the number-crunchers the data they need.

 

[Dutch01527]

I subscribed to the Windy pro app because it gave the option to see the forecasts of all the major weather services. It does that but the range of forecasts is massive. For example in Dartmouth next Tuesday the range of wind forecasts ranges from 3.7knts to 27knts - useless.

The conclusion I have reached is beyond 24 hours out the professional forecasts really do not have a clue. I suspect that weather is too complex for computer models and that is what the weather services are relying on. Asking a farmer or fisherman is probably the most accurate way.

 

[franksingleton]

I subscribed to the Windy pro app because it gave the option to see the forecasts of all the major weather services. It does that but the range of forecasts is massive. For example in Dartmouth next Tuesday the range of wind forecasts ranges from 3.7knts to 27knts - useless.

The conclusion I have reached is beyond 24 hours out the professional forecasts really do not have a clue. I suspect that weather is too complex for computer models and that is what the weather services are relying on. Asking a farmer or fisherman is probably the most accurate way.

Have you ever actually made a cross Channel passage or sailed for a few hours along the coast? If you have then you will know how much the wind actually does vary in space and time.

 

[Sandy]

With sailing I find a lot of people forget that forecast wind might be 1 Beaufort force different to apparent wind even if it’s correct due to boat speed and tide. Most of us remember the gusts rather than wind too

I had a fascinating chat with one of the people who make up prepare the Shipping Forecast at the Sailing and Weather Conference at Greenwich a few years back, the discussion was, 'Why does the Meteo France forecast reflect that I actually experience?' His reply was extremely informative - we need to forecast the maximums and minimums in any given sea/inshore area.

To this day I have borne this in mind.

p.s. A visit to the Met Office is well worth doing. If you can arrange one then it is well worth the trip to Exeter.

 

[franksingleton]

I had a fascinating chat with one of the people who make up prepare the Shipping Forecast at the Sailing and Weather Conference at Greenwich a few years back, the discussion was, 'Why does the Meteo France forecast reflect that I actually experience?' His reply was extremely informative - we need to forecast the maximums and minimums in any given sea/inshore area.

To this day I have borne this in mind.

p.s. A visit to the Met Office is well worth doing. If you can arrange one then it is well worth the trip to Exeter.

He was not quite correct. The shipping forecast must warn of the worst that might happen, while giving a broad general description of the weather over all.

Also, it has a strict word limit of 330 word, not counting the phrase, “Issued by the Met Office on behalf of the Maritime and Coastguard Agency.” That word limit is a two edged sword. It makes it difficult to write but it does help to focus the mind of the writer on the essentials. Many years ago they tried to automate its writing but failed. I wonder whether AI has yet been tried.

 

[boomerangben]

I subscribed to the Windy pro app because it gave the option to see the forecasts of all the major weather services. It does that but the range of forecasts is massive. For example in Dartmouth next Tuesday the range of wind forecasts ranges from 3.7knts to 27knts - useless.

The conclusion I have reached is beyond 24 hours out the professional forecasts really do not have a clue. I suspect that weather is too complex for computer models and that is what the weather services are relying on. Asking a farmer or fisherman is probably the most accurate way.

I think it is as much our expectation of forecasts that is at fault rather than the forecasters. Wind in the coastal context is hugely complex. For instance, I sail in the Minch and know full well that if I sail past the entrance to a sea loch, the wind will be funnelled in that small region requiring a reef, when sail another mile and I can shake it out. Forecasting that is impossible. Local knowledge and understanding/imagining the effects on topography is key.

As for long term forecasts - I think you are being too harsh. In my experience as a professional aviator, forecasts out to 3 days are pretty good. But once again localised effects can mean a relatively small area can get wayward weather

 

[lustyd]

For instance, I sail in the Minch and know full well that if I sail past the entrance to a sea loch, the wind will be funnelled in that small region requiring a reef, when sail another mile and I can shake it out. Forecasting that is impossible

That's not a forecast though, that's a local effect. We can easily model that stuff based on a forecast but nobody has because it's not financially viable - you're not going to pay for that information, and there are maybe 100 people interested in that area. It might be viable in the Solent with 10,000 racers wanting more detail, but realistically that's not going to pay enough to make a business out of it.
In theory we do now have sufficiently detailed information available, much of it for free, to automate the process. We have good topographic maps of the coastline in fine detail, as well as the seabed. We know what direction everything faces and can model the sun and tide effects so taking a weather report and translating to local effects could be done.

And then people would still think it's wrong because they're sailing at 7kt into the wind and don't understand apparent wind.

 

[franksingleton]

There is much truth in the last two posts. From my perspective as both a user and one-time provider of forecasts, it is always rather more shaded. Yes, small scale topographical effects can be modelled but they can be critically dependent on the large scale pattern. A small error or change in the large scale can have marked local effects. Obviously, there are costs involved and organisers of a large event might be able to get short term highly detailed forecasts but I would not guarantee total success. Small weather features have short lifetimes. To have any realistic chance of success in the short term necessitates hourly updates. Many so-called detailed forecasts are run too infrequently to be useful. A further point to bear in mind is the smoothing necessary in all NWP models. Effective resolution is about 5 grid lengths.

For domestically critical decisions, I make great use of the Met Office app. We live near the Surrey/Hants/Berks border. For us the app give good guidance with near uncanny accuracy 4 or 5 days ahead. Because we do not now drive, I sometimes have to organise transport for medical appointments. I have just done so for Monday morning and, probably could have done so yesterday.

For longer term planning when cruising, I rely mostly on the GFS with sideways looks at ECMWF. Sometimes for the next 7 or 8 days ahead, the forecasts are so consistent day to day that we are pretty sure that we can plan ahead with confidence. On other occasions that day to day comparison is so variable that we either wait where we are or make other plans. To me the ability to know whether or not we will be crossing the Cannel several days ahead is the biggest change in weather prediction. That was not possible 30 years ago. 20 to 25 years ago I was getting more confident at that kind of planning. Over the past 10 years, we have rarely got it wrong. And that includes some fairly critical decisions.

 

[lustyd]

Many so-called detailed forecasts are run too infrequently to be useful

I think this will start to change eventually. I just checked and you can now rent a single virtual machine with over 300 TFLOPS of compute for $4.59/hour, and this will more than double when the new hardware lands this year as well as dropping in price. To put that in perspective the MET XC40 was only 14 PFLOPS (so about 46 times more powerful). I think it won't be long before these things do become financially viable to run more frequently, and possibly even on real time readings. Whether anyone will bother, I don't know, but when compute is this cheap and available we see a lot of cool things start to happen.

 

[franksingleton]

I think this will start to change eventually. I just checked and you can now rent a single virtual machine with over 300 TFLOPS of compute for $4.59/hour, and this will more than double when the new hardware lands this year as well as dropping in price. To put that in perspective the MET XC40 was only 14 PFLOPS (so about 46 times more powerful). I think it won't be long before these things do become financially viable to run more frequently, and possibly even on real time readings. Whether anyone will bother, I don't know, but when compute is this cheap and available we see a lot of cool things start to happen.

Do you know who is running AI weather models?

The only one I know right now is ECMWF. They used 40 years of their own historical data, ERA5, for the learning process and their own, IFS data analyses for their trial runs. They are still only replicating a subset of their operational data but that is only a matter of time. The big question will come when they try to use the raw data. Do you know if anyone has tried that yet?

As with conventional NWP, any Tom, Dick or Harry can get hold of global model coding and limited area models, such as WRF. It is quite another matter populating those models. Global models are not a great problem, GFS, ICON and, probably, other data are readily available at model grid points. Limited area models require fine scale data and, as far as I know, people such as PredictWind, OpenSkiron, Theyr.com etc cannot handle and process the data necessary. The U.K. and other Met services have access to and can process radar and detailed satellite data.

There is more to the problem than having massive processing power.

 

[lustyd]

Do you know who is running AI weather models?

Lots of folk but it’s early days so I wouldn’t expect much for a while. We partner with a bunch of weather services around the world, I suspect partly because building supercomputers is less viable than it once was. We also work with research orgs and do some stuff internally. I wouldn’t expect a traditional weather service to crack this, as with many breakthroughs the incumbents tend to be too reserved and being frank I don’t think they have the budget to crack the problem in this way. Much like OS would have laughed in your face if you’d suggested Google street view in 2000, then Google bought three cars, some GPS and some cameras and drove down every street. Same with LLMs, it took a leap of faith to point the worlds largest cluster at the worlds largest data set and cross your fingers for six months while it processes.

There is more to the problem than having massive processing power.

Indeed, I was just highlighting how cheap the power is getting which will allow running localised models much more often if someone decides to. What seemed absurdly expensive 5 years ago is now entirely within reach. Since you said they are run too infrequently to be useful I thought it worth mentioning that that almost certainly will change in the near future.

 

[Dutch01527]

Have you ever actually made a cross Channel passage or sailed for a few hours along the coast? If you have then you will know how much the wind actually does vary in space and time.

80,000+ sea miles so the answer is yes. The point was that the range of forecasts for the same place and time between different for forcasting services is massive. . I do not understand your “ space and time” comment. Please explain.

 

[franksingleton]

80,000+ sea miles so the answer is yes. The point was that the range of forecasts for the same place and time between different for forcasting services is massive. . I do not understand your “ space and time” comment. Please explain.

My 60,000+ falls a bit short, I admit.
The forecaster has very few words to use in any forecast for it to be understandable. There is no way in which many forecasts can accurately cover a 24 hour period for a sea or coastal area. I am sure that you are well aware that the wind is often far more variable than the forecast says.
As regards forecasters not having a clue, a band of rain is currently forecast to cross England on Monday. I have little doubt it will. I could have made that statement last Thursday. For my domestic planning that app rarely lets me down seriously.
Your photo from Winy compares chalk and cheese. It is one of Windy’s sillier presentations. There are 5 or 6 forecasts. Only two, ECMWF and the GFS, are global models that have much relevance that far ahead. Usually, they are very similar. If they differ markedly. It tells me that the forecast at that time is uncertain 6 days ahead. That happens from time to time as I have often said.