Why is the Met Office weather forecast wrong?

[LittleSister]

And is it?

Every Tuesday the Met Office publishes on YouTube a 'Deep Dive' - a 20-ish minute discursive and detailed look at the emerging weather for the next week or so, delving into and explaining some interesting/obscure aspects of the current situation or particular weather features or forecasting challenges.

This week's episode is spent answering a query sent in by a viewer asking why the 'Deep Dive' forecast from a couple of weeks ago was 'wrong'. I thought this might be of general interest, but especially for those who think the Met Office or forecasting more generally is of poor quality.

(Note also the availability of another long-format but more focused '10 Day Trend' forecast published on YouTube by the Met Office on Wednesdays.)

 

[Sandy]

I am really enjoying the Met Office YouTube channel. Some really good content and loved how they highlighted the Andrew Neil tweet.

Not so sure about the choice of shirt!

 

[franksingleton]

As ever journalists and the general public have faulty memories.Very many years ago, a journalist presented me with 5 or 6 “wrong” forecasts. During our discussion, I gave the dates to an assistant and asked for the transcripts of our forecasts. None of the “wrong” forecasts was what we had said.

The Deep Dive man was interesting in emphasise the fine margins berween a good forecast and a poor one.

As I have said before, there are times when the weather is less predictable than at others. That is why, when I am making cruising decisions, I look for forecasts from one source that are consistent from one day to the next. Comparing different models on the same day does not tell you much. It is too small an ensemble.

 

[lustyd]

I look for forecasts from one source that are consistent from one day to the next

Blimey if you find one like that at the moment let us know, they all seem to be swinging wildly at the moment and often seem to change based on time of day as if the model doesn't understand day and night on the input side. It took me a while to realise that my partner always checked weather in the evening and thought it looked good, while I checked in the morning and thought it looked downright dangerous!
Fingers crossed that with advances in processing capability and scale of AI models we'll start to see some real improvements over the next few years, although it might take some vision to change the way the models are created and even more vision to spend the money on a much, much larger model as we've seen with LLMs and similar over the last year.

 

[capnsensible]

I wonder sometimes if the xbox generation expect too much from forecasting accuracy. No matter what revelations there are to come in 1s and 0s, nature is, I reckon, far too random to predict in the way that humans dream of.

Plus it helps to look out of the window now and then!

 

[franksingleton]

Blimey if you find one like that at the moment let us know, they all seem to be swinging wildly at the moment and often seem to change based on time of day as if the model doesn't understand day and night on the input side. It took me a while to realise that my partner always checked weather in the evening and thought it looked good, while I checked in the morning and thought it looked downright dangerous!
Fingers crossed that with advances in processing capability and scale of AI models we'll start to see some real improvements over the next few years, although it might take some vision to change the way the models are created and even more vision to spend the money on a much, much larger model as we've seen with LLMs and similar over the last year.

Quite. I did not say that I can always identify a window. Just thst when forecasts are inconsistent one day to the next, I cannot plan far ahead.

 

[lustyd]

I wonder sometimes if the xbox generation expect too much from forecasting accuracy. No matter what revelations there are to come in 1s and 0s, nature is, I reckon, far too random to predict in the way that humans dream of.

Plus it helps to look out of the window now and then!

I wonder sometimes if the hoop and stick generation are oblivious to advances in technology. Just because you don't understand it, doesn't mean it's not possible.
We're building a new data centre in London that's dedicated to GPU processing for AI (just one in a global network of data centres). The compute in each 19" rack will beat what the Met supercomputer had previously, but the facility has hundreds of racks on many floors. We've developed methods to train models on trillions of parameters, taking up to 6 months to complete their training, and that's just what we did last year before really getting going. We've trained language models that can hold a conversation, computer vision models that are better than humans at recognising things, and systems that can generate photo quality images of anything you can imagine in seconds.
Weather prediction is on the list and when it's tackled we will certainly see a change in quality of predictions by orders of magnitude. It's likely that the human element will be removed from the process because people aren't capable of working at that kind of scale. The training data will consist of every measurement ever taken of the planet, weather, atmosphere, and likely thousands of other sources you might not even consider relevant.

It helps to open your mind now and then.

 

[capnsensible]

I wonder sometimes if the hoop and stick generation are oblivious to advances in technology. Just because you don't understand it, doesn't mean it's not possible.
We're building a new data centre in London that's dedicated to GPU processing for AI (just one in a global network of data centres). The compute in each 19" rack will beat what the Met supercomputer had previously, but the facility has hundreds of racks on many floors. We've developed methods to train models on trillions of parameters, taking up to 6 months to complete their training, and that's just what we did last year before really getting going. We've trained language models that can hold a conversation, computer vision models that are better than humans at recognising things, and systems that can generate photo quality images of anything you can imagine in seconds.
Weather prediction is on the list and when it's tackled we will certainly see a change in quality of predictions by orders of magnitude. It's likely that the human element will be removed from the process because people aren't capable of working at that kind of scale. The training data will consist of every measurement ever taken of the planet, weather, atmosphere, and likely thousands of other sources you might not even consider relevant.

It helps to open your mind now and then.

It will still get the weather forecast wrong....

 

[capnsensible]

 

[lustyd]

It will still get the weather forecast wrong....

Not anywhere near as wrong as right now.

You obviously know nothing about the subject so your sarcastic answers seem misplaced.

 

[capnsensible]

Not anywhere near as wrong as right now.

You obviously know nothing about the subject so your sarcastic answers seem misplaced.

Touchy.

 

[capnsensible]

Actually, in the real world, I've generally found forecasts for sailing pretty good for up to 3 or 4 days. All over a great deal of the world. The problems seem to be generally associated with the randomness of low pressure frontal systems. Not so much so, but a bit too with tropical revolving storms. Historical data fed into a 1 0 box no matter how smart it becomes, is in my opinion, unlikely to outsmart nature in prediction.

 

[lustyd]

The problems seem to be generally associated with the randomness of low pressure frontal systems

They only look random to the uninformed. In reality they follow physics quite reliably. We've just never been able to take into account enough information before.

Your opinion on this topic is entirely uninformed so you'll forgive me for ignoring it since I work in the field of AI for the largest AI company in the world who have the largest compute resources available. The Met office work with us on this stuff, along with various other weather organisations around the world.

I'll try to dumb it down a bit. Imagine that once upon a time we had temperature measurements all across the UK and looked for patterns. Then some bright spark realised that temperature varies with altitude so our 2D map had to become 3D to be useful. Next, we realised that different surface types generate heat at differing rates, so we had to make the map have another dimension. Then someone realised our square map won't work because the planet is a spheroid. Then we realised the spheroid is spinning so we add in friction and Coriolis force, as well as accounting for the tilt of the earth which not only changes the amount of sunlight at different times of the year but also the wavelengths hitting the surface, and hence the heating effect.

We're only up to 10 parameters right now and you're probably zoning out. The models we work on have over 1,500,000,0000,000 parameters (so far, that we've released to the public). To say they're a bit more sophisticated than your feeling that "these modern computer things won't work" is an enormous understatement.

Not only that, but for weather we've created a system to collect all of the data from all of the sensors both on and off of the planet, along with other machine learning to ensure spurious results or bad sensors don't cause issues, so not only are we processing more types of data than ever, but we have access to more of that data than ever and have the compute resources available to process it in a timely manner.

But yeah, you've got a feeling it won't work so we may as well give up and just pocket the billions of dollars of investment.

 

[franksingleton]

I wonder sometimes if the hoop and stick generation are oblivious to advances in technology. Just because you don't understand it, doesn't mean it's not possible.
We're building a new data centre in London that's dedicated to GPU processing for AI (just one in a global network of data centres). The compute in each 19" rack will beat what the Met supercomputer had previously, but the facility has hundreds of racks on many floors. We've developed methods to train models on trillions of parameters, taking up to 6 months to complete their training, and that's just what we did last year before really getting going. We've trained language models that can hold a conversation, computer vision models that are better than humans at recognising things, and systems that can generate photo quality images of anything you can imagine in seconds.
Weather prediction is on the list and when it's tackled we will certainly see a change in quality of predictions by orders of magnitude. It's likely that the human element will be removed from the process because people aren't capable of working at that kind of scale. The training data will consist of every measurement ever taken of the planet, weather, atmosphere, and likely thousands of other sources you might not even consider relevant.

It helps to open your mind now and then.

AI weather prediction would, basically, be a statistical model. Weather models are based on the physics of the atmosphere. They are different beasts. Attempts to use statistical techniques have been tried. They have never come anyway near to numerical modelling. Personally, I do not think any statistical model will do better. AI might do as well for, say, predicting rain a few hours ahead at a specific location. I cannot imagine that AI would do any better at predicting rainfall at every grid point over the globe, winds for every airline route and flight time, winds at every grid point over the globe. Where I can see AI being beneficial is in relating computer output to actual weather at specific locations. Even then, I have to wonder how it would cope with probability, ie ensembles.
Revising Reeds Weather Handbook, I have been looking at ensembles, specifically, the AROME 16-member ensemble. On one example, the deterministic forecast for one grid point at one time was for 23 kts, F6. But, small variations in the input data gave forecasts ranging from bottom F5 to middle gale F8. I f you think about it, with a F6 forecast, I expect some F7, perhaps a touch of F8 and some F5. An AI system would be unlikely to do any better. The atmosphere does not know itself very well. Why should AI know the atmosphere any better?
Further, there are so many differing requirements for forecasts that an AI system would have to be so varied in its output that it would not cope. I think the answer will be for NWP output to be made available to users for hem to apply AI to optimise output to specific needs and applications.

 

[capnsensible]

The mis Trust amongst us great unwashed stems from the claims that the systems being developed are in some way infallible. It may be so but you guys need to win the publicity war.

 

[franksingleton]

The mis Trust amongst us great unwashed stems from the claims that the systems being developed are in some way infallible. It may be so but you guys need to win the publicity war.

I doubt very much that those who develop NWP make any such claims. It is the third parties, such a Predicteind who make extravagant claims. It is also some (many?) users also. I have seen many claims by forumites that the service they use is very accurate. I have said many times that the atmosphere is not precise, so neither can be forecasts. When it comes to the next 7 - 10 days, I hear forecasters on radio/TV saying how uncertain is the outlook on some occasions but how sure they are at other times. There is much misunderstanding and selective memories that I discount much that anyone says about forecast accuracy.
What I do know is that forecasts are now far better than when I was in the hot, top Met Office seat.

 

[capnsensible]

I doubt very much that those who develop NWP make any such claims. It is the third parties, such a Predicteind who make extravagant claims. It is also some (many?) users also. I have seen many claims by forumites that the service they use is very accurate. I have said many times that the atmosphere is not precise, so neither can be forecasts. When it comes to the next 7 - 10 days, I hear forecasters on radio/TV saying how uncertain is the outlook on some occasions but how sure they are at other times. There is much misunderstanding and selective memories that I discount much that anyone says about forecast accuracy.
What I do know is that forecasts are now far better than when I was in the hot, top Met Office seat.

I think I've said much the same sort of thing......

However weather forecasting is but one part of the cure the world of all what ails it AI claims. I remain unconvinced. As I said before, they are not winning people over with arrogance. Softly softly generally seems to work best....

 

[lustyd]

Attempts to use statistical techniques have been tried. They have never come anyway near to numerical modelling

The difference here is size. You may have noticed that language understanding models have been tried in the past and failed, yet with current LLMs being trained at previously unimaginable scales they are extremely successful and impossible to tell apart from speaking to a very well educated human who knows every language on the planet. We may not be there yet, but weather prediction will be addressed, and even in the short term, the LLMs will allow humans to do their work faster than ever before through speeding up coding (try GitHub Copilot, you'll be amazed how easy coding is), or by summarising documents, or even suggesting things to try or how to automate adding more data.
As I said, one rack in the new data centre holds the same compute as the entire Met office supercomputer had, so regardless of how the problem is approached, the sheer scale of available processing will change the game quite drastically, and the quantity of data we're making available to those models will itself enable more nuance.

We also have to remember that progress in this arena has been far from linear. As technology has improved so the performance has increased exponentially, and will continue to do so.

 

[capnsensible]

The difference here is size. You may have noticed that language understanding models have been tried in the past and failed, yet with current LLMs being trained at previously unimaginable scales they are extremely successful and impossible to tell apart from speaking to a very well educated human who knows every language on the planet. We may not be there yet, but weather prediction will be addressed, and even in the short term, the LLMs will allow humans to do their work faster than ever before through speeding up coding (try GitHub Copilot, you'll be amazed how easy coding is), or by summarising documents, or even suggesting things to try or how to automate adding more data.
As I said, one rack in the new data centre holds the same compute as the entire Met office supercomputer had, so regardless of how the problem is approached, the sheer scale of available processing will change the game quite drastically, and the quantity of data we're making available to those models will itself enable more nuance.

We also have to remember that progress in this arena has been far from linear. As technology has improved so the performance has increased exponentially, and will continue to do so.

I am, however, impressed with your explanations. Just gotta convince us joe publics that it will make our lives better.

 

[lustyd]

For the record I’m not claiming it’ll make your life better. It’ll make certain things easier or more accurate but at the end of the day it’s just a tool.
I can now ask Word to write a document covering some points I outline, and I can ask it to refine a doc I write. Autistic or dyslexic people can ask it to normalise what they write too, giving them more opportunity in life. I can ask GitHub to write some code to perform a function or help debug my function. The clever bit is still on me though.
If and when we improve weather forecasting then the forecast will be better but a human will still need to decide to go to sea or not. We’re probably already at the stage where a weather report can be written automatically from the data so in theory we can cut out the “middle men” but does that make life better? Personally I prefer the human touch.