Why is the Met Office weather forecast wrong?

[capnsensible]

For the record I’m not claiming it’ll make your life better. It’ll make certain things easier or more accurate but at the end of the day it’s just a tool.
I can now ask Word to write a document covering some points I outline, and I can ask it to refine a doc I write. Autistic or dyslexic people can ask it to normalise what they write too, giving them more opportunity in life. I can ask GitHub to write some code to perform a function or help debug my function. The clever bit is still on me though.
If and when we improve weather forecasting then the forecast will be better but a human will still need to decide to go to sea or not. We’re probably already at the stage where a weather report can be written automatically from the data so in theory we can cut out the “middle men” but does that make life better? Personally I prefer the human touch.

Like.

 

[franksingleton]

The difference here is size. You may have noticed that language understanding models have been tried in the past and failed, yet with current LLMs being trained at previously unimaginable scales they are extremely successful and impossible to tell apart from speaking to a very well educated human who knows every language on the planet. We may not be there yet, but weather prediction will be addressed, and even in the short term, the LLMs will allow humans to do their work faster than ever before through speeding up coding (try GitHub Copilot, you'll be amazed how easy coding is), or by summarising documents, or even suggesting things to try or how to automate adding more data.
As I said, one rack in the new data centre holds the same compute as the entire Met office supercomputer had, so regardless of how the problem is approached, the sheer scale of available processing will change the game quite drastically, and the quantity of data we're making available to those models will itself enable more nuance.

We also have to remember that progress in this arena has been far from linear. As technology has improved so the performance has increased exponentially, and will continue to do so.

I have no doubt that AI will produce some very clever results. Language is certainly a great challenge. Weather prediction is a very different problem. The number of variables and the complexity of the atmosphere are, I believe, beyond the capability of AI. For example, AI might be able to identify two days with exactly the same weather conditions. But, if they are not on precisely the same day of the year, then the input from the sun will be different and the two situations will have different outcomes. The complexities of language are enormous but lack the fluid physical interactions that drive the atmosphere.

 

[Roberto]

OT

 

[oldgit]

All some of us mere rarely out of sight of shore mortals want to know is as to wether we are going to get an earful from the "crew" when we get yet another hammering from the wind arguing with the tide during some long awaited trip to somewhere/anywhere.
95 % of the time the 05.00 BBC forecast gets its right enough for me to risk going and one of the internet local forecasts is the icing on the cake.

 

[lustyd]

I have no doubt that AI will produce some very clever results. Language is certainly a great challenge. Weather prediction is a very different problem. The number of variables and the complexity of the atmosphere are, I believe, beyond the capability of AI. For example, AI might be able to identify two days with exactly the same weather conditions. But, if they are not on precisely the same day of the year, then the input from the sun will be different and the two situations will have different outcomes. The complexities of language are enormous but lack the fluid physical interactions that drive the atmosphere.

Day of year is just another parameter though, and having the capability to account for trillions of parameters enables these things to work. It's also entirely possible to have a model trained for each section of the grid with such compute resources. If anything, I think it'll be the lack of sensors holding this back.

 

[capnsensible]

Day of year is just another parameter though, and having the capability to account for trillions of parameters enables these things to work. It's also entirely possible to have a model trained for each section of the grid with such compute resources. If anything, I think it'll be the lack of sensors holding this back.

From me being somewhat sceptical, you have certainly prodded my interest. Its good to hear from people who are actually doing stuff.

 

[franksingleton]

Day of year is just another parameter though, and having the capability to account for trillions of parameters enables these things to work. It's also entirely possible to have a model trained for each section of the grid with such compute resources. If anything, I think it'll be the lack of sensors holding this back.

Like anyone deeply into a new and developing conceptual system you underestimate the problems. AI is about developing logic based on experience. In its simplest form it will come down to a question of “if this happens then that will follow.” I realise thst is a gross simplification. In the case of the atmosphere, I think that the physical interactions and feedbacks will still result in the physical modelling outperforming what can only, in effect, be a highly complex statistical approach.

 

[lustyd]

In its simplest form it will come down to a question of “if this happens then that will follow.”

Indeed, that's how the system works and whether you approach it from physics modelling or statistical modelling, the same is true. Given a set of inputs for a given location the same thing will result every time. The challenge is and always has been getting a sufficiently large number of inputs processed in a given timeframe.
Ultimately physics might work better but will require lots of people to write lots of code and get very precise data which we don't generally have. This makes it a challenge in the short term but potentially not in the longer term. The physics approach also has the distinct disadvantage that it will always need enormous processing power to create the next data set because it has to perform physics calculations for every location and every property every time.

Statistics (aka AI) doesn't need anyone to write anything beyond the training algorithm, which is a known entity at this point. Nor does it require anyone to really understand the physics since the results of physics are statistically very consistent so we can replace physics calculations with if this then that kind of basic processing. If the sun shines at a given latitude with no cloud cover on a given day the heat passed to the ground/sea/air is the same so it makes no difference if we understand why. If we include enough additional inputs such as solar activity, pollution, pollen, air pressure, humidity, and a billion or so others then we'll get much better answers even for the things we can't or haven't modelled in physics yet, and even for things we didn't think had an influence but does. We could even include the migratory patterns of whales just for kicks, and their presence might have an influence we'd not accounted for in physics models.
We just need a sufficiently large set of data to train the model. With a sufficiently large dataset the model will take months to train, but once it/they are trained results will not require heavy processing despite taking more parameters into account because at that point it's essentially a lookup vector table. This also means we can rerun every few minutes if we want to, and we can run for specific areas.

I imagine both methods will remain going forwards, but the output quality will certainly see a step change over the coming decade.

 

[franksingleton]

Indeed, that's how the system works and whether you approach it from physics modelling or statistical modelling, the same is true. Given a set of inputs for a given location the same thing will result every time.

Maybe, if you had precisely the same initial data. But, we know that small differences in initial conditions can lead to significantly different end results. Meteorological statistics describe what has happened in the past. They have not been good predictors for the future.

The challenge is and always has been getting a sufficiently large number of inputs processed in a given timeframe.

That has always been the problem in meteorology and oceanography.

Ultimately physics might work better but will require lots of people to write lots of code and get very precise data which we don't generally have.

No. Newton’s third law is well known. The method of forward integration is well understood. It is a relatively simple matter to try different mathematical methods.

This makes it a challenge in the short term but potentially not in the longer term. The physics approach also has the distinct disadvantage that it will always need enormous processing power to create the next data set because it has to perform physics calculations for every location and every property every time.

The data analysis is a multivariate optimal fitting process using all the data available in the forms in which they are measured, ie infrared radiances, microwave emissions, GPS signal bending angles as well as in situ or remotely measured meteorological data. A significant proportion of the increase in forecasting skill arises from improvements in the satellite data and their use.

Statistics (aka AI) doesn't need anyone to write anything beyond the training algorithm, which is a known entity at this point. Nor does it require anyone to really understand the physics since the results of physics are statistically very consistent so we can replace physics calculations with if this then that kind of basic processing.

That would apply with a well ordered system, such as planetary orbits. It is not so for an, essentially, turbulent system.

If the sun shines at a given latitude with no cloud cover on a given day the heat passed to the ground/sea/air is the same so it makes no difference if we understand why. If we include enough additional inputs such as solar activity, pollution, pollen, air pressure, humidity, and a billion or so others then we'll get much better answers even for the things we can't or haven't modelled in physics yet, and even for things we didn't think had an influence but does. We could even include the migratory patterns of whales just for kicks, and their presence might have an influence we'd not accounted for in physics models.
We just need a sufficiently large set of data to train the model. With a sufficiently large dataset the model will take months to train, but once it/they are trained results will not require heavy processing despite taking more parameters into account because at that point it's essentially a lookup vector table. This also means we can rerun every few minutes if we want to, and we can run for specific areas.

I imagine both methods will remain going forwards, but the output quality will certainly see a step change over the coming decade.

The physical approach concentrates on understanding those processes that determine our weather for the coming days. The AI approach throws vast amounts of data into the hat and will come up with answers having large error bars. Users will have little understanding of the reasons for the outcomes and no means of assessing their value. The turbulent nature of the atmosphere on all scales will defeat the AI approach. The physical approach will never be perfect but its deficiencies will be well understood.
I am unlikely to live long enough to see a viable AI approach to weather prediction. Even not knowing your age, I guess, neither will you.

 

[Bodach na mara]

I am enjoying the discussion between Frank and Lustyd which has reminded me of one of the Institute of Physics conferences at Stirling University. In the chair for the conference was the Scottish IoP regional chair, Heather ("the weather") Reid who was at that time a well-known presenter of the forecast after the early evening news. The post-lunch presentation was an illustrated talk about chaos theory, which was so interesting that most of the delegates stayed awake. At the end of this Heather asked the first question. In the course of her nightly forecast she gave a prediction for the following four day but from what we had just heard this was unlikely to be of much use. Was this correct? The short answer was "yes." I noticed that Heather dropped her four day outlook shortly after and confined herself to a two- day outlook.

Although a physicist myself who was aware of the numerical methods of forecasting based on Bjerknes' baroclynic engine theory and massive banks of computers solving the necessary differential equations, I always felt that better results were achieved by forecasters who used instinctive understanding of the movements of depressions and anticyclones.

 

[capnsensible]

I blame that pesky butterfly always stirring storms, the over dressed moth.

 

[lustyd]

Users will have little understanding of the reasons for the outcomes and no means of assessing their value.

Indeed, but the point being that if the answers are more consistently right, then it makes no difference to normal users whether they understand it or not. It's only of academic interest to the weather folk, but nobody else cares about anything except the outcome.

This is where tech, and specifically AI has been making enormous progress, precisely because it removes the experts from the equation and focuses on results. We've seen it happen in many industries and weather will be no different. The self driving car folks thought they could build the perfect system through engineering and were eventually replaced with machine learning which has been orders of magnitude better as well as being cheaper and faster to implement.

 

[lustyd]

I am enjoying the discussion between Frank and Lustyd

It's important to remember that we're both right here. We absolutely do need proper science to understand the weather and the physics of it all, and we absolutely should build models to test and understand that. That doesn't preclude a simpler and faster approach solving the immediate problem of what things will look like over the coming week, they are certainly not mutually exclusive approaches and the "best" one will largely come down to how much compute, data and manpower are available at the time. Right now that's swinging the argument towards AI being the pragmatic approach but that will likely change in the future.

 

[capnsensible]

So armed with Google and a new interest in AI I'm now finding a zillion ways it's being used. As an avid cricket fan, I'm somewhat nonplussed to discover England cricket selectors are....and have been using AI to select squads based on a quarter million or more simulations.

I think I need to go back to sea!!!!

 

[franksingleton]

Well, perhaps I should get up earlier in the morning. Lustyd was ahead of me re AI. ECMWF, the Met Office and others are studying AI and ML. I had taken my eye off the ball. There are, of course, some problems. Resolution of current AI models is not as good as NWP, I had not thought of that. Confidence is a problem, as I had thought. How do you perturb the input data to get a probability?
To me, the black box approach is anethma but the thinking seems to be how to get the best out of the two approaches. Something that I had not thought about is that AI requires a lot of historical data to ensure that extremes are sampled. However, with climate change, we are getting more extreme weather so helping AI.
A last thought. I wonder what AI will make of climate change. Hopefully, it will produce results that will engender strong public and corporate reaction.

 

[mattonthesea]

Thanks Lusty and Frank. (I've always considered myself a Metgeek but the Deep Dive has corrected me; I'm a Methead!) This discussion has added a whole new understanding of the approaches to this science. I will watch for the developments with even more interest.

 

[lustyd]

I wonder what AI will make of climate change

It’ll likely be worse for AI than physics. Assuming we had a perfect physics model it would continue to work because nothing changes with climate change other than variables. With AI the results will be different than in the data set so it will either gradually fail or will need retraining often.
Obviously we’re far from having a perfect model so in reality the physics will fail with climate change too.

 

[franksingleton]

It’ll likely be worse for AI than physics. Assuming we had a perfect physics model it would continue to work because nothing changes with climate change other than variables. With AI the results will be different than in the data set so it will either gradually fail or will need retraining often.
Obviously we’re far from having a perfect model so in reality the physics will fail with climate change too.

I would have expected AI to be very positive about climate change. It must have vast amounts of CO2 data both in recent years as well as for millions of years ago. It must have masses of temperature data both instrumental and before. It must have masses if data about fuel usage. There can only be one answer - the current climate change/global warming can have only one prime cause. I can see those data counting more with most people than the modelling.

 

[lustyd]

I guess it would depend on the model. As you said, AI models don’t “understand” anything but yes perhaps they could extrapolate enough from what’s happened if given all of the data.
I think my reply was a bit of a knee jerk “but climate change is new” but I guess in the scheme of things what I said about the physics also applies and the AI would have data with rising CO2 and patterns changing. Whether it could extrapolate out to uninhabitable tropics and massive storms elsewhere I don’t know as there are no stats for it to lean on but in theory those are all just rules playing out if you zoom out enough.

 

[GHA]

Well, perhaps I should get up earlier in the morning. Lustyd was ahead of me re AI. ECMWF, the Met Office and others are studying AI and ML. I had taken my eye off the ball. There are, of course, some problems. Resolution of current AI models is not as good as NWP, I had not thought of that. Confidence is a problem, as I had thought. How do you perturb the input data to get a probability?
To me, the black box approach is anethma but the thinking seems to be how to get the best out of the two approaches. Something that I had not thought about is that AI requires a lot of historical data to ensure that extremes are sampled. However, with climate change, we are getting more extreme weather so helping AI.
A last thought. I wonder what AI will make of climate change. Hopefully, it will produce results that will engender strong public and corporate reaction.

Search - Consensus: AI Search Engine for Research can be an interesting place to while away some time.. though often with paywalls in the way.

Can deep learning beat numerical weather prediction? - Consensus

The recent hype about artificial intelligence has sparked renewed interest in applying the successful deep learning (DL) methods for image recognition, speech recognition, robotics, strategic games and other application areas to the field of meteorology. There is some evidence that better weather forecasts can be produced by introducing big data mining and neural networks into the weather prediction workflow. Here, we discuss the question of whether it is possible to completely replace the current numerical weather models and data assimilation systems with DL approaches. This discussion entails a review of state-of-the-art machine learning concepts and their applicability to weather data with its pertinent statistical properties. We think that it is not inconceivable that numerical weather models may one day become obsolete, but a number of fundamental breakthroughs are needed before this goal comes into reach. This article is part of the theme issue ‘Machine learning for weather and climate modelling’.

interesting there's hardly a peep in the msm about 5g mobile networks potential adverse effect of forecasting.