Met Office model upgrade

[franksingleton]

Met Office launches major upgrade to forecasting system announces a major upgrade to the UK UM and UKV models although computational grid lengths remain at 10 and 1.5km. The reasons for the upgrade appear to be in the treatment of physical processes such as details of processes in clouds and effects of turbulence on cloud and fog formation. These are clearly important on a small scale, local level for localised flood events, ice on roads, visibility. It is not clear, to me at least, how these improved models will impact on the larger scales that affect the cruising yachtsman/woman.

 

[lustyd]

I suspect there’s more to come as this is off the back of greatly expanded compute. They mention their new “supercompter” in the text but not that it’s cloud based and much more easily expandable (and extendable with other services). It should also reduce cost, making more budget available for mucking about with new stuff.

 

[Boathook]

If this upgrade is the same quality as that done with the app we are going to have problems !

 

[franksingleton]

I suspect there’s more to come as this is off the back of greatly expanded compute. They mention their new “supercompter” in the text but not that it’s cloud based and much more easily expandable (and extendable with other services). It should also reduce cost, making more budget available for mucking about with new stuff.

Rather surprisingly, there is no mention of AI and asking AI questions of Google and others confirms that this is all about physical modelling. From Better forecasts ahead as Met Office transitions to a supercomputer in Azure cloud and other statements over the last year, it is clear that they are working in the cloud. My last direct contact with senior members of staff was 10+ ‘years ago and they were talking about cloud computing even then. It will be clearer to you than me why it has taken so long to implement. Part, at least, of the reason is probably the operational
, time critical role of the Met Office. I get the impression that, although AI is on the way, physical modelling is expected to continue to have a role.

 

[Supertramp]

Rather surprisingly, there is no mention of AI and asking AI questions of Google and others confirms that this is all about physical modelling. From Better forecasts ahead as Met Office transitions to a supercomputer in Azure cloud and other statements over the last year, it is clear that they are working in the cloud. My last direct contact with senior members of staff was 10+ ‘years ago and they were talking about cloud computing even then. It will be clearer to you than me why it has taken so long to implement. Part, at least, of the reason is probably the operational
, time critical role of the Met Office. I get the impression that, although AI is on the way, physical modelling is expected to continue to have a role.

As a mere user I will be glad if the focus is on physical modelling (and hopefully associated real data) rather than AI. Last thing I want is a computer generated best guess. I am always impressed with the information Met Eireann make available from weather stations/buoys plus the detail and promptness of their coastguard and web transmitted small craft warnings which add usefully to the routine forecasts.

 

[lustyd]

Rather surprisingly, there is no mention of AI

My feeling is that AI is the “mucking about” they’re free to do with more budget and compute. Obviously job one is continuing to provide and improve existing methods until AI is ready so I’d hope they do focus on that mostly.
The time to implement is because they’re coming from very specialised specific implementation of hardware and software. Although they are (I think) still basing it on Cray hardware, there will be a huge bunch of peripheral stuff to change and improve, and anything that can be offloaded now should be offloaded, probably to Linux on Intel or Databricks clusters which are cheaper and more flexible. Some of that would have been on other platforms before too I’d imagine.

 

[franksingleton]

As a mere user I will be glad if the focus is on physical modelling (and hopefully associated real data) rather than AI.

With due respect, I do mot think that you understand the basis ofAI forecasts. Lusrtyd can put it better but, at heart it pattern matching based on history. The ECMWF AIFS slightly but consistently outperforms their physics based IFS. There are some caveats and uncertainties.

Last thing I want is a computer generated best guess.

Physical models have their shortcomings some of which seem to be being addressed with this latest Met Office model but there will always be uncertainties - known unknowns and, probably, some unknown unknowns.

I am always impressed with the information Met Eireann make available from weather stations/buoys plus the detail and promptness of their coastguard and web transmitted small craft warnings which add usefully to the routine forecasts.

Met Eireann uses the same models as the UK, The charts they show on their site are the European HARMONIE as used by Spain and some other countries. Like all physical models it uses all available data both in situ, ships, land stations, radio sondes, data buoys as well as a massively greater volume of data observed from space.
To what extent the future will ne based on machine learning. To me a better description than AI, I have no idea. I do not know what is happening in other parts of the world, I do not know. The fact that the Met Office is working with Turing while still putting substantial effort into physical models tells me tha that they are thinking about some amalgam. My gut feeling, knowing what I do about the atmosphere and the data, is that ML will not be the end result. I do know that I am unlikely ever to know.

 

[lustyd]

The fact that the Met Office is working with Turing while still putting substantial effort into physical models tells me tha that they are thinking about some amalgam.

I don’t see why both can’t be worked on. Until AI is fully set up we need physical models so they may as well keep refining them. AI doesn’t need lots of experts working on it, it’s mostly data plumbing which is a different skill entirely. Eventually the weather experts and forecasting will probably split in my opinion, we will always want the science and understanding but forecasting accurately will probably outstrip that at sone point. That’s a good thing too, weather and climate experts are wasted making forecasts and having to answer to the public about their models, their time is probably better spent learning and refining our understanding. My main fear is funding for the science bit, but hopefully that will remain.

 

[Stemar]

The one thing that I always wonder about with working in the cloud is resilience. Where is this cloud? What happens if the ruler of the country decide they don't like us - or another someone doesn't like them enough to take out their data centres? OK, for the Met Office, it would "only" be inconvenient, but I hope people in the MOD and the NHS are thinking about it.

Or am I just an old fart who doesn't understand how anything works any more?

 

[franksingleton]

I don’t see why both can’t be worked on. Until AI is fully set up we need physical models so they may as well keep refining them. AI doesn’t need lots of experts working on it, it’s mostly data plumbing which is a different skill entirely. Eventually the weather experts and forecasting will probably split in my opinion, we will always want the science and understanding but forecasting accurately will probably outstrip that at sone point. That’s a good thing too, weather and climate experts are wasted making forecasts and having to answer to the public about their models, their time is probably better spent learning and refining our understanding. My main fear is funding for the science bit, but hopefully that will remain.

When I see the word accurate(ly) I always have to ask how it is defined and measured in a way meaningful to the majority user.

 

[franksingleton]

The one thing that I always wonder about with working in the cloud is resilience. Where is this cloud? What happens if the ruler of the country decide they don't like us - or another someone doesn't like them enough to take out their data centres? OK, for the Met Office, it would "only" be inconvenient, but I hope people in the MOD and the NHS are thinking about it.

Or am I just an old fart who doesn't understand how anything works any more?

Join the club!
However, remember that MOD and much else is weather dependent.

 

[lustyd]

The one thing that I always wonder about with working in the cloud is resilience. Where is this cloud? What happens if the ruler of the country decide they don't like us - or another someone doesn't like them enough to take out their data centres? OK, for the Met Office, it would "only" be inconvenient, but I hope people in the MOD and the NHS are thinking about it.

Or am I just an old fart who doesn't understand how anything works any more?

Locations for cloud data centres are well defined. Here is a visualised map Explore | Azure global infrastructure experience
If your spend is over about £50M/year you can get the postcodes of them under NDA, and even visit one for a tour (Dublin usually if you're in the UK). Other locations in the UK are Swansea, Slough, London and all are interconnected with very high bandwidth lines. The London data centres are connected with hollow optic fibre connections, which are faster than standard fibre links because the light travels through air rather than glass - this helps with real time financial applications.
Resilience-wise it utterly trounces the meagre installation that MET office used to have, which itself was fairly resilient but constrained by budget.
Some clouds have "sovereign" cloud data centres. These use identical technology but are purposefully run by local operators so that the US CLOUD Act cannot be enforced and data extracted. The trust centre for each cloud provider will tell you more than you ever wanted to know on this subject. If said "ruler" decides not to play ball the legal process would be longer than their remaining term in office and it's unlikely the next administration would carry it on. We have seen this play out in some very public cases (I think one in Ireland) and nothing untoward happened.
Yes, the MOD and NHS thought long and hard about it, and did it anyway. We've already seen enormous (and well documented) benefits from them doing so.
You don't have to be that old to not understand this. Cloud was only a big thing for the last 15 years or so and AI only the last 5. When I worked at a big tech firm I often told people not to trust "AI Experts" as the tools were only a year old so expertise would be essentially impossible. That is changing, but lots of charlatans still around.

Anyway, long story short, all of your worries have been thought through long and hard and cloud is more secure, faster, cheaper and endlessly more capable while having a well defined regulatory framework around it. The only tripping point I saw in my tenure was when I pointed out (as a regs nerd) that UK DPA (aka GDPR) prevented UK companies using UK data centres following Brexit since our own law explicitely defined us as a third country. That was subsequently fixed with the help of our legal department.

When I see the word accurate(ly) I always have to ask how it is defined and measured in a way meaningful to the majority user.

In this instance, probably data scientists who do not need to be weather experts. It could be weather experts, but that would be a waste of resources and training.

 

[Supertramp]

With due respect, I do mot think that you understand the basis ofAI forecasts. Lusrtyd can put it better but, at heart it pattern matching based on history. The ECMWF AIFS slightly but consistently outperforms their physics based IFS. There are some caveats and uncertainties.

Physical models have their shortcomings some of which seem to be being addressed with this latest Met Office model but there will always be uncertainties - known unknowns and, probably, some unknown unknowns.

Met Eireann uses the same models as the UK, The charts they show on their site are the European HARMONIE as used by Spain and some other countries. Like all physical models it uses all available data both in situ, ships, land stations, radio sondes, data buoys as well as a massively greater volume of data observed from space.
To what extent the future will ne based on machine learning. To me a better description than AI, I have no idea. I do not know what is happening in other parts of the world, I do not know. The fact that the Met Office is working with Turing while still putting substantial effort into physical models tells me tha that they are thinking about some amalgam. My gut feeling, knowing what I do about the atmosphere and the data, is that ML will not be the end result. I do know that I am unlikely ever to know.

I agree that I am ignorant of the mechanisms used to turn observation data into an accurate forecast. I don't use Starlink or Inreach so rely on picking up internet info when possible and the VHF broadcasts every few hours (UK and Eire waters).

I keep a log of the forecast over the days when a system is developing which I guess is what AI or algorithms would do.

Met Eireann uses the same info as other areas but the coastguard service seem better at communicating local changes or developments than the UK. Perhaps because they have a more local effects and more small boat fishermen.

A bit like many aspects of digital technology, I just want it to work but having some understanding of how it works helps the trust process.

 

[lustyd]

having some understanding of how it works helps the trust process.

It's actually very simple how it works. I'll use a non-weather example of house prices. (sorry this got a bit long)

Take the number of bedrooms of all houses on the market and plot this on an X/Y graph against their sale price. Draw a line along the average and you'll get a line roughly from zero with increasing price and bedrooms. The angle of this line is essentially the relationship ratio and so you can draw a line from any number of bedrooms to find the expected cost, or vice versa.

Now, we can add in square footage as the Z axis and end up with a 3D version of the same chart. Slightly more useful, slightly more accurate. Next, add in dimensions for distance from school, quality of view, priximity to London, expected wages. You can no longer imagine the graph as it has more dimensions than you could comprehend, but the concept scales well with computers so we can keep adding bits and bobs in to improve things, right down to whether there's a cat flap, or how often the neighbor mows the lawn.

Large scale models take this to the extreme, and add in all of the data and let the model decide what's important. ChatGPT has billions of parameters and as a result not only can understand nuanced language (which is hard) but can do it in every language both reading and writing and can add emotion into the mix very well.

Move over to weather and the process is much like what weather forecasters have done. On day one you have the weather stone method (if rock is wet, it's raining...). Add in pressure, temperature, humidity and we start to get to early 20th century stuff. Add in satelite observations and historical records from ice cores and tree rings and things improve more. Physical models have been limited by two things, primarily, and that's speed of science and computer time. Having to understand how a new parameter affects the weather can hold things back for years or decades, and sometimes data is discarded because we're not sure if it does affect weather. Compute is expensive at scale, and weather models are some of the most demanding compute use-cases we have (or they were, until AI!). As such, adding too many parameters to a physical model can mean that the model won't finish processing in time for the output to be useful. We might be able to predict the weather next Tuesday very accurately, but if we don't get the forecast until a week next Friday it's of no use. This is because in a physical model we compute points in the atmosphere and how they interact over time with various expected inputs. Lots of points, lots of data, lots of processing.
With AI, we bypass a lot of the issues by pre-computing everything. The technique is a time-memory tradeoff and costs unimaginable sums to create but then gets cheaper and faster to run. With AI we put all of the data in a pot and let the machine do it's multi-dimensional thing and back comes a model that doesn't need to compute every point the long way, we just do a lookup for the inputs. This means that we can include all of the parameters up front, even seemingly trivial stuff like adoption of heat pumps which will reduce moisture in the atmosphere and cease producing "new heat" in an area. It'll be a small effect, but an effect nonetheless.

We also set up automated systems to test the models using current and past data as well as other quality improvement techniques. Once it's at a good level, the process reverses and we back-engineer the science to try to explain why a parameter has an effect, hence my saying that the science side will continue and probably grow.

 

[Supertramp]

It's actually very simple how it works. I'll use a non-weather example of house prices. (sorry this got a bit long)

Take the number of bedrooms of all houses on the market and plot this on an X/Y graph against their sale price. Draw a line along the average and you'll get a line roughly from zero with increasing price and bedrooms. The angle of this line is essentially the relationship ratio and so you can draw a line from any number of bedrooms to find the expected cost, or vice versa.

Now, we can add in square footage as the Z axis and end up with a 3D version of the same chart. Slightly more useful, slightly more accurate. Next, add in dimensions for distance from school, quality of view, priximity to London, expected wages. You can no longer imagine the graph as it has more dimensions than you could comprehend, but the concept scales well with computers so we can keep adding bits and bobs in to improve things, right down to whether there's a cat flap, or how often the neighbor mows the lawn.

Large scale models take this to the extreme, and add in all of the data and let the model decide what's important. ChatGPT has billions of parameters and as a result not only can understand nuanced language (which is hard) but can do it in every language both reading and writing and can add emotion into the mix very well.

Move over to weather and the process is much like what weather forecasters have done. On day one you have the weather stone method (if rock is wet, it's raining...). Add in pressure, temperature, humidity and we start to get to early 20th century stuff. Add in satelite observations and historical records from ice cores and tree rings and things improve more. Physical models have been limited by two things, primarily, and that's speed of science and computer time. Having to understand how a new parameter affects the weather can hold things back for years or decades, and sometimes data is discarded because we're not sure if it does affect weather. Compute is expensive at scale, and weather models are some of the most demanding compute use-cases we have (or they were, until AI!). As such, adding too many parameters to a physical model can mean that the model won't finish processing in time for the output to be useful. We might be able to predict the weather next Tuesday very accurately, but if we don't get the forecast until a week next Friday it's of no use. This is because in a physical model we compute points in the atmosphere and how they interact over time with various expected inputs. Lots of points, lots of data, lots of processing.
With AI, we bypass a lot of the issues by pre-computing everything. The technique is a time-memory tradeoff and costs unimaginable sums to create but then gets cheaper and faster to run. With AI we put all of the data in a pot and let the machine do it's multi-dimensional thing and back comes a model that doesn't need to compute every point the long way, we just do a lookup for the inputs. This means that we can include all of the parameters up front, even seemingly trivial stuff like adoption of heat pumps which will reduce moisture in the atmosphere and cease producing "new heat" in an area. It'll be a small effect, but an effect nonetheless.

We also set up automated systems to test the models using current and past data as well as other quality improvement techniques. Once it's at a good level, the process reverses and we back-engineer the science to try to explain why a parameter has an effect, hence my saying that the science side will continue and probably grow.

Thanks for this - I now understand the difference AI brings to the physical model!

Is there an accuracy figure acessible for the different models in terms of wind strength and direction? Or is that intrinsic to the AI part?!

I often use UKV2 (in Windy) as it shows more local detail which is very useful up the Irish Sea/Scotland and ECMWF for the overview.

It was(n't!) simpler in the old days when you made your own isobar map and wind forecast off the LW forecast.........and then blamed yourself if you got it wrong.

 

[lustyd]

Is there an accuracy figure acessible for the different models in terms of wind strength and direction? Or is that intrinsic to the AI part?!

Right now AI is being used to refine physical models, it’s still a while before we see large models finish getting trained, and even then there will be a bedding in period to work out how best to use them. Even then, it may be a while before they get fully used. ChatGPT took well over three years after release to be used in a useful way and five to commercialise properly and embed in solutions.

It’s worth noting that the output of a large forecast model probably won’t be in standard GRIB format so the tooling may need completely rewriting. GRIB is a result of the processing rather than a desirable format.

 

[franksingleton]

Forecasters use measures of model (physical and AI) such as RMS of surface pressure, temperature and heights of geopotential levels - eg 500, 300hPa height and RMSVE of wind. These will not mean much to us users but do give goo indicators of effect of changes to models. These measures were used by ECMWF in deciding to publish results of their AIFS output along with their physical model, IFS.
AIFS uses learning based on ERA5. This is a data set produced by ECMWF of data reanalyses from the start of satellite data soundings, late 1979. It is widely acknowledged as the best source of data for training of AI/ML purposes. The US AIGFS uses ERA5.


So, at present, an AI forecast has to start by running a physical model data analysis. That takes about 3 hours from a nominal data time but the actual forecast out to 10 days takes a further 2 or so hours. Similarly for operational forecast models run by the U.K. DWD, Meteo France etc.
The big problem, as I see it, running a full AI/ML system is that we do not have the data. The vast majority of data comes from satellite soundings. There are a relatively tiny amount of in situ tempersture, humidity, wind data. Satellite infra-d soundings are only available over oceans and large seas. They measure the effects of air temperature and humidity on infrared radiation from the earth. However the same sounding data can come from infinitely many temperature profiles. The radiative equations cannot be inverted.
Microwave soundings measure tiny radio signals. The vertical resolution is poor. Again.
, the same same sounding data can come from different temperature or humidity profiles. In both, measurements are from satellites in LEO and not continuously available, nor at the same time over large areas.

There was a lot of excitement some while ago on GPS Radio Occultation data and these are certainly valuable. There are no instrumental drift problems as bedevil satellite borne instruments. They measure the bending of radio signals passing through the earth’s atmosphere and, hence, the profile of air density. The effects of temperature and humidity on density can it be separated.
The result of these various factors is enormously complex. No doubt some powerful brain power will address the problems which are far from trivial.
That will do for now, as lusty found it is not easy to discuss sensibly and briefly this complex topic.
Nevertheless, I should add that pattern matching is nota new concept in meteorology. It is only recently tat it has become possible and, even then, by making use of some physical modelling data.
E &OE!

 

[Stemar]

Wow, I didn't expect all that. Thank you, Frank and Lustyd I almost feel like I understand it now.

I have an OU diploma in computing, but it's the best part of 25 years old, and things have come along a bit since then...

 

[lustyd]

There was a lot of excitement some while ago on GPS Radio Occultation data and these are certainly valuable. There are no instrumental drift problems as bedevil satellite borne instruments. They measure the bending of radio signals passing through the earth’s atmosphere and, hence, the profile of air density. The effects of temperature and humidity on density can it be separated.
The result of these various factors is enormously complex. No doubt some powerful brain power will address the problems which are far from trivial.

This is an interesting one and demonstrates something I touched on. Whether they are useful or not, and whether they are complete or not, AI can use them in training. The complexity of the actual problem doesn't matter because the AI model doesn't need to know the why, it just knows that there is a relationship and as long as the data is provided in a consistent way it can be used. We'd never dream of doing that with traditional approaches for obvious reasons.

A great example is the Cornish language. ChatGPT didn't get much information written in Cornish because there isn't much out there. When you speak to ChatGPT in English, it doesn't know or need to know that the Cornish stuff is irrelevant. If you ask it a question in Cornish though, it will understand because that data becomes relevant. It's not translating, it just knows all of the languages natively.

A weather person will strive to convert raw data to something they understand as useful, such as radio signals to density etc. while a trained AI model can just use the radio signal data directly, it doesn't care that it's related to pressure before the process starts, even if that data is used for pressure purposes. Hopefully that makes sense, it's an important distinction. A similar idea might be using colour of light instead of temperature of stars. Because of science we do know they're related, but even if we didn't we could use colour or temperature quite interchangeably when discussing the particulars of a given star.

 

[yotter]

I preferred the old model.....there was less rain