Weather advancements

[RobbieW]

But we don’t and current techniques make it unlikely we ever will. Don’t lose sight of the fact that we’re not trying to model the atmosphere, we’re trying to predict the weather.
With ML we can concentrate on the exam question of predicting the weather. That means we don’t need to understand, we just need to test predictions and be right more often. It’s not helpful to keep circling back to current techniques or to how the data might be useful, all that matters is whether it produces better results.

With my tongue somewhat in my cheek, what your saying is that the weather tomorrow will be be as it is today. Thats long been the most accurate predictor and we dont need AI to make that prediction. However, climate change is making weather less predictable and more extreme so the past becomes less of a guide to the future in meteorology. That makes me wonder what sort of dataset would be useful in understanding greater chaos and how an AI system would be trained to use it.

 

[franksingleton]

But we don’t and current techniques make it unlikely we ever will. Don’t lose sight of the fact that we’re not trying to model the atmosphere, we’re trying to predict the weather.
With ML we can concentrate on the exam question of predicting the weather. That means we don’t need to understand, we just need to test predictions and be right more often. It’s not helpful to keep circling back to current techniques or to how the data might be useful, all that matters is whether it produces better results.

You and I agree that ML has the potential to become an important component in weather prediction. We differ on the matter of understanding both the atmosphere and the roles of data. You seem to want to put every scrap of information which has some connection with weather into a massive pot and see what comes out. But, where do you stop? Presumably, you do not follow the Lorenz rhetorical question and look at every butterfly. What about a murmurstion of starlings? Thousands of big butterflies acting in unison? Do you build that into your data? I do not know the amount of energy input. What about the various forms of transport. Massive energy inputs. Etc etc.
You make a big thing of mobile phone data without appreciating the fact that these will show up a mass of short lived noise and, in any case, weather moves and changes while doing do.
Your approach would result in collection of masses of data that add little to weather prediction. The scientific approach is to proceed in a logical manner. The first ECMWF paper looked at data on a 500km grid and showed that there was some skill,in the predictions. They then went to a finer grid and showed that there was more success. Obviously, they can go to finer resolutions.
At some stage, they will have to use raw data. I do not know what raw data are available for ML, but you have to remember that the whole system is interconnected. I can imagine a hierarchy of AI models starting with large scales working down, progressively to smaller scales. ML on a global scale will enable prediction of major storms and other large scale weather for days and weeks ahead. A more focussed AI would be needed to identify regional risks. Coming down to city, town or village scales would require appropriate scales of AI.
At present we have a system that works pretty well, in fact far better than I could have ever imagined 50 years ago when the Met Office was in the IBM 195/158 era. I have been wondering how far modelling would take us as we seemed to be in the diminishing returns stage. AI is, obviously the next stage. But, if we are to carry governments who are both funders and major customers with us as well as industry, commerce and the general public, then it will be necessary to work methodically. The multitude of users and myriads of interests have all to be considered.

 

[lustyd]

understanding greater chaos and how an AI system would be trained to use it.

At a macro level, yes climate change causes great disruption. On a more local level, and when talking specifically about weather, not the case. Weather is caused by various things moving about, and the same inputs will produce the same outputs. We don't need to know why something happens to predict it.

The same issues arise from "renewables". We're taking vast quantities of energy from one place and releasing it in another as heat when it's consumed. With data about where the energy is taken and used though, we can spot trends quite quickly. Global warming is fundamentally just a small tweak to one of many parameters, whether a physical model or ML model, but it's still linked in the same way to how the weather fundamentally works.

 

[lustyd]

You seem to want to put every scrap of information which has some connection with weather into a massive pot and see what comes out. But, where do you stop?

Why would you stop? It's a fully automated process that highlights where there are relationships. The real question is, why would you hold that back and exclude things that might be useful? You seem to be stuck in the traditional approach where a team need to work on the model and come up with theories then implement them as code. That's nothing like what I'm talking about.

What about a murmurstion of starlings? Thousands of big butterflies acting in unison? Do you build that into your data?

Absolutely, yes! Why would you ignore the movement of birds when it's fundamentally based on weather? You're not smarter than nature. Same with flowers blooming, it's a great indicator of start conditions for the upcoming season.

You make a big thing of mobile phone data without appreciating the fact that these will show up a mass of short lived noise

Mobile phones are just a good example for people that don't understand the subject. There are tens of millions of them in the UK alone, and they are by their nature where the people are. They also have sensors and connectivity and can report very precisely their position with an exact timestamp in UTC so they are an excellent data source. They are capable of measuring at any frequency, so not short lived at all, and they offer much richer views of data than traditional interpolation techniques which are deeply flawed.
Your dismissal of this says more than my enthusiasm. As I said, that demonstrates your lack of understanding of data science, and you're not even open minded enough to consider what I'm (as an expert in the field) telling you.

Your approach would result in collection of masses of data that add little to weather prediction.

You have no basis at all for that statement, since you're unwilling to even consider the benefits. Collecting data costs essentially zero these days for the storage and organisation. Experimentation will quickly tell us what's useful and what's not, then we can exclude the non-useful things from future processing.
What we absolutely must not do, is allow individuals to dictate what is and is not useful based on their limited view of the world and experience in completely unrelated techniques.

But, if we are to carry governments who are both funders and major customers with us as well as industry, commerce and the general public, then it will be necessary to work methodically

We don't need to consider those folk at all. They didn't participate in Google Maps or Streetview, they were dragged kicking and screaming into the 21st century by forward thinking organisations. The same will happen here, I imagine, one of the big three tech companies will produce a model that works orders of magnitude better. The government will likely cease funding of traditional weather organisations, which will return to university projects and research, while the tech companies monetise the weather data and make trillions.

 

[Marsali_1]

But we don’t and current techniques make it unlikely we ever will. Don’t lose sight of the fact that we’re not trying to model the atmosphere, we’re trying to predict the weather.
With ML we can concentrate on the exam question of predicting the weather. That means we don’t need to understand, we just need to test predictions and be right more often. It’s not helpful to keep circling back to current techniques or to how the data might be useful, all that matters is whether it produces better results.

I disagree with your assertion that "...we don't need to understand, we just need to test predictions and be right more often." AI/ML technology will, no doubt, speed up the analysis of large amounts of data but if you don't understand which variables are producing the "right more often" predicted outcomes and why they are producing those results then you're left with something akin to "It works...don't ask me why...but it works". That's not likely to carry much weight when it comes to convincing funding bodies to maintain the funding.

Current weather forecasting is based on a variety of observational data coupled with atmospheric modelling and seems to be doing a resonably good job. Improvements in data collection (accuracy and volume) and faster analysis have improved the quality of the forecasts over time. However the point that has been made more than once is that there would be benefit to more data being collected from the atmosphere above the oceans since events can happen there which have potentially predictable effects "downwind" in the areas where people want an accurate forecast. Since there are gaps in sampling over the oceans there may be missed signals of later consequence which is why filling in those data gaps coupled with faster data processing will improve weather forecasting. However the Fastnet disaster showed that, even if the analysis happens faster and with more data, the transmission of the resultant product becomes the limiting factor. Forecasting, by definition, is predicting something before it happens and cell phone data from every phone in the UK won't improve forecasting in the UK since it is not, for the most part, generating data from regions and atmospheric layers that are lacking in data.

 

[Laser310]

Mobile phones are just a good example for people that don't understand the subject. There are tens of millions of them in the UK alone, and they are by their nature where the people are. They also have sensors and connectivity and can report very precisely their position with an exact timestamp in UTC so they are an excellent data source. They are capable of measuring at any frequency, so not short lived at all, and they offer much richer views of data than traditional interpolation techniques which are deeply flawed.

what are you going to measure with them?

as i mentioned above, they have good barometers, but need regular calibration.

what else can they measure?

 

[lustyd]

AI/ML technology will, no doubt, speed up the analysis of large amounts of data but if you don't understand which variables are producing the "right more often" predicted outcomes and why they are producing those results then you're left with something akin to "It works...don't ask me why...but it works". That's not likely to carry much weight when it comes to convincing funding bodies to maintain the funding.

The organisations who will build and run ML models don't answer to funding bodies, they have more money than the countries those bodies represent, and will make more by training and monetising weather models. It doesn't matter how they work, it matters that they're better. I recall people saying it was impossible to photograph every street in Britain and would be cost prohibitive. Google did it for about £250k by buying two cars and some cameras. There are ML techniques that automatically choose the useful hyper parameters, people just need to provide the data.

Forecasting, by definition, is predicting something before it happens and cell phone data from every phone in the UK won't improve forecasting in the UK since it is not, for the most part, generating data from regions and atmospheric layers that are lacking in data.

Have you read any of my replies? We don't have this data at the kind of frequency or resolution that would be possible, and that data would absolutely be useful to inform an ML model. Mobile phones aren't the only additional useful source, I've mentioned many others on the thread. I chose mobile phones because it's easy for people to understand, although apparently not in this instance.

 

[Marsali_1]

what are you going to measure with them?

as i mentioned above, they have good barometers, but need regular calibration.

what else can they measure?

To be fair to lustyd, I'm not as concerned about his suggestion of cell phones as data collection devices as I am about the limited areas from which they would collect the data. My elderly cell phone has, I think, a 6 MP camera in it which is quite a high resolution compared to many film cameras that came before it. The limitation with it and its 12 MP and higher successors is the quality of its lens and the fact that it is a fixed focal length. The point being that technology (in this case cell phone cameras) has improved and will continue to as demand drives the change. Given how compact the camera is and how big some of the portable devices most people favour today are, it is not unreasonable to see other small sized sampling devices being incorporated in the future. However, if there is no direct benefit to the owner/operator there is unlikely to be a sufficient demand to warrant the manufacturer designing and installing said device and, thereby, increasing its retail price.

With regards to calibration, a network of fixed and accurately calibrated ground truthing sites of an appropriate density that are continuously sampling would allow for corrections of an individual phone's data, regardess of the variable being measured, while the phone is within the zone of the ground truthing site. Logically, since we are using a cell phone, these ground truthing sites would be located at cell tower sites. Mind you, the density of cell towers is far higher than the Met Office's recording locations so those new data points may be of sufficient statistical density to make redundant the collection of data from individual phones. Maybe the data expert can advise on this. Regardless, even if the phone happens to be at the end of a long selfy stick, it is unlikely to be more than 10 feet above local ground level if it is outdoors and indoor observations from the 70th floor of the Shard or someone's 20th floor flat in Manchester is not going to have much relevance to weather forecasting for someone wanting to sail out of the Solent and across to France.

 

[lustyd]

what else can they measure?

It varies by device. Most modern phones have temperature sensors, light sensors, GPS, barometer, some have IR cameras and other sensors. Then add in smart watches that can connect and add in depth sensors and other measurements. We can use those to enrich other data sets as I said earlier in the thread, all of which might have great input to ML weather models

 

[lustyd]

I'm not as concerned about his suggestion of cell phones as data collection devices as I am about the limited areas from which they would collect the data. My elderly cell phone has, I think, a 6 MP camera in it which is quite a high resolution compared to many film cameras that came before it

The "limited areas" are the ones which have a much larger affect on the weather, so exponentially more useful as a result. They are also in the locations we want more accurate forecasts for, since the people are in those locations.
You make a good point about the camera - it would be trivial for Apple to detect cloud cover and type and create a data set (they don't need your consent for this) based on images taken, processed in the phone. Apple own a weather company, so this would be of value to them.

With regards to calibration, a network of fixed and accurately calibrated ground truthing sites

That's not how the ML approach works. Tightly calibrated instruments is necessary when you have a handful of sites. When you have 30 million it's less necessary and you just use data science techniques to remove outliers. We can also apply corrections per device, so if a single device consistently reports higher than others in a given location, then just adjust that device. The scale approach has been proven to work, but needs a different attitude. This isn't liked by scientists, but that's why science is so much slower an approach, because they are trying to understand something not just produce a result.

it is unlikely to be more than 10 feet above local ground level if it is outdoors and indoor observations from the 70th floor of the Shard or someone's 20th floor flat in Manchester is not going to have much relevance to weather forecasting for someone wanting to sail out of the Solent and across to France.

I've never needed a forecast at that altitude other than when I've been up a mountain, and even then, that's "ground level". Irrelevant in ML modelling, I say again we are not trying to model the atmosphere, we're trying to predict the weather.

 

[franksingleton]

Why would you stop? It's a fully automated process that highlights where there are relationships. The real question is, why would you hold that back and exclude things that might be useful? You seem to be stuck in the traditional approach where a team need to work on the model and come up with theories then implement them as code. That's nothing like what I'm talking about.

We know that AIFS might do better than current NWP. But AIFS depends upon a NWP model. Do you really think that you can produce forecasts without satellite data? Well, I suppose that you might be able to produce forecasts but of what quality?

Absolutely, yes! Why would you ignore the movement of birds when it's fundamentally based on weather? You're not smarter than nature. Same with flowers blooming, it's a great indicator of start conditions for the upcoming season.

How are you going to collect data from bird activity, flower and plant growth, animal behaviour?

Mobile phones are just a good example for people that don't understand the subject. There are tens of millions of them in the UK alone, and they are by their nature where the people are. They also have sensors and connectivity and can report very precisely their position with an exact timestamp in UTC so they are an excellent data source. They are capable of measuring at any frequency, so not short lived at all, and they offer much richer views of data than traditional interpolation techniques which are deeply flawed.

Great for nowcasting but weather is 4 dimensional.

Your dismissal of this says more than my enthusiasm. As I said, that demonstrates your lack of understanding of data science, and you're not even open minded enough to consider what I'm (as an expert in the field) telling you.

I am a realist. I am trying to understand, inter alia, how such an AI system would be developed while maintaining existing, albeit imperfect, services.

You have no basis at all for that statement, since you're unwilling to even consider the benefits. Collecting data costs essentially zero these days for the storage and organisation. Experimentation will quickly tell us what's useful and what's not, then we can exclude the non-useful things from future processing.

That is my opinion. You are going to have to demonstrate that AI works. If it is so easy and major tech organisations have the capability and the cash in their back pockets, then give it a go. Remove all the satellite data, collect all your cell phone and other data and see how it goes. I doubt any airline would be willing to use your product without a thorough trial. Nor anyone concerned with safety at sea or issuing warnings of severe weather over land.

What we absolutely must not do, is allow individuals to dictate what is and is not useful based on their limited view of the world and experience in completely unrelated techniques.

But, we must see proof that your concept of an AI system will do at least as well as the current system.

We don't need to consider those folk at all.

You will have to demonstrate that AI will do at least as well as the current NWP with all its data inputs.

They didn't participate in Google Maps or Streetview, they were dragged kicking and screaming into the 21st century by forward thinking organisations.

Those are hardly comparable with the prediction of a dynamical system with multiple feedbacks and interactions.

The same will happen here, I imagine, one of the big three tech companies will produce a model that works orders of magnitude better. The government will likely cease funding of traditional weather organisations, which will return to university projects and research, while the tech companies monetise the weather data and make trillions.

As I have said above, get the big techs on board and see. You keep making a big play for data science, about which I probably know as little as you seem to do about atmospheric physics.

 

[Marsali_1]

The "limited areas" are the ones which have a much larger affect on the weather, so exponentially more useful as a result. They are also in the locations we want more accurate forecasts for, since the people are in those locations.

There is no doubt that urban heat islands and urban humidity islands as well as urban smog islands affect local weather and, if large enough, perhaps more than just local weather, but increased heating of the oceans is having a similar and far greater impact on global weather. The transfer of energy to the overlying atmosphere is what drives the weather process. If you haven't got the sampling density over the oceans then you can't describe the consequences of this changing energy transfer with enough precision to predict down wind effects on the weather at a scale to suit the consumer. Missed events over the oceans due to lack of data in that area will have consequences which could have been predicted were that data available. Collecting more data now from a location were something is happening now does not help predict that event.

No question that people want more accurate forecasts of the weather where they are but, as you pointed out previously (yes I do read your replies) we live in a global environment where everywhere is downwind from somewhere else. Therefore it is very useful to know what is happening in the atmosphere elsewhere, including over the oceans, because of the impact that has on the local weather. Even though you dismiss it with virtually every post, modelling and understanding the atmosphere is integral to accurate weather forecasting because it is the medium in which the weather is happening.

 

[RobbieW]

The "limited areas" are the ones which have a much larger affect on the weather, so exponentially more useful as a result....

Are they though ? I understood something like, say, an El Nino year to be generated by the water temperature in the East Pacific. Similarly with hurricanes forming in East Atlantic with a combination of a tropical wave plus warm water. Some of the UKs weather originates in the US midwest, from a combination of warm Gulf air meeting cold Canadian air. None of those areas have high densities of population.

 

[franksingleton]

Are they though ? I understood something like, say, an El Nino year to be generated by the water temperature in the East Pacific. Similarly with hurricanes forming in East Atlantic with a combination of a tropical wave plus warm water. Some of the UKs weather originates in the US midwest, from a combination of warm Gulf air meeting cold Canadian air. None of those areas have high densities of population.

The atmosphere is a big heat engine. The tropical oceans are the main source of heat. Some years ago, there was an international observation program entitled TOGA -,Tropical Ocean Global Armosphere.

 

[franksingleton]

There is no doubt that urban heat islands and urban humidity islands as well as urban smog islands affect local weather and, if large enough, perhaps more than just local weather, but increased heating of the oceans is having a similar and far greater impact on global weather. The transfer of energy to the overlying atmosphere is what drives the weather process. If you haven't got the sampling density over the oceans then you can't describe the consequences of this changing energy transfer with enough precision to predict down wind effects on the weather at a scale to suit the consumer. Missed events over the oceans due to lack of data in that area will have consequences which could have been predicted were that data available. Collecting more data now from a location were something is happening now does not help predict that event.

No question that people want more accurate forecasts of the weather where they are but, as you pointed out previously (yes I do read your replies) we live in a global environment where everywhere is downwind from somewhere else. Therefore it is very useful to know what is happening in the atmosphere elsewhere, including over the oceans, because of the impact that has on the local weather. Even though you dismiss it with virtually every post, modelling and understanding the atmosphere is integral to accurate weather forecasting because it is the medium in which the weather is happening.

In a nutshell, to know about weather somewhere, you have to know about weather everywhere. That truism will apply whether you are using NWP or AI. No doubt an oversimplification but a fact that lustyd seems incapable of understanding.

 

[lustyd]

How are you going to collect data from bird activity, flower and plant growth, animal behaviour?

These data sets already exist, they just aren't being used for this purpose.

Great for nowcasting but weather is 4 dimensional.

You've clearly not read anything I've written, or you're purposefully ignoring what I've written so I won't carry on responding to you. I've addressed this point multiple times on this thread and never once suggested it would be used for immediate forecasting.

 

[lustyd]

In a nutshell, to know about weather somewhere, you have to know about weather everywhere. That truism will apply whether you are using NWP or AI. No doubt an oversimplification but a fact that lustyd seems incapable of understanding.

simply not true. It's entirely possible to predict the weather in the midlands tomorrow while only knowing the conditions for the UK today. You're still incapable to thinking outside of the well trodden path you've made a career of, which is why you'll never understand these new techniques.

 

[lustyd]

Are they though ? I understood something like, say, an El Nino year to be generated by the water temperature in the East Pacific. Similarly with hurricanes forming in East Atlantic with a combination of a tropical wave plus warm water. Some of the UKs weather originates in the US midwest, from a combination of warm Gulf air meeting cold Canadian air. None of those areas have high densities of population.

Yes, they are. If the planet were entirely ocean we'd have very consistent weather. We know this because we can see relatively featureless planets that do. The complexities come from surface type and terrain, and the movement of energy and matter around the world. El Nino is a predictable event and sets the environment in which weather progresses, but it's one of the most simple.
Building out massive housing estates, replacing flood planes and causing water movements to change will have a huge effect on weather. Sadly, the industry just says "turbulent atmosphere, we can never understand what caused this change".
Yes, UK weather is obviously affected by the US, the planet continues to spin. In reality there are multiple levels of forecasting with climate being the big one, then weather patterns, but then there are masses of local effects which are ignored on purpose by traditional forecasting because they're too hard. ML can solve many of those by bringing in new and different data at higher resolutions. A flood accross the UK will put more moisture into the atmosphere which will end up somewhere and alter the weather for that other place. Home heating and car charging/use put enormous amounts of heat energy into the atmosphere which will end up somewhere else and alter the weather for that other place. All of this is relevant and can be used to make forecasting better. Crucially, it can only do so if those working the problem have an open mind.
Thankfully big tech (where I currently work) is working the problem with an open mind. Traditional forecasters are still doing traditional things, and will return to university research projects as they are replaced by better forecasting products.

 

[franksingleton]

simply not true. It's entirely possible to predict the weather in the midlands tomorrow while only knowing the conditions for the UK today. You're still incapable to thinking outside of the well trodden path you've made a career of, which is why you'll never understand these new techniques.

Sorry, but you are showing your blinkered attitude. If what you say is correct then you, as a data science expert could set up a totally automated service. Put you money where your mouth is. Then give a thought to all the other forecasts and users of forecasts.

 

[lustyd]

but increased heating of the oceans is having a similar and far greater impact on global weather. The transfer of energy to the overlying atmosphere is what drives the weather process

At a macro level you're right, but weather isn't a macro level thing. Experienced weather is affected enormously by local effects, and forecasting being at 10s of KM resolution means that most people get an incorrect interpolated weather forecast. That interpolated forecast is often wrong because the movement of a weather system is affected by small, local things while forecasters say it's "unpredictable turbulence". If we can understand why a system moves north in certain conditions rather than south then longer term forecasts become more reliable. That's where more detailed data comes in, and where it comes in locally too. Weather over oceans is considerably more predictable because there are almost no features.