Laser310
Well-known member
how are you going to crowd source at 500mb?
Weather advancements
- Thread starter lustyd
- Start date
how are you going to crowd source at 500mb?
Laser310
Well-known member
but better data to 15km would still be a big help.
but anyway - the question I am asking is; which method is likely to result in the greatest increase in forecasting skill; basic science, or AI?
I fear it is AI.., and I am unhappy about what that means for the future of basic science funding; it will necessarily be less than if it was perceived as the most effective way to increase forecasting skill.
franksingleton
Well-known member
Yes, but remember that, roughly, a little over 10% of the mass of our atmosphere is above 15 km.
The learning so far has, I understand, used the ERA5 data set. I do not know if the raw data, ie those used to generate ERA5, have also been archived. If they have, it must be a truly astronomical data set, many orders of magnitude larger than ERA5. It is not at all clear to me that a fully automated AI system will emerge.
lustyd
Well-known member
Any data is useful data, higher resolution is better. You’re describing the problem with the scientists, if it’s not perfect they discount the approach completely. This is why other skills need to be brought in to make some progress and try new things.
If you want to achieve results not seen before, you must employ methods not tried before.
lustyd
Well-known member
So you want to ignore a data set because it’s not exactly what you want? Can you see why progress isn’t being made?
franksingleton
Well-known member
Whether it is pre-computer synoptic data of the current mix of radiances, cloud motion vectors, scatterometer wind data, experimental data such as from Aeolus, meteorologists are well accustomed to working with variable data quality and quantity, using models that are invariably best endeavours. They do not expect perfect dat or perfect forecasts. As I have said many times, the atmosphere is not precise. Neither are or ever will be forecasts.
lustyd
Well-known member
And yet you seem very dismissive of using high resolution realtime data from smartphones despite the many obvious advantages.
franksingleton
Well-known member
That is an odd interpretation of what I said. Yes, all data are useful. There must be some useful information there. I cannot help pointing out that over 70%of the earth is covered by the oceans where there are few mobile phones. I could see such data being useful for short term, local area forecasts but add not very much to the overall data analyses. The cover would be very patchy compared to Meteosat 12. I think the data would be a useful enhancement for local interpretation of national or regional scale predictions.scale. Much of the detail would have very short lifetimes and have disappeared by the time the information was made available to users.
lustyd
Well-known member
I wholeheartedly disagree. Just because the data doesn’t cover the oceans doesn’t mean it’s not worthwhile. The oceans are relatively featureless from a prediction perspective and the consumers of the forecast are mostly on land.
Using realtime high frequency and high resolution data about pressure, sunlight and temperature would be enormously beneficial for both checking efficacy of models as well as adding to the ability to predict. Satellite data has its own flaws, and often scientists ignore them because it seems more scientifically pure a way to collect data. Unfortunately this attitude holds back the whole industry.
Using realtime high frequency and high resolution data about pressure, sunlight and temperature would be enormously beneficial for both checking efficacy of models as well as adding to the ability to predict. Satellite data has its own flaws, and often scientists ignore them because it seems more scientifically pure a way to collect data. Unfortunately this attitude holds back the whole industry.
franksingleton
Well-known member
How useful would temperature be from a centrally heated house? What use would pressure be from a 4th floor flat? Such data might well be useful for now-casting but have little to add aver and above the synoptic networks coupled with satellite observing systems for any modelling whether AI or NWP. You have to think about predictability and representivity. Just look at UK rainfall radar map - Met Office to see the lifetimes of small detail. I think that you are losing sight of what happens in the real world as opposed to an AI one.
Not so. The scientists are all too aware of data limitations. Much of the effort in using all data forms is about maximising data use. This involves realism in what can usefully be predicted. AI might well improve prediction but will still be constrained by noise on all levels from gusts to hurricanes.
Performance of NWP models depends on just how the many data types are combined and on how best to estimate the physical processes, primarily heat transfer. On an experimental basis, these effects are modified in models to give the best results. There are limitations to the amount of testing and the combination of the uncertainties. Ensembles are run to try to see the level of uncertainty.
The machine learning that is the basis of AI forecasting should, in effect, result in better assessments of these model uncertainties by looking at how the real atmosphere has worked in the past, ML will, in effect, be running real models. As there will have been, in effect, more testing I can see that AI should out-perform NWP models. However, being based on a finite data set, the models will still not be perfect.
AI run as in the ECMWF AIFS trial will have the main benefit of saving several hours supercomputer time and provide forecasts 2 or 3 hours earlier than now. If the ML can lead to use of the raw data, then the need to generate an IFS analysis will be avoided. That would save more supercomputer time and forecasts being available very quickly after nominal data times.
lustyd
Well-known member
Sorry Frank, but your response just shows your lack of understanding of the subject. Just as with our previous conversation on AI/ML you need to do some research and understand the topic before being so dismissive.
As I've said multiple times, this is the problem with the incumbent professionals, they're far too set in their ways and unwilling to listen to suggestions. Thankfully, as with other industries, we don't actually need to bring them on the journey with us. Ordnance Survey saw this when Google Maps appeared and then Street-view, luckily they had the good sense to get on board and are now world leading in their use of AI for mapping.
It'll happen with or without the current industry players.
I stated in my previous post that anomalies can be removed. I'm not suggesting a weather man sits in a room and crosses out odd data, this is 2024 and we can fully automate this at planetary scale. It's not even a difficult task these days. That having been said, I can see enormous benefit in home temperatures for weather data purposes since the actual air temperature of homes will vary based on humidity and rainfall which both affect that value. If you're measuring at sufficient frequency and properly processing then there's a lot you can infer from all kinds of data.
As I've said multiple times, this is the problem with the incumbent professionals, they're far too set in their ways and unwilling to listen to suggestions. Thankfully, as with other industries, we don't actually need to bring them on the journey with us. Ordnance Survey saw this when Google Maps appeared and then Street-view, luckily they had the good sense to get on board and are now world leading in their use of AI for mapping.
It'll happen with or without the current industry players.
Marsali_1
Member
"...I can see enormous benefit in home temperatures for weather data purposes since the actual air temperature of homes will vary based on humidity and rainfall which both affect that value. If you're measuring at sufficient frequency and properly processing then there's a lot you can infer from all kinds of data...."
I can't see how you can extrapolate surface measurements in an urban environment into predictions of upper atmospheric conditions over an ocean, particularly when the ocean is upwind from the urban environment as would be the case for a lot of Europe and Asia. There is no question that modelling can be improved with progressively finer sampling, whether that is more frequent timewise or spatially. From what I understand from Frank's discussions there are, at the moment, serious limitations caused by lack of sufficient data from the various atmospheric levels over oceans. I can't see cell phone data filling that void.
I can't see how you can extrapolate surface measurements in an urban environment into predictions of upper atmospheric conditions over an ocean, particularly when the ocean is upwind from the urban environment as would be the case for a lot of Europe and Asia. There is no question that modelling can be improved with progressively finer sampling, whether that is more frequent timewise or spatially. From what I understand from Frank's discussions there are, at the moment, serious limitations caused by lack of sufficient data from the various atmospheric levels over oceans. I can't see cell phone data filling that void.
lustyd
Well-known member
In global weather, there isn't really an "upwind", the planet is round and weather endlessly goes around it (or rather, the planet spins under it).
I didn't say that you could extrapolate upper atmospheric conditions, you made that part up. What I said, was that the internal temperature of houses changes as a direct result of rainfall and humidity. Wet walls lose heat faster, so at times when the heating isn't running the house will lose heat more quickly. This applies even when the heating is "on" since heating is almost always controlled by a simple thermostat (whether it's smart or not, it's still a simple mechanism).
Frank's comments on lack of data are centred around traditional physics modelling. He doesn't know the subject of data science like he knows weather, and I've never said his weather knowledge is anything but superb. This is a very different topic though, and requires a different skillset and mindset. Phone data is just one example of using the data we do have to improve things. With enough data a machine learning model will produce superb results even though nobody would directly understand the mechanism. As I've previously said, even crop yields from prior years could be useful since they will indirectly indicate global conditions. Since we have crop yield data for most of the land on the planet, and land has a greater impact on weather than oceans, this could lead to a huge step up in predictability.
franksingleton
Well-known member
Clearly, you did not follow my train of thought. ML uses algorithms that learn from observed data to make predictions. Whether you like it or not, it is describing the results of the physical processes that drive the atmosphere. There are physical reasons for everything that happens. So, starting from what is happening now, to predict the future AI is using the observed effects of physical processes. When developing NWP models, the scientists simply do not have the resources to test all the possible combinations of data analysis issues and model formulation uncertainty. ML can look at far larger numbers of actual events during which the atmosphere has developed physically. You seem to be treating AI as a large black box which generates forecasts as if by magic. I am saying that AI is using the observed effects of physical processes.
Your arrogance is astounding. An ECMWF paper written in 2018 described an early trial of AI prediction. Before that they had been using neural networks in data analysis. The Met Office works closely with ECMWF. On the path to delivering next generation UK weather forecasts suggests that they see AI as the future. I know that private sector scientists like to stick pins in government institutions. In the case of meteorology, there has long been close liaison with government scientists in the UK, and other countries, particularly the US taking the lead. Much research work is done under contract. ECMWF and the Met Office both attract good scientists. There is far more fluidity than in the far off days when I was a young scientist.
lustyd
Well-known member
Experience, not arrogance. I’m sure it does look that way to you but it’s really not.
Yes, they are dipping their toes in the water but that’s a world away from seriously embracing the technology or even something simple like using different data sets from what they’re used to. People like me have to fight every day to enable progress and in return I get experts mansplaining why I’m wrong right up until we force them to listen by showing them it’s better.
Weather follows patterns whether you agree or not, and it’s not helpful to explain away everything you don’t understand as “turbulence”. That turbulence is just a finer grain than you’re modelling but it’s still a pattern. Where ML excels is that we don’t need to understand what’s causing the effect, and we don’t need direct data to predict it, we just need enough data to predict when it’s going to happen. It’s entirely possible that ice-cream sales across the country correlate better than your physical models to what’s coming next. You’d never know because you’d never try.
Marsali_1
Member
""...I didn't say that you could extrapolate upper atmospheric conditions, you made that part up..."
Maybe I should have used the term "one could" instead of "you could". However, in the context of the prior discussion, it did seem a logical way of phrasing the point. To clarify my use of "upwind" I meant that the cell phone data collected would have no bearing on predictions on what is coming from upwind. Obviously if everyone in Europe and Asia had their thermostats up on bust that heat would eventually arrive back on the continent following at least one revolution of the Earth (assuming some unpredicted perturbation didn't divert it towards the Arctic, for example) and the previously collected cell phone data could then be said to have some predictive value. Otherwise it is surface data just telling you that people have their heaters on because it is cold and I still don't see how that helps in climate modelling or weather forecasting.
Maybe I should have used the term "one could" instead of "you could". However, in the context of the prior discussion, it did seem a logical way of phrasing the point. To clarify my use of "upwind" I meant that the cell phone data collected would have no bearing on predictions on what is coming from upwind. Obviously if everyone in Europe and Asia had their thermostats up on bust that heat would eventually arrive back on the continent following at least one revolution of the Earth (assuming some unpredicted perturbation didn't divert it towards the Arctic, for example) and the previously collected cell phone data could then be said to have some predictive value. Otherwise it is surface data just telling you that people have their heaters on because it is cold and I still don't see how that helps in climate modelling or weather forecasting.
lustyd
Well-known member
You missed my point again. Everywhere is upwind because upwind is infinite. Look at earth :: a global map of wind, weather, and ocean conditions and look down from the poles and you’ll see that what happens in London now absolutely affects America later despite America being “upwind”. Weather just isn’t that immediate.
Again you’re misunderstanding. I’m not suggesting that hotter homes will affect the weather I’m telling you we can use the rate of change of home temperatures to determine heat, humidity and rainfall outside the home. With 30 million homes in the UK that’s a far better data set than the few weather stations measuring the same thing directly. That gives us better resolution of data at a higher frequency, and that will certainly deliver better local forecasts in the places people want them.
If we knew how much surface area was wet, and how wet it was, and how long it took to dry, we’d have a far better idea of how much energy was extracted from the system.
Marsali_1
Member
"...If we knew how much surface area was wet, and how wet it was, and how long it took to dry, we’d have a far better idea of how much energy was extracted from the system...."
If we fully understood the mechanics of the atmosphere or, at the very least had data from locations through out the 3 dimensions where there is no data being collected in quantity, then the modelling that leads to weather forecasts would do a better job. Collecting denser amounts of data from an urban surface environment will not achieve that because it is not measuring atmospheric conditions enough in advance (both spatially and timewise) to make predictions for that immediate location. By the time you measure the variables and analyse the data you are past the event you are wanting to predict.
With regards to your comment about heat movement related to evaporation from wet surfaces, I am surprised that there is need for cell phone data when satellite systems have been measuring surface moisture conditions for some years now at a resolution much finer than Met Office (or their equivalent) surface stations.
I am well aware that what happens in London will eventually affect the Americas as, similarly, do events in the Americas create effects in London. However, for it to be measureable, it would need to be a large event hence the "thermostat analogy". Maybe a better example would be the potential from Three Mile Island and the actual from Chernobyl instead the one I did use.
If we fully understood the mechanics of the atmosphere or, at the very least had data from locations through out the 3 dimensions where there is no data being collected in quantity, then the modelling that leads to weather forecasts would do a better job. Collecting denser amounts of data from an urban surface environment will not achieve that because it is not measuring atmospheric conditions enough in advance (both spatially and timewise) to make predictions for that immediate location. By the time you measure the variables and analyse the data you are past the event you are wanting to predict.
With regards to your comment about heat movement related to evaporation from wet surfaces, I am surprised that there is need for cell phone data when satellite systems have been measuring surface moisture conditions for some years now at a resolution much finer than Met Office (or their equivalent) surface stations.
I am well aware that what happens in London will eventually affect the Americas as, similarly, do events in the Americas create effects in London. However, for it to be measureable, it would need to be a large event hence the "thermostat analogy". Maybe a better example would be the potential from Three Mile Island and the actual from Chernobyl instead the one I did use.
franksingleton
Well-known member
You seem to forget that the Met Office, NOAA, DWD etc are operational organisation. They have to run services to meet many needs from Avistion, shipping, general public forecasts and warnings etc. Any development or changes to the forecasting system has to be carried out carefully and methodically. That applies to new observing system as well as to forecast production. It is quite clear that AI is likely to become an integral part of the system. Clearly, there will be gains in efficiency in terms of computer power. It is less clear that there will be great improvements in forecast accuracy. None of these organisations is going to act hastily. They learned that lesson a long time ago. ECMWF is in a slightly different position. It was set up, primarily to research forecasting weeks ahead. It has morphed into a service provider to its owners - including the Met Office. So, it, too, will proceed slowly and cautiously.
In overall terms, you are correct. Turbulence and showers can be predicted. But, just look at a showery situation over the sea. Do you really think that AI will be able to predict even the location of individual clouds, never mind precipitation intensity?
I think that you and other proponents of AI as a weather forecasting tool are getting carried away by what is, in all probability, a powerful tool. I think that you do not appreciate the limitations inherent in the atmosphere.
lustyd
Well-known member
But we don’t and current techniques make it unlikely we ever will. Don’t lose sight of the fact that we’re not trying to model the atmosphere, we’re trying to predict the weather.
With ML we can concentrate on the exam question of predicting the weather. That means we don’t need to understand, we just need to test predictions and be right more often. It’s not helpful to keep circling back to current techniques or to how the data might be useful, all that matters is whether it produces better results.