lustyd
Well-Known Member
No, it’s composed by an expert who worked at the largest tech company in the world and specialised in data and AI.
Weather all’s
- Thread starter franksingleton
- Start date
franksingleton
Well-Known Member
As we know, ECMWF uses their ERA5 set of historical analyses and, so, have to use their physical model to create a starting point for AIFS. As I expected, starting from the raw data seems to be a different kettle of fish. The raw observational data vary enormously in their characteristics. At one end of the scale, there are data that should give accurate in situ values of temperature, humidity, wind, pressure at the surface and above it. At the other end of the scale there are large volumes of data that show in the effects of the atmosphere on transmission of radiation in infrared, microwave and VHF wavelengths. Some of these data are only available over the oceans.
These data cannot provide profiles or vales of meteorological elements.
In between there are global high level wind data from tracking cloud and water vapour using infra red and visible light. There are swathes of wind data measured over the sea using scattering of radar beams from LEOs. New types of sensor are continually being developed. Even when sensors are replaced by new ones of the same design, it takes a lengthy period before the data can be used. With some exceptions, notably radio occultation, instrumental drift is a perpetual concern.
Weighting of the various data types has long been a problem that has never been solved satisfactorily. Quite likely, all these problems will be solved in time although I suspect that progress will be slower than trials such as AIFS might suggest. I have seen too many early success stories to get over excited in ivory tower.
Both hECMWF and the Met Office talk about composite systems without saying, as far as I have seen, what that means. It might be that AI could generate large scale models while local detail is best done using physical models. Could AI be used to improve data weighting in physical models? I see that there is work on AI for seasonal forecasts. Those of us of a certain age are scarred my memories of abysmal failures of monthly forecasts.
The above is based on my hands-on knowledge of the data. We have at least one forumite who works for ECMWF. It would be interesting to see an insider view of progress in AI with some indication of direction and. Speed of travel. I hope that we are not se ring a false dawn.
lustyd
Well-Known Member
It’s interesting because many/most of those issues go away with AI. Incomplete data isn’t actually a problem, the model can use what it has where it has it and effectively assume similar will happen elsewhere.
Weighting is entirely automated in large models so again that problem goes away entirely. More data means better weighting.
As we’ve seen with GPT, hallucinations will be a problem. If you feed the AI just the temperature data for an area and ask for a forecast it could well confidently make a prediction with insufficient data. Thankfully those problems have been solved on less important projects like GPT.
From my perspective the problem is and always will be funding. It’s a do or do not do type thing. Either give it all of the data and spend a year on training or don’t bother. Google are one of the few with such vision and luckily they are involved. Microsoft have the hardware and skill but won’t fund it. The weather orgs lack the funding and hardware to pull it off, and I’m not yet convinced they have the vision to think that big organisationally, nor do they understand the monetisation approach required (which is similar with most science based orgs).
One thing is for sure, progress will be slow for a while but then hopefully will see a sudden jump
Weighting is entirely automated in large models so again that problem goes away entirely. More data means better weighting.
As we’ve seen with GPT, hallucinations will be a problem. If you feed the AI just the temperature data for an area and ask for a forecast it could well confidently make a prediction with insufficient data. Thankfully those problems have been solved on less important projects like GPT.
From my perspective the problem is and always will be funding. It’s a do or do not do type thing. Either give it all of the data and spend a year on training or don’t bother. Google are one of the few with such vision and luckily they are involved. Microsoft have the hardware and skill but won’t fund it. The weather orgs lack the funding and hardware to pull it off, and I’m not yet convinced they have the vision to think that big organisationally, nor do they understand the monetisation approach required (which is similar with most science based orgs).
One thing is for sure, progress will be slow for a while but then hopefully will see a sudden jump
Arcady
Well-Known Member
I tend to look at them all and get then go with the one I most like the look of ;-)
franksingleton
Well-Known Member
Totally agree although maybe not for quite the same reasons. National Met services support many aspects of life, some with serious safety connotations. New systems are trialled thoroughly with operational back-up systems.
AAMOI, what other physical systems are modelled by AI?
lustyd
Well-Known Member
There are various projects ongoing, not a lot completed as such widescale training is a relatively new and expensive approach.
I worked on one where we were using AI to predict where rocks would have fewer faults using historical data from core samples. This was for two reasons, first core samples are expensive as heck and second, core samples make holes in the rock which makes it unusable for the intended purpose.
Most of the projects aren’t public so can’t be discussed. Ironically the more important ones tend to be more open about what they’re up to.
As you’d expect though, trivial projects in retail are the biggest use case and are paying the bills to advance the techniques. I also wrote a fashion app that helps people work out an outfit to wear and accessorise it.
franksingleton
Well-Known Member
Your geological project was no doubt challenging and complex but totally different from weather modelling. It is a truism to say that, to know about weather somewhere, you have to know about weather everywhere. Roughly speaking, 1/5th of the atmosphere is above the tropopause. So, sone change at, say, 30km over Japan might be related to weather over Europe at some future time. No doubt, you will say that such effects will come out in the wash with AI. I will not disagree. However, it highlights the magnitude of the challenge in weather prediction whether by physical modelling or AI.
Are there any AI examples with cause and effect being modelled over time and space scales varying from hours upwards to days, weeks, months, years and, spatially, from what is happening to my boat in mid channel to heatwaves and floods affecting continents.
I am not trying to debunk AI. It is a valid approach which, in far more limited circumstances, has been tried in the past. We now have the tools to make it work. It may or may not have more potential than physical modelling. The suggestion of composite approach is interesting. You are right to say that it will be a slow process. I suspect that progress will be incremental, quite likely, non-linear. It will never be perfect and the main improvements may well be in probability prediction and in timeliness. ?It is unfortunate that those who do not know that AIFS is a halfway house to a true AI system, have hailed it as a major breakthrough and assumed that weather forecasting will improve greatly. They do not seem to realise that, however it is predicted, the atmosphere is driven by the laws of physics. There is chaos on many scales and that forecasts cannot be precise.
Last edited:
lustyd
Well-Known Member
This is entirely incorrect, and is the root of the issues in the industry I keep mentioning. People deeply set in their ways explaining why there’s only one solution have been eventually displaced and proven wrong in every industry. Ordnance Survey would have told you photographing every street in the world was unachievable 20 years ago, and that’s literally their job.
The problem is that weather people aren’t trying different ways of solving the problem.
franksingleton
Well-Known Member
Wrong on several accounts. Meteorologists were looking forward to modelling back as far as 1908. Ever since the advent of computers they have been implementing developments in technology. As AI developed they have been studying how to use it - at a rate that caught me on the hop. They have always been prepared to use new tools but are always having to be careful as there are many real time users.
As regards my truism, you seem to have no understanding of the atmosphere. I realise that ChatGPT is not infallible but it is a good starting point and, as I have found, pretty objective. Take a look.
lustyd
Well-Known Member
Exactly my point, you don’t understand what I’m saying so you revert to type and start attacking with what you know. Absolutely pointless conversation as you simply have no grasp of how progress can happen outside of your narrow expertise.
My expertise means I don’t need an understanding of the atmosphere to make progress. Modern techniques mean that a billion computers work it out so we don’t have to. Your techniques rely entirely on individuals trying things one at a time for thousands of years, or on unimaginable compute being used to model the planet at the atomic scale…which is why you and your colleagues have failed to make much progress.
franksingleton
Well-Known Member
I rest my case.
lustyd
Well-Known Member
As do I. If you want to see results never before seen, you must be willing to try methods never before tried. Experts hold back fields just as often as they progress them, if not more so. At a certain point, education becomes indoctrination and it takes a disruptor to move things forwards. History calls those disruptors geniuses, while their contemporaries often call them lunatics.
I certainly recall our discussion where you denied AI was even being tested and were confident it would never work or help. I dragged you forwards and then you wrote an article claiming expertise as a result, yet here we are again with you insulting me and trying to educate me on a field I know orders of magnitude more about than you. I’m sure in time you’ll enjoy some more humble pie.
I certainly recall our discussion where you denied AI was even being tested and were confident it would never work or help. I dragged you forwards and then you wrote an article claiming expertise as a result, yet here we are again with you insulting me and trying to educate me on a field I know orders of magnitude more about than you. I’m sure in time you’ll enjoy some more humble pie.
Marsali_1
Well-Known Member
"My expertise means I don’t need an understanding of the atmosphere to make progress." I wonder if I can make progress in the medical field by adopting this line of thinking?
Chiara’s slave
Well-Known Member
Whilst the AI approach will lead to better forecasting, I can’t quite see it taking over, it’ll still need the modelling data surely? Or can AI forecast weather without any data input? I doubt if it’ll be forecasting the time of arrival of the sea breeze with any accuracy 24 hours in advance. What we might end up with is a much more regularly updated forecast, say, in time to get the washing in before it rains ‘unexpectedly’. Things that would help me as a sailor would be accurate gust forecasting, and some kind of frequency index, and a few more days in advance. UKV2 is pretty good, but if AI could improve just that, I’d be a believer. I’ve seen some magnificent cock ups it’s performed though, so before relying on it, I’d want to monitor it in action for a few years.
lustyd
Well-Known Member
Yes, many people are doing just that. Quite a lot of progress in cancer treatment and detection has been made using AI and done by people with no medical training whatsoever. Meanwhile the trained doctors have continued to refine ways of cutting bits out of humans. It’s a classic example of indoctrination holding up progress.
lustyd
Well-Known Member
Data will always be needed, but the modelling phase will eventually be replaced, making the updates faster and less processor intensive. That’s assuming someone decides to fund it properly and allows access to data. Often the incumbent experts get testy and block access to data to try and prevent progress.
franksingleton
Well-Known Member
These two posts demonstrate your blinkered mind.
But only with input from medical experts.
That is one possible outcome. Meteorologists are, wisely, keeping their options open as to whether forecasting will be entirely AI or still have a physical modelling component.
You are showing your ignorance of how meteorology in its widest sense has been developing over the last 100 years.
lustyd
Well-Known Member
Unfortunately your complete lack of knowledge of the area is showing again Frank. If you’re going to weigh in on AI conversations with experts then you’ll need to do some reading and try to understand the subject from at least a basic point of view.
Medical experts are used to evaluate models, but are largely unnecessary for the training since their input was recorded years or decades ago in the training dataset.
Your lack of understanding prevents you from acknowledging that to predict the weather we don’t need to understand the weather or the physics of it, we just need a system able to interpret the data. We don’t even need complete datasets because many things can be inferred from others fully automatically. The quantity of data your brain is able to process is tiny and you’d be looking at perhaps hundreds of parameters. A modern AI will account for all available parameters and has no limitations in terms of relationships, data set size, data set completeness etc. the only limitation is how long it takes to train and how much that costs. Those will be measured in years and billions.
Weather science will continue, but weather scientists will be gradually removed from the field of forecasting as they just aren’t necessary. Sorry if that offends you, but it’s happening in all fields.
franksingleton
Well-Known Member
Sorry, but you deal in partial truths in both fields. Rather than me trying to argue the toss, I suggest that asking ChatGPT about your claims produces answers that brings up some interesting facts. I am not saying that AI will never remove the need for physical models or the need for um a forecasters. The time is not yet. Similarly with medicine. AI is likely to become increasingly valuable but the need for doctors will continue into the foreseeable future.
As regards the need for data, try reading about teleconnections.
As regards the need for data, try reading about teleconnections.
lustyd
Well-Known Member
Doctors are already changing the way they work. They’re more hands on than before thanks to the technology.
I don’t need to read about teleconnections, that’s the point. AI modelling will find thise relationships without ever having to explain them.