Weather all’s

Frank, I've come to the conclusion that Dave is someone who will never give up arguing a point even when that point is not in dispute and will ignore or circle, at his convenience, points which are in dispute and, for which, he has not got a valid counter argument. He's claimed he doesn't need to kniow or read about anything because AI will do it for him. I hope his blind trust in a machine doesn't lead him onto the rocks. Heaven knows there are lots of them in the Hebrides and I doubt even AI can find all of them.
 
Marsali_1 said:
He's claimed he doesn't need to kniow or read about anything because AI will do it for him. I hope his blind trust in a machine doesn't lead him onto the rocks. Heaven knows there are lots of them in the Hebrides and I doubt even AI can find all of them.
Click to expand...
I stated that I don't need to know the mechanism because training the AI replaces that requirement. At a simple level, if you had a bunch of measurements of circle radii and circumferences then you could use machine learning to make a model to translate any radius or circumference to the other. Doing this requires no knowledge whatsoever of Pi or maths, and certainly doesn't need a mathematician.

For the record, points are not in dispute, they are not being understood. There's an important distinction. I could explain in more detail, but to a layperson the explanation would be no more useful. This would be similar if Frank were explaining in detail how weather systems work, at some point you just have to trust the expert.

It isn't blind faith, it's expertise. You have to understand that we aren't discussing Frank's expertise here, we're discussing mine. Frank continues trying to derail the conversation with expertise from another field which is frankly irrelevant to how AI and ML function. It's a problem I continually faced in my career with incumbents slowing progress until they either learned about AI or were replaced. Either way, the AI projects were always successful with or without the former experts.

In this instance we're speaking about forecasting. The purpose of which is to furnish people with information about the most likely weather over the coming days. Those people care little for how it was derived or whether we understand how it came to be. They just want reliable predictions. AI can do that and it can do it well if trained on sufficient data. Not only that but it can do it faster and more often for less money. It's not like the incumbents are getting the forecast perfect on a regular basis, so there's a pretty low bar in terms of success.

I'm happy to have an in depth discussion about this, my day rate is £2500 plus expenses. Otherwise, this is a public forum and you get public forum answers.
 
On the question of AI, I think that we agree that it will come, but slowly. David knows far mor about AI than many of us. Clearly, he knows much more about dealing with large data sets. Likewise, I probably know far more than many about the meteorological and oceanographic data. AI is a data driven system. I know that he says that all the data problems will be dealt with by the ML. The meteorologists are less sure. There is concern that the variability in the atmosphere, the noise in the data, is so great that AI might not deal well with extreme events, an increasingly important consideration today. Less than 40 years of ERA5 data are used by the AIFS. We, I at least, can only speculate on the effect of having 100 years of ERA5.

AIFS training used the well ordered ERA5. Raw dat is proving to be a different problem. The data characteristics will be continually changing. With physical modelling, new instruments and even replacement sensors are only used operationally after testing and evaluation. What will happen in an AI system? I am not trying to debunk AI. I am just interested in the practical aspects of making the system robust in an operational environment where lives and safety can easily be jeopardised

Meanwhile, the various national weather services will carry on providing services to a great range of needs and introduce new methods carefully as they have done for many years now. ECMWF has more freedom as its main raison is to concentrate on the 3 to 15 day timescale for prediction while developing prediction techniques. There is less real time pressure. The larger state services use ECMWF for their outlooks beyond the first 2 or 3 days and, of course, forecasters are always watching output from other centres, including ECMWF.


Where I differ most from David is in his perception of the approach of national weather services to new ideas and techniques. I can only speak for the UK Met Office, but, even before computers existed, they were thinking about NWP. After WW2, progress in the UK was affected by the destruction of Colossus. It took until the late 1970s before we caught up with and surpassed the US. The NWP developers were continually looking ahead to the next computer even before the latest one was being installed. Similarly with observing systems, especially from space. In the mid 1990s, the Met Office and ECMWF were studying Radio occultation which was not introduced into NWP models until 2007. That might seem a long time but there always has to be great care when allowing the implementation of any new data or prediction system.

I may not be around when a full AI system is introduced but I wish it all success and will watch developments with interest - at a very modest hourly rate.
 
Last edited:
There is not enough money in weather forecasting for the ai lot to be interested in the short term.
The companies will be chasing the easy stuff like fashion and gambling.

It is new and just like years ago Computers were the new thing to free everyone from work.
Well that didn't happen, working from home is a good example.

AI is the new dot com and time will tell how it develops.
 
Mister E said:
There is not enough money in weather forecasting for the ai lot to be interested in the short term.
The companies will be chasing the easy stuff like fashion and gambling.

It is new and just like years ago Computers were the new thing to free everyone from work.
Well that didn't happen, working from home is a good example.

AI is the new dot com and time will tell how it develops.
Click to expand...
Well, major national Met services individually and through ECMWF have been able to obtain massive computer power in the past. Running those computers is costly. If AI can save the Met Office about 20 computer hours a day and ECMWF maybe a similar amount, the business case is strong. A good weather service is a must for the national economy, we all gain.

Even before the Rayner review under Thatcher, the Met Office has had to look at its value to the nation. A quick search will give you data for 2015.

PS, More recent data are at Met Office delivers £56 billion of economic value to the UK - London Economics and Met Office delivers £56bn of economic value to the UK, finds new report - Emergency Services Times.
 
Last edited:
I thought it was quite self explanatory from the post.
AI has been used against extraordinarily large sets of data to produce weather forecasts better than physics models can due to their ability to use millions or billions of parameters. Something traditional approaches could never hope to do.
This project aims to extract algorithms from that approach to inform more traditional weather science in how weather works. Instead of trying to understand the physics and then model it, this gives you a model of how it works to reverse engineer, and tells you which parameters were contributing or important.

The approach has been used in many fields to drastically increase the productivity of scientists in understanding their subject. Medical science in particular has moved exponentially faster with these techniques and led to breakthroughs we couldn’t have imagined just ten years ago. Physics has also benefited in a similar way.
 
lustyd said:
I thought it was quite self explanatory from the post.
AI has been used against extraordinarily large sets of data to produce weather forecasts better than physics models can due to their ability to use millions or billions of parameters. Something traditional approaches could never hope to do.
This project aims to extract algorithms from that approach to inform more traditional weather science in how weather works. Instead of trying to understand the physics and then model it, this gives you a model of how it works to reverse engineer, and tells you which parameters were contributing or important.

The approach has been used in many fields to drastically increase the productivity of scientists in understanding their subject. Medical science in particular has moved exponentially faster with these techniques and led to breakthroughs we couldn’t have imagined just ten years ago. Physics has also benefited in a similar way.
Click to expand...
Having had several attempts to read this article, it is no longer accessible but there seemed to be some muddled thinking. In suggesting looking for some relationships it referred to pressure and density separately. But, if you know about density, you know about pressure. In the same list as pressures density, temperature, it mentioned vorticity but this can only be calculated from wind data. Winds are only available global around jet stream levels. It just seemed an odd grouping.

AI will surely come and have an important part to play in making more efficient use of computer hardware but is unlikely to be a catch all. Because it learns using actual data, it does not sample enough extremes and seems to do less well with unusual events. Detailed, local area forecasts, seem to do less well using AI. There will be a problem with a changing climate as well as with newer generations of sensors. The learning will have to be kept under continual review. Met Office and Turing are looking at some forms of hybrid approach. Knowing what I do about meteorology, that seems a sensible, pragmatic approach. Weather prediction has always had to be tackled using the art of the possible. AI will not change that.

The real gain with AI will come when it can use raw data. I have said before that this will be a different problem from starting with the ERA5 data set or the new ERA6. The latest I can find about use of raw data is An update on AI–DOP: skilful weather forecasts produced directly from observations. Clearly, there is some way to go. The list of data used is interesting.
 
franksingleton said:
It just seemed an odd grouping
Click to expand...
Ironically thats the whole point. It’s probably not an odd grouping because the AI has found connections and possibly more subtle interactions. You may not be aware of those connections due to the limited way these things have previously been studied. Working with more data and more parameters these things can be identified and extracted. Weather scientists might very well spend a decade working on the science to explain the algorithm, but this way in the mean time we can benefit from it immediately
 
lustyd said:
Ironically thats the whole point. It’s probably not an odd grouping because the AI has found connections and possibly more subtle interactions. You may not be aware of those connections due to the limited way these things have previously been studied. Working with more data and more parameters these things can be identified and extracted. Weather scientists might very well spend a decade working on the science to explain the algorithm, but this way in the mean time we can benefit from it immediately
Click to expand...
I was not doubting that AI might reveal some unexpected relationships. I was trying to point out that quoting pressure and density was repetition. Knowing density implies knowledge of pressure. Also, that vorticity was not observable in the sense of, say, temperature but had to be calculated knowing winds over the globe. Apart from tracking cloud and water vapour at high levels, wind cannot be observed globally. Quoting vorticity implying that its observation is on a par with pressure and wind is odd.
I am trying to point out that AI learning using an ERA5/6 data sets is one matter and may not need much knowledge of the dats, using raw data is a different ball park. Not understanding the data could result in going up blind alleys.
 
Came across this quote earlier...
“The greatest enemy of knowledge is not ignorance; it is the illusion of knowledge.”
Stephen Hawking

Seems to me that this highlights one the dilemmas of using AI to produce results. It helps with human understanding if we can understand the basis of correlations found even if we don't understand the subtleties. Blind faith has never seemed rational to me.
 
franksingleton said:
I was trying to point out that quoting pressure and density was repetition
Click to expand...
They may be related but they certainly aren’t the same thing by a long way. Treating them interchangeably would be a big mistake.

Modern AI techniques take all data. It makes no difference whether you think there is a downside to raw data, the learning algorithms can use it just the same since the relationships are there when enough data and parameters are used. Physics models were produced in the same way, just by humans over hundreds of years using a handful of parameters at a time. We spot patterns, find relationships and try to explain them. All this does is increase parameters and data, and use them all at once at previously unimaginable scale. Humans just aren’t necessary to that part of the process, if anything our limitations hold it back.
 
RobbieW said:
Came across this quote earlier...
“The greatest enemy of knowledge is not ignorance; it is the illusion of knowledge.”
Stephen Hawking

Seems to me that this highlights one the dilemmas of using AI to produce results. It helps with human understanding if we can understand the basis of correlations found even if we don't understand the subtleties. Blind faith has never seemed rational to me.
Click to expand...
There are two things happening here. A model is produced that more accurately than ever can predict weather and for longer periods. This we must have faith in and monitor results since we currently cannot understand what’s going on.
The second is what the link described, whereby the model is documented and scientists then have to work on understanding the mechanics. Imagine being given e=mc^2 and having to then reverse engineer the processes, this is like that but with far more complex formulas and algorithms, each having potentially thousands of parameters. Eventually we will understand the detail, but it’s not necessary for using the results of the model. I’ve said it before, scientists will be removed from forecasting within a decade and their focus will move entirely to understanding the science.


Also, it’s not that we don’t understand what’s going on and how the AI works. Don’t mistake your lack of understanding for “nobody knows”. The people working on AI models know exactly how they’re being produced and how they should be used, just as we saw with ChatGPT.
 
The predict wind AIFS was pretty crap at forecasting wind strength for storm Amy, around the Oban area today. From a few days ago to last night it predicted F7-8 and even today, was predicting F7 when other models were consistent around the F9. I guess it will learn from this.
 
RunAgroundHard said:
The predict wind AIFS was pretty crap at forecasting wind strength for storm Amy, around the Oban area today. From a few days ago to last night it predicted F7-8 and even today, was predicting F7 when other models were consistent around the F9. I guess it will learn from this.
Click to expand...
What was the IFS saying?
GRIBs will invariably under predict because of smoothing in the model formulation. Effective grid length for IFS is about 50 km. It cannot predict the strongest winds.
 
RunAgroundHard said:
The predict wind AIFS was pretty crap at forecasting wind strength for storm Amy, around the Oban area today. From a few days ago to last night it predicted F7-8 and even today, was predicting F7 when other models were consistent around the F9. I guess it will learn from this.
Click to expand...
I saw a video of Loch Aline and am glad we went south, currently in Howth with a measly 58kt wind.
 
If IFS is ECMF then it was a point higher BF which was lower than other models as well. All the models under predicted, with F9/10 steady winds. On the validation section of PW, based on last week, AIFS ranked 1st for wind speed. Today’s storm is worse than all the models predicted.
 
Last edited:
lustyd said:
I saw a video of Loch Aline and am glad we went south, currently in Howth with a measly 58kt wind.
Click to expand...

It is quite wild, has been so since 4 pm with huge seas in the Sound of Jura and gusts that blew me over, even when braced. The sea spray from the breaking waves stung. Not a pleasant day. Pressure dropped to 965 mBar.
 
You must log in or register to reply here.

Share this page

Top