ECMWF AI

[franksingleton]

Whether prediction is by AI or NWP, the starting point has to be knowledge of what is happening mow. Even with our current, largely satellite based, observing systems, we never know the precise starting point. Nor do we have sufficient or sufficiently precise data to feed into the machine learning process to predict small scale events. Given the small scale of features that can develop, lack of detail must limit predictability. I am not a butterfly flapping wings person, I recognise that Lorenz’s question was purely rhetorical.
I have little doubt that AI will become a significant factor in weather prediction. I am simply trying to introduce a note of caution. Some while ago, miniPiddy said that she worked at ECMWF. I am now a distant bystander. It woul be interesting to get an insider view. My main source of information is ECMWF Newsletters - eg my quote at #17.

 

[lustyd]

I think you’re making a lot of assumptions there that aren’t necessarily true. I agree that’s the status quo but it’s not necessary that it’s the only approach. If a model is sufficiently good at self learning and correction there’s zero reason we need to start with a full and accurate representation of current weather. Many AI systems use a process of iteration and refinement and this would be an idea approach with weather, comparing predicted with actual and tweaking. This kind of approach may ultimately lead to an ability to fill in gaps in measurement as the model understands more about what causes effects.

 

[franksingleton]

Time will tell but it would be good to have an insider view - if miniPiddy is reading this.

PS In the meantime, I suggest that it is quite possible that two identical situations could have different outcomes. There is a stochastic element to the atmosphere. That is why I believe that an AI model will generate better ensembles but not, necessarily, better deterministic forecasts.

 

[miniPiddy]

Hello
Interesting discussion. As franksingleton noted, most current AI systems are trained on reanalyses like ERA5 — so the "starting point" is still grounded in a physics-based framework and observational data assimilation, even if it's not explicit in the AI design.

There’s growing interest in going further and using raw observations directly for learning or as a starting point. Projects like AI-DOP are exploring that space, though there's still a lot to understand — especially when observational coverage is sparse or uneven. Satellite data will likely play a key role here too; ESA’s recent investment in AI-ready HPC infrastructure points to that shift and EUMETSAT are also investigating this.

Personally, I’m not convinced we’re close to replacing physics-based models entirely, but using them to complement or inform weaker areas is a great chance. But observation-driven AI could prove powerful in contexts like nowcasting, data-denied regions, or as part of hybrid systems and I agree there’s real promise in iterative AI approaches.. It might be that the strength lies in blending all three approaches: physics-based modelling, reanalysis-informed systems, and observation-driven AI.

That said, with the speed that AIFS picked up, I could well be totally wrong! I also hope that humans are still interested enough to understand why AI might perform better (when it does). It's a great opportunity to let something work operationally, while scientists can dig into the physics behind it to learn more, maybe...

But these are just my views (not ECMWF), of course!

 

[franksingleton]

Thank you. As a bystander, I would comment that, however it is predicted, we are dealing with a physical system. Knowledge of the system is necessary to understanding the output, even if that is from a black box.
I do have some direct interest as I am near finalising a 3rd edition of Reeds Weather Handbook. I am having to refer to AI. I also have the uncertainty regarding NOAA and what they will be doing in the future.