zyGrib has been forked and is now continuing as XyGrib

GHA

Well-known member
Joined
26 Jun 2013
Messages
12,104
Location
Hopefully somewhere warm
Visit site
For a viewer I've gone over to opencpn these days, overlaying synoptics /sat images over gribs is really useful, also having the option of putting in a waypoint with actual pressure to check how a front is moving against the forecast.

This weeks favourite book :cool:
https://www.amazon.co.uk/Barometer-...=1523221583&sr=1-1&keywords=sailing+barometer

6C2351D.png
 

franksingleton

Well-known member
Joined
27 Oct 2002
Messages
3,385
Location
UK when not sailing
weather.mailasail.com
.

Are we at that limit for meteorology yet? As a layman, I doubt it. I expect that finer detail and longer ranges will both be possible ... and since finer detail runs into the problem of unpredictable Scottish glider pilots starting thermals, I think range is the way to go. Maybe an ultimate goal of forecasts a week ahead as good as we currently have for a day ahead?

It really all depends on what level of detail you are trying to predict. We know that there are tele-connections - ENSO and the Indian Ocean oscillations are linked to our weather and better understanding might lead to useful monthly or seasonal type predictions although I have seen many false dawns here. More specifically, the lifetimes of weather structures that affect us directly depend on the drivers. The Indian Ocean SW Monsoon is caused by heating over a large area. The onset and severity may be difficult to predict but, once it has been seen to be forming, the weather is fairly predictable. You could0 write a TAF for Salalah a month ahead and get it right- the 100 day fog along the S coast of Arabia.

The depressions that dominate weather across the N Atlantic are formed by large scale ascent associated with the polar front. Seeing one start to form its development is predictable in general terms for days ahead - a week or more. That is why we can often, not always, see a weather window opening 7 or 8 days ahead. Not the detail within the systems, of course. Details, such as troughs in surface isobars associated with or caused by thundery activity have lifetimes of around 2 days. Once identified, they are predictable on that time scale. Your large cumin was driven initially by local convection. Once identified it is, in principle, predictable for around 6 hours. Your small fine weather cumulus so loved by glider pilots have lifetimes of minutes - up to an hour or so.

No matter how good are the data, you cannot get away from the theoretical predictability. I am sure that meteorologists will strive towards those limits.

As regards better data, Antarctic Pilot is in a better position than I to know what is on the observational horizon. Satellites make precise measurements but these cannot be used to define the atmosphere precisely. Effective use of the data is an ongoing problem. Observing resolution vertically and horizontally will improve in time.

How much the model physics can be improved is uncertain. Radiative transfer, so important in operational forecasting and in climate change studies, is always going to be a major problem. There is no currently foreseeable way of calculating the radiative effects of every bit of cloud. OK when you have total cover of a uniform layer but dealing with radiation with large convective clouds must be nigh impossible. That is why they use parameterisation - a long word for intelligent guesswork.

Improvements in our operational forecasts will be incremental and asymptotic in nature. I suspect that we are nearing the law of diminishing returns. Some while ago, one of my former colleagues estimated that the atmospheric computer was more powerful than the latest machines by a factor of 10 to the power of 35. I am not sure what Moore’s law would say about that. The constraints may reduce “simply” to the observational issue and predictability.

Following your gliding exploits, we were once chased by a large cumin from Les Minquiers to St Helier., we were in a motor boat doing 30 knots. It was keeping pace with us.
 
Last edited:

RobbieW

Well-known member
Joined
24 Jun 2007
Messages
4,597
Location
On land for now
Visit site
I'm just trying to visualise the effect of a cumin layer on wind but I'm rather hoping its your spell checker that has substituted that for cunim ?
 

Joker

Active member
Joined
2 Jul 2010
Messages
1,081
Location
location location ...
Visit site
'The mentioned Gribs are now readable in the update.'

That's wonderful. I am currently involved in an update to 'Cruising Guide to Germany & Denmark', and that is a really useful resource. I'll make sure I recommend it widely.
 

AntarcticPilot

Well-known member
Joined
4 May 2007
Messages
10,011
Location
Cambridge, UK
www.cooperandyau.co.uk
It really all depends on what level of detail you are trying to predict. We know that there are tele-connections - ENSO and the Indian Ocean oscillations are linked to our weather and better understanding might lead to useful monthly or seasonal type predictions although I have seen many false dawns here. More specifically, the lifetimes of weather structures that affect us directly depend on the drivers. The Indian Ocean SW Monsoon is caused by heating over a large area. The onset and severity may be difficult to predict but, once it has been seen to be forming, the weather is fairly predictable. You could0 write a TAF for Salalah a month ahead and get it right- the 100 day fog along the S coast of Arabia.

The depressions that dominate weather across the N Atlantic are formed by large scale ascent associated with the polar front. Seeing one start to form its development is predictable in general terms for days ahead - a week or more. That is why we can often, not always, see a weather window opening 7 or 8 days ahead. Not the detail within the systems, of course. Details, such as troughs in surface isobars associated with or caused by thundery activity have lifetimes of around 2 days. Once identified, they are predictable on that time scale. Your large cumin was driven initially by local convection. Once identified it is, in principle, predictable for around 6 hours. Your small fine weather so loved by glider pilots have lifetimes of minutes - up to an hour or so.

No matter how good are the data, you cannot get away from the theoretical predictability. I am sure that meteorologists will strive towards those limits.

As regards better data, Antarctic Pilot is in a better position than I to know what is on the observational horizon. Satellites make precise measurements but these cannot be used to define the atmosphere precisely. Effective use of the data is an ongoing problem. Observing resolution vertically and horizontally will improve in time.

How much the model physics can be improved is uncertain. Radiative transfer, so important in operational forecasting and in climate change studies, is always going to be a major problem. There is no currently foreseeable way of calculating the radiative effects of every bit of cloud. OK when you have total cover of a uniform layer but dealing with radiation with large convective clouds must be nigh impossible. That is why they use parameterisation - a long word for intelligent guesswork.

Improvements in our operational forecasts will be incremental and asymptotic in nature. I suspect that we are nearing the law of diminishing returns. Some while ago, one of my former colleagues estimated that the atmospheric computer was more powerful than the latest machines by a factor of 10 to the power of 35. I am not sure what Moore’s law would say about that. The constraints may reduce “simply” to the observational issue and predictability.

Following your gliding exploits, we were once chased by a large cumin from Les Minquiers to St Helier., we were in a motor boat doing 30 knots. It was keeping pace with us.

I've just got back from holiday, and caught up with this discussion. JD and Frank, thanks for an interesting and enlightening debate! I'd just like to emphasize a couple of mathematical issues that both Frank and JD hint at. First, most natural systems are fractal in nature, and weather is no exception. There is self-similar detail at all scales from the smallest to the largest, as is shown by the fact that you can generate extremely realistic images of clouds using fractal techniques. And the fractal nature of natural systems allows me to get all pedantic about the impossibility of measuring the length of a coastline or the number of islands in an archipelago - and that same impossibility means that in some senses the detail of weather is unmeasurable without careful definition and understanding of the parameter being measured. I would always temper a "how long" measurement by stating that the coast of X is Y kilometres at a length scale of Z metres - because the length increases infinitely as Z decreases! JD's dust-devil that became a thermal is a beautiful example of a small-scale phenomenon growing to a larger scale. Second, and I know that here JD and Frank are better placed to elaborate than I am, weather is an intrinsically chaotic system. What that means is that tiny changes in initial conditions can result in very large changes in later conditions. The usual analogy is the butterfly in the Amazon flapping it's wings and creating a tornado in Kansas! Tiny changes can put you one side or another of a "fork" between two different situations, and the number of influences and forks is such that it is often impossible to predict what the ultimate outcome will be. If JD had pitched his tent a metre away from where it was, the dust devil might have collapsed rather than growing! Frank mentions situations where a weather system, once established, is very stable - but such situations rarely occur in the British Isles, where we have weather and not climate!

I'm afraid I am less able to describe satellite observation systems than I once was; since retiring I don't have the same access to things that I used to have. But all satellite systems have limitations; often the problem is that they measure things over footprints that are too large or too small! And often you don't measure the actual thing you want to measure, but another parameter that relates to it. For example, Sea Surface Temperature is pretty reliable, but what you measure is actually the infrared emissivity of the surface, which is then converted to a temperature using a modified black-body equation, which has been verified empirically.But I am sure there are situation where the relationship breaks down!

Finally, you'd think that it would be easy to automatically delineate the extent of exposed rock in Antarctica using satellite images; after all rock is black and snow is white! But it actually turns out to be very difficult when you take into acount the problem of shadows - and most exposed rock occurs on steep slopes where shadows on slopes facing away from the sun include about 50% of the outcrops. It is only fairly recently that new satellites with greater spectral resolution and higher resolution have allowed rock outcrop to be delineated automatically with better accuracy than manual tracing - which was prohibitively time consuming except in small areas (I've had to do it!).
 

JumbleDuck

Well-known member
Joined
8 Aug 2013
Messages
24,167
Location
SW Scotland
Visit site
It really all depends on what level of detail you are trying to predict ...

Many thanks for your detailed and interesting answer. I expect it's complicated by the different type of detail as well - some people might really want to know if they will be in a patchy shower area, but be less concerned about whether rain falls on a particular spot [1] while others will care only about one spot [2].

[1] When I had some building work done in a river, it would have been useful to know whether any rain was likely in the main catchment area of about 200 square miles, but exactly whereabouts in that area would have been irrelevant.

[2] Tennis people probably care a lot whether rain falls on a couple of acres of Wimbledon, but don't care what happens half a mile to one side

Following your gliding exploits, we were once chased by a large cumin from Les Minquiers to St Helier., we were in a motor boat doing 30 knots. It was keeping pace with us.

Scary buggers, ain't they?
 

JumbleDuck

Well-known member
Joined
8 Aug 2013
Messages
24,167
Location
SW Scotland
Visit site
Second, and I know that here JD and Frank are better placed to elaborate than I am, weather is an intrinsically chaotic system. What that means is that tiny changes in initial conditions can result in very large changes in later conditions. The usual analogy is the butterfly in the Amazon flapping it's wings and creating a tornado in Kansas! Tiny changes can put you one side or another of a "fork" between two different situations, and the number of influences and forks is such that it is often impossible to predict what the ultimate outcome will be.

We used to call these "ill-conditioned" problems, the idea being that if d(solution)/d(boundary condition) wasn't finite, an infinitesimal change in BC gave a finite change in solution and you were, mathematically, stuffed. "Chaotic" seems to be the new, trendy name for the same effect. Harumph. Kids today. Get off my lawn.

I've never worked in that area, but when I have done numerical modelling it has been one of the things to check and avoid. As Frank has said, all sorts of nasty things happen when you try to model on a scale inappropriate to the physical reality - equally nasty things can happen as purely mathematical artefacts. Get your finite difference grid too coarse and perfectly model-able systems go all unstable on you.
 

AntarcticPilot

Well-known member
Joined
4 May 2007
Messages
10,011
Location
Cambridge, UK
www.cooperandyau.co.uk
We used to call these "ill-conditioned" problems, the idea being that if d(solution)/d(boundary condition) wasn't finite, an infinitesimal change in BC gave a finite change in solution and you were, mathematically, stuffed. "Chaotic" seems to be the new, trendy name for the same effect. Harumph. Kids today. Get off my lawn.

I've never worked in that area, but when I have done numerical modelling it has been one of the things to check and avoid. As Frank has said, all sorts of nasty things happen when you try to model on a scale inappropriate to the physical reality - equally nasty things can happen as purely mathematical artefacts. Get your finite difference grid too coarse and perfectly model-able systems go all unstable on you.

Chaotic also refers to systems where the outcome of an operation is a) unpredictable without carrying the operation out and b) has two or more dichotomous outcomes. The classic mathematical example is the Mandelbrot set. You can only find whether the function diverges or not by evaluating it; there is no expression that predicts divergence or convergence for any particular starting point. IT people like it because you can generate pretty pictures with it!
322px-Mandel_zoom_00_mandelbrot_set.jpg
 

JumbleDuck

Well-known member
Joined
8 Aug 2013
Messages
24,167
Location
SW Scotland
Visit site
Chaotic also refers to systems where the outcome of an operation is a) unpredictable without carrying the operation out and b) has two or more dichotomous outcomes.

I think that's the same thing as I said, basically, because in the Mandelbrot or Julia sets the outcome (convergent or not) changes finitely with an infinitesimal change of boundary conditions. Is this the most semantic digression ever on these forums?
 

AntarcticPilot

Well-known member
Joined
4 May 2007
Messages
10,011
Location
Cambridge, UK
www.cooperandyau.co.uk
I think that's the same thing as I said, basically, because in the Mandelbrot or Julia sets the outcome (convergent or not) changes finitely with an infinitesimal change of boundary conditions. Is this the most semantic digression ever on these forums?

Probably! You're putting it in terms of physical systems; I put it in mathematical ones (rather boldly, as I'm far from being a mathematician!)
 

franksingleton

Well-known member
Joined
27 Oct 2002
Messages
3,385
Location
UK when not sailing
weather.mailasail.com
All very interesting but however you look at it in academic terms, the practicality is that wearher forecasting is an initial value problem. Leaving aside the questions of real or theoretical predictability, the starting point for any non-trivial forecast has to be observational data.

Paul made a good point when he said that satellites often do not measure what we really want. Another good example is that of measuring the temperature structure of the atmosphere. This is fundamental in models. From a well defined surface pressure/wind field, knowing the vertical and horizontal temperature fields tells you about the flow at all levels. Over the oceans there are few in situ observations from radio-sondes launched from islands and none or virtually none from ships since the regrettable demise of the ocean weather ships. Satellites measure the absorption of infra-red by CO2. The absorption depends on the air temperature structure - CO2 is a well mixed gas. However the equations that relate temperature to absorption cannot be inverted. The measurements are highly accurate but using them to drive models is a problem to which there is no unique answer. Indeed, the same is so for the initial data analysis generally.


There are myriads of data from several widely differing data sources - see https://www.ecmwf.int/en/forecasts/...18041000,0,2018041000&obs=synop-ship&Flag=all. There are in situ data at fixed “synoptic” hours. There are in situ data on a catch as catch can basis. There are satellite images and some, mainly high level wind data, at fixed times from geostationary orbits and there are both images and numerical data from low earth orbiters continuously available varying in space and time. There is no unique way of producing a definitive data analysis to describe the atmosphere. Met Service modellers use whatever combination of data gives the best overall results for their particular purposes.

Further improvements will come as observational resolution and accuracy improve, as analysis techniques evolve to handle the improved data, as models are improved to use the improved analyses. How much improvement will be obvious to the man in the street or the sailor on his/her boat is not at all clear. Given the natural variability that we observe, we may not see much significant improvement. Even now, we see people lauding “derailed” models run with no detailed input. So, why are meteorologists putting so much effort into improving models? Some answers -

* good seasonal, annual, decadal prediction is still perceived as economically desirable,

* improved deterministic forecasts with model ensembles over a period probably around 10 days seems feasible and, again, economically desirable,

* short period (hours) warnings of locally severe weather is still an urgent need.
 

franksingleton

Well-known member
Joined
27 Oct 2002
Messages
3,385
Location
UK when not sailing
weather.mailasail.com
All very interesting but however you look at it in academic terms, the practicality is that weather forecasting is an initial value problem. Leaving aside the questions of real or theoretical predictability, the starting point for any non-trivial forecast has to be observational data.

Paul made a good point when he said that satellites often do not measure what we really want. Another good example is that of measuring the temperature structure of the atmosphere. This is fundamental in models. From a well defined surface pressure/wind field, knowing the vertical and horizontal temperature fields tells you about the flow at all levels. Over the oceans there are few in situ observations from radio-sondes launched from islands and none or virtually none from ships since the regrettable demise of the ocean weather ships. Satellites measure the absorption of infra-red by CO2. The absorption depends on the air temperature structure - CO2 is a well mixed gas. However the equations that relate temperature to absorption cannot be inverted. The measurements are highly accurate but using them to drive models is a problem to which there is no unique answer. Indeed, the same is so for the initial data analysis generally.


There are myriads of data from several widely differing data sources - see https://www.ecmwf.int/en/forecasts/...18041000,0,2018041000&obs=synop-ship&Flag=all. There are in situ data at fixed “synoptic” hours. There are in situ data on a catch as catch can basis. There are satellite images and some, mainly high level wind data, at fixed times from geostationary orbits and there are both images and numerical data from low earth orbiters continuously available varying in space and time. There is no unique way of producing a definitive data analysis to describe the atmosphere. Met Service modellers use whatever combination of data gives the best overall results for their particular purposes.

Further improvements will come as observational resolution and accuracy improve, as analysis techniques evolve to handle the improved data, as models are improved to use the improved analyses. How much improvement will be obvious to the man in the street or the sailor on his/her boat is not at all clear. Given the natural variability that we observe, we may not see much significant improvement. Even now, we see people lauding “derailed” models run with no detailed input. So, why are meteorologists putting so much effort into improving models? Some answers -

1.Good seasonal, annual, decadal prediction is still perceived as economically desirable,

2. Deterministic forecasts with model ensembles over a period probably around 10 days seems feasible and, again, economically desirable,


3. Short period (hours) warnings of locally severe weather is still an urgent need.
 
Last edited:

franksingleton

Well-known member
Joined
27 Oct 2002
Messages
3,385
Location
UK when not sailing
weather.mailasail.com
After that diversion, back to the OP. David, congratulations on a site that is becoming a significant resource. Particularly welcome are the ICON-EU and the KNMI HARMONIE output. Do you have the FMI and KNMI HIRLAM in your sights? These appear on the Meteociel.fr pages but only in a streamline format.

At some stage, I still hope to see the U.K. NWP output. Maybe the availability of the various GRIB products on your site will encourage sailors and others to ask why the U.K. does not make its model data equally and readily available. Maybe just now with all the interest on Syria is not s good time but MPs’ cages could be rattled. We
 

davidgal

New member
Joined
4 Nov 2011
Messages
15
Visit site
Do you have the FMI and KNMI HIRLAM in your sights?

We are looking at the KNMI HIRLAM as it is now available with the EU open data legislation.

FMI is also to be considered after we get our new subset-server operational.

Second your hope to see UK NWP gribs as open data.
 

NOHOH

Member
Joined
27 Aug 2003
Messages
516
Location
uk
Visit site
Any chance that any of you `Sheldons` could just simply provide a web address from where I can download the `best` version that equates to simple old Zygrib....er...please
 

franksingleton

Well-known member
Joined
27 Oct 2002
Messages
3,385
Location
UK when not sailing
weather.mailasail.com
XyGrib now provides forecasts, twice daily, from DWD-ICON and the Meteo France Arpege. I still live in hope that sailors, or those representing our interests (RYA, CA) will pressure the Met Office to make its data available similarly. It is the next best model to ECMWF but run 4 times a day and about 3 hours earlier.
 

franksingleton

Well-known member
Joined
27 Oct 2002
Messages
3,385
Location
UK when not sailing
weather.mailasail.com
I see that there is now DWD COSMO 2 km model output at http://openskiron.org/en/cosmo-gribs. As well as the usual elements and parameters, this includes reflectivity. Essentially, this is what the weather radar will see. High values are where thunder and lightning are possible. An advance on CAPE which shows lightning potential rather than a more specific prediction. Together with HARMOMIE, https://www.euroszeilen.utwente.nl/weer/grib/ giving output from the KNMI 2.5 km model covering the Dover Strait to the IJsselmeer, we now have the most soundly based Weather model data available from the Dover Strait to Bornholm. Together with ICON-EU from Open Skiron at http://openskiron.org/en/icon-gribs, and Ventusky at https://www.ventusky.com/?p=48.9;2.8;4&l=temperature-2m&m=icon_eu, we are seeing a significant increase in availability of weather information.

Met Office, wake up!
 
Last edited:
Top