Problems downloading ECMWF forecast

By way of introduction, Piddy is my Dad (Andy) and I work at ECMWF, so if you have questions, I'm happy to answer them (wherever I can!) :)
Ah! Thank you for that, I had detected some real knowledge. Like the curate’s egg, I was good in parts regarding the 06/18 runs. It has always seemed to me that using ECMWF for LAM boundary conditions is rather late in the day. As far as I know, the U.K., DWD and Meteo-France all run their LAMs with much more recent data. If you do not mind, I might come back to you via PM at some stage.

If it is not a commercial-in-confidence mater, can you confirm my guess that Windy.com pays for the ECMWF data on their site?
 
Question for Andy’s son. Predictwind provide ECMWF data out to 10 days to their paying customers on, I am told, the model grid (or resolution if not the precise grid.). As a non-subscriber, I can get ECMWF data from PW on a 50 km grid out to 5 days. I understand that is to be extended to 10 days. Is all that in accordance with the ECMWF policy?
 
Question for Andy’s son. Predictwind provide ECMWF data out to 10 days to their paying customers on, I am told, the model grid (or resolution if not the precise grid.). As a non-subscriber, I can get ECMWF data from PW on a 50 km grid out to 5 days. I understand that is to be extended to 10 days. Is all that in accordance with the ECMWF policy?

They are not providing anything that windy isn't providing...

okay - you don't need to use a web interface.

it's not really of much use if you can't import it into a good nav program.

what i don't understand is why does EC care what program I view it in?

So - I download an ECMWF file in the PW app.., and I can view it in the (useless) PW app.., but it's encrypted so i can't view it in any other program.

Why should EC care whether I import the file to another program?

it just seems needlessly restrictive
 
You're welcome to PM, of course.
I can confirm that Windy pays for a licence, yes.

Question for Andy’s son. Predictwind provide ECMWF data out to 10 days to their paying customers on, I am told, the model grid (or resolution if not the precise grid.). As a non-subscriber, I can get ECMWF data from PW on a 50 km grid out to 5 days. I understand that is to be extended to 10 days. Is all that in accordance with the ECMWF policy?

Andy's daughter :-)

From what I know of PredictWind, the ECMWF 9km data is encrypted to their apps and extends to the full 10 days. The 50KM resolution should only be the WMO essential data as this is free of charge from ECMWF anyway.
 
From what I know of PredictWind, the ECMWF 9km data is encrypted to their apps and extends to the full 10 days. The 50KM resolution should only be the WMO essential data as this is free of charge from ECMWF anyway.

The free 50km includes 10m winds

but - as i mentioned - you can only view it in the PW app
 
If it's only visible in the application, then it is a 'non-retrievable value added service' and is compliant.

i guess what i am trying to find out.., is why does the EC care what program I view the file in?

so.., previously.., i was subscribing (monthly fee) to Squid, and PW.., and downloading the ECMWF from them in GRIB format.., and viewing the files in a full-featured navigation program.

I happen to use Expedition.., but there are others - Adrena is probably the main competitor.

Now, I can no longer do that.

Squid has stopped providing the ECMWF for download in any form (Actually, I haven't checked recently - they may be providing the WMO essential ).., and PW is now providing only encrypted file downloads that can not be viewed in a nav program.

What interest doe the EC have in whether I view the downloaded file in PW or Expedition?
 
I am still perplexed as to why there is so much desire to be able to get ECMWF GRIB data. For most of the time, using other sources, you can download useful GRIB data based on more recent data than the latest available ECMWF fotecast.

I write as someone who has been ""using GRIBs ever since Saildocs GFS was the sole source. In the recent past I have been looking at windy.com as well as GRIBs on viewers such as XyGrib and PocketGrib. I do not use them with navigation software. I have not done a thorough statistical study but it is my impression that, for the first few (4 or 5) days, there is little to choose between models. Beyond that ECMWF leads the rest - as would be expected as their remit is, at least in part, to study modelling with particular stress on the medium range ie up to 10 days in their original raison d’etre.

Of more interest/concern to me is why we cannot get, easily, global and LAM data from the UK Met Office. Their global model grid length, 10 km (0.1 degree) is the same as ECMWF and their 4 km model is, I believe, run 3-hourly. It extends further west than ICON-EU and, therefore, should be more reliable than ICON-EU for UK waters.

I think that I know the reasons why UK GRIBs are not freely/easily available. That dates back to the Thatcher era and the Rayner review of the Met Office. The philosophy then was that “Information of general public interest should be free at the point of delivery but that sectional interests should not benefit at the taxpayer’s expense.”

Mariners, collectively, are regarded as a sectional interest. I stand to be corrected but I think that was behind the original reasoning for charging for ECMWF data as agreed by member nations. Most countries have moved away from that philosophy. The UK has not until recently when it has begun to make data free of charge with some restrictions and a user interface as difficult to navigate as the labyrinth of Greek mythology. Ariadne, where is that ball of thread? I have a Minotaur to kill.
 
Last edited:
I am still perplexed as to why there is so much desire to be able to get ECMWF GRIB data. For most of the time, using other sources, you can download useful GRIB data based on more recent data than the latest available ECMWF fotecast.

I basically agree - and i have been using GFS as long as you!

Here's how I see GFS vs Euro...

Both models are excellent. The recent GFS upgrade has (mostly) been a substantial improvement. The forecasting skill of these models, compared to say 10 years ago is amazing.

It is true that on some measures of forecasting skill the Euro performs marginally better. Users should be aware that the measurement of forecasting skill is itself subject to considerable uncertainty and to some extent reflects the same biases as the models themselves.

The differences in skill are small.., and, critically, are only apparent when looking over many forecasts

It is also true that the models tend to converge for short term forecasts - say 72hrs - usually there is little difference at 48hrs.

When looking at longer time periods - say 7-10 days - there can be considerable difference between the models.

The important thing is that because the difference in forecasting skill between the models is so small.., that difference is not a reason for the user to prefer one long term forecast over another.

typically - when we see a big difference in the long term forecast for a particular weather system between the two models.., one is closer to being right than the other.., and generally, for our purposes, it's as likely to be the GFS as the Euro.

Maybe if we looked at a 100 weather systems, the euro would have a slight advantage - but that really doesn't help us choose one or the other for a particular system.

If you read the NOAA meteorologists "discussions" - issued multiple times per day.., they will often tell you which model they prefer for a given system, and why. Usually they go with a blend of the models.., UK MET being another model they often like. I am unaware of any similar discussion products available from other weather services.

So - back to the original question: why do i want the euro? I am navigating races that can be several days to a week.., and i want to start preparing well ahead of time. If the long term forecasts are different - and they usually are - I want to understand the full range of possible conditions, and have a plan.
 
What interest doe the EC have in whether I view the downloaded file in PW or Expedition?

The data policy of no redistribution needs to be considered separately from the industry itself. ECMWF is not specifically seeking to prevent sailors from using a particular software package (or any user type), nor do we focus on preventing sailors from using the data. There is nothing to prevent any service provider from any industry (or indeed a sailor) from requesting a licence via ECMWF or one of its Member and Co-operating States. With no redistribution, our licence is designed to encourage the creation and provision of value added services, where data format and delivery are not always/usually considered to be a value added service.

The data policy (which is decided and agreed by the 34 Member States and Co-operating States), has been written to prohibit redistribution for all industries and users of ECMWF real-time data. This policy affects all industries and is not targeted based on a ‘care’ or ‘interest’ about how the users want to use the data. At this point, to permit certain companies to redistribute data and not others would be anti-competitive.

Instead, as I mentioned in an earlier post, ECMWF is actively working towards an open data policy which will eventually permit redistribution. A great deal of work is being done towards this, as we acknowledge that user requirements, industries and service providers change over time. We are considering new approaches over the coming years, but it is not an overnight solution or switch.

It is worth highlighting, however, that even when data is ‘open’, this does not mean all data will be ‘free’. In the same way that users may have paid for a subscription to access ECMWF data via other platforms, there may be a subscription to access ECMWF data via our own servers (as there currently is for the archive data, for example). Similarly, when data is open, service providers may still choose to implement their own business model and charge for access.
 
If the long term forecasts are different - and they usually are - I want to understand the full range of possible conditions, and have a plan.

Thanks for the explanation. It's useful to understand different viewpoints and requirements. As a coastal sailor (who previously raced in the Solent/coastal waters), I don't have much use for GRIB files in any software and prefer to use visualisations and probabilistic Meteograms. I don't have much cause for routing, either. In short races, there's little gain when the time steps are usually broader than the actual race itself! Modern now-casting solutions might be more effective in my case, but less so for ocean racing or longer-term passages. Similarly, Frank is right that ICON and higher resolution models might be more useful, but even so the GRIB would be more time consuming for my variety of sailing.

We find that a lot of sailors have requested the short- and medium-range GRIB files where the forecast skill is notably more accurate than a long-range forecast. We conducted a survey and many requested the first 6-days only.
Many of the current service providers (like Windy) don't provide much in the way of long-range forecasts. I'm curious though: if they did, would this be useful to you for planning ahead? Or does this specifically pair with a service in a software that you use, and hence the need for the GRIBs?
 
I basically agree - and i have been using GFS as long as you!

Here's how I see GFS vs Euro...

Both models are excellent. The recent GFS upgrade has (mostly) been a substantial improvement. The forecasting skill of these models, compared to say 10 years ago is amazing.

It is true that on some measures of forecasting skill the Euro performs marginally better. Users should be aware that the measurement of forecasting skill is itself subject to considerable uncertainty and to some extent reflects the same biases as the models themselves.

The differences in skill are small.., and, critically, are only apparent when looking over many forecasts

It is also true that the models tend to converge for short term forecasts - say 72hrs - usually there is little difference at 48hrs.

When looking at longer time periods - say 7-10 days - there can be considerable difference between the models.

The important thing is that because the difference in forecasting skill between the models is so small.., that difference is not a reason for the user to prefer one long term forecast over another.

typically - when we see a big difference in the long term forecast for a particular weather system between the two models.., one is closer to being right than the other.., and generally, for our purposes, it's as likely to be the GFS as the Euro.

Maybe if we looked at a 100 weather systems, the euro would have a slight advantage - but that really doesn't help us choose one or the other for a particular system.

If you read the NOAA meteorologists "discussions" - issued multiple times per day.., they will often tell you which model they prefer for a given system, and why. Usually they go with a blend of the models.., UK MET being another model they often like. I am unaware of any similar discussion products available from other weather services.

So - back to the original question: why do i want the euro? I am navigating races that can be several days to a week.., and i want to start preparing well ahead of time. If the long term forecasts are different - and they usually are - I want to understand the full range of possible conditions, and have a plan.

Thank you for that. I hope that you do not think me to be condescending but your post shows far more understanding of the realities of weather prediction than I have seen from the majority on these forums. I have a few rather pithy sayings that I trot out from time to time.

Weather does not know itself to within at least one Beaufort Force.
Weather is not precise so neither can forecasts be precise.
The only certainty in any forecast is the date - as long as you leave the theologians out of the argument.
Better to bein port wishing you were at sea than at sea wishing you were in port.

But, broadly, I agree with much of what you say. As a cruising sailor, mainly coastal, I am always looking ahead for the next week. Is there a weather window coming? When and for how long? Can I or should I move on? What are the consequences of staying put? To those ends, I monitor forecasts at least daily for the next 10 days. The GFS can give me 15 days but that is too far ahead to be sensibly useful. I look for consistency in forecasts. When I see consistency, I plan ahead for up to 7 days. Sometimes, lack of consistency means that we only think a day or so ahead.

This is an example a couple of years ago showing parts of (then) zyGrib meteograms.
ninedaystppdart.png

these are forecasts for a point near Channel LV for 7 September. By the 1st we were fairly sure that the crossing from St Peter Port to Dartmouth would be OK with winds in a sailable direction, not too strong but enough to sail. Obviously, we watched the forecast each day but planned on that basis - eg warning our marina to keep our berth available, provisioning etc.
 
The data policy of no redistribution needs to be considered separately from the industry itself. ECMWF is not specifically seeking to prevent sailors from using a particular software package (or any user type), nor do we focus on preventing sailors from using the data. There is nothing to prevent any service provider from any industry (or indeed a sailor) from requesting a licence via ECMWF or one of its Member and Co-operating States. With no redistribution, our licence is designed to encourage the creation and provision of value added services, where data format and delivery are not always/usually considered to be a value added service.

The data policy (which is decided and agreed by the 34 Member States and Co-operating States), has been written to prohibit redistribution for all industries and users of ECMWF real-time data. This policy affects all industries and is not targeted based on a ‘care’ or ‘interest’ about how the users want to use the data. At this point, to permit certain companies to redistribute data and not others would be anti-competitive.

Instead, as I mentioned in an earlier post, ECMWF is actively working towards an open data policy which will eventually permit redistribution. A great deal of work is being done towards this, as we acknowledge that user requirements, industries and service providers change over time. We are considering new approaches over the coming years, but it is not an overnight solution or switch.

It is worth highlighting, however, that even when data is ‘open’, this does not mean all data will be ‘free’. In the same way that users may have paid for a subscription to access ECMWF data via other platforms, there may be a subscription to access ECMWF data via our own servers (as there currently is for the archive data, for example). Similarly, when data is open, service providers may still choose to implement their own business model and charge for access.
Thank you for that input. This is becoming a useful discussion. One point about using forecasts that you do not mention is “effective resolution.” Models have to have some smoothing to avoid going unstable as used to happen in the early days of NWP. A model can only represent weather features on a scale of about 5 grid lengths. So, global models really only define weather on a scale of about 50 to 60 km. Effectively, wavelengths smaller than these values are filtered out.
 
Effectively, wavelengths smaller than these values are filtered out.

This is correct and also one of the reasons that, for me, ECMWF data is only useful for evaluating larger-scale weather formations which might have a localised impact on me in the days to come! Many users of ECMWF data take ECMWF as input for another model, such as WRF. One of the things I love most is seeing the effect of downscaling (using models like WRF) from ECMWF data to a finer resolution where more detailed topography is used. I've seen examples of 500m grids for localised forecasts with incredible features.

P.S. - you might like this article: Impact of orographic drag on forecast skill
 
This is correct and also one of the reasons that, for me, ECMWF data is only useful for evaluating larger-scale weather formations which might have a localised impact on me in the days to come! Many users of ECMWF data take ECMWF as input for another model, such as WRF. One of the things I love most is seeing the effect of downscaling (using models like WRF) from ECMWF data to a finer resolution where more detailed topography is used. I've seen examples of 500m grids for localised forecasts with incredible features.





P.S. - you might like this article: Impact of orographic drag on forecast skill


Thank you. That illustrates (one of) the problems with NWP, energy diffusion. “Big whorls have little whorls .......”



I am always a little sceptical about LAMs that are available. People such as PW, OpenWTF, MeteoConsult, Poseidon, WindGuru etc can only initialise by Ii interpolation from global model grid points, usually the GFS. Granted, they do use detailed topography but, surely, their results are compromised by lack of detailed meteorological data. On the other hand, although the U.K., DWD, AEMet etc do use highly detailed analyses based on satellite and radar, small weather features have short lifetimes. By the time forecasts are available, several hours after the base time, detail less than about 30 km will have come and gone.



Looking at LAM output, I never see the fine detail that I know exists. These are two examples of LAMS.


LAMAEmet.png



From AEMet using a model with grid around 3 or 4 km but displayed at about the effective resolution.


LAMWRF.png

This an OpenWRF Forecast ar the same time with data on a 4 km grid.


Like you, I use NWP output to give the best answer possible while recognising that noise in the system is, effectively, unpredictable. On a Channel crossing, if models say F4-5, I assume that I will see some F3 and some F6. During the September Medicane, the global models and ICON-EU gave at least 60 hours warning. Poseidon was at least a day behind.


I would appreciate your comments. Am I missing something?
 
This is correct and also one of the reasons that, for me, ECMWF data is only useful for evaluating larger-scale weather formations which might have a localised impact on me in the days to come! Many users of ECMWF data take ECMWF as input for another model, such as WRF. One of the things I love most is seeing the effect of downscaling (using models like WRF) from ECMWF data to a finer resolution where more detailed topography is used. I've seen examples of 500m grids for localised forecasts with incredible features.

P.S. - you might like this article: Impact of orographic drag on forecast skill

everyone and their grandmother is running a WRF model these days....

not all implementations of WRF are the same.., and not everyone is doing an equally good job of it

simple WRF models can be run on a laptop!

However, I make frequent use of hi-res models when racing

Two that I have had success with are the HRRR from NOAA and AROME from meteo france - I think they are both based on WRF.

I use AROME in the med and french caribbean - i think it's worth looking at for the Caribbean 600

HRRR obviously in the USA - I have seen it be spectacularly good.., but it's inconsistent.

HRRR is really amazing considering the enormous domain - it's 3km over the whole USA.., and is available with 15min time steps.

But the thing that really sets HRRR apart is that there is a new model run every hour - 24 times a day - with new assimilation/initialization.

It is the only model i know of that uses weather radar data for the initialization - their ultimate goal is to forecast individual convective cells.

what i find is that sometimes when i download HRRR.., the first time step matches what i see on the water (wind speed and direction) , and sometimes it doesn't. If it matches pretty well, I use it pretty much exclusively for as long as that's the case.
 
HITLAM, I believe, has been using radar imagery for some while now, as has the U.K. and, no doubt most National weather services around Europe. The U.K. runs its UK model hourly. Just a great pity that they do not issue their detailed model in the same way that some others do.
As a sceptic, I have to query accuracy claims in any forecast of wind. First, as I have said several, even many, times, models have a built in smoothing. Structures less than about 5 grid lengths cannot be predicted. So on the most detailed models features less than between 5 and 20 km will not be predicted.
A typical thunderstorm has a lifetime of a few hours. If you watch the U.K. weather app, it is not very good at predicting the development of existing thunder clouds. Therefore, it cannot be “accurate” for wind prediction. Back to my over-used, “Weather does not know itself precisely.”

PS. I should have checked. HRRR uses a 3 km grid. It’s effective resolution is about 15 km.
 
  • Like
Reactions: GHA
If you read the NOAA meteorologists "discussions" - issued multiple times per day.., they will often tell you which model they prefer for a given system, and why. Usually they go with a blend of the models..,

that was one of the products I queried through email when things looked odd (in west east transats), unfortunately in the recent years they have begun concentrating only in US offshore waters a few hundred miles from the US coastline, the NT1 and NT2 as they call them, barely mentioning the metarea IV "high seas", say bermuda to (almost) Azores.
The Metarea 4 high seas text bulletin has also become essentially gmdss/safety, indication of significant weather but often ending "elsewhere winds below 20 kt".
 
that was one of the products I queried through email when things looked odd (in west east transats), unfortunately in the recent years they have begun concentrating only in US offshore waters a few hundred miles from the US coastline, the NT1 and NT2 as they call them, barely mentioning the metarea IV "high seas", say bermuda to (almost) Azores.
The Metarea 4 high seas text bulletin has also become essentially gmdss/safety, indication of significant weather but often ending "elsewhere winds below 20 kt".


NOAA has many discussion products.

note that discussions are not forecasts - the forecasts are separate products - the discussions are meant to give an insight into how the forecast was arrived at.

the one most applicable to the trade wind crossing is the Tropical Weather discussion - It covers as far east as West Africa - it is not always particularly useful

Atlantic Tropical Weather Discussion

Anyone wanting an idea of what a more in-depth discussion looks like should try the NOAA ocean Prediction Center's Marine Weather Discussion - but as Roberto notes, it only covers as far east as 50W - This is updated several times per day, and sometime provides great insights into how forecasters use weather models.

North Atlantic Marine Weather Discussion

i am copying it here, because that one will disappear - sorry for the length, but i think few people appreciate that such an in-depth discussion is readily available from the meteorologists who actually are doing the forecasting.

note that when he talks about "populating the grids", he is talking about the NOAA NDFD grib.., which can be thought of as a human-made blend of many models - including ECMWF. It basically uses the forecasters experience to choose which model results to use or blend for a given weather feature. local NOAA offices all over the country are each responsible for their own area, but it is a seamless product. The north atlantic is done by the Ocean Prediction Center.

Every local NOAA office also produces a discussion for its region

i read the discussions regularly.

AGNT40 KWNM 161451
MIMATN

Marine Weather Discussion for N Atlantic Ocean
NWS Ocean Prediction Center Washington DC
1051 AM EDT Fri Oct 16 2020

.FORECAST DISCUSSION...Major features/winds/seas/significant
.weather for the North Atlantic Ocean W of 50W from 30N to 50N.

Other than populating the grids with the hires 12z Warw through
00z tonight, I will make no other changes to the ongoing grids in
place. The current warning headlines will remain as is in this
intermediate morning update.

Please refer to the previous discussion below for additional
information.

-----------------------------------------------------------------
PREVIOUS DISCUSSION...

The latest NCEP surface analysis shows a high pressure ridge
axis across the offshore waters this morning, and a cold front
just W of the area approaching the coast. Current surface
observations indicate winds have increased in the S to SW flow
ahead of the front along the New England coast, with highest
conditions in the Gulf of Maine where a few reports show up to 25
kt. This is slightly higher from last night's 02Z Ascat wind
retrievals which indicated up to 20 kt in the Gulf of Maine at
that time. The 00Z GFS winds are initialized well when compared
with the data, while the 00Z ECMWF is about 5 kt low along the
New England coast. The 00Z models agree well, however, that the
front will move offshore early today as low pressure develops
along the front while moving off the Mid Atlc coast. The guidance
then indicates that the low will move across nrn NT2 and NT1
later today into Sat while pulling the front across the offshore
waters. The models had been trending slightly faster over the
past day or so with the progression of the low, though the 00Z
model cycle has remained consistent from the previous cycle. The
00Z GFS and ECMWF are in good agreement on the overall sequence
of events over the next 48 hours, and indicate that the winds
will increase in the cold advection behind the front. There has
been an increasingly stronger signal in the guidance over the
past few days for the development of gales in this region behind
the front, and this makes sense as the GFS indicates unstable
conditions behind the boundary which should allow stronger winds
aloft to mix down to the surface. The previous forecast had gales
up consistent with the guidance, and confidence remains just
above average as a result of the good agreement between the GFS
and ECMWF. In addition, the UKMET and GEM show 30 kt in the cold
advection, which supports the gales when their underperformance
in cold advection is taken into consideration. As such, am
planning on continuing the headlines in next forecast package
with only minor changes. Am also planning on blending in the 00Z
GFS with the previous grids over the short term to incorporate
the minor changes between the 00Z models and the previous run.

The 00Z GFS then indicates that the front will pass E of the
offshore waters late Sat, and then stall and weaken E of the
area on Sun while high pressure builds across the W Atlc. The
models then show the weakened front or low pressure trough
drifting back W towards the area Sun into Mon as an area of low
pressure moves slowly E towards the area into Tue. However the
GFS/ECMWF indicate winds will remain fairly light as the high
maintains a ridge axis over the offshore waters through Tue.
There are then some differences between the models on another
low approaching the area from the W late in the forecast period.
The 00Z GFS/GEM have been consistently faster than the
UKMET/ECMWF solutions, so there is a fair amount of spread and
uncertainty with the guidance. As such, am planning on using a
compromise solution late in the forecast period to account for
the differences and model spread. As such, will continue with the
blend of the 00Z GFS and previous grids into late Mon night, then
add the 00Z ECMWF into the blend early Tue through the rest of
the forecast period. As was hinted at above, winds are expected
to remain below warning criteria after Sat with above average
confidence since there is nothing to support anything above 25
kt, so not planning on having any hazard headlines in the next
forecast beyond Sat.

Seas...Since am planning on using a blend of the 00Z GFS with the
previous grids for the winds, will stay consistent and use a
blend of the 00Z Wavewatch with the previous wave height grids
into late Mon night. Will then add in the 00Z ECMWF WAM into the
blend from early Tue through the rest of the forecast period.

Extratropical Storm Surge Guidance: Both the 00Z ESTOFS and ETSS
show positive surge values of over .5 along the SE coast Sat
afternoon into Sun in the strong NE flow behind the cold front.
With most guidance showing similar wind values after fropa, these
values appear reasonable. Please monitor products from coastal
National Weather Service offices for detailed water level
information.
 
Top