ECMWF EPS Mean/std Archive

[devsjc]

As part of the GDM work, we require an archive of ECMWF Ensemble data - not full members, but rather mean and standard deviation values. The Operational Archive contains within it a data stream specific to EPS (Ensemble Prediction System). Here follows a summary of the data quality we can expect from the different products available on MARS for EPS:

Derived Probability Products: Ensemble Mean/Std

• 🔴 3 hourly steps
• 🔴 few parameters: 2 metre temperature, 10 metre wind speed, 100 metre wind speed, mean sea level pressure
• 🟢 usual run times (00:00, 12:00)
• 🟢 comprehensive archive (1994-present)
• 🔴 no radiation parameters

Derived Probability Products: Time-averaged Ensemble Mean/Std

• 🟢 hourly steps
• 🟢 many parameters
• 🔴 only one init time: 00:00
• 🔴 limited archive (stops at June 2023, only every 3rd day of the month available)
• 🔴 no radiation parameters

Neither of these is completely ideal! There is also an ENSEMBLES product available but none of the data on there is near the 2020s. @Sukh-P, and @peterdudfield, which type do you think suits us better? And @Sukh-P, what would you want ideally pulling in terms of region, steps, parameters, years? Or, do you know of any other MARS product that you had in mind with the ensemble data work?

[Sukh-P]

Thanks for this @devsjc! Is the ENSEMBLES product you mentioned without archive data from 2020 the same as the ENS one https://confluence.ecmwf.int/display/FUG/Section+2.1.2.1+ENS+-+Ensemble+Forecasts?

[devsjc]

I think the ENSEMBLES project is something else entirely and not particularly useful to us. However I am a bit confused now: the links on that page take you to a real-time order page where set iii: ENS is clearly defined as the 15- day ensemble model; I've been looking at the archive datasets on MARS, none of which seem to reference anything called ENS, only EPS...

[Sukh-P]

Yeah I also had a look and I am quite confused, it seems like the ENS doesn't have an archive available but maybe worth contacting ECMWF about this to make sure?

[devsjc]

Yes, I don't think they do.

[devsjc]

In discussion with Sukh:

• Desirable to have at least 64 steps, 70-84 step
• Region that covers UK and India if possible
• Most important variables were temperature, mean sea pressure, wind speed componenets

It would also be beneficial to have probability boundaries if possible (P10, P25, P50(median), P75. P90) in an ens_stat dimension.

[devsjc]

I believe GDM are referring to use of the top option specified in the first comment, so that's what I'll pull. It does not, unfortunately, seem to have probability boundaries - that is in a seperate archive with much lower temporal resolution. I'll get what I can for now.

[devsjc]

Pulling the following from ECMWF:

Derived Probability Products: Ensemble Mean/Std

🔴 3 hourly steps
🔴 few parameters: 2 metre temperature, 10 metre wind speed, 100 metre wind speed, mean sea level pressure
🟢 usual run times (00:00, 12:00)
🟢 comprehensive archive (1994-present)
🔴 no radiation parameters

Added into Consumer with #203

[devsjc]

Coding complete, so closing this. Tracking download at openclimatefix/dagster-dags#141