One of my favourite data discoveries this year: Google's mind-blowing ARCO-ERA5 dataset: hourly data for ~300 climate variables, available globally from 1940! 🤯 Loadable with a single line of Python code from a single cloud-friendly Zarr file! Below: a month of wind waves + swell: 🌊 - ThreadSky

satellitesci.bsky.social • 87 days ago

One of my favourite data discoveries this year: Google's mind-blowing ARCO-ERA5 dataset: hourly data for ~300 climate variables, available globally from 1940! 🤯

Loadable with a single line of Python code from a single cloud-friendly Zarr file! Below: a month of wind waves + swell: 🌊

Comments

raulnanclares.bsky.social•87 days ago

📌

mktippett.bsky.social•87 days ago

is it up to date? the climate data store is not great lately

satellitesci.bsky.social•87 days ago

It's currently updated monthly with a three month delay - I think there's plans to update it daily with a five day delay:

https://github.com/google-research/arco-era5/pull/89

amadestam.bsky.social•87 days ago

📌

brackenlab.bsky.social•87 days ago

📌

stephenbaines.bsky.social•87 days ago

📌

rowanwinsemius.bsky.social•87 days ago

A similar product is the World Ocean Atlas from NOAA - it has info like water temperature at diff depths over time.
It's netcdf and I once wrote a horribly long python script to download and process it 😂
https://www.ncei.noaa.gov/products/world-ocean-atlas

rowanwinsemius.bsky.social•87 days ago

To be fair the actual netcdf part wasn't too bad, it was post-processing we did that was the hard part, particularly with the World Ocean Database, which is a massive point dataset of water data used to generate the Atlas raster.

drkjoyce.bsky.social•87 days ago

Wow, now that’s super cool!!

pinkychow10107.bsky.social•87 days ago

Look amazing! Wondering are there free query limits like big query public datasets or is the loading without constraints? 😊

satellitesci.bsky.social•87 days ago

I've done some pretty giant queries and not had any issues with limits yet. I'm guessing if they are limits they come from Google Cloud itself rather than anything specific to ARCO-ERA5 - perhaps @stephanhoyer.com knows?

stephanhoyer.com•86 days ago

To my knowledge, there are no limits on accessing ARCO-ERA5 or other public datasets stored in Google Cloud Storage. You don't even have to be logged in!

troach.bsky.social•87 days ago

📌

darrenwiens.github.io•87 days ago

📌

nick-golledge.bsky.social•87 days ago

📌

nphuffeldt.bsky.social•87 days ago

📌

miskaknapek.bsky.social•86 days ago

Excellent tip - thanks you very much!

jack-kelly.com•87 days ago

I agree, this ARCO-ERA5 dataset is fantastic!

The Zarr conversion is the awesome work of, amongst others: @alexmerose.com, @wundersooner.bsky.social, and @stephanhoyer.com.

And, of course, it's built on @zarrdev.bsky.social 😀

Also check out https://dynamical.org for NWPs in Zarr format.

obsidianfinger.neocities.org•68 days ago

📌

aridideas.bsky.social•87 days ago

📌

anero.id•86 days ago

Yes, staggering work. My only gripe is that the chunking scheme doesn't allow you to query by pressure level. Makes calculating something like a 30 year normal for 850 hPa temperature very expensive.

satellitesci.bsky.social•86 days ago

Yeah, I think there are a few open Github issues around that, e.g. https://github.com/google-research/arco-era5/issues/69

anero.id•86 days ago

I'm watching that one. I understand though that the folks running the show have limited time, and most interest is probably in single-level parameters.

gpetrick.bsky.social•87 days ago

Now I just need to figure out how to use it as a historian! Code is not my strong strength.

johnpill.bsky.social•87 days ago

That’s quite interesting as my current team are thinking about ways to make things such as this more accessible with low code approaches as the entry bar is high if you don’t code.

gpetrick.bsky.social•86 days ago

Very cool. I have a bunch of ideas for digital history projects, but need help with some of the infrastructure. I can see the outcomes, identified data, but the in-between is the hard part.

jplaka.bsky.social•87 days ago

📌

laugesen.com.au•87 days ago

That is mind blowing

ejolson.bsky.social•87 days ago

📌

t0mmiec00kie.bsky.social•87 days ago

📌

knowyouronion.bsky.social•86 days ago

📌

nick-silkstone.bsky.social•86 days ago

📌

abostriker.bsky.social•86 days ago

📌

acefujiwara.bsky.social•87 days ago

excellent post! single glance at where the massive renewable energy reserves really are! as they say in Wall Street, go where the money REALLY is. #energysky #hydrogen #climate

ericleer.bsky.social•87 days ago

The amount of climate data that is freely available is truly incredible. I think it is really one of the things underpinning the current boom in ai weather models!

fabriceardhuin.bsky.social•87 days ago

Looks cool but do not forget that ERA5 used an old version of IFS that is not so great for waves (low bias ...)
ECMWF actually made a better hindcast, and CMEMS Waverys is pretty good too.

granmatxin.bsky.social•87 days ago

Just curious, is there an easy way to obtain metocean data from ECMWF? Thanks in advance

satellitesci.bsky.social•86 days ago

Thanks @fabriceardhuin.bsky.social, I'd love to know more about this - any paper/resource you could point me to about these limitations of ERA5? (and by low bias, do you mean ERA5 wave heights underestimate reality?)

schijvenaars.name•87 days ago

📌

jay-kay-arm.bsky.social•87 days ago

📌

satellitesci.bsky.social•87 days ago

Github repo: https://github.com/google-research/arco-era5

Python code snippet below:

satellitesci.bsky.social•87 days ago

(The ERA5 data underpinning this dataset is credit to the incredible work of @copernicusecmwf.bsky.social. But Google's Zarr file packaging makes it incredibly easy to access huge amounts of data directly from the cloud, all without ever having to download any files locally!)

lalonsander.bsky.social•86 days ago

This sounds amazing 😍

helenajambor.bsky.social•87 days ago

Good you clarify, the data, for once, is not google... !

barbie4science.bsky.social•87 days ago

Wow! Thank you Robbi.

So much action along the roaring 40s (southern hemisphere)...

pukite.com•75 days ago

Badly distorted 2D representation due to the Mercator projection used

satellitesci.bsky.social•75 days ago

Yep, this was just a simple/easy plot of the data in its original EPSG:4326 projection - reprojecting it would make the workflow a bit more complicated!

ausp.io•87 days ago

So cool!

Comments

Posting Rules

Reply