One of my favorite "go to" patterns in my Elixir apps for read only data is to store it all in DETS (erlang.org/doc/apps/std...), commit it to the repo with Git LFS (git-lfs.com), and then bundle it into the Docker container. Data fetching doesn't much faster than that ⚡ #ElixirLang - ThreadSky

akoutmos.bsky.social • 99 days ago

One of my favorite "go to" patterns in my Elixir apps for read only data is to store it all in DETS (https://erlang.org/doc/apps/stdlib/dets), commit it to the repo with Git LFS (https://git-lfs.com), and then bundle it into the Docker container. Data fetching doesn't much faster than that ⚡ #ElixirLang

Comments

uldiscode.bsky.social•98 days ago

Have you compared ets:file2tab and ets:tab2file with your DETS -> ETS method?

akoutmos.bsky.social•97 days ago

I actually have not run that benchmark to see what's faster. I would suspect that :ets.file2tab is faster....but I have absolutely 0 data to back up that claim haha. Someone should totally put together a benchee comparison of the 2 😄

david.bernheisel.com•99 days ago

I have some exchange rates to load upon boot, so I have a committed CSV that warms the ETS cache and then schedules an API request to freshen them.

Are you saying I could skip the CSV loading if I start the ETS table with a DETS file?

akoutmos.bsky.social•98 days ago

That is exactly what I am saying! I used to do something similar with CSVs....but my CSVs actually got so large that it was stalling application start up by a few seconds. Loading DETS -> ETS was instant!

pragtob.bsky.social•99 days ago

Why not in the config or a module attribute? Do you use the queries in DETS that much? 👀

akoutmos.bsky.social•98 days ago

For https://eaglemms.com I have a whole bunch of read only data like year/make/model data for cars. Some of my DETS files are 100+ MB and I load them from DETS -> ETS on application init. It's data that only needs to be freshened up once a quarter and it is a hot path for everything else in the app.

pragtob.bsky.social•98 days ago

I understand that, I'd still ask why DETS/ETS over compiling them as data in a module or put it into the config.

To be fair I've never done it with data that big and it may have compile time impacts.

But depending on lookup you could use the data in a form you like or also into ETS.

peregrine.bsky.social•98 days ago

Just need to be careful about too large of Dockerfiles, basically adds lag to every deploy and scale. But yeah I have done similar things, could also compile it into a module that doesn’t change often.

peregrine.bsky.social•98 days ago

Could also skip dets entirely and do :erlang.term_to_binary/binary_to_term

Comments

Posting Rules

Reply