SQLite is really opening up doors for distributed database architectures due to it being a single file. Single files are easy to manage, backup, move around, etc. You also have the flexibility of 1 database per customer.
The object store trend seems very strong. I don’t think there is anything as obvious when it comes to programming models? There’s a lot of experimentation with durable execution models at the moment but it’s unclear where or if they will land.
For my applications, I rely on object storage as the single source of truth. Each user has their own S3 prefix and SQLite database, with downloadable WebAssembly (WASM) business logic.
"A Note on Distributed Computing" by Waldo, et. al. https://scholar.harvard.edu/files/waldo/files/waldo-94.pdf is the foundational paper for this space. In the early 2000s Jini was created to address all these challenges. It had object databases as backing stores, transaction managers, leases, and more. Built on Java.
I guess Jini failed to revolutionise distributed computing — why? Also, the object stores of today are very different from the object databases of the 2000s
At the time, there were some very impressive systems deployed using Jini technology. But things move fast in our industry. Even at the time there were a lot of competing ideas. Plus, Jini was a Java implementation and that was unpalatable for many. Jini lives on as Apache River. People still use it.
The underlying storage is definitely very different and much, much better. If anything, it seems like the implementations have finally caught up to allow the theory—i.e., distributed computation built on an "object space"—to be used effectively.
As distributed consistency decentralised, we'll see object storage increasingly as a compatibility layer on top of commoditised hyperscaling. I've seen this happen at enterprises even without edge to centre consistency, but this will speed this process up
Nice article by Colin. I’m of course biased but can’t avoid to think that we are addressing a lot of the issues described with Akka 3. Separated business logic, built in multi cloud support and high-level abstractions that hide and solve hard issues with DistSys. And it’s still fun to build stuff :D
I agree, a migration path is hugely important. Large corporations will have legacy applications and few opportunities for greenfield projects. Startups are always encouraged to ship a minimum viable product and avoid premature optimization. Both cases work against consideration for a paradigm shift.
Comments
Cost is also very low.
Spot on insights about changing the model/primitives/contract we use to be less screwed when things change.
(That was also one of my favourite parts of Designing Data-Intensive Applications too.)
You’re very good at explaining the present landscape, how we got here, and where we might be headed.
You’re a bit like the Yuval Noah Harari of tech.
https://www.youtube.com/watch?v=ei0wwTy6_G4
For my applications, I rely on object storage as the single source of truth. Each user has their own S3 prefix and SQLite database, with downloadable WebAssembly (WASM) business logic.
Works like a charm!
Every era has its own needs and its own solutions.
No standard is ever going to win, and when your customer has a 30 year old database your fancy APIs won’t help
In healthcare there’s a ton of data duplication, and a push for better standards that other verticals can use - is still a clusterfuck but 🐯