An unordered / non-exhaustive list of things that helped scale Bluesky's infra efficiently:
+ Exiting the cloud (colocation)
+ HAProxy w/many Node backends
+ Go w/clever code
+ ScyllaDB
+ SQLite w/per user databases
+ Redis w/many instances
+ AMD servers w/many cores
+ Purchasing bandwidth directly
+ Exiting the cloud (colocation)
+ HAProxy w/many Node backends
+ Go w/clever code
+ ScyllaDB
+ SQLite w/per user databases
+ Redis w/many instances
+ AMD servers w/many cores
+ Purchasing bandwidth directly
Comments
And amazing choices especially Go.
With this setup, when you have just two locations, you can't go over 50% capacity in either, but things get nicer as you add more.
Gonna re-investigate this at work.
One novelish aspect is there’s no routers per se just Linux boxes
How do you deal with multiple instances reading/writing to it? How about replication or backup?
🤔
Reads like ancient Greek for us non-technies
What is some of the cleverness in the Go code?
Among many other things like Bloom filters, restructuring workloads to favor very high concurrency, etc.
What are you using to orchestrate server workloads behind HAproxy? (Eg, containers, k8s or just plain VMs/metal?)
Curious, How do you do SQLite persitence?
https://open.substack.com/pub/pragmaticengineer/p/bluesky?utm_source=share&utm_medium=android&r=ch6l7
That's the main limitation with Redis on a machine with many cores, you can't really leverage them!
So Bluesky uses sharding and large number of Redis instances on a single physical host, to utilize the cores.
https://valkey.io/blog/valkey-8-0-0-rc1/
Easily one of the best storage engines out there.
Did you know it even supports window functions and CTEs?
Bonus points for issues being contained to that user.
Most of my experience is based on wazuh's design, it now uses per-agent sqlite, and frequently runs into problems. But it's very write intensive.
I've written my own ledgers based on ext4 directly, when mariadb is overkill, but time to give sqlite a go
More DBs is always better if you don't _need_ to have all your data in one sqlite database file. If you have multiple tables and don't have to run transactions modifying both of them, use more than one sqlite file each with their own writer!
https://atproto.com/articles/atproto-for-distsys-engineers
Makes sense for this use case in large part because each users atproto repository is self contained, with links to other repos, like a website.
On that service it’s stored ScyllaDB, which is on the other extreme of database systems 😏
2. How are you storing the Sqlite files? On Sycalla? Or NFS? Hosted on top of what file storage solution?
Spoiler: the faster and closer to the PCIe bus, the better.
SycallaDB can cluster at the application level, so local storage all the way
2. SQLite is used on the PDS, stored on the local (NVMe SSD). Each (of us) users has our own SQLite database for our atproto repository data.
Here's the original PR where @dholms.xyz and @divy.zone did this work: https://github.com/bluesky-social/atproto/pull/1705
(I’m not on the team as well)
The SQLites are just stored on disk for the PDSs (not in the same place as where Scylla runs) and have backups plus litestream on the PDS-wide DBs which allow us to recover from outages by playing back the event stream for the PDS.
that
true?
Write the same reply but try to copy the whole text before you post it.
If you paste it into another new post, the blank lines disappear.
I like to copy a reply and repost it in other similar threads.
It's no big deal. I can reinsert the blank line. It's just annoying.
You may be right about the browser being the culprit. I use firefox.
Probably will get fixed one day!
https://atproto.com/articles/atproto-for-distsys-engineers