Published a new blog post: Building a Distributed Log Using S3 (Under 150 Lines of Go) The log is a powerful storage abstraction. Using a log, you can build a database, message queue, or event streaming system. S3 makes it all better and easier! avi.im/blag/2024/s3... - ThreadSky

About ThreadSky

avi.im • 85 days ago

Published a new blog post: Building a Distributed Log Using S3 (Under 150 Lines of Go)

The log is a powerful storage abstraction. Using a log, you can build a database, message queue, or event streaming system. S3 makes it all better and easier!

https://avi.im/blag/2024/s3-log

Comments

growth.wtf•85 days ago

Very cool. I'd be interested in reading a step further -- what's stopping this from being appropriate in production? Where are the gaps and how would you fix them? What system might you be trying to replace specifically? It's a great concept and would like to learn more where it fits in.

avi.im•85 days ago

It’s not a replacement for existing system. s3-log on its own isn’t useful. It’s a library and is a crucial building block for data systems.

Using this, building Kafka would be very simple as durability and transactional properties are already handled

avi.im•85 days ago

Tests! I need a lot of them. I need to add tests with multiple writers, multiple nodes etc. Tests which check for data integrity

I need to add some way of doing compaction

Lastly a way to buffer writes so that we can increase throughput

avi.im•85 days ago

The code is open source, comes with tests and open issues to contribute: s3-log

https://github.com/avinassh/s3-log

dharmjit.bsky.social•84 days ago

Awesome and thanks for sharing it. I will try to keep an eye out for learning purposes and also any opportunities to contribute back :)

avi.im•84 days ago

thank you!

there are some open issues already!

aparker.io•85 days ago

would love to see an exploration on the query side of this; in practice it’s where I see most object storage log systems fall down

ahmet.dev•85 days ago

You wouldn’t query the log, right? You’d use the log to build a data structure and query that instead. e.g. if you build LSMs, it can be quite object-store friendly in the query path?

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply