1/11. ParadeDB is now integrated with Postgres block storage. As far as we know, no one has integrated a search and analytics engine with Postgres storage before. This is a big deal. Here's why we did it, how we did it, and why you should care. 🧵 - ThreadSky

philippemnoel.bsky.social • 39 days ago

1/11. ParadeDB is now integrated with Postgres block storage. As far as we know, no one has integrated a search and analytics engine with Postgres storage before. This is a big deal.

Here's why we did it, how we did it, and why you should care. 🧵

Reposted from ParadeDB

ParadeDB is now integrated with Postgres block storage.

Today, we are releasing Part 1 of our 3-part blog series on how we designed a new storage mechanism for search and analytics in Postgres.

www.paradedb.com/blog/block_s...

Comments

philippemnoel.bsky.social•39 days ago

2/11. For the newcomers, ParadeDB is building a modern alternative to Elasticsearch on Postgres.

We're building the features of Elasticsearch (the database), e.g. text search and columnar analytics, inside Postgres. This frees enterprises from complex ETL and denormalized, unreliable ES nodes.

philippemnoel.bsky.social•39 days ago

3/11. Until now, ParadeDB worked by integrating with the Postgres query planner, but we stored data files on disk. Our underlying search library, Tantivy, is based on immutable segments which are hard to reconcile with the block-based Postgres storage system.

This worked well, until it didn't.

philippemnoel.bsky.social•39 days ago

4/11. As we started working with ever larger enterprises, the need for high availability in ParadeDB became overwhelming. In Postgres, this meant writing data operations to write-ahead logs so they can replicated to standby ParadeDB instances.

This process is called physical replication.

philippemnoel.bsky.social•39 days ago

5/11. Postgres is amazing. When you integrate with its storage system, you get write-ahead logs integration, buffer caching and other critical features pretty much for free.

Our enterprise customers needed these features to go live, and we knew block storage was the way to go. But could it be done?

philippemnoel.bsky.social•39 days ago

6/11. The answer is yes! But we had our doubts. In fact, it wasn't until a conversation with @fulmicoton.bsky.social, the main developer of Tantivy, that we started to have some hope.

philippemnoel.bsky.social•39 days ago

7/11. Three months, a new Postgres block storage layout and much time spent reconciling immutable segments with Postgres' MVCC later, we've done it.

And by we, I mean Ming, our CTO, and Eric, our resident Postgres expert. We'll be releasing a 3-part blog series to document it. Part 1 is out today.

calistrum.bsky.social•30 days ago

A fusion between a DB and a search engine is a cool idea. I think crateDB does it with Lucene. With native compilation (GraalVM) performance may be on par with Tantivy, but with Lucene you also get vector search.

philippemnoel.bsky.social•29 days ago

We get our vector search from pgvector in Postgres :)

calistrum.bsky.social•29 days ago

And how do you aggregate if one wants to combine a lexical (Tantivy) with a vector clause?

philippemnoel.bsky.social•29 days ago

Standard SQL clauses. ParadeDB has deeply integrated Tantivy into Postgres. So everything is just a SQL statement. You can combine it with any aggregate or statements from other extensions

damien-henry.bsky.social•35 days ago

Congrats!

Comments

Posting Rules

Reply