New blog post! Big data isn’t dead; it’s just going incremental. But bad things happen when uncontrolled changes collide with incremental jobs. Reacting to changes is a losing strategy. jack-vanlightly.com/... - ThreadSky

vanlightly.bsky.social • 107 days ago

New blog post! Big data isn’t dead; it’s just going incremental. But bad things happen when uncontrolled changes collide with incremental jobs. Reacting to changes is a losing strategy.

https://jack-vanlightly.com/blog/2024/11/13/incremental-jobs-and-data-quality-are-on-a-collision-course-part-1-the-problem

Comments

lzaugg.bsky.social•106 days ago

Thanks a lot for the awesome blog posts! Reminded me back in the days when I came to the same conclusion, although from a slightly different angle: https://medium.com/swlh/apache-kafka-pyramid-f475d8840596

vanlightly.bsky.social•106 days ago

This is great, I'm sharing it with colleagues 😄 We have something similar called the Confluent Maturity Curve, but I think your pyramid has a lot more personality.

loppsean.bsky.social•107 days ago

I enjoyed this series, esp part 2. Lots of folks talk about the need for data products/contracts, but incremental processing as a motivator was a good addition.

loppsean.bsky.social•107 days ago

Why not re-use the tools from swe directly? Have downstream "consumers" use API interfaces to access upstreams instead of direct queries? You get auth/versioning for "free". I understand its not tractable for huge datasets, but here its all incremental anyway

vanlightly.bsky.social•107 days ago

I've seen this kind of pull method for syncing using a paged-based API to return data changes, not comfortable for data engineers, but the software team knew APIs so they built that.

loppsean.bsky.social•107 days ago

I think with the right tooling it could become comfortable

I am just skeptical any data mesh / process / people soln can resolve "SELECT * FROM upstream_out_of_my_control_v2"

Whereas "SELECT * FROM API(...)" at least has some of the problems already solved by SWE systems

jobaeyaert.com•106 days ago

Interesting article. It feels like we really need to rethink our data architecture. Shifting to modular, event-driven designs could be a game-changer, cutting down on pipeline breaks and letting us adapt on the fly as data sources change.

michaelayoub.bsky.social•106 days ago

I never thought about it this way but of course, it makes perfect sense with the dual purpose of so many of these jobs: "When the output feeds into other systems that take actions based on the data, these arbitrary delays caused by periodic batch jobs make less sense."

lalithsuresh.bsky.social•106 days ago

Nice article! I agree, incremental processing is everywhere. Do give us at Feldera a look sometime -- fully incremental compute for arbitrarily complex SQL and open-source under an MIT license (https://www.feldera.com/).

runnin-l8.bsky.social•106 days ago

📌

larziman.bsky.social•106 days ago

Really appreciated your blogposts. The Gunnar Morling post is also very interesting.

Just joined Bsky and first content I found and digged :-)

Comments

Posting Rules

Reply