New features in the State Reader API of Spark Structured Streaming for auditing, exploring, and visualizing state changes. Interesting stuff also needed for Apache Flink (hopefully FLIP-423 will help with that making the state queryable again) www.databricks.com/blog/announc... - ThreadSky

About ThreadSky

galiglobal.com • 21 days ago

New features in the State Reader API of Spark Structured Streaming for auditing, exploring, and visualizing state changes. Interesting stuff also needed for Apache Flink (hopefully FLIP-423 will help with that making the state queryable again) https://www.databricks.com/blog/announcing-simplified-state-tracking-apache-sparktm-structured-streaming

Comments

2symbols.com•21 days ago

I read this more as changes to what would be the equivalent of the State Processor API in Flink?

galiglobal.com•21 days ago

That's also my understanding. Snapshots are equivalent to Flink Savepoints, but as far as I know there's no Flink equivalent to the Change feed feature, right?. Datorios offers some interesting state evolution tracking capabilities and it's something I would like to explore at some point

2symbols.com•21 days ago

The main challenge is always that savepoint and checkpoints can also contain data that hasn’t been emitted to the sink yet. You’re basically looking at committed and uncommitted data from internal state. Is that really interesting for users? Or just the committed part?

galiglobal.com•20 days ago

It may be interesting for debugging during development. I found myself in situations where checkpointing was frozen after a while and there was no useful info in the thread dump to debug the issue, being able to see the state evolution in real time may help to understand the issue

2symbols.com•20 days ago

Yeah I can see it being useful when you’ve implemented your own operator, but I also see users making assumptions how some of the built-in operators work and try to change/fix things where they don’t know exactly what they’re doing

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply