milicevica23.bsky.social
Data/Software engineer building a data platform for Magenta (German Telekom in Austria) interested in database internals, modern data stack, and rust
Located in Vienna, Austria
111 posts
195 followers
463 following
Prolific Poster
Conversation Starter
comment in response to
post
Then, you have a hard time orchestrating the workload?
comment in response to
post
We use it to wrap everything inside
I am annoyed by the size of the image 2-3 GB depending on the features inside. But also we didn't try to optimize so hard
comment in response to
post
The nice thing about this integration is that it is concept is the same as other dagster integration
So it is very easy to get the context and make it work for you
We are currently exploring Tableau but there is some stuff missing for us to be useful
comment in response to
post
I've heard that Gitlab has some remote work guides online maybe it helps
comment in response to
post
Already watched the full courses from multiple years and read all of the provided papers 😃
comment in response to
post
Thanks, let me check
comment in response to
post
Nice, let me try it out
comment in response to
post
We dont have so much complex code in the body functions therefore i was never really interested in debuging it, but i would try this in our environment.
What we were debuging is the load of the code location itself beecause we have some code generators which needed some debuging 😃
comment in response to
post
Interesting to see what dbt will adopt from the newer tools. It makes a lot sense to use sql parser and not string formater for all sql relations
comment in response to
post
If you know dbt, then you already know what asset-based means. We can also ask ourselves what is the benefit of an orchestrator where the base unit is an asset.
1. We can attach tests to an asset
2. We can do declarative orchestration -> because one asset is one unit in the dagster
comment in response to
post
Now we can ask ourselves what an asset is, and given a bit of refactored dagster example, we can show it. The most important thing to do is to think about what is implicit in the execution flow and what is the base unit of an orchestrator. The base unit can be a task or an asset. Focus on colors
comment in response to
post
Imagine an asset and its lifecycle
There are two phases:
1. Loading asset in a code location -> It reads the metadata and sends it to the UI, which is saved in the state db
2. Somebody selects and clicks materialize (run) for an asset -> daemons are responsible for listening and running the function
comment in response to
post
Technically, dagster is an orchestration ecosystem with an ui app, few working threads (deamons), a state database, and clever usage of the decorators
When you run dagster dev all of the above starts and has its functionality in the system
comment in response to
post
Last 6 months, my team and I have developed and tried to learn about dagster and its concepts. I will try to summarize in a thread how we see dagster
comment in response to
post
Oh damn there’s also this: github.com/hafenkran/du...
comment in response to
post
Somebody also has to make hands dirty?