andygrove.io
Apache Arrow & DataFusion PMC Member. Original creator of Apache DataFusion.
37 posts
2,571 followers
77 following
Prolific Poster
Conversation Starter
comment in response to
post
Is this using Arrow and/or DataFusion? If so, our Discord is probably a good place to ask.
datafusion.apache.org/contributor-...
comment in response to
post
Honestly, I have no idea. I wonder if @timsaucer.bsky.social knows
comment in response to
post
Data is stored locally on each node at the moment. I will probably set up minio when I have some spare time.
comment in response to
post
I've been using the scripts in github.com/apache/dataf... to generate the csv data and then convert to Parquet.
comment in response to
post
Yes, k3s.
comment in response to
post
I'm currently using the k8s cluster for benchmarking of TPC-H and TPC-DS with various Apache DataFusion subprojects, such as DataFusion Comet, which is a Spark accelerator.
comment in response to
post
Not currently, no. I am not currently doing any GPU acceleration, but I can always add GPUs later.
comment in response to
post
My first work PC had closer to 256KB of RAM, and a 20MB hard drive, IIRC (IBM PC AT).
comment in response to
post
I really shouldn't do anything technical before my 3rd cup of coffee 😂
comment in response to
post
More RAM would be nice but these consumer desktops are pretty limited.
comment in response to
post
Having a Rust wrapper around cuDF would open up many opportunities for GPU-accelerated Arrow & DataFusion and all the systems now building on this foundation.
comment in response to
post
What kind of database? OLTP, OLAP, HTAP, time-series, streaming, something else?