Apparently the real time logs from Amazon CloudFront into Kinesis are in *TAB SEPARATED VALUE* format 🙄 - ThreadSky

ThreadSky

About ThreadSky

darko.rup12.net • 22 days ago

Apparently the real time logs from Amazon CloudFront into Kinesis are in *TAB SEPARATED VALUE* format 🙄

Comments

tobilg.com•22 days ago

Yup… I‘m using this when processing web tracking data

https://docs.ownstats.com/architecture/processing-layer

darko.rup12.net•22 days ago

Do you think the TSV format is for the sake of performance or efficiency? It seems like a very odd choice for data to be processed :)

tobilg.com•22 days ago

It’s odd, yes, and I‘m not sure why it was probably chosen. But I guess the main reason could eventually be that you can parse/split it very efficiently in probably all runtimes, e.g. compared to JSON…

darko.rup12.net•22 days ago

You are probably right! For me JSON seem so much easier to parse / deserialize! But I also guess that JSON has a lot more overhead in the terms of raw bytes being sent over.

tobilg.com•22 days ago

I think parsing single records as JSON for potentially thousands of records/sec could make a huge difference… Also regarding costs.

focusotter.bsky.social•22 days ago

That is gross! I’m assuming this is for cross system support, but still surprised there isn’t an AWS client to handle this in node.

If it helps, there’re a few npm packages that do it.

https://www.npmjs.com/package/tsv-json

darko.rup12.net•21 days ago

My thinking is: It's less data being sent across vs JSON! But I need to do some testing!

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply