wow! this is super resource-efficiency-brained, and seems very far along for a single-person project - ThreadSky

bnewbold.net • 46 days ago

wow! this is super resource-efficiency-brained, and seems very far along for a single-person project

I open-sourced AppViewLite, a self-hostable AppView.

It's still a work-in-progress, but the main features are there (besides being read-only)

Comments

knuxbbs.com•46 days ago

@drasticactions.dev

drasticactions.dev•46 days ago

Yep, I’m aware of it! They’ve committed some PRs for perf stuff and I’ve been monitoring it for quality of life features I can add to the library.

davd.dev•33 days ago

📌

bnewbold.net•46 days ago

> Post data is compressed by turning it into GPT/Tiktoken tokens, then encoding the 18-bit tokens using a variable-length bit representation, and then serializing everything into a Protobuf message (along with other metadata), which is then Brotli-compressed

wow

filippo.abyssdomain.expert•46 days ago

uh, excuse me?

mackuba.eu•46 days ago

I can't tell if this is a real thing or trolling

justinm.one•46 days ago

I recall seeing something on hackernews about this - the tokenizer used by AI models happens to be a _very_ efficient compression algorithm for readable text

khanwinter.com•46 days ago

Oh funny

bnewbold.net•46 days ago

needs blocks implementation (a bit complex), and label ingestion (simple, but do need to juggle multiple websockets/subscriptions).

does kind of feel like a "label relay" would help projects like this

piss.beauty•46 days ago

what would this "label relay" do in order to help? same socket aggregation as firehose apis, but for labels?

bnewbold.net•46 days ago

yeah, just re-broadcast labels from many services. would handle "discovery" (knowing what labelers exist), apply rate-limits, and would be a single downstream. might "help" with verifying signatures as well.

saves a bunch of reliable networking code which folks might not bother with

bnewbold.net•46 days ago

fwiw, I haven't confirmed the 200 GB/day quote for raw firehose. but if it is something scary-large like that, note that actual repos are much smaller. eg, the firehose history for any single account is way way bigger than the final repo. intermediate MST nodes add up fast! and don't compress well

mackuba.eu•46 days ago

Yeah, it's more or less like that

evbogue.com•46 days ago

Why is it so much?

bnewbold.net•46 days ago

it's a lot because you are getting a lot: complete authenticated synchronization of repo data structure, which has completeness+enumerability, and can do key rotation or multiple ops with a single cryptographic signature

vitriolix.mastodon.social.ap.brid.gy•46 days ago

@bnewbold.net it would be fun to run a firehose watcher of some kind, home gigabit should have plenty of bandwidth to keep up right? might need to explore making one

bnewbold.net•46 days ago

yeah, definitely. you can use the 'goat' CLI and just 'goat firehose' as a minimal example

vitriolix.mastodon.social.ap.brid.gy•46 days ago

@bnewbold.net oh perfect thanks

Comments

Posting Rules

Reply