it's been a rough couple hours for the firehose! it has been flapping around in volume and offset. the bsky app has been smoother because we are using an alternative internal relay (starting a couple days ago). 🧵 - ThreadSky

bnewbold.net • 192 days ago

it's been a rough couple hours for the firehose!

it has been flapping around in volume and offset. the bsky app has been smoother because we are using an alternative internal relay (starting a couple days ago). 🧵

Comments

tiletony.bsky.social•192 days ago

Whatever you are doing, keep up the good work. And, thank you!

euphoricfuzion.bsky.social•192 days ago

You're doing an incredible job with this unexpected torrent of activity.

We all, the ones with hearts anyway, are happy to be patient and helpful as your team works through the major load increase 🫶🏼

snarfed.org•192 days ago

#hugops! you all got this, we're all rooting for you ❤️

bnewbold.net•192 days ago

we have thrown out our earlier plans for now, and have cut over the https://bsky.network public firehose endpoint to our alternative relay.

we have rolled the cursor back about 4 hours of wall-time, and it is catching up to current.

this is a new cursor sequence! 251476547 is a random recent seq

andwhyisit.bugsiteguardian.com•192 days ago

So is the plan to eventually rework the firehose so that it scales better, then switch back?

natanael.bsky.social•192 days ago

If the hardware and servers is near equivalent, I'd assume they stick to the current one (annoying to switch often) but make the prior one the new alternate server after a full restore.

bnewbold.net•192 days ago

if you are a firehose consumer, you'll need to update cursor to this new sequence space. you will probably see duplicate/overlapping events. there may be a few other bumps in there, like tooBig, and a spike from a couple PDS reconnects.

mikebeas.com•192 days ago

presumably no impact for jetstream right?

nickwrightdata.ntw.app•192 days ago

📌

andy.wrestlejoy.com•192 days ago

For Jet Stream, do we just wait?

bnewbold.net•192 days ago

very sorry about all this! it may take a day or two to clean up.

will have some stories to tell and new tools coming out of this.

finnthepanther.com•192 days ago

As a developer, I really appreciate the transparency and updates in real-time. Companies much larger can learn a thing or two from the team here 💜

flicknow.xyz•192 days ago

appreciate all the hard work you all are putting into this 🩵 amazing things had held up as well as they have tbh!

mackuba.eu•192 days ago

🍿

unnecessary.tech•192 days ago

Thanks for the hard work and transparency. It’s really interesting to get a view into such a high traffic system.

johnrees.bsky.social•192 days ago

your efforts are appreciated!

roblugg.bsky.social•192 days ago

I have no idea what any of this means but it all sounds very impressive so keep up the good work!

cloudhunter.co.uk•192 days ago

Just gotta say that I love the transparency y'all give with this

#HugOps

iame.li•192 days ago

say I'm a developer and the data I'm using could be got at through the firehose or through hitting the PDSses directly. would the latter be helpful at the moment?

natanael.bsky.social•192 days ago

Depends on what exactly you're doing. If you're mostly just accessing data from your own service's users, while they're using it, then sure.

natanael.bsky.social•192 days ago

Tldr, the PDS is intended to be cheap to run (easier to self host), if a bunch of things start polling every PDS continously then the bandwidth use starts increasing significantly. Stuff like feed builders should use the relay.

iame.li•192 days ago

I think the main difference for me is that I'm exclusively scanning User X's PDS while they're using the app to set up a linking between an ATProto user and a livestream -- I shouldn't need to be querying every PDS like a feed would need to be.

iame.li•192 days ago

Oh I just realized that's what you said in your first message 🤦 We're on the same page.

walkingalchemy.com•192 days ago

Siiiicckkk!!!

Comments

Posting Rules

Reply