Every time we do a 1.5 hour session with the highlights for that particular year - it’s PNLNMS-1035, it should be in the recordings. We don’t redo from scratch so the full story by now can probably take the whole day 😂😂
Comments
Log in with your Bluesky account to leave a comment
(.. don’t redo the design from scratch.). The network itself is pretty much like a sand castle on the seaside - build it, run for a week, then pack up and until next year :-)
Total NOC crew ~ 60 enthusiasts internally manage to beg, borrow and steal themselves from their day job for this week :-), of which slightly over half are dealing with the mechanics of rolling out at peak 220+ switches over a course of a day, and packing them down over a course of 2 hours.
The remainder are dealing with the various aspects of ~1200-device network - core, switching, wireless, SOC (we even have a pentester in residence who ensures nobody uses a simple password and that all ACLs are in place and tight).
Everyone has a ton of fun doing their part, eg DC folks used Llama model to tweak the webex bot that is used to query about IP addresses and entities into a “LLM with tools” setup, which had the added benefit of entertaining us with short stories or poems about the stuff we were asking it.
Heavy use of products from $work, but also a lot of open source - grafana, librenms, netbox, etc. I built and run a custom Rust based automation thing that drives the config and changes of all IOS XE fleet, and does asset and deployment/teardown tracking - some of the tiny chunks of it on my GitHub.
L3 wise, this year we ran 120 9300s, all in OSPF area 0 with “no autostate” - very little churn in the routing table at the expense of the very TCAM-hungry two byte addressing scheme keyed mostly on “(service number, L3 node number)”
Comments