Total NOC crew ~ 60 enthusiasts internally manage to beg, borrow and steal themselves from their day job for this week :-), of which slightly over half are dealing with the mechanics of rolling out at peak 220+ switches over a course of a day, and packing them down over a course of 2 hours. - ThreadSky

ayourtch.bsky.social • 9 days ago

Total NOC crew ~ 60 enthusiasts internally manage to beg, borrow and steal themselves from their day job for this week :-), of which slightly over half are dealing with the mechanics of rolling out at peak 220+ switches over a course of a day, and packing them down over a course of 2 hours.

Comments

ayourtch.bsky.social•9 days ago

The remainder are dealing with the various aspects of ~1200-device network - core, switching, wireless, SOC (we even have a pentester in residence who ensures nobody uses a simple password and that all ACLs are in place and tight).

ayourtch.bsky.social•9 days ago

Everyone has a ton of fun doing their part, eg DC folks used Llama model to tweak the webex bot that is used to query about IP addresses and entities into a “LLM with tools” setup, which had the added benefit of entertaining us with short stories or poems about the stuff we were asking it.

ayourtch.bsky.social•9 days ago

Heavy use of products from $work, but also a lot of open source - grafana, librenms, netbox, etc. I built and run a custom Rust based automation thing that drives the config and changes of all IOS XE fleet, and does asset and deployment/teardown tracking - some of the tiny chunks of it on my GitHub.

ayourtch.bsky.social•9 days ago

L3 wise, this year we ran 120 9300s, all in OSPF area 0 with “no autostate” - very little churn in the routing table at the expense of the very TCAM-hungry two byte addressing scheme keyed mostly on “(service number, L3 node number)”

ayourtch.bsky.social•9 days ago

L2 wise ~350 9200s, new this year; quite happy with them. Nice and bricky - so compared to 3560cx *much* easier to provision en masse, mechanically, and certainly faster to ssh into - the port changes generally completed within 3-5 seconds, as opposed to about 10 sec or so on CX series :-)

ayourtch.bsky.social•9 days ago

Another peculiarity - the addressing for uplinks is dynamic, with a local pool configured for the downlink interfaces, this gives a very Lego-like property of being able to easily adapt the design and not having to track the exact wiring - if both ends of the cable end on our infra = 👍

spoofnet.bsky.social•9 days ago

Wow thanks for all that! Need to do me some reading and watching

ayourtch.bsky.social•9 days ago

I will also post here as i toy around with various ideas for modernizing things… thinking of finishing https://github.com/ayourtch/textfsm-rs and using it as part of the engine that would allow to implement various provisioning workflows that are easy to tweak - the existing nearly 8 year old code is… crufty 😂

ayourtch.bsky.social•9 days ago

Oh and one thing I forgot - git for templates with hooks for regeneration + kickoff the changes - while trivial things like service changes on the port are just a dropdown + button click away for anyone of the 60 people, the things like ACL editing and push are limited to a group of ~4 people.

Comments

Posting Rules

Reply