The Upgrade Was the Outage
16 November 2026 · @Domaincracy
The 3 AM alarm
On my way home from NamesCon in Miami, my nonstop to Phoenix was cancelled. The airline rerouted me through Los Angeles. I made it to LAX; the connection to Phoenix was then delayed, cancelled, delayed, cancelled again, and rebooked.
By then I had caught a cold and was running a fever, so I took a hotel room to sleep and shower. When the alarm went off, I checked the newest 6 AM flight. It was delayed another 14 hours. I rented a car and was home six hours later.
I was one of the lucky ones. In both Miami and Los Angeles I saw huge lines of people waiting to be rebooked: families with toddlers, people in wheelchairs. Not everyone in those lines could rent a car and drive home.
What we know, and what we don't
On November 13 the FAA halted departures nationwide for more than four hours. The investigation is not finished, but officials have described the outline:
- Two sites flooded the network with traffic: a radio facility in El Dorado, Arkansas, and a radar installation in Amarillo, Texas. Each was opening roughly 1,000 connections every four to five seconds (WSJ, via Yahoo Finance).
- The Transportation Secretary said it began with work to move the Arkansas site from copper lines to fiber (Yahoo News).
- Blocking the two sites stabilized the network. The router manufacturer says its equipment did not fail, and no root cause has been established (IBTimes UK).
So this is not a diagnosis. It is what the story looks like to someone who has spent a career connecting old systems to new ones.
The view from layer 7
I earned an MCSE in 2000, so I still think in OSI layers. My working life, though, has been at the top of the stack: the application layer, where the job is getting decades-old platforms and modern ones to cooperate.
What happened on November 13 lives lower down, in routing and transport. I won't pretend to debug it from the driver's seat of a rental car. But the pattern is one every integration person recognizes, and it does not depend on the layer.
The migration is the dangerous moment
The popular story is aging equipment: copper lines and routers old enough to vote. Old gear is a real problem. But by the official account so far, the old gear did not fail on its own; the trouble started during the work to replace it.
That matches everything I have seen. A legacy system in steady state is ugly but predictable, and the people who run it know its limits. The dangerous moment is the cutover, when a new component with modern defaults meets an environment built on assumptions nobody wrote down.
New equipment is faster, chattier, and more capable. The old environment was never protected against a neighbor like that, because for decades it never had one. The test that matters is what the new thing does to everything it touches, and that test is the one that gets skipped.
Least privilege, for devices
In access management the first principle is least privilege. A new account gets what it needs and nothing else, so that when it misbehaves the damage has a boundary.
The same question applies to devices. Why could two edge sites reach enough of a national network to destabilize it? Blocking them restored stability, which means containment was possible. It was manual, and it came after the planes were already on the ground.
My first instinct was an application-layer one: put a broker between the old world and the new, to translate and mediate. On reflection that is the wrong tool. A broker solves a translation problem; this looks like a containment problem, and a central broker is one more thing whose failure takes everything with it. The right tools are older and duller: segmentation, rate limits at every boundary, and automatic quarantine for any node that starts shouting.
Why a domain investor cares
The domain name system has the same property. It is shared infrastructure that everyone depends on and almost nobody thinks about, and a small mistake in one place can reach everyone at once. A bad signing key at a registry, or a routing error at a large DNS provider, can make every name behind it unreachable, and it has happened more than once.
Every domain's value rests on the assumption that it resolves. I spend part of my time on DNS policy, and the questions there are the ones above: who can affect what, how far a mistake can travel, and whether containment is automatic or a phone call.
Questions for the root cause analysis
Whatever the final report says about the routers, these are the questions I would want answered:
- Was the cutover rehearsed against a realistic copy of the production network, or was production the test?
- What limits how much traffic a single site can send, and who verified those limits after the new equipment went in?
- When a site misbehaves, is isolating it automatic, or does it take people on a conference call?
- Was there a rollback plan, and how long did it take to execute?
- A much larger modernization is now being proposed. How many more cutovers like this one are ahead, and what changes before the next?
Each of these has a boring answer when the work is done right. The cost of skipping the boring work is not measured in routers. It was standing in line in Miami and Los Angeles, holding a toddler or sitting in a wheelchair, waiting to be rebooked.