Incident Response: The Real Enemy in a Major Incident Is Chaos

Earlier you learned on-call mitigation and systematic troubleshooting — but that’s “one person against one problem”. When a major incident erupts (many people involved, big impact, high time pressure), you’ll find the biggest enemy is often not the technical problem itself, but chaos.

The real enemy in a major incident is “chaos”

Technical problems always get fixed; but the chaos of “five people changing things at once, nobody knowing what the others are doing” stretches a ten-minute problem into two hours, or even breeds a new disaster:

The incident command system: one role per person

To tame chaos, SRE borrowed the Incident Command System (ICS) straight from firefighting and disaster response: clear role divisions, each person carrying exactly one thing:

A few key moves

With the roles in place, a few more things make incident response run smoothly:

• Declare “this is an incident” early. Too many teams drag their feet about admitting something’s wrong (let’s wait a bit, it should be fine soon), and miss the moment to start coordinating. Declaring an incident isn’t conceding defeat; it’s starting a mechanism that helps you resolve it faster.
• One shared communication channel (a war room / chat channel): everyone aligns in one place; don’t let information scatter into DMs.
• Explicit handoff: when the IC needs to leave or can’t go on, hand command to a named person, out loud; never disappear silently.
• Practise in peacetime: don’t let the real major incident be the first time you use this process.
• Track outages afterwards (Ch16): record, classify and trend every incident (which kinds happen most, is MTTR improving) — only with data can you talk about improvement.

Reflections

The bottleneck in a major incident is often coordination, not technology

I’ve seen plenty of incidents where the technical fix was actually simple (roll back, restart, shift traffic), and what dragged out the time was the chaos of “five people each doing their own thing, nobody holding the whole picture” — duplicated actions, conflicting changes, even someone breaking what someone else had just fixed. It taught me: chaos is itself a kind of failure, and a man-made one that process can avoid. In a fire, a clear command structure often speeds things up more than two more brilliant engineers — because what it solves isn’t the technical problem but the harder one of “too many hands”.

The IC’s most counter-intuitive trait: they don’t touch anything

The mistake a new IC most easily makes is jumping in to debug — and then nobody watches the whole picture, the others lose their coordination centre, and it gets messier. The IC’s value lies precisely in “not touching, only coordinating”: holding the whole picture, assigning tasks, deciding, shielding everyone from interruptions. Let the best fixer focus on fixing, and the best coordinator focus on coordinating. It’s exactly my experience leading a team: when the leader can’t resist jumping in as the strongest individual contributor, the team loses its brain. Holding back and lifting yourself to the coordination layer is the hardest, and most necessary, lesson of being a commander (and a manager).

Admit “this is an incident” early

Delaying the declaration is the most common and most expensive mistake I’ve seen. People want to save effort, avoid making a fuss, bet it’ll fix itself — and by the time they’re forced to admit it, everything is a mess and the golden window for coordination has passed. My principle now: better to declare and find it was minor than to drag it out until it’s major and panic. And whether that’s achievable comes back to the previous post‘s blameless culture — only when “declaring an incident” is safe, encouraged, and not held against you later will people dare to raise the alarm early. Technical process and culture are tied together here.