Effective Troubleshooting: Debugging Is a Method, Not a Talent

· tech

#sre#incident

📑 Contents

The previous post said on-call stops the bleeding first; but once the bleeding has stopped, you still have to find why. This post is about troubleshooting — and its single most important idea is: debugging doesn’t rely on talent or luck; it’s a systematic method you can learn. The difference between a beginner and a veteran isn’t “knowing the answer” but having a process that closes in on it.

Recognise the anti-patterns first: guessing and swapping parts

What does method-less debugging look like? Changing things at random to see if it gets better (part-swapping debugging), checking only the places you know, changing a pile of settings at once, being led by the nose by “something was touched recently, I think”. These don’t work because they aren’t narrowing the problem — you’re just gambling: lucky and you stumble on it, unlucky and you make it worse, and afterwards you have no idea what actually fixed it (because you changed too much at once).

The systematic troubleshooting process

Debugging with a method is an ordered loop:

① Triage keep the system alive (root cause can wait) ② Examine monitoring / logs four golden signals ③ Diagnose hypothesise → test → eliminate bisect to close in ④ Treat one variable at a time reversibly not fixed? new hypothesis Every step "narrows the range"; the anti-patterns (guessing, swapping parts, changing many things) never narrow, only gamble
Mitigate first so users feel nothing, then examine the monitoring (four golden signals), then close in step by step with "hypothesise → test → eliminate", and finally fix carefully. The only difference from the anti-patterns: every step narrows the problem's range

Diagnosis (③) is the heart of the process, and its technique is one sentence: form a hypothesis, find a way to confirm or refute it, and so eliminate part of the possibilities. It’s exactly binary search — every test cuts the suspect range in half.

The core weapon: divide and conquer (bisection)

The sharpest move in diagnosis is divide and conquer: along the request path, don’t blindly try segment by segment; measure the middle first — ask “is the problem before this point, or after?” and cut half away in one stroke:

Client LB App Cache DBslow ❌ ① measure the middle (App) ② slow only after App → narrow to the back → DB Halve along the path each time → located in a few steps, not blind trials from the start User report: slow. Don't guess "probably X" — measure, bisect
Bisection turns "needle in a haystack" into "located in a few steps": measure the middle of the path, decide whether the problem is in the front half or the back, cut half away, repeat on what's left. The move works in any layered system

A few iron rules

Beyond the process, a few iron rules keep debugging from going astray:

  • Change one variable at a time. Change three things at once and it works, and you’ll never know which one fixed it, or whether the other two planted mines.
  • Trust data, not intuition. Look at monitoring, logs, the golden signals; use data to refute hypotheses rather than intuition to confirm prejudices.
  • Ask “what changed”. Most failures relate to “the most recent change” (a deploy, a config, traffic) — check the change log first and it’s often solved in a second; but don’t let it hijack you into looking nowhere else.
  • Record what you did. It makes backtracking and handover easy, and it’s the raw material for the postmortem (next post).

Reflections

Debugging is a method, not a talent

When I started out, I thought senior engineers debugged by some sixth sense — one glance and they knew where it was broken. Watching up close, I found it wasn’t a sixth sense but a steady method of closing in: look at the data first, then hypothesise, then bisect to narrow. They didn’t “know the answer”; they “had a way to force the answer out in a few steps”. That realisation affected me a lot — it turned debugging from “mysticism that depends on inspiration” into “a skill that can be deliberately practised”. It’s also the first thing I teach new people: don’t rush to guess, look first; don’t change things at random, narrow first. With the right method, anyone can debug reliably.

Bisection is the universal key to debugging

“Cut half away along the data flow each time” works wherever I use it. Its power is that every step halves the problem space — a ten-segment path is located in three or four steps, not by trying from one end to the other. Once it clicked, I noticed it’s the same “shrink the search space” thinking I use reading SQL execution plans to find bottlenecks, narrowing problems in gaps and islands, even finding bugs in code review. Learning to bisect is worth more than memorising the fix for any particular bug, because it applies to every problem you haven’t met yet.

”Trust data, not intuition” — look, don’t guess

The greatest enemy of debugging is really the prejudice of “I think it’s probably X”. Prejudice is dangerous because it makes you look only for evidence that supports it and automatically ignore contradicting clues, so you dig ever deeper in the wrong direction. The value of a systematic process is that it forces you to “look” — at monitoring, at logs, at the execution plan — using facts to refute hypotheses rather than intuition to confirm them. It’s the same belief I keep repeating in SLIs that track user experience and reading EXPLAIN instead of guessing: let the facts speak. What engineers most need to train isn’t guessing accurately; it’s the discipline of holding back the guess and looking first.