Eliminating Toil: Treat Repetitive Operations as Bugs to Be Killed

· tech

#sre#reliability

📑 Contents

The first post said SRE’s core is “operations can be engineered”. The toil in this post is the thing to be engineered away. Many people think toil means “hard work”; it doesn’t — it’s a class of work with clear traits, and if you don’t actively cut it, it naturally expands until it eats all the time you have for engineering.

What toil is (and what it isn’t)

Toil is work “tied to running a Production service, with the traits below”. The more it matches, the more it’s toil:

Is it toil? Check six traits ✓Manual — a person does it, step by step ✓Repetitive — done many times, and will be done again ✓Automatable — a machine could do it; nobody has written it yet ✓No enduring value — the system is no better afterwards ✓Scales linearly — the service grows, so does it ✓Reactive — done when triggered, not planned More matches → more toil. Note: meetings, planning, documentation are overhead, not toil
The six traits of toil. It isn't "tiring", it's "nothing left behind afterwards" — manual, repetitive, and the system no better for it. Meetings, planning and email are annoying too, but they're overhead, not toil

One distinction matters: toil ≠ all unpleasant work. Meetings, documentation, planning, answering email are overhead — they take time, but they’re not toil. Toil specifically means operations that are manual, repetitive, something a machine could already do, and that leave the system no better — manually restarting a service, manually changing a setting, manually handling the same alert every time. Nor does it mean zero toil is required; a little is acceptable. The point is not letting it expand.

A common grey area: “does manual testing count as toil?” — the answer is it depends which kind, and the line falls exactly on “no enduring value”. Manually clicking through the same regression flow every release hits all six traits; it’s textbook toil and should be automated away. But exploratory testing (a person poking around by experience to find new edge cases), the first test of a brand-new feature, UX testing — those need human judgement, explore something new every time, and are hard to automate; they accumulate, so they’re not toil. One test: “after this manual run, will next time have to be exactly the same again?” If yes, toil; if not, and every run is judgement-driven exploration, it’s valuable human effort.

Why it must be cut: it grows linearly with scale

Toil’s most dangerous trait is “grows linearly with service scale”. Double the service and the manual operations roughly double too. Leave it alone and the headcount required climbs with scale until it drowns the team:

service scale → people needed ↑ left alone: headcount grows linearly → drowned automation: headcount decouples from scale SRE guardrail: time on toil < 50%; the other half must go to engineering that "reduces future toil"
Left alone, toil grows linearly with scale and drowns the team (red); investing in automation costs a little more up front, then headcount decouples from scale (green). That's the core of SRE's "operations as software engineering": headcount no longer tracks scale

Worse, there’s a vicious cycle: more toil → less time to write automation → toil keeps accumulating → even less time… Because toil is always “urgent” (an alert is firing, a service needs restarting) while automation is always “important but not urgent”, forever pushed to tomorrow by firefighting. To break the cycle, Google set a famous guardrail: SREs should spend less than 50% of their time on toil, and the other half must go to engineering that “reduces future toil”. The line is deliberate — without forcibly fencing off engineering time, toil will certainly eat it all.

How to cut it (but not all of it)

The main weapon against toil is automation: turn repetitive manual operations into code, into self-service tools, into systems that heal themselves. But one important premise — not all toil is worth automating. Automation has its own cost, and you have to do the sums: the investment in automation vs the future toil saved × how often it happens. An operation done once a year isn’t usually worth two weeks of automation; one done daily, ten minutes each time, has an extremely high return. So cutting toil isn’t mindlessly automating everything; it’s going after the highest return first.

Reflections

The essence of toil isn’t “tiring”, it’s “nothing accumulates”

I used to equate “this work is annoying and tiring” with “this is toil”, and later realised the point isn’t tiredness at all but whether anything is left behind afterwards. Building a new feature is tiring, but it accumulates, the system gets better — that’s not toil; manually restarting a service for the hundredth time is just as tiring, but the system is unchanged and you’ll do it again next time — that is toil. The distinction sharpened what I protect — I guard the time that “accumulates”, and cut the time that’s “pure consumption”. Put people where things accumulate and hand pure consumption to machines: that’s what SRE really means by “engineering” it.

The 50% cap is a “deliberate guardrail”, not an ideal

What I appreciate most about the 50% line is that it admits a reality: unless engineering time is forcibly fenced off, firefighting will certainly eat it. Toil is always more “urgent”, automation always more “important but not urgent”, and urgent always wins. 50% doesn’t mean “ideally spend half on toil”; it’s a cap — a guardrail that forces you to protect the important-but-not-urgent. It’s the same discipline as any long-term investment I make: important but not urgent things never happen unless you actively fence off time for them.

Don’t automate for automation’s sake

Cutting toil can also go too far — not all toil is worth automating. I’ve seen someone spend a month automating a process that ran once a quarter and took five minutes, purely because “manual isn’t very SRE”. The sums simply don’t add up. Automation is a means, not a faith, and the criterion is always that ROI: investment vs toil saved × frequency. Which brings us back to confirm the pain first, then bring the heavy weapons — first check how much this toil actually hurts and how often, and automate only when it’s worth it. Spending effort on the few things that hurt every day is far more real than a purist pursuit of “zero manual work”.