Data Pipelines and Data Integrity: Having a Backup Doesn't Mean You Can Restore

Beyond the “service stays up” reliability covered so far, a data system has a more fundamental layer — the data itself must not go missing or go wrong. This post covers two things: the reliability of data pipelines, and an idea that will upend your intuition: “having a backup” isn’t “being able to restore”.

Data pipelines: the hidden traps of periodic pipelines

Pipeline reliability differs from online-service reliability. The most common hidden trap is backlog in a periodic pipeline: data volume grows slowly → each run takes longer → it misses its scheduling window → backlog piles up → the next run has to chew through a giant batch (thundering herd) → slower still. And pipelines are usually several stages chained together, so any one stage that sticks or emits bad data stalls the whole chain, or quietly pollutes downstream.

So pipeline reliability has different concerns from a service: an SLA on data freshness (how fresh must the output be), monitoring on every stage (don’t wait until the end to discover the middle rotted), and the most critical — idempotent and re-runnable (see my Airflow post): when it breaks you can safely re-run it and get the same result, rather than duplicates or an explosion.

”Having a backup” isn’t “being able to restore”

Now the one line this post most wants you to remember. Everyone’s instinct about data safety is “we have backups, relax” — but that reassurance may well be false:

Data integrity: assume every layer will leak

Another counter-intuitive fact: data loss is mostly not hardware failure; it’s bugs and people — a bug writes a column wrong, one wrong command deletes the wrong thing, a batch of bad upstream data quietly pollutes everything downstream. So the defence can’t be just “back up against hardware failure”; it has to be defense in depth, several layers stacked:

Reflections

”We have backups” is the most dangerous sense of safety I’ve seen

Backups give a solid feeling of reassurance, but that reassurance is often false — a backup you’ve never restored from is Schrödinger’s backup: you don’t know whether it’s alive or dead until you open it. The backup file may have been silently corrupt for six months, the restore procedure may live in a document nobody reads and has never been run, the real restore may be too slow for what the business can tolerate. So now when I hear “we have backups”, my reflex is one question: “When did you last actually run a restore drill? How long did it take?” No answer, and that sense of safety is made of paper. It’s consistent with SRE’s standing spirit: don’t assume, verify — backups need a real restore periodically, just as tests need a real run periodically.

Data loss is mostly not hardware; it’s bugs and people

When people think of data loss they picture a disk burning out, but far more common in the real world is a bug writing a whole column wrong, one slip deleting the wrong table, a batch of dirty upstream data silently polluting the whole downstream. “Hardware redundancy” can’t block these; only defense in depth can: soft delete for time to regret, early detection (reconciliation, validation) to catch it before users do, backup + restore as the last net. And you have to assume every layer will leak to stack them thick enough. It’s the same pessimism as the cascading-failures post — good reliability engineering is built on “assume it will break”, never on “hope it won’t”.

Pipeline reliability is half the craft of data engineering

This chapter felt especially close to home, because data pipelines are what I do every day. It reminded me that a pipeline’s reliability is far more than “did today’s run succeed” — it’s whether the data is fresh enough (a freshness SLA), whether a re-run breaks anything (idempotent and re-runnable), whether the backlog can catch up, whether bad data quietly pollutes downstream. The idempotency I talked about along the Airflow line, the reliability in DDIA, and this chapter are all the same thing: keep the data “always there, and correct” under every kind of failure. A dead service comes back with a restart, but wrong or lost data often never comes back — so the reliability of data deserves a bit more paranoia than the reliability of services.