Easy to Upload, Hard to Delete: The Life Cycle of Images and Resources

A detour into the least glamorous topic here: product images. “Isn’t that just file upload?” — this chapter spends half its length on uploading and the other half on something far harder: deleting. Uploading takes an afternoon; deleting takes a lifetime.

Birth: an upload pipeline designed for the live floor

Look at this pipeline’s real usage context first, because the whole design grew from it. #9 said the person operating the platform during a stream is the assistant — the same person also driving the stream’s settings. Where do product images come from? Screenshot the stream, paste into a dialog on our front end, confirm, upload. The host holds the goods up on camera, the assistant grabs the frame, Ctrl+V, and there’s the product image — the source of product images is the stream itself.

Both ends doing webp processing looks like duplicated work, and is in fact two completely different things:

The front end cropping to a square and converting to webp is optimisation — screenshots come in every aspect ratio and squares lay out cleanly; bandwidth on set is precious, so one webp pass makes the upload fast and the experience good. But everything the front end does can be bypassed, which is why —

The back end doesn’t trust the front end and always re-encodes, which is a guarantee. Before writing this chapter I went back to the code: even an incoming webp is re-encoded, and the originally uploaded bytes never land. That design is deeper than it looks: file validation is a ladder — check the extension, check the declared Content-Type, check the magic number, actually decode, re-encode — each rung stronger than the last, and re-encoding is the strongest: a file that won’t decode fails on its own; anything that survives decode+encode is necessarily a real image; and any payload hidden inside the file or EXIF riding along is destroyed by the re-encode.

We never wrote a single “check whether this file is safe” if. Validation isn’t a gate before the normal flow, it’s a by-product of it — the same philosophy as #3‘s “the lookup is the validation” (a parsed key not found in the DB is naturally discarded): transcoding is validation.

One last small, pretty detail: the DB stores a path (the fact), and the API resolves it into a full URL (the derivation) at response time — with images actually served from GCS behind a CDN. Change CDN or bucket one day and you change one line of resolution logic, with zero migrations. #14 said product images were carried by the CDN, so what the DB had to bear at peak was only the list API — the image-serving path never went through our machines at all.

Life: one table, and an insurance policy never claimed on

The image is stored; who remembers it? image_metadata: the GCS path plus content type + object id — the generic foreign key‘s fourth appearance in this system (after cart sources, blocklists and comment provenance). Meanwhile product and style also store the image_metadata id directly.

Note that this is a bidirectional reference, and the two directions exist for completely different reasons: the forward one (product → image) serves reads and is used every time product data goes out; the reverse one (image → owner) exists for exactly one thing — cleanup. GCS is outside the database’s jurisdiction and a foreign key constraint can’t protect it; this table is effectively extending FK discipline by hand onto blob storage — the unbundled database restoring another piece: Postgres manages TOAST and vacuum for itself, and since our “large objects” live in GCS, we have to keep their books and reclaim them ourselves.

Was the reverse reference ever actually used? Honesty time: no. The query “look up from image_metadata whether any product or style still uses this” was never written. It’s an insurance policy never claimed on — the reason for taking it out was entirely genuine, and the day of the claim never came.

Death (1): the daily sweep

So how did cleanup run? The real mechanism is simpler than I remembered, and more interesting:

1. When an assistant replaces an image, the system marks the old image_metadata for deletion.
2. A daily scheduled sweep: for everything marked, actually delete the GCS object and clear the metadata.

In garbage-collection language: this isn’t a tracing GC scanning globally for live references, it’s a tombstone mechanism that knows at the moment of change who died — the instant an image is replaced, the old one’s death is a settled fact, so mark it; the daily sweep only executes, it never judges. The judgment (checking back whether anyone still uses it) should in theory exist, and in practice was skipped — and nothing ever went wrong.

Why not? Because there was exactly one entrance to deletion: replacing an image. The only place that marks anything for deletion is the place that definitively knows the old image has been superseded, so the mark is always right and the reverse lookup is redundant. The last chapter‘s sentence pattern shows up again — “not built, because the structure made it unnecessary” — but this time I have to add a but: this is the lucky version. Last chapter’s “unnecessary” was designed (redundancy deliberately squeezed out); here it merely happens to hold — the day a second deletion entrance appears (a bulk import, a product duplication, a back-office cleanup tool), a sweep with no reverse lookup starts killing innocents. The rebuild’s one-line conclusion: check back once before sweeping — one cheap query, upgrading luck into a guarantee.

Death (2): the bug generator I wrote with my own hands

That’s death for a resource; now death for data — soft deletion. This part is my honest failure, and it originated with me.

I defined a SoftDeleteModel: add an is_deleted column with a custom manager — two methods, actived and deleted, with the default queryset returning only actived and the original objects renamed to all_objects. And then every model inherited it. It was comfortable: every query automatically filtered out deleted rows. Clean.

The collapse chain is in the diagram. One day we wanted to record deleted_at, but changing the base model would touch every model — so some models dropped the inheritance, and implicit and explicit filtering started being mixed. From then on every call site had to think first: “which camp is this model in? Do I need all_objects? Do I filter myself?” The mental load grew day by day, and an engineer’s rational choice is to converge on the one answer that needs no thought: use all_objects everywhere — safe, never loses rows. And then the queries that forget to filter is_deleted started appearing. A bug generator is my epitaph for it.

The root cause isn’t at any step of the collapse chain, it’s on day one: objects lied. Its name promises “all objects” and its behaviour is “objects that aren’t deleted” — every call site that believed that lie is a buried mine, and mixing merely attached the fuses. A half-built abstraction is more expensive than none: fully implicit or fully explicit can both live; mixing makes every call site pay for one more judgment, and faced with repeated judgment a human will find a shortcut, and the shortcut will pick the wrong side. #10‘s three-act permission play is the same structure — a framework’s magic half-used hurts more than not using it.

The rebuild: give every kind of death a proper name

Three rules for a rebuild, all variations on “explicit”:

One: store a fact in the column, not a flag. deleted_at (a nullable timestamp), not is_deleted. “Is it deleted?” is derived from deleted_at IS NOT NULL; “when was it deleted?” has an answer from day one — and the collapse that started with being unable to add deleted_at never happens. Append facts, derive status; this series’ iron law applies even to a deletion column.

Two: share a QuerySet, not a base model; and objects never lies.

class SoftDeleteQuerySet(models.QuerySet):
    def active(self) -> "SoftDeleteQuerySet":
        return self.filter(deleted_at__isnull=True)

    def deleted(self) -> "SoftDeleteQuerySet":
        return self.filter(deleted_at__isnull=False)


class Product(models.Model):
    deleted_at = models.DateTimeField(null=True, blank=True)  # declared explicitly on each model
    objects = SoftDeleteQuerySet.as_manager()                 # objects doesn't lie: the default is everything

    class Meta:
        constraints = [
            models.UniqueConstraint(
                fields=["keyword"],
                condition=Q(deleted_at__isnull=True),   # a soft-deleted row doesn't squat on the unique key
                name="uniq_active_keyword",
            ),
        ]

One rule remains: objects is always everything, and filtering means writing .active() — explicit at every call site, greppable, no judgment required. Inheritance shares policy; composition shares mechanism: everyone shares the mechanism (QuerySet methods) while each model declares its own facts (columns), so when somebody later wants deleted_by they add it themselves without touching anyone else. The conditional unique constraint alongside is a vaccine against a hidden pothole: without it, a soft-deleted keyword squats on the unique key forever and a product of the same name can never be created again.

Three: soft deletion is a per-table policy, not a global default. The decision that “every model inherits it” went wrong earlier than how the manager was written. This system’s data naturally falls into three classes, each with its own proper death:

The data’s role	How it dies	Examples
Facts	Never deleted — “deletion” doesn’t exist	order, orders payment, allocation log
Master data referenced by facts	Soft delete — hard deletion breaks history	product, style, image_metadata
Transient	Hard delete — cleared without apology	cart items

The interesting part is that the behaviour back then already followed that table: closing a round hard-deleted carts without apology, and nobody ever dared delete an order — it’s just that the base model flattened all three classes into one. The tables genuinely needing soft deletion number five or six; declaring them explicitly isn’t tiring at all.

Reflections

Uploading is a feature; deleting is a responsibility. Upload code is written in an afternoon, demos perfectly and ships to applause; but from the moment upload is pressed, every byte starts being billed, every reference can break, and every image eventually faces the day of “does anyone still want you?”. Few people think deletion through while building upload — we managed half: the table existed, the reverse lookup didn’t. A resource’s life cycle doesn’t end at “upload succeeded”, it begins there.

The best validation is no validation gate. Transcoding as validation, the lookup as validation — this system’s two toughest defences weren’t written as ifs, they came from designing the unavoidable flow into a natural filter. A gate can be bypassed, forgotten, or drift away from the main flow; a by-product can’t, because it is the main flow.

Implicit is borrowed convenience. The comfort of objects automatically filtering deleted rows is borrowed from the future — interest starts accruing the day you begin mixing, is paid in instalments by the judgment cost at every call site, and settles in one lump sum as a bug. Typing seven more characters (.active()) buys out never having to judge again. What’s that lesson worth? An epitaph reading “bug generator”, and a chapter’s length.

Peaks, operations, reconciliation, life cycles — the battles that cut across the system end here. Next, the people who built all of it: six engineers, and how they ran at the speed of twenty.