Confirm the Pain Before You Bring In the Heavy Weapons

This is a concept note — a judgment call I keep reusing, split out into its own post so other articles can link back to it.

In one sentence: before adopting any heavyweight tool or architecture, confirm that your pain has actually reached the scale that calls for it. If it hasn’t, don’t — because the cost of a heavy weapon isn’t in “setting it up”, it’s in “feeding it every day afterwards”.

What counts as a “heavy weapon”

I mean the things that are powerful but also expensive to operate: Airflow, Spark, Kafka, self-hosted clusters, multi-layer data architectures… Every one of them solves real, hard problems. But every one of them also brings a whole bundle of things to learn, monitor, be on call for, and debug. The power and the burden are sold as a package — you can’t take only the former.

The failure mode: treating complexity as achievement

The most expensive mistakes I’ve seen weren’t “failing to adopt the right tool” — they were adopting it too early, using it for the sake of using it:

• The data is still a few GB, and there’s already a Spark cluster — when local PySpark, or even dbt + a warehouse, would have solved it.
• Two or three services occasionally exchange a message, and someone hauls in a full Kafka deployment — when direct API calls or a lightweight queue are usually the better deal.
• There are only one or two reports, and the pipeline gets carved into a three-layer Medallion — when two layers (raw + reporting) are plenty.

What these decisions share is treating “we use impressive things” as an engineering achievement. But the complexity you can carry is a finite budget: spend it here, and there’s none left for the problems that actually differentiate you.

The yardstick: has the pain reached that scale?

Before bringing in a heavy weapon, I force myself to answer one concrete question: is the pain I have right now the kind of pain this thing exists to solve?

• Does a single machine genuinely no longer fit or finish the computation? → Only then does Spark get its turn.
• Are there genuinely multiple sources, multiple consumers, audit and rebuild requirements? → Only then do layered architectures / Kafka get their turn.
• Is there a genuine “scheduled, interdependent, retry-on-failure” orchestration need? → Only then does Airflow get its turn.

If the pain is concrete and you can point at it, go ahead; if it merely “sounds like something you should have” or “would look good on a résumé”, hold off.

But this is not “anti-tool”

The crucial flip side: once the pain is there, don’t hesitate. This principle doesn’t tell you to stay stuck with duct tape forever — it tells you to save the heavy weapons for the right moment. It also has exceptions: something like dbt — low adoption cost, small downside risk, high return — I’d say you can “almost always just adopt it”. The yardstick has always been the ratio of cost to pain, never “the fewer tools the better”.

Heavy weapons are a means, not an achievement. Complexity should be spent where it cuts — this is the ruler I run over almost every technology choice I make.