An SRE Parachuted into a 'Build Everything In-House' Company: Standing Firm in the First 90 Days

The most thrilling kind of job change is joining a company that builds almost everything itself: no off-the-shelf cloud services, and even Stack Overflow can’t help — because the infrastructure and deployment tools here are used by exactly one company in the world. Most of the “how to configure such-and-such tool” you’ve accumulated stops working; the knowledge isn’t on the internet, it’s hidden in the code, in a few people’s heads, and in past incident records.

How do you get up to speed in an environment like that? I chew through an unfamiliar system from three angles — outside-in, following one request; top-down, watching it live through observability; from zero, rebuilding it to force out every hidden dependency — and close with a rhythm. The core mindset condenses to one sentence: don’t rush to prove yourself; get the system’s map into your head first.

(As an aside: in-house companies usually don’t build monitoring from scratch too — many go straight to the open-source Grafana LGTM stack; and even if they really did build their own, LGTM’s mental model still fits, so I’ll use it as the example below.)

Move one: grab a real request and follow it all the way

If you can only do one thing, do this. The architecture diagram on the wiki is the ideal, and usually out of date; what truly forces you to understand a system is picking one real user request and tracing it by hand from entry to response, through every hop in between:

The power of this move is that it fills in three things at once: what the system looks like (components and topology), observability (where the monitoring and logs are), and failure imagination (the consequences of each hop breaking). And it’s active — you aren’t passively listening to a briefing, you’re digging yourself, and only what you’ve dug up truly takes root.

Seeing the system “alive”: the Grafana LGTM stack

While “following the request”, the question “how do I see its health?” needs a tool to answer it. Modern observability’s answer is three signals (metrics, logs, traces), and Grafana’s LGTM stack happens to pair each signal with one backend, then funnels them all into Grafana as the “pane of glass”:

For someone just starting, LGTM has a fixed routine that maps onto the four golden signals: first look at metrics to catch “where something’s off” (latency spiking, error rate rising), then flip to the logs for that time window to see “what exactly”, finally use traces to pin down “which hop, which service dragged the whole chain down”. Metrics tell you something happened, logs tell you what, traces tell you where — missing any one, you’ll be guessing blind in a 3am incident. So the first piece of infrastructure I want to understand at a new company is usually “what does our observability look like, and which pane of glass do I look at”.

The ultimate test of onboarding: can you rebuild the whole environment

The first two moves let you “see” the system, but there’s a harsher, more honest test that forces you to “truly understand” it: try to bring the whole environment up from zero, locally or in a sandbox. Reading docs, you skip the paragraphs you don’t understand without noticing; rebuilding doesn’t lie — miss any one layer of dependency and it simply won’t start, forcing you to dig out every hidden relationship:

In practice you don’t have to rebuild at Production scale; the point is to get the dependency chain working end to end: which services start first, who depends on whom, where config comes from, how data is seeded. Many companies have a docker-compose or a one-shot local-environment script — if so, run it once as written, then deliberately break one link and see how it fails; if not, writing that script for them is one of the most valuable contributions you can make during onboarding (more on this in the next section).

The rhythm of the first 90 days: understand first, act later

The mistake newcomers make most easily is rushing to “do something to prove myself” in week one — changing config, proposing refactors, criticising this and that. In a company with in-house systems this is almost guaranteed to step on a mine, because behind every odd-looking design there’s usually a bloody reason you haven’t seen yet. The rhythm I set for myself:

Of these, shadowing on-call + reading past postmortems is the stretch I consider highest return. Past postmortems amount to a condensed textbook of “how this system really breaks, and where” — worth more than any architecture briefing, because they describe real bloodshed rather than the designer’s wishful thinking. Watching “what normal looks like” in Grafana is the other half: you have to know what the system looks like when healthy before you can tell abnormal from normal when something breaks. As for trying to rebuild the environment in a sandbox, I put it in this stage rather than later because it’s best done during the honeymoon when you “still dare to ask dumb questions” — you’ll definitely get stuck rebuilding, and getting stuck is the perfect excuse to ask about, and understand, the dependency chain.

A few special plays for all-in-house companies

• Knowledge hides in three places: the code (the final truth), past postmortems (where it blows up), and that senior who “knows everything” (ask them, but don’t depend only on them — people leave). If it isn’t on the internet, dig in these three.
• Build a jargon sheet early: in-house tools have their own internal names, abbreviations, terms. Start recording in week one; two weeks later you’ll thank yourself.
• Treat “all the code is in your hands” as a bonus: with SaaS you can only guess at a black box, but every line of an in-house system is in your repo — you can actually read to the bottom, and actually change it. It’s the one place an in-house environment beats everyone else; use it well.
• Ask “why did they build it themselves” before saying “replace it”: don’t rush to shout “just swap this for open-source X”. They didn’t use the off-the-shelf thing back then usually for a reason you haven’t hit yet — understand first, evaluate second.

Reflections

A newcomer’s greatest asset is “not understanding” — don’t waste it

In the first few weeks at a company you own something you’ll never get back: a pair of eyes that “take nothing for granted”. Every place you get stuck while onboarding, every moment of “what is this? why has nobody written it down”, precisely marks a gap in the documentation — and that’s your best contribution list for month one. Every time I land in a new environment I open a “where I got stuck” note, and two months later it becomes the first batch of runbooks I fill in. Because a little later you’ll have “got used to it”, those pits become everyday things you no longer see, and that perspective is gone for good. Not understanding isn’t a weakness; it’s an asset with a shelf life.

Understand first, act later isn’t slow; it’s respect for the system’s complexity

When I was younger I badly wanted to “fix something in week one” to prove I deserved the hire, and the result was often that I changed a design I thought redundant, only to discover it was guarding an edge case I hadn’t seen. In-house systems especially — those seemingly stupid special cases are often scars from some 3am incident. So my discipline now: when I see something odd, I first ask “how did it get this way”, not “this is terrible”. It’s the same spirit as troubleshooting — trust the evidence, don’t guess on intuition; and consistent with the underlying assumption of blameless: the design in front of you isn’t there because your predecessors were stupid, but because they faced situations you haven’t yet. Understanding first, criticism second.

Whether you can rebuild it by hand is the mirror of whether you understand it

Whether I dare say “I understand” a system has exactly one standard: can I bring it up from zero myself. After reading the docs and hearing the briefing, you’ll have the illusion of “I more or less get it” — an illusion that gets punctured, layer by layer, the moment you actually rebuild: oh, this service depends on that internal API nobody mentioned? Where does this environment variable come from? Who manages this secret? Every place you get stuck is evidence that what you “thought you understood” you didn’t. I love this test, because what it forces out is exactly what an SRE most needs to master: the complete dependency chain, the startup order, the sources of config — the very same things you’ll have to rebuild by hand, at night, under pressure, in disaster recovery (DR). Rebuilding the environment during onboarding is a rehearsal of the worst case in advance; and leaving the rebuild script (docker-compose, one-shot environment) in good shape for those who come after turns a one-off pain into an asset for the whole team.

Reading postmortems is the most efficient onboarding material I’ve seen

If I could recommend one thing to a parachuted-in SRE, it’s read every postmortem from the past six months to a year. A good incident report condenses the system’s most fragile joints, its most misunderstood parts, and how people actually behave under real pressure — things the new-hire deck will never tell you, because they’re too real and too unflattering. The week I spent reading postmortems taught me more about the system than the previous month of briefings. It also deepened my conviction about the value of monitoring and a postmortem culture: an organisation willing to record honestly how it breaks has pre-written the most precious map for every future newcomer.