NoOps on a Bad Night: A Problem-First Diagnosis

NoOps on a Bad Night: A Problem-First Diagnosis

The ticket arrived at 2:14 a.m.: checkout service down, revenue leaking, nobody paged. That last detail was the strange one. The company had celebrated going NoOps fourteen months earlier — ops team dissolved, everyone “owning quality end to end,” a slide deck to prove it. Yet here was a production failure that three different teams each assumed somebody else was watching. Engineers joined the bridge one by one, each asking a variation of the same question: whose service is this? The answer, it turned out, was nobody’s. By dawn the incident was fixed, the retrospective room was full, and the whiteboard carried a question someone finally wrote out loud: when we removed the operations team, what exactly did we think we removed?

If your organisation has scars like this, the diagnosis below will feel familiar. It is written the way a good incident review should be: symptoms first, causes second, treatment last. And it uses the definition of NoOps from the NoOps write-up over at the Ops Atlas — an aspiration of automating and buying operations until no dedicated team is needed — because the gap between that sentence and what most companies implement is precisely where the 2 a.m. failures live.

The Symptoms We Keep Seeing

Teams that announced the end of operations tend to present a recognizable cluster of complaints within a year:

  • Incidents with no owner — pages that route to whoever is least lucky, or to no one at all.
  • Alert fatigue that peaked exactly when headcount for triage went to zero.
  • A restore attempt that failed, revealing the managed database was never backed up the way everyone assumed.
  • An outage caused by a provider, met with genuine surprise that “the cloud” could go down at all.
  • Leadership metrics that show automation rising while nothing measures what the remaining duties cost.

Each symptom feels like a separate stroke of bad luck. It is not. They are downstream of three specific misdiagnoses about what operations was — and each misdiagnosis has a matching correction, which is what the rest of this piece supplies.

The Diagnosis: Where the NoOps Plan Actually Breaks

Misdiagnosis one: operations was a team, so deleting the team deletes the work

Operations was never primarily headcount. It was a set of duties — some performed by people, some by providers, some by pipelines. Dissolving the team redistributes the duties to the cloud provider, to any surviving platform staff, into automation code, onto product engineers, and into vendor management. If the redistribution is not designed, the duties design themselves, and they design themselves badly: unowned, unmeasured, invisible until failure. The treatment is an inventory. Every recurring duty gets written down and given a name, and the share with no name — track it as a number, target zero — is your leading indicator of the next bad night. In the checkout incident, the root cause turned out to be an expired certificate for a component whose renewal had been “somebody in ops” for five years, and nobody in particular since.

Misdiagnosis two: the provider’s availability was mistaken for ours

The 2:14 a.m. failure traced back to an upstream the team had never treated as theirs: an identity service outage that cascaded into checkout. Here is the arithmetic they had skipped. A service is up only when its whole chain is up, so its availability is the combination of every critical dependency — never the strongest figure among them. The provider’s service-level agreement describes one link; customers experience the product of them all. Treatment: derive the public uptime promise from the weakest chain you actually run, list the dependencies in writing, and rehearse what happens when each one degrades. SRE teams formalise this by treating the provider’s SLA as an input, never as their SLO — the distinction sounds academic until the first cascade makes it commercial. The checkout team, to their credit, rebuilt their availability target within the month and put every dependency on a one-page diagram that now hangs in the team area like a family portrait.

Misdiagnosis three: “managed” was heard as “backed up”

During the same quarter, a quieter failure surfaced: a restore test failed. Nothing was corrupted; there was simply nothing recent to restore. The platform was managed, and the team had read that as a recovery guarantee. It is not — the provider handles replication inside their service, while the backup strategy, the retention choices, and the proof that restores work remain with the customer. AWS’s own guidance says as much. Treatment: scheduled restore drills, on the calendar, with the results circulated. A backup nobody has restored from is a rumour.

The Treatment Plan for a NoOps Ambition

None of this means abandoning automation — the gains are real, and the checkout team in this story automated its way out of several classes of failure for good. It means sequencing the ambition correctly:

  1. Count before cutting. Inventory every recurring operational duty with hours attached, before any team is restructured or any tool is bought.
  2. Name every owner. Provider, platform group, product team, script, or vendor manager — each duty gets a human answerable for it. Zero unnamed duties, enforced as a rule rather than an aspiration.
  3. Promise the chain’s uptime, not the best link’s. Rebuild customer-facing availability targets from the dependency list, then build slack for the difference.
  4. Drill the recovery. Quarterly, timed, circulated. Untested recovery is a story, not a capability.
  5. Let machines assist, with signatures. Modern tooling shortens diagnosis dramatically — the AIOps approach to incident management and faster resolution is worth adopting — but keep a human approving production changes, because accountability cannot be delegated to software without someone inheriting it.
  6. Watch the humans carrying the pager. Absorbing on-call into product roles only works while alert load stays humane; the moment pages climb past what a responder can absorb in a shift, rotate more people in or buy back the automation that would have caught the problem first.

Teams that want the deeper playbook behind items four and five will find the reasoning in a complete guide to AIOps for operations teams: instrument first, automate the correlation, keep the judgement human where the blast radius is wide.

The Verdict After the Diagnosis

The whiteboard question from the opening story deserves its answer. When that company removed its operations team, it removed the people while keeping every duty — and duties without stewards do not idle quietly; they fail loudly, at the least convenient hour, in front of customers. Operations is the one discipline that cannot be skipped, only scheduled: pay for it in planned registers, named owners, and rehearsed failures, or pay for it in unplanned outages at multiplied rates. Nobody chooses the second option explicitly. Companies drift into it one well-intentioned announcement at a time, which is why the diagnosis matters more than the slogan it corrects.

NoOps, honestly pursued, is a reduction of operational toil under full accounting — and the teams that get closest to it are the ones most expert in what they are automating. That is the quiet irony of the whole idea: the road to needing no operators runs directly through deep respect for operations. For teams rebuilding their practices from first principles, the complete guide to DevOps principles and workflows remains the sturdier starting point — collaboration first, automation second, declarations last.