{"id":1176,"date":"2026-11-04T23:30:00","date_gmt":"2026-11-04T23:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1176"},"modified":"2026-11-04T23:30:00","modified_gmt":"2026-11-04T23:30:00","slug":"acceptable-failure","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/acceptable-failure\/","title":{"rendered":"Acceptable Failure, Quantified: A Plain-Language Explainer"},"content":{"rendered":"<p>Every system that serves people fails sometimes. That is not pessimism; it is the one empirical constant in operations. The interesting question \u2014 the one this explainer unpacks \u2014 is what happens when an organisation stops treating that fact as a secret and starts treating it as a quantity. The practice of defining acceptable failure, assigning it a number, and managing work against the remainder has become the backbone of modern reliability engineering, and the logic behind it is simple enough to explain without a single diagram. Here is the whole idea, from first principles, in plain language.<\/p>\n<h2>The core move: acceptable failure becomes a quantity<\/h2>\n<p>Start with an observation anyone who has run software can verify. When a team has no stated tolerance for failure, it does not get zero failures \u2014 it gets undefined ones. Each incident becomes a fresh negotiation about whether it was &#8220;too much,&#8221; conducted under pressure, by whoever shouts most convincingly. The reliability discipline replaces that negotiation with a decision made in advance. The team declares a target: this service will work, as the user sees it, some stated fraction of the time \u2014 perhaps 99.5% across a month. Subtract that from 100% and you get the permitted shortfall: the half percent of attempts that may fail before the team is in breach of its own word.<\/p>\n<p>That shortfall is the budget. The word is chosen deliberately, because every property of a financial budget transfers. It is finite. It is spent by identifiable events \u2014 a bad release, a dependency wobble, an operator mistake. It can be conserved when reserves run low and spent boldly when they are full. And its balance at any moment is a fact, not an opinion, which is what makes it so effective at ending arguments.<\/p>\n<h2>Why a target below perfection is the rational choice<\/h2>\n<p>Two pieces of evidence settle this, and both are checkable against your own experience. The first is the cost curve. Moving a service from unreliable to solid involves fixing real defects \u2014 valuable work at reasonable cost. Moving it from solid to near-perfect requires redundancy for events that almost never happen: duplicate infrastructure, elaborate failover, releases slowed to a crawl. The price of every further nine rises steeply while its visible benefit shrinks toward zero, because past some threshold users cannot distinguish rare failure from none. Spending beyond that threshold is, in the strict sense, waste \u2014 resources burned to purchase a difference nobody can perceive.<\/p>\n<p>The second is behavioural. A target of perfection cannot be met, and targets that cannot be met do not discipline behaviour \u2014 they distort it. Teams held to impossible standards under-report incidents, avoid risky-but-valuable changes, and burn their most conscientious people. A meetable target does the opposite: it makes honesty cheap, since admitting a failure merely debits a budget everyone agreed to, and it makes courage affordable, since a well-stocked budget is standing permission to attempt the ambitious migration. The number is not a lowered bar. It is the first bar whose height anyone actually measured.<\/p>\n<h2>The mechanics, end to end<\/h2>\n<p>In practice the system has four moving parts, and naming them removes most of the mystery.<\/p>\n<ol>\n<li><strong>A user-facing measure.<\/strong> The thing counted must be something a user would recognise as success or failure: a request answered correctly, a page delivered in time, a job completed. Internal machine health matters for diagnosis but does not denominate the budget.<\/li>\n<li><strong>A window.<\/strong> The target applies over a stated period \u2014 commonly a month \u2014 after which the budget refills. The window converts reliability from an eternal moral quality into a periodic, renewable account.<\/li>\n<li><strong>A ledger.<\/strong> Someone, or something, tracks consumption continuously. The evidence from adopting teams is unambiguous on one point: an untracked budget changes nothing. The tracking can be humble \u2014 scheduled checks and a spreadsheet beat an elaborate system nobody trusts.<\/li>\n<li><strong>A consequence.<\/strong> Agreed before it is ever needed: when the budget empties, risk-adding work pauses and the freed effort goes to stabilisation until the measure recovers. This clause is where the practice lives or dies. A budget with no consequence is a chart; a budget with one is a governor on the whole delivery machine.<\/li>\n<\/ol>\n<p>A worked miniature makes the arithmetic concrete. Suppose a booking service commits to 99.5% successful requests over a calendar month, and it handles a few hundred thousand requests in that time. The permitted shortfall is one request in two hundred. A botched Tuesday deploy that fails requests for twenty minutes might consume a third of the month&#8217;s allowance; the ledger says so the same day. With two-thirds remaining and three weeks to go, the team can proceed normally but defers the riskiest planned migration to next month \u2014 a decision made in a minute, by arithmetic, that would once have consumed an hour of meeting and a week of resentment.<\/p>\n<p>Notice what the mechanics do to incident response. When failures are withdrawals from a known account, the response to one is proportionate by construction \u2014 a small debit gets a fix and a note, a large one triggers the agreed slowdown. Teams that pair the ledger with fast diagnosis get a compounding advantage, which is why budget practice sits so comfortably alongside <a href=\"https:\/\/cotocus.org\/blog\/mastering-incident-management-with-aiops-for-faster-resolution\/\">approaches that shorten incident resolution with machine assistance<\/a>: the faster an incident is understood and closed, the less budget it consumes, and the more freedom the team retains for deliberate risk.<\/p>\n<h2>Objections, examined<\/h2>\n<p><em>&#8220;This licenses sloppiness.&#8221;<\/em> The record says otherwise. Sloppiness thrives in ambiguity, and the budget removes ambiguity. A team that blows through its allowance monthly is exposed by the ledger far more mercilessly than by any slogan about excellence. What the budget licenses is risk \u2014 a different thing \u2014 and it licenses it in measured doses.<\/p>\n<p><em>&#8220;Our business demands 100%.&#8221;<\/em> Then your business is describing its wish, not its system. The honest versions of this objection are really statements about criticality, and the framework accommodates them: critical paths get strict targets and generous engineering investment; peripheral features get looser ones. What no business can buy, at any price, is zero failure \u2014 and contracts pretending otherwise simply relocate the failure into legal language.<\/p>\n<p><em>&#8220;We are too small for this.&#8221;<\/em> The method scales down further than almost any other operations practice, because its minimum viable form is a sentence and a tally. The analytical habit matters more than the tooling, and that habit is teachable: the progression is laid out in formal training \u2014 <a href=\"https:\/\/sreschool.com\/\">SRESchool structures the subject across certification levels from newcomer to reliability manager<\/a> \u2014 precisely because the reasoning, not the software, is the substance. Operations teams planning broader capability building can treat budget literacy the way they treat any roadmap item; it slots naturally beside the staged adoption described in <a href=\"https:\/\/cotocus.org\/blog\/aiops-roadmap-a-practical-guide-for-devops-sre-and-operations-teams\/\">a staged guide for bringing intelligent operations into a team<\/a>.<\/p>\n<h2>What to take away<\/h2>\n<p>The entire discipline compresses into three sentences. Decide, while calm, how much failure each service may have, and write the number where everyone can see it. Count failures against that number continuously, using a measure a user would recognise. Let the remaining balance \u2014 not mood, seniority or volume \u2014 govern how boldly the team changes the system. Everything else in reliability practice is elaboration on those sentences. The evidence from teams that adopt them is remarkably consistent: fewer arguments, faster releases, calmer incidents, and an end to the strange organisational theatre in which everyone promises perfection while privately budgeting for its absence. The budget was always there, implicit in every system that ever failed and kept its users anyway. Writing it down just made it honest \u2014 and made the trade-offs it governs, for the first time, something a whole organisation could reason about together.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>What happens when a team stops hiding failure and starts counting it? The logic of reliability budgets, from first principles, without jargon or diagrams.<\/p>\n","protected":false},"author":4,"featured_media":1177,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[129],"tags":[],"class_list":["post-1176","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1176"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1176\/revisions"}],"predecessor-version":[{"id":1180,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1176\/revisions\/1180"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1177"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1176"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}