{"id":1361,"date":"2026-11-15T19:30:00","date_gmt":"2026-11-15T19:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1361"},"modified":"2026-11-17T14:30:00","modified_gmt":"2026-11-17T14:30:00","slug":"ticket-queue-to-error-budget","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/ticket-queue-to-error-budget\/","title":{"rendered":"A Year With an Imaginary Ops Team That Learned SRE"},"content":{"rendered":"<p>None of the people here are real, nor is their employer; this is a made-up illustration stitched together from habits common to small operations groups. Picture six engineers keeping an online course platform alive. They are friendly, overworked and proud of their quick fixes. Over one year they change how they work, and the changes line up neatly with ideas in <a href=\"https:\/\/www.xopsschool.com\/sre\/\">the SRE entry in the Ops Atlas<\/a>. Think of the tale as a way to see the concepts moving, not as evidence of what any real organisation achieved.<\/p>\n<h2>January: the week nobody could finish anything<\/h2>\n<p>It starts with a time sheet. The team lead, tired of hearing &#8220;we&#8217;re slammed,&#8221; asks everyone to log their hours for one month. Six people, eighty hours each, makes 480 hours. When the totals come in, 190 hours went to restarting stuck jobs, re-running reports, resetting accounts and clearing disk space by hand. That is roughly 40 percent of their time on chores that a script could handle.<\/p>\n<p>Nobody is shocked, but seeing it on a chart helps. The lead has just read that the discipline calls such work toil and suggests keeping it under half of the team&#8217;s time. They are inside the guideline, yet only just, and the trend is upward as the platform grows. The group agrees to automate the top three chores before starting any new feature.<\/p>\n<h2>February: arguing about numbers instead of feelings<\/h2>\n<p>The next meeting is livelier. The product manager complains that the site &#8220;keeps being down.&#8221; The engineers insist it is &#8220;mostly fine.&#8221; Both are right, depending on what you count. The lead proposes measuring one thing: the share of lesson page loads that succeed, among all genuine learner requests. Good events over valid events, taken at the edge where learners actually connect, not from inside the data centre.<\/p>\n<p>They choose a target of 99.9 percent over a rolling four weeks. The product manager initially asks for 100 percent, and the lead explains, with the Atlas on screen, that the cost curve is brutal and that users cannot perceive the difference. Everyone settles on three nines. The allowance works out to about forty minutes of lost service per 28 days. It is the first time the argument has a number in it.<\/p>\n<h2>March: the budget gets a rule<\/h2>\n<p>A number alone changes little. In March a large course launch goes badly, and the team spends nearly a quarter of the month&#8217;s allowance in a single afternoon. The lead drafts a one-page policy modelled on the published example. If the four-week allowance runs out, nothing ships except emergency and security fixes. If one incident eats over a fifth of the budget, there will be a written review with a priority action. The company&#8217;s director signs the page. The team notes with some amusement that the signature, not the maths, is what changes behaviour.<\/p>\n<p>The first review is awkward. The lead insists that blameless means nobody is hunted, but also that every action gets an owner and a date. Two of the five actions are finished within a week. A third lingers for a month and is publicly flagged until done.<\/p>\n<h2>April: a shelf of short reviews<\/h2>\n<p>Between launches, the team keeps its postmortems in one shared folder, each limited to two pages. A newcomer reads three of them in her first week and learns more about the platform than from any diagram. The lead adds a monthly ten-minute slot to check which actions are still open. It is a small ritual, yet it is the thing that stops the reviews from becoming a graveyard of good intentions. By the end of the month the open list is empty for the first time.<\/p>\n<h2>May: alerts that stop waking people<\/h2>\n<p>By spring, the on-call rota has five people and bad nights are common. The pager fires for disk warnings, slow queries, certificate notices and anything else somebody once thought important. The team replaces most of it with burn-rate rules. Fast burns over short windows page the person on call. Slow, steady leaks create tickets for daytime. They test the rules against two old incidents; the first would have paged within minutes, the second only a ticket, which the group decides is acceptable.<\/p>\n<p>The Atlas treats a shift with more than two incidents per half-day as too heavy, and counts on roughly eight people to staff a single site continuously. The team is far short of eight, so they negotiate with a neighbouring group to share cover. Perhaps unsurprisingly, resignations stop being a topic at lunch.<\/p>\n<h2>July: capacity and the fear of success<\/h2>\n<p>Enrolment peaks in the autumn term, so the team forecasts demand and load-tests the platform before the surge. They read about spare-capacity rules such as keeping enough replicas that losing two still leaves the service healthy, and that this limits safe utilisation to about a third of capacity with three replicas. They conclude their database tier has too few replicas for that safety margin and add some. A boring spreadsheet becomes the most useful document of the year.<\/p>\n<h2>September: the quiet month<\/h2>\n<p>Term starts. Traffic doubles. The budget barely moves. Instead of celebrating, the lead frowns, because the discipline warns that a nearly full budget can mean a target is too loose or that releases are being held back more than necessary. The group tightens its release cadence, ships two long-delayed improvements and watches the allowance fall to a healthier middle. Reliability, they remind themselves, exists to be spent.<\/p>\n<h2>November: what stuck<\/h2>\n<p>At year end, the lead lists what survived. Toil dropped from about forty percent to nearer a quarter after automation. The policy was invoked twice, once pausing releases for a week. Postmortems closed most of their actions. The on-call rota grew to seven. Nothing was perfect, and none of this makes the platform immune to bad days. But the arguments with the product manager now begin with a graph.<\/p>\n<h2>Reading the story with care<\/h2>\n<p>Because the team is fictional, treat its numbers as illustrations. The method, however, is real: measure from the user&#8217;s seat, choose a target below perfect, give the allowance a signed rule, alert on burn, protect sleep, and review honestly. If you are weighing where to begin, our <a href=\"https:\/\/cotocus.org\/blog\/the-complete-guide-to-devops-principles-workflows-and-tools\/\">guide to DevOps principles, workflows and tools<\/a> frames the culture that makes the steps easier, and the <a href=\"https:\/\/cotocus.org\/blog\/aiops-roadmap-a-practical-guide-for-devops-sre-and-operations-teams\/\">practical roadmap for DevOps, SRE and operations teams<\/a> shows how neighbouring practices fit around it.<\/p>\n<p>One last caution echoes the Atlas: do not simply rename an operations group and call it finished. In our story the name never changed at all. What changed was the work.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>An invented story of six engineers who swap firefighting for targets, budgets and calmer on-call nights, month by month, as an illustration of SRE.<\/p>\n","protected":false},"author":4,"featured_media":1362,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[130],"tags":[],"class_list":["post-1361","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-it-operations"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1361","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1361"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1361\/revisions"}],"predecessor-version":[{"id":1469,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1361\/revisions\/1469"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1362"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1361"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1361"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1361"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}