Why Incident Time Averages Mislead, and What to Track Instead

Why Incident Time Averages Mislead, and What to Track Instead

The symptom is familiar. A quarterly operations review opens with a slide showing mean time to resolve, the number is up by a third, and an hour of discussion follows about what the team did wrong. Then someone checks the data and finds that one ugly overnight outage explains almost all of it. This post diagnoses why averages mislead in IncidentOps, the craft of organising a response to live service disruptions, and proposes a measurement set that holds up when you have only a dozen incidents a quarter. The evidence comes from the Ops Atlas, which summarises published analyses and runs its own small simulations.

Symptom one: the same acronym means four things

Diagnosis begins with vocabulary. The Atlas notes that the familiar “MTTR” abbreviation is used for at least four different intervals: repair, recovery, respond and resolve. Each starts and stops the clock at a different moment. A team that reports “MTTR is 90 minutes” may be measuring the time until a rollback finished, while another team in the same company is measuring the time until the incident ticket closed days later.

Treatment: never publish the bare abbreviation. State the phase (detect, acknowledge, mitigate or restore), state the population (which severities, which services) and state the period. A footnote of ten words removes most of the arguments.

Symptom two: the number jumps for no reason

The second problem is statistical. Incident durations are positively skewed, meaning many short ones and a few very long ones. In that kind of data the mean is dominated by the tail. The Atlas describes a lab example with eleven incidents in which the average time to mitigate was 78.4 minutes while the median was 32. A single top-severity event moved the mean by 45.5 minutes on its own. When the authors simulated quarters with no real change in performance, the average moved by at least twenty percent in 71 percent of them.

That is a spectacular amount of false drama. It means a manager who reacts to every wobble will chase ghosts, reward luck and punish bad draws.

Treatment: pair every average with a median and the full list of durations. With eleven points you can simply show the dots.

Symptom three: improvements that cannot be seen

The reverse problem is just as damaging. The Atlas leans on a published analysis by a Google reliability researcher, who built simulations from data at three companies. When incident durations were truly shortened by ten percent, the measured average still came out lower than the baseline only some of the time. In the three datasets, the difference fell below zero in 38, 40 and 20 percent of runs respectively. And with no change at all, there was a 19 percent chance of seeing a spurious half-hour improvement. The author’s conclusion, quoted in the Atlas, is that the default stance toward such metrics for trend analysis should be rejection.

In plain terms, you could do real, successful work and see no improvement in the chart, or do nothing and appear to improve. Neither outcome tells you anything about the team.

Differential diagnosis: what is actually worth tracking?

If averages fail, the Atlas suggests a replacement panel built from four groups of measurements.

  1. Phase distributions. Time to detect, to acknowledge, to mitigate and to restore, each shown as a spread with percentiles, not as a single value.
  2. Impact on service targets. How many minutes of error budget each incident consumed, which links the response directly to what customers feel.
  3. Pager load. A trailing 21-day average of pages per person, a leading indicator of burnout and of poor alert quality.
  4. Action closure. The share of agreed follow-ups finished by their due date, which tells you whether reviews change anything.

Each of these is harder to game than a mean and more tied to decisions. If detection time is long, invest in alerting. If mitigation is slow, rehearse rollbacks. If closure rate is poor, shrink the action list.

Complication: the cost side of the ledger

Measurement also has a workload dimension. The Atlas quotes a rough Google figure of six hours of average effort per incident. Multiplied across a quarter, that is about seventy-two person-hours for a modest volume. Count that against any proposed improvement, because a metrics programme that adds ten hours of manual spreadsheet work per incident is itself a source of drag. When you are looking at how automation might shorten this loop, the discussion in how AIOps helps teams resolve incidents faster offers a useful contrast.

A worked example you can try tonight

Take ten of your own recent incidents and write each mitigation time on a card. Lay them out in order from shortest to longest. Now remove the longest card and recompute the average. If the number drops sharply, you have just seen the tail effect with your own eyes. Next, ask what the longest incident had in common with the others. Was it detected late, escalated late or mitigated slowly once everyone knew? Every answer leads to its own remedy, and a single mean cannot tell those remedies apart.

This ten-minute exercise also makes a persuasive demo for sceptical managers. Nobody needs a statistics course to understand a row of cards with one very long outlier at the end.

Prescription: a monthly routine for small samples

Here is a routine that works with small numbers. First, record the four phase timestamps for every incident, even minor ones, in a shared sheet. Second, once a month, plot the individual durations on a simple strip chart, one dot per incident, coloured by severity. Third, annotate the three longest dots with a one-line reason each. Fourth, check pager load and action closure. Fifth, write one sentence about what to try next and revisit it after the next batch of incidents.

You will notice that no step compares this month’s average with last month’s. That is intentional. With so few observations, a trend line is a story you tell yourself. A set of annotated dots is a record you can defend.

Follow-up: what to avoid

Do not tie bonuses or performance reviews to a single duration figure. Do not celebrate a falling average without checking the tail. Do not drop incidents from the data because they are “outliers”: the outliers are the lesson. And do not let the arithmetic crowd out the human story, since the review is still the place where you ask what conditions made the failure possible and stop short of blaming a person. Our broader overview of intelligent operations for IT teams covers where measurement fits into the larger toolkit.

A note on communicating the change

Switching metrics can unsettle people who are used to a familiar headline figure. Introduce the new panel alongside the old one for a quarter so that nobody feels something has been hidden. Explain, with the card exercise, why the old number swings. Then retire it quietly once the new view has earned trust. Leaders generally welcome the change when they see that the new panel points to specific actions, such as faster detection or lighter pager load, rather than to a vague sense that things are better or worse.

When to re-test

After a quarter of keeping the new panel, compare how many review conversations were spent arguing about whether a number was real. If the answer is fewer, the treatment is working. Detailed sources, formulas and caveats sit in the measurement section of the Atlas IncidentOps entry, which is worth reading before you redesign any dashboard. Start with one incident from last month and plot its phases, and you will see the point immediately.