A DetectionOps Maturity Audit: Five Sections, One Score

A DetectionOps Maturity Audit: Five Sections, One Score

An audit is only useful if it produces a score you can repeat. This checklist lets an operations or security lead assess how mature their detection management is, using the DetectionOps maturity ladder described by the Ops Atlas as the spine. Answer each item yes or no, count the yeses per section, and compare against the thresholds at the end. Run it twice, a quarter apart, and you will have a trend instead of an opinion.

How to use this audit

Pick a sample, not the entire library: twenty rules chosen at random, plus the ten noisiest. Record the evidence for each yes, such as a commit link, a test name or a dashboard figure. An answer without evidence counts as a no. The Ops Atlas coverage of DetectionOps defines five levels from ad hoc through versioned, tested and validated to a managed portfolio, and the sections below follow those levels in order.

Section A: Versioned (level one)

  • Every rule in the sample exists as a file in a repository.
  • Production configuration matches the repository, with no console-only edits.
  • Every change records who made it, who approved it and why.
  • A lint or syntax check runs automatically on every change.
  • The toolchain versions are pinned and listed somewhere visible.

The Atlas describes the target here as zero drift. If production and repository disagree even once in the sample, record the cause. Console hot-fixes are the usual culprit, because the next deployment silently overwrites them.

Section B: Tested (level two)

  • At least ninety percent of the sampled rules have tests; this is the Atlas threshold for level two.
  • Each tested rule has both a positive and a negative case.
  • Tests derive from the written requirement, not from the rule’s own logic.
  • Accepted blind spots appear as “known gap” tests with an owner and an expiry.
  • Analyst dispositions (true positive, benign, false positive) are recorded in a structured form.

Scoring tip: if only some tests come from the requirement, count the item as a no. Tests that pass by construction create false confidence, which is worse than having none.

Section C: Validated (level three)

  • You maintain a prioritised list of behaviours that matter to your organisation.
  • Each priority has been reproduced in an emulation exercise within the last ninety days.
  • Coverage is reported per implementation, with a date and the share actually detected.
  • No coverage claim relies solely on technique tags.
  • New rules run silently for about a week before they are allowed to page staff.

Be strict on the third item. In the Atlas example, a tag heat map said one hundred percent while emulation showed eighteen of thirty priorities, sixty percent, actually detected. The Atlas’s guidance, echoing a public research programme, is that a label from the framework proves nothing about what the system actually catches.

Section D: Managed portfolio (level four)

  • Rule health, meaning rules that are tested, current and not failing, is at or above ninety-five percent.
  • Filters carry evidence and expiry dates, so filter debt is close to zero.
  • Change failure rate is tracked, and review happens within a defined window for emergency changes.
  • Requests from threat analysis and misses from incident handling flow into a visible backlog.
  • Retired rules are marked deprecated and removed from the active set.

Few teams will score well here, and that is fine. The Atlas cites a published industry maturity model in warning that rushing up the tiers causes burnout, and recommends reaching level three for the top threats before attempting level four across the board.

Section E: Dependencies and data

  • For each sampled rule you can name the log source it depends on and confirm it is currently flowing.
  • Required audit settings are documented, since some events exist only when a specific policy is enabled.
  • Monitoring-platform end-of-support dates are tracked; the Atlas flags 31 March 2027 for one cloud monitoring portal.
  • Data gaps found during rule work are handed to whoever owns log collection.

Scoring and what to do next

Count yeses per section. Fewer than four in section A means start there and nowhere else. Four or five in A and fewer than four in B means your next investment is tests for the twenty noisiest rules. Four or five in both A and B but fewer than four in C means plan an emulation exercise with five priority techniques. The Atlas proposes the same ordering across ninety days: repository and lint checks in the first month, tests in the second, and shadow mode plus a first coverage report in the third.

Record the figures you can compute at the same time: precision for each sampled rule, test completeness, change failure rate and analyst minutes per genuine finding. The Atlas’s worked month gives reference shapes, with 70 percent test completeness and a six percent change failure rate, but your own numbers matter more than anyone else’s.

A worked scoring example

Suppose a team samples thirty rules and finds the following. Section A: four of five yeses, because the toolchain is not pinned. Section B: two of five, since only half the rules have tests and none carry known-gap entries. Section C: zero of five, as no emulation has ever been run. Section D and E are skipped for now, because there is no point auditing a portfolio level on a foundation that is not yet tested.

The score tells the team exactly what to do. First pin the toolchain, which takes an afternoon. Then write paired tests for the twenty noisiest rules, which takes a few weeks of part-time effort. Only then schedule the first emulation of five priority techniques. Resist the temptation to jump to the interesting part; an emulation run against untested rules mostly shows you what you already suspect, and costs a lot more.

Re-run the audit after ninety days using a fresh random sample, not the same rules you already improved. Fresh samples reveal whether habits have spread or whether you only polished the items everyone was looking at. A trend line of section scores over three or four quarters is also the most persuasive thing you can show a budget holder.

Where this fits with adjacent practices

An audit of this kind overlaps with incident handling: misses found during incidents should create rule requests, and our write-up on managing incidents with AIOps for quicker resolution shows how the same feedback idea works on the response side. For the wider picture of how operations teams structure automation, this broader explainer on AIOps for operations teams offers useful context on where detection rules sit relative to event correlation.

Finally, share the results. Audit findings that stay in a drawer change nothing; findings published to the team with an owner for each gap get fixed.