{"id":1395,"date":"2026-11-17T00:30:00","date_gmt":"2026-11-17T00:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1395"},"modified":"2026-11-20T04:30:00","modified_gmt":"2026-11-20T04:30:00","slug":"aiops-diagnosis-fixes","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/aiops-diagnosis-fixes\/","title":{"rendered":"Six AIOps Symptoms Diagnosed: Causes and Fixes for Noisy Alerting"},"content":{"rendered":"<p>When an alerting setup goes wrong, the symptoms are rarely subtle: pages at all hours, an outage nobody was told about, a clever platform that somehow makes things worse. This piece works backwards from six symptoms to likely causes and practical fixes. The evidence comes from the documented cases and lab runs behind <a href=\"https:\/\/www.xopsschool.com\/aiops\/\">an AIOps reference page published in the Ops Atlas<\/a>, and the advice stays blunt.<\/p>\n<p>A quick framing note: every symptom below has a boring root, and every fix is within reach of a small team. None needs a new platform. That matters, because the instinct under pressure is to buy something, and purchases tend to inherit whatever mess they are connected to.<\/p>\n<h2>Symptom 1: Page storms after a single failure<\/h2>\n<p><strong>Likely cause.<\/strong> One component failed and everything that depends on it complained separately. The Atlas describes the standard picture from alerting documentation: a network partition makes hundreds of instances raise alerts about the same database.<\/p>\n<p><strong>Fix.<\/strong> Group by the labels that identify the failing thing, and add an inhibition rule so dependants stay quiet while the primary alert is open. Before relying on it, confirm that every alert has the labels the rule matches on. Our guide to <a href=\"https:\/\/cotocus.org\/blog\/mastering-incident-management-with-aiops-for-faster-resolution\/\">handling incidents faster with AIOps<\/a> goes deeper on the response side.<\/p>\n<h2>Symptom 2: A real outage nobody heard about<\/h2>\n<p><strong>Likely cause.<\/strong> Over-eager inhibition. The Atlas points to a worked runbook where a label named cluster meant two things at once, the cluster affected and the server doing the evaluating. An inhibition rule matched on it and silenced an outage that was genuine. It also notes a tool quirk: if all the labels named in the rule&#8217;s equality list are missing from both the muting and the muted alert, the rule still applies.<\/p>\n<p><strong>Fix.<\/strong> Give each meaning its own label, in this case one for the affected cluster and another for the evaluating server. Test every inhibition rule with missing labels. Start the outage runbook with a query that shows which alerts are currently being held back, so a responder knows what the quiet dashboard may be concealing.<\/p>\n<h2>Symptom 3: Alerts fire on holidays and quiet days<\/h2>\n<p><strong>Likely cause.<\/strong> A seasonal detector that does not know about the calendar. In the Atlas lab, an anomaly baseline found every one of four planted faults and also fired 34 spurious alerts, of which 32 fell on a public holiday missing from its calendar.<\/p>\n<p><strong>Fix.<\/strong> Add the missing calendar. One new calendar line was enough to remove all 32 of those. Keep a shared list of holidays, launches and maintenance windows, and make updating it part of your release checklist.<\/p>\n<h2>Symptom 4: Numbers look great but incidents feel worse<\/h2>\n<p><strong>Likely cause.<\/strong> Grouping tuned for compression. A plain time window of five minutes shrank the Atlas lab&#8217;s alert stream 5.5-fold, but it merged unrelated incidents, so one dashboard line now hides two problems.<\/p>\n<p><strong>Fix.<\/strong> Stop reporting reduction as the headline. Collect a set of past incidents with the correct alert membership, then score your grouping on pairwise precision and recall, with F1 as the summary. Topology matters here: the Atlas lab saw related-pair recall drop to 0.53 from 0.93 once a single connection vanished from the map, so keep the map fresh. For planning an improvement programme, see our <a href=\"https:\/\/cotocus.org\/blog\/aiops-roadmap-a-practical-guide-for-devops-sre-and-operations-teams\/\">practical AIOps roadmap for operations teams<\/a>.<\/p>\n<h2>Symptom 5: The learned grouping never improves<\/h2>\n<p><strong>Likely cause.<\/strong> The learning signal is thin. The Atlas notes that one popular paging service learned its grouping only from manual merges made in its web application, and not from merges done through its programming interface. If your team does most of its merging by script, the model sees almost nothing. More generally, the Atlas suggests having roughly one hundred labelled incidents before trialling learned grouping at all.<\/p>\n<p><strong>Fix.<\/strong> Check how feedback reaches the system, and build the labelled set first. Trial any learned component quietly first, scoring answers without acting on them, then expose it to one or two services before letting it run everywhere. Agree rollback triggers in advance.<\/p>\n<h2>Symptom 6: Automation made the incident bigger<\/h2>\n<p><strong>Likely cause.<\/strong> Unbounded actions. The Atlas retells an episode from a big internet company in which a decommissioning script erased disks that served content. The follow-up was sanity checks, throttling, and rewriting the workflow so a repeat run did no extra harm.<\/p>\n<p><strong>Fix.<\/strong> Copy the guarded pattern in the Atlas lab&#8217;s log-rotation example. Only approved actions run, each can be undone, a host gets at most three runs in a day and the whole fleet at most five in an hour, the result is checked, and failure pages a person. Run it as the owner of the logs. The Ansible documentation itself warns that privilege escalation cannot be restricted to particular commands, so root is the wrong default.<\/p>\n<h2>Bonus symptom: Someone else can silence your alerts<\/h2>\n<p><strong>Likely cause.<\/strong> Open control surfaces. The Atlas observes that the alert manager&#8217;s API answers with a wildcard cross-origin header, so a single request from a web page could silence an outage. It adds that ingest webhooks need authentication, the remediation runner holds production credentials, and the rule-driven automation source in one popular collection listens over unauthenticated HTTP and belongs on a loopback address only.<\/p>\n<p><strong>Fix.<\/strong> Put the control plane behind authentication, keep automation sources local, and review who and what can reach them. Treat alert text as untrusted input too, particularly before any language model reads it.<\/p>\n<h2>How to prioritise the fixes<\/h2>\n<p>Write each symptom on a card with its suspected cause, then arrange the cards by risk before you touch a single rule.<\/p>\n<p>You will probably recognise more than one of these symptoms, and fixing all of them at once is unrealistic. Rank them by blast radius. A silenced real outage outranks a noisy page storm, and unsafe automation outranks both, because the damage compounds without anyone noticing. Fix in that order: safety first, then missed signals, then noise, then measurement polish.<\/p>\n<p>Whichever you pick, assign a single owner and a date. A diagnosis without an owner is a complaint, and a fix without a date is a hope. Check again two weeks later and compare the incident log before and after.<\/p>\n<h2>Observer problems and the two-vantage rule<\/h2>\n<p>One last diagnosis deserves its own section. If the server that watches a cluster loses its own connectivity, it may report that the whole cluster is unreachable when only the observer is cut off. Its suggested guard is to raise the unreachable alert only when two independent viewpoints both see the failure. That is an inexpensive rule against a very noisy kind of false alarm.<\/p>\n<h2>A routine to stop the symptoms returning<\/h2>\n<p>Most of the fixes above are one-off; the habit that keeps them in place is a rhythm. The Atlas suggests a short noise review every week, a scorecard each month and a platform review each quarter, with a first maturity gate set at 95 percent label compliance. Add one line to your incident template asking whether any alert was silenced wrongly or any real alert merged away. If the answer is yes, a symptom from this list has just returned, and you know where to look.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>From page storms to silenced outages, six common alerting symptoms traced to their causes, with fixes drawn from documented AIOps lab results and cases.<\/p>\n","protected":false},"author":4,"featured_media":1396,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[130],"tags":[],"class_list":["post-1395","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-it-operations"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1395","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1395"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1395\/revisions"}],"predecessor-version":[{"id":1518,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1395\/revisions\/1518"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1396"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1395"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1395"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1395"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}