{"id":1365,"date":"2026-11-16T10:30:00","date_gmt":"2026-11-16T10:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1365"},"modified":"2026-11-18T23:30:00","modified_gmt":"2026-11-18T23:30:00","slug":"weather-stations-observability-plain-language","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/weather-stations-observability-plain-language\/","title":{"rendered":"ObservabilityOps Explained With a Town and Its Weather Stations"},"content":{"rendered":"<p>Think of a small town that runs its own weather stations. There is one on the roof of the school, one by the river and one at the grain store. Each sends readings to a screen in the town hall, and the mayor uses that screen to decide when to warn farmers about frost. One cold morning the river station goes quiet. The screen still shows a calm line, because the last reading is still sitting there. Nobody notices for a day. This little story is the heart of ObservabilityOps: the everyday work of keeping the instruments, the wires and the screen trustworthy so the people reading them are not misled. Let us unpack it in everyday language.<\/p>\n<h2>Two jobs that sound alike<\/h2>\n<p>It is easy to blur two different jobs. The first is reading the screen and deciding what to do about the frost. The second is making sure the stations work, the messages arrive, the screen is current and the electricity bill for it all is sensible. The <a href=\"https:\/\/www.xopsschool.com\/observabilityops\/\">Ops Atlas explanation of ObservabilityOps<\/a> is about the second job. It describes the subject as operating the observability system itself, which is different from using that system to run a service.<\/p>\n<p>The distinction matters because the first job already has plenty of names and owners: reliability engineers, on-call responders, analysts. The second job often belongs to nobody in particular, so it falls through gaps. Naming it is the first step to giving it a home.<\/p>\n<h2>What &#8220;observability&#8221; means, without the jargon<\/h2>\n<p>OpenTelemetry, the open standard most teams now use, describes observability as the ability to understand the internal state of a system by examining its outputs. In the town, that means being able to work out what is going on in the valley from the readings alone. The Atlas adds a contrast taken from DORA research: ordinary monitoring checks things you decided on in advance, such as whether the temperature has dropped below zero, while observability keeps enough detail to investigate things you did not predict.<\/p>\n<p>The idea has a long history. It borrows a concept from control theory dating to 1960, and over the decades engineers built habits on top of it. A 2012 method looks at utilisation, saturation and errors for each resource. A 2015 method watches rate, errors and duration for each service. In 2016 a famous reliability handbook popularised four so-called golden signals. All of these are recipes for deciding which weather stations to build first.<\/p>\n<h2>The three kinds of reading<\/h2>\n<p>Most systems produce three broad kinds of output. <strong>Metrics<\/strong> are numbers over time, like the temperature by the river every minute. <strong>Logs<\/strong> are written notes, like the grain-store keeper&#8217;s diary of who came and went. <strong>Traces<\/strong> follow a single request through the town, from the farmer&#8217;s order at the shop to the delivery at the barn, showing where it slowed down. A newer fourth kind, profiles, shows where a program spends its effort and has been in public alpha in the OpenTelemetry project since March 2026.<\/p>\n<p>The trio is often nicknamed the three pillars, though a cloud native community paper from 2023 favours the term primary signals, since pillars hint at three separate buildings when the payoff lies in linking the readings. A log line that carries the trace identifier can be matched with the journey that produced it.<\/p>\n<h2>Why the station network needs its own manager<\/h2>\n<p>Left alone, the network grows. Each farmer adds a station, and every station adds new labels to its readings: which field, which crop, which shift, which dog walker. The Atlas explains that in metric systems every unique combination of labels becomes a brand-new series to store. A label with unlimited values, such as a person&#8217;s name, can multiply storage far beyond what anyone planned. It recommends keeping the number of values per label small, under about ten, and investigating once it passes a hundred.<\/p>\n<p>The same story applies to cost, to noise and to trust. The town hall pays for storing every reading, and if the screen flashes red for trivial reasons, people stop looking at it. Someone has to say which stations stay, how long readings are kept, and what counts as a warning worth waking the mayor for.<\/p>\n<h2>The duties of the network manager<\/h2>\n<p>The Atlas lists the responsibilities fairly plainly: agree how stations are built and named, run the pipes that carry messages, look after the places where readings are stored, set limits on labels and sampling and retention, design the dashboards and the warnings, watch the watchers, and keep an eye on cost. It deliberately excludes a few things that belong elsewhere, such as setting reliability targets, assigning incident roles, using machine learning to correlate events, security readings and the cloud bill.<\/p>\n<p>That last exclusion is worth a pause, because machine learning is often sold as the cure for alert overload. If you want to see how automated correlation fits into incident handling, this piece on <a href=\"https:\/\/cotocus.org\/blog\/mastering-incident-management-with-aiops-for-faster-resolution\/\">handling incidents faster with AIOps<\/a> covers that neighbouring subject, and the practical <a href=\"https:\/\/cotocus.org\/blog\/aiops-roadmap-a-practical-guide-for-devops-sre-and-operations-teams\/\">AIOps roadmap for operations teams<\/a> shows how teams sequence it. Both work far better when the underlying readings are clean, which is exactly the network manager&#8217;s job.<\/p>\n<h2>Three small habits that make a big difference<\/h2>\n<p><strong>A heartbeat.<\/strong> Back to the town hall. Suppose a tiny device on the roof sends a signal every minute that says, I am alive. If the screen stops receiving it, a bell rings in the fire station, which is outside the town hall&#8217;s own wiring. The Atlas calls this a dead-man&#8217;s switch or watchdog, and it would rather see one full-path test than a pile of small component checks.<\/p>\n<p><strong>A named owner.<\/strong> Somebody&#8217;s name goes on the rota for the network, and everybody knows it. The Atlas suggests that a smaller organisation with one or two teams needs this mostly when it runs its own storage or is regulated; larger ones may need a platform team or a compact standards group.<\/p>\n<p><strong>A monthly price tag per unit.<\/strong> Divide the monthly bill for the screen and stations by something people understand, such as the number of requests served or forecasts issued. Watching that single figure over time shows whether the network is getting more efficient or merely bigger.<\/p>\n<h2>A few everyday words, translated<\/h2>\n<ul>\n<li><strong>Cardinality:<\/strong> how many different values a label can take. Ten crops is fine; every farmer&#8217;s phone number is not.<\/li>\n<li><strong>Sampling:<\/strong> keeping only some of the journeys you could record, such as every failed delivery but just a handful of smooth ones.<\/li>\n<li><strong>Retention:<\/strong> how long the town hall keeps old readings before throwing them away or filing them in the cellar.<\/li>\n<li><strong>Scrape:<\/strong> the moment the screen goes round collecting the latest readings from every station.<\/li>\n<li><strong>Showback:<\/strong> a monthly note telling each farmer how much of the network&#8217;s bill their stations account for, without actually charging them.<\/li>\n<\/ul>\n<p>Once those five terms click, most conversations about telemetry stop sounding like a foreign language. They are ordinary questions about how much to measure, how long to remember and who should pay.<\/p>\n<h2>Where people go wrong<\/h2>\n<p>Several traps recur. Some teams keep every reading forever on the theory that it might help later, and discover that storage costs rise faster than usefulness. Others assume that installing the instrumentation kit finishes the job, when traces can still break at queues or at proxies that drop headers. A few believe that running open source tools themselves is free, forgetting upgrades and the person on call. And some hand all alerts to a single central group, even though DORA research argues monitoring should not sit with only one team.<\/p>\n<h2>Back in the town hall<\/h2>\n<p>The mayor, in our story, did not need a degree in meteorology. She needed three things: a heartbeat that rings when a station dies, a clipboard that says who looks after the stations, and an annual note of what the whole screen costs. Once those were in place, the next frost warning went out on time. That is the promise of the discipline in plain language: fewer surprises from the tools you rely on to avoid surprises.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A plain-language explainer of why the system that watches your software needs its own care, told through a small town&#8217;s network of weather stations.<\/p>\n","protected":false},"author":4,"featured_media":1366,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[130],"tags":[],"class_list":["post-1365","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-it-operations"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1365","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1365"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1365\/revisions"}],"predecessor-version":[{"id":1496,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1365\/revisions\/1496"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1366"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1365"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1365"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1365"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}