{"id":1187,"date":"2026-11-05T14:30:00","date_gmt":"2026-11-05T14:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1187"},"modified":"2026-11-05T14:30:00","modified_gmt":"2026-11-05T14:30:00","slug":"myths-about-evidence-based-tech-decisions","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/myths-about-evidence-based-tech-decisions\/","title":{"rendered":"Six Myths About Evidence-Based Tech Decisions, Examined"},"content":{"rendered":"<p>Ask most engineers whether they make evidence-based tech decisions and they will say yes. Watch a typical architecture review, though, and you will hear plenty of &#8220;I read that&#8221;, &#8220;everyone uses it&#8221; and &#8220;the vendor says&#8221;. The gap is not hypocrisy; it comes from a handful of persistent myths about what evidence-first work actually involves. This piece takes six of those myths apart, with an analyst&#8217;s preference for what can be shown over what merely sounds plausible.<\/p>\n<h2>Myth 1: &#8220;Evidence-based tech decisions are too slow for real operations&#8221;<\/h2>\n<p>The belief is that checking sources and testing claims belongs in academia, while production needs speed. In practice, the expensive part of a wrong decision is rarely the decision itself; it is the rollback, the incident review and the weeks of distrust that follow. Reading the release notes and running a quick reproduction usually takes an hour or two. Undoing a misguided rollout can consume days.<\/p>\n<p><strong>What holds up instead:<\/strong> evidence-first work front-loads a small, predictable cost to avoid a large, unpredictable one. Genuine emergencies still exist, but even those are typically triggered by a primary source such as an official advisory, not a viral post.<\/p>\n<h2>Myth 2: &#8220;If many people say it, it has been verified&#8221;<\/h2>\n<p>Popularity feels like peer review. It is not. A recommendation repeated across ten articles may rest on one original benchmark, run once, on hardware unlike yours. Repetition multiplies reach, not evidence.<\/p>\n<p><strong>What holds up instead:<\/strong> count independent origins, not mentions. If every article traces to the same source, you have one data point. Go and read it.<\/p>\n<h2>Myth 3: &#8220;Benchmarks published by projects are objective&#8221;<\/h2>\n<p>Project maintainers usually publish honest numbers. Honest is not the same as transferable. A benchmark answers a specific question under specific conditions, and those conditions are often chosen to show the software at its best \u2014 not out of malice, simply because that is the use case the authors care about.<\/p>\n<p><strong>What holds up instead:<\/strong> treat a published benchmark as a hypothesis about your environment. Read its methodology section, note the workload and configuration, then reproduce the part that matters to you. The same logic applies to machine-learning systems, where results on one dataset can say little about another; our primer on <a href=\"https:\/\/cotocus.org\/blog\/model-monitoring-basics-every-mlops-learner-should-know\/\">the fundamentals of watching models in production<\/a> touches on why ongoing measurement beats one-time claims.<\/p>\n<h2>Myth 4: &#8220;Experience makes evidence unnecessary&#8221;<\/h2>\n<p>Senior engineers carry valuable pattern recognition. Their instincts are often right. The myth is that instinct should therefore replace checking. Systems change: versions shift, defaults get updated, workloads grow. An intuition formed three years ago on a different stack can be confidently wrong today.<\/p>\n<p><strong>What holds up instead:<\/strong> experience is a strong source of hypotheses and a good guide for what to test first. Label it as such. &#8220;In my experience, this flag helps&#8221; followed by &#8220;let us confirm on staging&#8221; combines the best of both.<\/p>\n<h2>Myth 5: &#8220;Admitting unknowns weakens a proposal&#8221;<\/h2>\n<p>Many engineers fear that listing open questions will get their design rejected. The analysis points the other way. Reviewers who see no unknowns tend to suspect the author has not looked hard enough, and they go hunting for gaps themselves. A proposal that names its own limits reads as thorough.<\/p>\n<p><strong>What holds up instead:<\/strong> include a short &#8220;what we do not know yet&#8221; section in every significant design. It focuses review time on the real risks, and it creates a natural list of follow-up checks. For teams under growth pressure, our article on <a href=\"https:\/\/cotocus.org\/blog\/a-practical-framework-for-managing-sre-work-during-rapid-application-growth\/\">organising SRE effort while an application scales quickly<\/a> shows how to keep such follow-ups from slipping.<\/p>\n<h2>Myth 6: &#8220;This is only an engineering problem&#8221;<\/h2>\n<p>Following a statement back to where it started is a universal skill. It shows up wherever polished messaging competes with documented evidence, and health information is a vivid case. Readers exploring cosmetic treatment encounter persuasive imagery and confident promises everywhere. <a href=\"https:\/\/www.bestcosmetichospitals.com\/\">Best Cosmetic Hospitals, which offers patients no-cost guides on aesthetic procedures and treatment travel<\/a>, describes its guides as drawing on references from organisations such as the NHS and ASPS, and it makes clear its content is informational, not medical advice. The parallel to engineering is direct: credible material shows where it came from and states what it does not cover.<\/p>\n<p><strong>What holds up instead:<\/strong> once you recognise the pattern, you can borrow practices across fields. Good health resources model citation and scope statements; good engineering teams model reproduction and post-incident timelines. Each can learn from the other.<\/p>\n<h2>A short note on cost<\/h2>\n<p>Teams sometimes ask how much rigour is enough. A reasonable rule is to scale the checking with the blast radius. A change to a developer&#8217;s local tooling deserves a glance at the docs. A change to the payment path deserves the primary record, a reproduction and a written list of open questions. Matching effort to stakes keeps the practice sustainable rather than bureaucratic.<\/p>\n<h2>What the myths have in common<\/h2>\n<p>Look at the six together and a theme emerges. Each myth treats some proxy \u2014 speed, popularity, authorship, seniority, confidence, domain \u2014 as a substitute for the actual question: <em>what is the evidence, and where does it come from?<\/em> Proxies are convenient because they let us skip the reading. They are risky for the same reason.<\/p>\n<h2>A practical starting point<\/h2>\n<p>If your team wants to move beyond the myths, try three small changes for a month:<\/p>\n<ol>\n<li>Every design or change proposal names the primary source behind its key claim.<\/li>\n<li>Every proposal includes one line on what remains unknown.<\/li>\n<li>Any claimed performance improvement gets a short local reproduction before rollout.<\/li>\n<\/ol>\n<p>Measure what happens. You may find fewer reversals, shorter reviews and a noticeable change in how people talk about risk. Or you may not \u2014 in which case you will have evidence about that, too, which is rather the point.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Speed, popularity and seniority often stand in for real evidence. Six common myths about evidence-based tech decisions, examined with an analyst eye.<\/p>\n","protected":false},"author":4,"featured_media":1188,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[129],"tags":[],"class_list":["post-1187","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1187","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1187"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1187\/revisions"}],"predecessor-version":[{"id":1197,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1187\/revisions\/1197"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1188"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1187"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1187"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}