{"id":1369,"date":"2026-11-16T04:30:00","date_gmt":"2026-11-16T04:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1369"},"modified":"2026-11-18T09:30:00","modified_gmt":"2026-11-18T09:30:00","slug":"first-failover-drill-steps","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/first-failover-drill-steps\/","title":{"rendered":"Your First Failover Drill: A Step-by-Step DisasterRecoveryOps Guide"},"content":{"rendered":"<p>A recovery plan that has never been run is a hypothesis. This guide walks an operations team through its first timed failover drill, step by step, using the method described in the Ops Atlas entry on DisasterRecoveryOps. The goal of a first drill is not to pass. It is to produce honest numbers and a list of things to repair, and to do so without harming production.<\/p>\n<h2>Before you begin: what you need<\/h2>\n<ul>\n<li>One service, ideally mid-importance, with a defined owner.<\/li>\n<li>A staging or isolated environment where a switchover can be exercised safely. The Atlas places the first staging switchover in the second month of its 30\/60\/90 plan.<\/li>\n<li>A timekeeper who is not performing any steps.<\/li>\n<li>Two to four hours of uninterrupted calendar time.<\/li>\n<\/ul>\n<h2>Step 1: Record the objectives<\/h2>\n<p>Write down the recovery time objective and the recovery point objective for the service. If nobody has agreed them, propose a pair and have the service owner approve it before the drill. A recovery point objective is a moment in time to return to, according to NIST, so express it as a time, for example &#8220;no earlier than five minutes before the failure&#8221;.<\/p>\n<h2>Step 2: Compute a decision deadline<\/h2>\n<p>Take the time objective, subtract the slowest technical recovery you have observed, and subtract a small margin. If you have never measured one, use the drill itself to create the first figure and compute the deadline afterwards. The Atlas example arrives at twenty-two minutes from a sixty-minute objective, a thirty-three minute technical run and a five-minute margin.<\/p>\n<h2>Step 3: Define the trigger and the approver<\/h2>\n<p>Write down what condition prompts a failover decision, and who may approve it. Name a deputy. Add the means by which the approver is contacted, and the fallback if they cannot be reached within a few minutes. This is the step most plans omit, and it is the one that decides the human share of the clock.<\/p>\n<h2>Step 4: Brief the participants<\/h2>\n<p>Share the scenario in one paragraph: what has failed, what is unavailable, and what is still working. Keep the facts simple, such as &#8220;primary region unreachable, replica healthy&#8221;. Tell people that the stopwatch will be running, and that results will be used to improve the plan rather than to evaluate individuals.<\/p>\n<p>Choose one observer to take notes on friction rather than on timing: unclear ownership, missing access, instructions that assumed knowledge the reader lacked. Those notes become the fix list later, and they are usually longer than the timing table. A shared document that every participant can edit live keeps the record accurate and avoids reconstructing events from memory.<\/p>\n<h2>Step 5: Start the clock at the failure, not at the first keystroke<\/h2>\n<p>Announce the failure and log three timestamps as they occur: when the problem was noticed, when the approver decided to fail over, and when the service was confirmed working at the new location. The Atlas sample splits its drill into six minutes of detection, twenty-five minutes of deciding and thirty-three minutes of systems work. Your own split will differ, but the discipline of recording it is the point.<\/p>\n<h2>Step 6: Execute in the safe order<\/h2>\n<ol>\n<li><strong>Fence.<\/strong> Prevent the old side from accepting new writes. Include batch jobs and queue consumers, because switching DNS only moves some of the traffic.<\/li>\n<li><strong>Promote.<\/strong> Let the replica at the destination take over as the writable copy of record.<\/li>\n<li><strong>Steer.<\/strong> Move user traffic, then verify with a real transaction.<\/li>\n<\/ol>\n<p>Prefer scripted or runbook-driven steps over console clicking, and note each place where a person had to improvise. The Atlas calls ad hoc console recovery an anti-pattern because it cannot be reviewed or repeated at speed.<\/p>\n<h2>Step 7: Measure the data gap<\/h2>\n<p>Many first drills skip this because it is fiddly, and that is a mistake: it is the only step that tells you what the business actually lost.<\/p>\n<p>Compare the newest record on the new side with the last write accepted on the old side. The difference is your achieved data gap. Remember that the effective gap is governed by the worst lag among every store the service depends on, so check queues, caches and search indexes along with the database.<\/p>\n<h2>Step 8: Rehearse the return<\/h2>\n<p>Do not end the exercise at a working failover. In the Atlas view, coming back is a migration in its own right. Preserve the old primary first, work from a restored copy when comparing, and replay missing writes through the application in a way that tolerates repeats. Time this phase separately, and note how many manual interventions it required.<\/p>\n<h2>Step 9: Score and report<\/h2>\n<p>Complete a short table: total time against the objective, decision time against its allowance, data gap against the target, and the steps that deviated from the runbook. In the Atlas worked example, the total was sixty-four minutes against sixty, a four-minute miss, and the decision had overrun by nine. Present the results to the service owner soon after, before anyone forgets what was confusing. Include a candid paragraph on what surprised the team, since surprises are the real product of a first drill.<\/p>\n<h2>Step 10: Fix, schedule, repeat<\/h2>\n<p>Convert each deviation into a ticket with an owner. Schedule the next drill before the meeting ends. Teams who automate incident triage should also read <a href=\"https:\/\/cotocus.org\/blog\/mastering-incident-management-with-aiops-for-faster-resolution\/\">our notes on faster incident handling with AIOps<\/a>, since faster detection shrinks the first segment of the clock, and the planning habits in <a href=\"https:\/\/cotocus.org\/blog\/aiops-roadmap-a-practical-guide-for-devops-sre-and-operations-teams\/\">an operations roadmap for DevOps and SRE teams<\/a> pair well with a drill calendar.<\/p>\n<h2>Common stumbling blocks<\/h2>\n<ul>\n<li>The approver was unreachable, and nobody had named a deputy.<\/li>\n<li>Credentials for the recovery side lived inside the failed platform.<\/li>\n<li>A quota or image was out of date in the target region.<\/li>\n<li>Replication had been quietly broken for days.<\/li>\n<\/ul>\n<p>All four are common enough to be listed among the dependency checks in the Atlas, which also covers control-plane limits and corrupted data that replication faithfully copies. For the full background, see <a href=\"https:\/\/www.xopsschool.com\/disasterrecoveryops\/\">the reference page on DisasterRecoveryOps<\/a>.<\/p>\n<h2>What good looks like after three drills<\/h2>\n<p>Expect the first drill to feel messy and a little embarrassing. That is normal, and it is a sign that the exercise is finding real problems. Teams that persist usually see the largest improvement between the first and second attempts, when the obvious gaps close. By the third attempt the decision happens in a few minutes, the technical steps are scripted, the data gap is known and the return trip has been tried. At that stage you can move to the next tier of services, and eventually towards production switchovers on a schedule.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A ten-step walkthrough for running a timed failover drill, from recording objectives and a decision deadline to scoring the result and rehearsing failback.<\/p>\n","protected":false},"author":4,"featured_media":1370,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[130],"tags":[],"class_list":["post-1369","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-it-operations"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1369","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1369"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1369\/revisions"}],"predecessor-version":[{"id":1487,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1369\/revisions\/1487"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1370"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1369"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1369"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1369"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}