GitOps in a demo and GitOps at three in the morning are different sports. The demo shows the happy path; production is where merge conflicts, deletion cascades, and half-understood reconcilers live. The nine rules below come from the sharp edges of that second sport. Each one is short enough to enforce and concrete enough to check. Teams that adopt them early avoid the specific incidents that teach them late.
Nine rules that keep GitOps safe in production
Read them once, then argue with them — a rulebook you have argued with is one you will actually follow.
Rule 1: Keep one road to production
If changes can arrive by pull request and also by someone’s terminal, the terminal wins during every emergency, and then the record stops being true. Pick a single route — reviewed proposal, merge, reconcile — and make every other route either impossible or loudly temporary. Side doors are how drift begins and how audits fail. The whole model rests on the premise that the repository is the only place change originates; protect that premise like the load-bearing wall it is.
Rule 2: Pin every reference, promote by digest
Moving parts quietly undo the model. The :latest tag means the record never quite says what runs. An unpinned chart means a rendering today differs from last week’s identical file. A remote base pointed at somebody’s HEAD means your production depends on a stranger’s branch. Pin versions, pin charts, pin sources. When promoting a tested build between environments, reference its digest — the immutable fingerprint — rather than a tag that can be retagged. A declaration that can change without a commit is not a declaration; it is a rumour with good handwriting.
Rule 3: Give the agent least privilege
Reconcilers ship with broad permissions because breadth is convenient, and convenience is not a security policy. Scope the agent through application projects and through service accounts that impersonate real identities, or via the lockdown switches Flux provides, so a compromise of one application cannot become a compromise of the cluster. The same scepticism applies to speed features: a webhook only triggers an earlier pull, and the endpoint that receives it ships with no rate-limiting mechanism built in, so give it a shared secret before exposing it. Least privilege is boring until the day it is the only thing between an incident and a catastrophe.
Rule 4: Guard deletion before you allow it
Pruning — the agent removing whatever the record no longer mentions — is where GitOps earns both its efficiency and its horror stories. A merge conflict that quietly drops one file can cascade into deleting a namespace and its storage along with it. Set the guards first: disable automatic pruning for anything precious, set reclaim policies so storage outlives the resource that declared it, and add a pipeline check that flags any proposal which removes namespaces, storage claims, or other structural resources. Deletion under this model is powerful and reversible — right up until it is not.
Rule 5: Keep data out of the repository
The record describes structure, never contents. Customer rows, uploaded files, message queues mid-flight: none of it belongs in Git, both because the repository would balloon and because a copy of production data in a code repository is a compliance finding waiting to be written. Structure and data recover through different channels — one from the repository, the other from backups — and a team that confuses them discovers the gap during its worst possible week. Decide the boundary once, write it down, and enforce it in review.
Rule 6: Environments live in directories, never in branches
Branch-per-environment sounds tidy and ages terribly: differences between environments hide inside merge commits, promotion becomes a hope-based exercise, and long-lived branches accumulate changes nobody can explain. Put staging and production side by side as directories. Their differences become visible lines in a diff; promotion becomes copying a reviewed change across a known boundary; and the shared base stays shared. This single layout choice prevents a whole family of “but it works in staging” conversations.
Rule 7: One value change per overlay, rendered and reviewed
A pull request that changes eleven values at once cannot be reviewed, only approved on faith. Keep overlays small — one intentional change each — so the diff states its own purpose. Then render the configuration in CI and present the output for review rather than the input: reviewers see the exact YAML that will touch production, not a template they must mentally execute. Rendering also catches template errors before a merge instead of after one. Review quality is the true throughput ceiling of this whole practice; rules 1 through 6 just keep the queue worth reviewing. Teams building this muscle will recognise the overlap with the work described in this complete guide to CI/CD pipelines for delivery teams.
Rule 8: Write a drift policy per resource
Drift is not one phenomenon; it is at least four, and each wants its own response. Decide in advance: revert unauthorised edits automatically; tolerate the legitimate self-adjustment of autoscalers and webhook-filled fields by marking those differences ignored; alert without acting where a human should judge first; and protect structural resources from cleanup entirely. Without a written policy, every drift event becomes an improvisation, and improvised decisions during incidents trend badly. The policy costs an afternoon and repays itself the first time the reconciler and an autoscaler disagree about a replica count.
Rule 9: Keep a rehearsed break-glass exit
Somewhere, someday, the automation will fight your incident responder. Self-healing is a feature until a human is mid-surgery on a misbehaving system and the reconciler keeps restoring the fields the human changed. The exit is suspension: pause reconciliation, work the problem, then re-enter every manual change through proposals so the record regains the truth. Rehearse this in calm weather, twice a year. Incident responders who want the wider playbook for keeping clear heads during those events will find it in this guide to incident management with AIOps for faster resolution.
Making GitOps rules stick
If you are starting from zero, implement in this order — the cheap guards first, the cultural ones as you go:
- Rule 5, the data boundary, since it costs nothing and prevents a compliance finding.
- Rule 2, pinning, because every unpinned reference is already drifting today.
- Rule 4 and Rule 3 together, the deletion guards and the least-privilege scope.
- Rule 6 and Rule 7, layout and review discipline, next time the structure is touched.
- Rule 8 and Rule 9, the drift policy and the rehearsed exit, once alerts are flowing.
Rulebooks decay when they are aspirational; these nine are enforceable, which is what separates them from poster material. Review them quarterly alongside your roadmap and operations guides — the habit pairs naturally with maintaining a practical AIOps roadmap for operations teams, since both are really statements about where judgement lives in your system. When a rule grates, fix the rule in daylight through discussion rather than eroding it at midnight through exceptions. And when a rule earns its keep, say so out loud in the retro; enforced habits survive, admired ones do not. The full technical depth behind every item here — labs, runbooks, and the trade-off tables — is collected in this production GitOps reference, which makes a sensible companion for whoever on the team owns the rulebook. Nine rules, no drama, and a delivery system that behaves the same way at 3 a.m. as it did in the demo.
