Registry Versus Production: A Checklist to Reconcile Models

Registry Versus Production: A Checklist to Reconcile Models

Ask a machine learning team which model version is serving traffic and you will often get a confident answer from the registry. Ask the production environment and you may get a different one. A rollback that was never recorded, a hotfix deployed outside the normal path, a stale replica still running last quarter’s artefact: each is small, each is plausible and each leaves the documented state and the live state disagreeing. This checklist helps you reconcile the two on a fixed rhythm, so that every difference is resolved while it is still cheap to understand.

The principle in brief

The registry is a record of intent. Production is a record of fact. Reconciling means comparing intent with fact on a schedule and resolving every difference. The routine does not judge whether a model is good; that belongs to evaluation and monitoring. It simply confirms that you know what is running. Our introduction to building scalable, dependable AI products with MLOps sets out why that knowledge underpins everything else in the lifecycle.

Before you start

  • Choose a fixed day and a time box. Weekly is a good default for active systems.
  • Name one owner for the session and a deputy.
  • Decide where the results will be recorded, and use the same place every time.
  • Agree the definition of live: serving real requests, not merely deployed.

Checklist part one: collect the two records

  1. Export the registry view. For every service, note the model name, version, stage and the date it was promoted.
  2. Export the production view. From the serving layer, record the model identifier and version each endpoint actually loads.
  3. Include batch and scheduled consumers. Nightly scoring jobs often use older artefacts than the online service.
  4. Capture the configuration references. Note which feature definitions and thresholds each deployment points to.

Checklist part two: match and label

  1. Match each service to its registry entry. Mark it matched, mismatched, unregistered or retired-but-running.
  2. Check version agreement. A matched name with a different version is the most common mismatch.
  3. Check stage agreement. A model marked as archived in the registry but serving traffic needs immediate attention.
  4. Check replicas. Confirm every instance behind a single endpoint runs the same version.
  5. Check dependent artefacts. Confirm feature definitions and preprocessing code match what the registry records.

Checklist part three: resolve each gap

  1. Assign an owner and a date to every mismatch before the session ends.
  2. Decide the direction of the fix. Either update the registry to reflect reality, with a note explaining why, or redeploy so reality reflects the registry.
  3. Record the cause. Hotfix, rollback, incomplete rollout, manual change, naming error.
  4. Verify the fix. Re-check the affected endpoint after the change, not just the ticket status.
  5. Open the next session with the unresolved items.

The fourth step in this part deserves emphasis. A ticket marked as done does not prove that production changed. Look at the serving layer again.

A worked example

Imagine a recommendations service whose registry entry names the latest approved version as live. During the weekly match, the engineer on duty queries the serving layer and finds that two of three replicas load the approved version while the third still holds the previous one, left behind by a rollout that stalled overnight. The registry was never wrong about intent; the deployment simply never completed. The engineer logs the cause as incomplete rollout, restarts the stale replica, re-checks all three and records the result. Total elapsed time, a quarter of an hour. Left alone, that replica would have served a fraction of requests with outdated behaviour for weeks, and any complaint would have been almost impossible to trace.

Who does what

Keep roles light. The session owner runs the exports and leads the comparison. A second engineer, ideally from a different team, challenges any explanation that sounds too convenient. The service owner attends only for rows that concern their service and leaves with a dated action. This split keeps the meeting brief and prevents the common situation in which the person who made the change is also the only person judging whether it matches the record.

Why a fixed rhythm matters

Mismatches age badly. On the day a hotfix is applied, the engineer remembers why. A month later, the version string in production is a puzzle, and the person who deployed it is on another project. A fixed schedule keeps each investigation within days of the change. It also builds a record of causes, so patterns such as too many manual fixes become visible, and you can address them at the process level.

Reconciliation also sits comfortably beside monitoring. The fundamentals in our article on what every learner should understand about watching deployed models describe how to see behaviour; this checklist makes sure you know which model that behaviour belongs to.

The same discipline in finance

The matching logic has a direct parallel in revenue operations. Finance teams compare issued invoices with settlements that have actually arrived, and tools such as PayManagerPro exist to link invoices, payment links and reconciliation in one place. Whether the records are model versions or payments, the principle holds: compare intent with fact on a schedule, and resolve each difference.

Common failure points

  • Treating the registry as automatically correct. It records what people entered, not what was deployed.
  • Ignoring batch consumers. They quietly fall behind because no one watches them.
  • Fixing production without updating the record. The gap reopens at the next session.
  • Skipping the verification step. A fix that is assumed is not a fix.
  • Making the session optional. Optional sessions disappear during busy weeks, exactly when drift is highest.

Sign-off criteria for each session

Close every session by confirming that all services have been compared, every mismatch has an owner and a date, causes are recorded, and last session’s items have been reviewed. If any of those four is missing, the session is incomplete. Keep the sign-off short and factual, since its purpose is to leave a trail that someone else can pick up.

Summary

The checklist boils down to collecting two records, matching them with clear labels, resolving each gap with a named owner and verifying the fix in production. Run it weekly, record the causes and open the next session with unresolved items. A team that does so always knows which model is serving which traffic, and that is the foundation for every other claim it makes about quality. Schedule the first session this week and see how many services disagree.