Metric Definitions Explained: Why Your Reports Disagree

Metric Definitions Explained: Why Your Reports Disagree

Two reports land on the same desk on the same morning. One shows a healthy jump in new customers last quarter. The other shows a noticeably smaller figure for the very same period. Both were built by competent people, both pulled from the same warehouse, and neither contains a bug. This is what happens without metric definitions: written, agreed statements of exactly what a number counts. This explainer covers what a metric definition is, why numbers disagree without one, what a good one contains and how to get your organisation using them. No jargon beyond what is necessary.

What metric definitions actually are

A metric definition is a short, precise description of a number your business relies on. It answers a few blunt questions: what is being counted, over what period, from which source, with which exclusions, and who is responsible for keeping that answer correct. That is it. It is not a dashboard, not a data model and not a policy document. It is the agreed meaning behind a word like “customer,” “revenue” or “active user.”

Most organisations have dozens of such words floating around. Only a handful drive real decisions. Those are the ones worth defining.

Why the two reports disagreed

Back to the customer count. Here is what each analyst did:

  • Report A counted every new account created in the quarter, including free trials.
  • Report B counted accounts that made at least one payment in the quarter and had not previously paid.

Both are reasonable meanings of “new customer.” Neither analyst was wrong. The problem is that the word was never pinned down, so each person filled the gap with their own sensible assumption. Multiply that across sales, finance, marketing and product, and you get meetings where people argue about numbers instead of decisions.

This is one of the most common problems in analytics work; our guide to recurring obstacles in analytics projects lists it alongside several related issues.

Where the ambiguity hides

When you start looking, ambiguity turns up in predictable places:

  1. The unit. Is a customer a person, an account or a company? A household with three logins might be one customer or three.
  2. The time window. “Active” in the last seven days, thirty days or calendar month? Measured in which time zone?
  3. The event. Does revenue count when an order is placed, shipped, paid or past the refund window?
  4. The exclusions. Are internal test accounts, staff purchases, refunds and fraud removed? Who decided?
  5. The source. The billing system, the CRM and the product database often disagree. Which one wins?

Each of these is a decision. If nobody makes it explicitly, everybody makes it implicitly, and differently.

What a good metric definition contains

Keep it short enough to read in a minute. A practical template:

  • Name: the exact label people should use, for example “New paying customers.”
  • Plain meaning: one sentence a non-technical manager can understand.
  • Calculation: the rule, in words and, ideally, a link to the query or model that implements it.
  • Source: the system of record.
  • Exclusions: what is deliberately left out.
  • Owner: the person or role who approves changes.
  • Last changed: a date and a brief reason.

If you cannot fit a definition into that shape, you probably have two metrics pretending to be one. Split them and give each its own name. “New accounts” and “new paying customers” can coexist peacefully once they stop sharing a label.

A worked example: “active user”

Suppose a product team wants to report active users. A loose version might read “people who use the app.” A usable definition reads more like this:

  • Name: Monthly active users (MAU).
  • Plain meaning: distinct people who did something meaningful in the product during a calendar month.
  • Calculation: count of distinct user IDs with at least one qualifying event (create, edit, share or comment) between the first and last day of the month, UTC.
  • Source: the product event stream, not the login table.
  • Exclusions: staff accounts, automated test users and logins with no qualifying event.
  • Owner: head of product.

Notice what that does. Merely opening the app no longer counts, so the number reflects real use. The time zone is fixed, so two analysts in different offices get the same answer. And anyone who disagrees with the rule knows exactly whom to talk to.

Metric definitions versus a general glossary

People sometimes confuse the two. A glossary explains vocabulary, such as what “churn” means as a concept. A metric definition goes further: it fixes the exact calculation your organisation uses for a specific number. You can have both, and the glossary can link to the definitions. But only the definition ends arguments about figures, because only the definition is precise enough to reproduce.

Why ownership matters so much

A definition without an owner decays. Someone tweaks a filter for one report, someone else copies the tweaked version, and within months there are three variants again. An owner fixes that. The owner is usually the business lead closest to the decision the number supports: finance for revenue, product for active users, sales for qualified leads. The owner does not have to write SQL. They decide what the number should mean, approve changes and make sure people hear about them.

How definitions make pipelines better

Once a metric is defined, engineers can build it once and reuse it everywhere, instead of every analyst rebuilding their own variant. That single implementation can then be tested: does the count of new paying customers ever exceed the count of new accounts? If so, something upstream is broken. Definitions therefore improve both trust and delivery speed. Our article on raising pipeline quality while shipping data work faster covers how teams wire such checks into their workflow, and our piece on getting BI reports to decision-makers sooner shows the knock-on effect for reporting.

Getting people to actually use them

Writing definitions is the easy half. Adoption is harder. What works in practice:

  • Start with the five numbers leadership argues about. Defining those visibly earns goodwill for the rest.
  • Put the definition next to the number. A tooltip or footnote on the dashboard beats a separate wiki page.
  • Use the defined name in meetings. If someone says “customers,” ask which defined metric they mean. It feels pedantic twice and then becomes habit.
  • Announce changes. When a definition changes, post the old version, the new one and the effective date. Note the break on any trend chart.
  • Retire duplicates. If an old report uses a conflicting meaning, fix it or remove it. Leaving it live keeps the argument alive.

Common objections

“Our business changes too fast.” Then you need definitions more, not less, because change is exactly when meanings drift. Version them.

“We have a semantic layer or metrics tool already.” Good. That is an excellent place to store definitions. But a tool only enforces what someone decided; it does not decide for you.

“Nobody will read them.” They will if the definitions sit where the numbers are and if leaders refer to them.

Learning the practice properly

Metric definitions are one small, concrete piece of a wider discipline: data quality and governance. Teams that want to build that discipline systematically can look at DataOpsSchool’s training, certification and consulting for data teams, which treats governance and quality as part of a complete DataOps operating model rather than as separate side projects.

The short version

Numbers disagree because words are vague, not because people are careless. Pick the metrics that drive decisions. Write down what each one counts, from where, over what period, with what exclusions. Give each one an owner. Put the definition beside the number and announce any change. Do that, and the next time two reports land on the same desk, they will say the same thing, or they will carry different names that explain why they do not.