Metric definitions are code: how to end dashboard arguments

The meeting is always the same. Marketing reports active users up 12% this month; the product dashboard says flat; the spreadsheet a founder keeps says something else again. Nobody is wrong, and that is the problem. Each number is computed from a different, unwritten definition, and the company spends an hour arguing about arithmetic instead of deciding anything.

One name, three defensible numbers

Take “active user,” the most contested two words in SaaS. Before you can count it, you have to answer questions that sound pedantic and are not:

• Does “active” mean logged in, or performed the key action your product exists for? A user who signs in, stares at the dashboard, and leaves is active by one definition and dead weight by the other.
• Active within what window: 7 days or 30?
• Do trial users count? Internal accounts? The customer’s purchased seats that never log in?

Run those variants against the same database and you get a spread like this illustrative one: 4,200 (logged in, 30 days), 2,600 (key action, 30 days), and 1,100 (key action, 7 days). Every one is defensible. A board deck built on the first and a product dashboard built on the third will disagree by nearly 4x forever, and each team will be certain the other’s tooling is broken.

Multiply this by every metric you care about. Even Stripe, which computes MRR for you, makes the definition a setting: you can configure whether recurring and one-time discounts are subtracted from MRR, metered usage is excluded by default, and its “churned revenue” figure counts downgrades (contraction) alongside cancellations. If the billing system needs a configuration screen for one metric, your team needs a written definition for all of them. Without one, you get the standing argument most startups mistake for a data-quality problem.

Definitions belong in code, not in chart configs

The failure mode has a specific anatomy: the same metric gets defined inside a Metabase question, a Mixpanel report, and someone’s spreadsheet. Three places, each editable independently, none aware of the others. Drift is not a risk so much as a schedule. Someone tweaks a filter in one tool to answer a one-off question, and the numbers never agree again.

We’ve watched the endgame of this more than once: a contractor edits one BI tool’s definition of activation to exclude a noisy event, rolls off the project in the spring, and for two quarters the company reports an activation rate nobody can reproduce, until someone spends three days diffing chart configs by hand to find it. Three days, for what should have been a one-line pull request.

The fix is a rule more than a tool: every metric that matters is defined once, in version-controlled SQL, and charts only read from those definitions. In practice that means a dbt model (or equivalent) per metric, where a definition has four parts:

• The SQL itself: the one authoritative computation, in git.
• An owner: a named person who answers questions about it and reviews changes to it. dbt builds this in: every group has an owner with a name or email, and models can be made private to their group.
• A description: the plain-language contract, such as “a user who performed the key action within 30 days, excluding internal and trial accounts.” dbt’s description property puts it next to the code and into the generated docs.
• Tests: dbt ships built-in tests for unique, not-null, accepted values, and relationships, and you add reconciliation tests against a source of truth like Stripe.

The BI layer then becomes deliberately dumb: every chart points at the modeled table, never at raw event data with per-chart filters. dbt’s own pitch for its semantic layer makes the same argument, that moving metric definitions out of the BI layer and into the modeling layer means a change is refreshed everywhere the metric is used. Your BI tool turns into a display surface, which is the job it was always best at.

Arguments become pull requests

The payoff shows up in a ritual we call the metric review. A number moves unexpectedly (activation drops 9% week over week) and the first question is no longer “which dashboard is right?” but “did the definition change?” Git history answers that in about thirty seconds. If the definition changed, there’s a pull request with an author, a reviewer, and a stated reason, and the mystery is solved before the meeting starts. If it didn’t change, the movement is real, and you can spend the meeting on the actual business.

The deeper change is cultural. “I think churn shouldn’t count downgrades” stops being a recurring argument and becomes a pull request: a concrete diff, discussion attached to the code, a decision merged and recorded. Six months later, when a new hire asks why churn excludes downgrades, the answer has a date, a name, and a rationale instead of a shrug. Disagreements still happen. They just happen once, in writing, and produce a definition instead of a grudge.

When a shared doc is honestly enough

A five-person company where one person builds every chart does not need dbt to have consistent metrics. It needs a shared doc titled “Metric definitions” and the discipline to follow it. Version-controlled SQL is coordination infrastructure, and coordination problems require multiple people. The discipline starts paying the day the second and third person begin building charts, because that is the day drift begins.

The signals that you’ve crossed the line are concrete: two dashboards disagree on a number that matters and reconciling them takes days; a metric you put in an investor update can’t be reproduced the following quarter; “what changed?” conversations happen weekly and end without an answer. Any one of these costs more in credibility than the modeling layer costs in time. In our experience, an experienced person can stand up tested definitions for a startup’s core metrics in one to two weeks.

This modeling layer is the center of every data stack we build, and the retainer includes owning it: definitions reviewed, tested, and evolved as the questions change. If your next board meeting is going to feature two versions of the same number, write down the three metrics you argue about most. That list is where we’d start.

Sources

• Billing analytics, Stripe Documentation
• dbt Semantic Layer, dbt Developer Hub
• Add groups to your DAG, dbt Developer Hub
• description, dbt Developer Hub
• Add data tests to your DAG, dbt Developer Hub