Metric definitions are code: how to end dashboard arguments

The meeting is always the same. Marketing reports active users up 12% this month; the product dashboard says flat; the spreadsheet a founder keeps says something else again. Nobody is wrong, and that is the problem — each number is computed from a different, unwritten definition, and the company spends an hour arguing about arithmetic instead of deciding anything.

One name, three defensible numbers

Take “active user,” the most contested two words in SaaS. Before you can count it, you have to answer questions that sound pedantic and are not:

• Does “active” mean logged in, or performed the key action your product exists for? A user who signs in, stares at the dashboard, and leaves is active by one definition and dead weight by the other.
• Active within what window — 7 days or 30?
• Do trial users count? Internal accounts? The customer’s purchased seats that never log in?

Run those variants against the same database and you get numbers like 4,200 (logged in, 30 days), 2,600 (key action, 30 days), and 1,100 (key action, 7 days). Every one is defensible. A board deck built on the first and a product dashboard built on the third will disagree by 4x forever, and each team will be certain the other’s tooling is broken. Multiply this by every metric you care about — churn (does a downgrade count?), MRR (do one-time fees? paused subscriptions?), activation — and you have the standing argument most startups mistake for a data-quality problem.

Definitions belong in code, not in chart configs

The failure mode has a specific anatomy: the same metric gets defined inside a Metabase question, a Mixpanel report, and someone’s spreadsheet — three places, each editable independently, none aware of the others. Drift isn’t a risk; it’s a schedule. Someone tweaks a filter in one tool to answer a one-off question, and the numbers never agree again.

We’ve watched the endgame of this more than once: a contractor edits one BI tool’s definition of activation to exclude a noisy event, rolls off the project in the spring, and for two quarters the company reports an activation rate nobody can reproduce — until someone spends three days diffing chart configs by hand to find it. Three days, for what should have been a one-line pull request.

The fix is a rule, not a tool: every metric that matters is defined once, in version-controlled SQL, and charts only read from those definitions. In practice that means a dbt model (or equivalent) per metric, where a definition is four things:

• The SQL itself — the one authoritative computation, in git.
• An owner — a named person who answers questions about it and reviews changes to it.
• A description — the plain-language contract: “a user who performed the key action within 30 days, excluding internal and trial accounts.”
• Tests — unique keys, accepted values, and reconciliation against a source of truth like Stripe.

The BI layer then becomes deliberately dumb: every chart points at the modeled table, never at raw event data with per-chart filters. Your BI tool turns into a display surface — which is the only job it was ever good at.

Arguments become pull requests

The payoff shows up in a ritual we call the metric review. A number moves unexpectedly — activation drops 9% week over week — and the first question is no longer “which dashboard is right?” but “did the definition change?” Git history answers that in about thirty seconds. If the definition changed, there’s a pull request with an author, a reviewer, and a stated reason; the mystery is solved before the meeting starts. If it didn’t change, the movement is real, and you can spend the meeting on the actual business.

The deeper change is cultural. “I think churn shouldn’t count downgrades” stops being a recurring argument and becomes a pull request: a concrete diff, discussion attached to the code, a decision merged and recorded. Six months later, when a new hire asks why churn excludes downgrades, the answer has a date, a name, and a rationale instead of a shrug. Disagreements still happen — they just happen once, in writing, and produce a definition instead of a grudge.

When a shared doc is honestly enough

A five-person company where one person builds every chart does not need dbt to have consistent metrics — it needs a shared doc titled “Metric definitions” and the discipline to follow it. Version-controlled SQL is coordination infrastructure, and coordination problems require multiple people. The discipline starts paying the day the second and third person begin building charts, because that is the day drift begins.

The signals that you’ve crossed the line are concrete: two dashboards disagree on a number that matters and reconciling them takes days; a metric you put in an investor update can’t be reproduced the following quarter; “what changed?” conversations happen weekly and end without an answer. Any one of these costs more in credibility than the modeling layer costs in time — an experienced person can stand up tested definitions for a startup’s core metrics in one to two weeks.

This modeling layer is the center of every data stack we build, and the retainer includes owning it: definitions reviewed, tested, and evolved as the questions change, so the numbers in the board deck stay boring.