Four simple rules for identifying a good metric

Avinash Kaushik — the man, the myth, the legend — had another excellent post yesterday. He titled it Web Metrics Demystified, which is a take on Eric Peterson’s Analytics Demystified (book, brand, catchphrase). Avinash has a background in data that extends into the broader world of BI and data warehousing. So, typically, his posts talk about “web” metrics and “web” data…but the “web” can be removed and you’ve got insightful thinking that is much broader than the world of web analytics.

In this post, Avinash laid out four attributes by which any metric should be judged. The metric must meet all four criteria to be a good metric — no ORs in this evaluation:

1. Uncomplex (or…um…Simple…but Avinash feels like the term “simple” has a “semantic implication” that he wanted to avoid) — all too often, we head down a road of trying to limit the number of metrics we’re looking at, so we combine multiple metrics into a single metric (I just saw one today: “annualized revenue by role based on current month’s revenue divided by number of people in the role and multiplied by 12″…OUCH!); or, we feel like a metric is too simple and doesn’t sufficiently reflect the nuances of our business…so we add in adjustments and tweaks that, in the end, just make the metric much harder to understand while only getting incrementally closer to an accurate reflection of reality
2. Relevant — it seems like this would go without saying…but it’s critical; have you ever found yourself or your company reporting on something simply because “we’ve always reported that?” It brings to mind a case at my last company where new functionality had been rolled out on the web site that was expected to offload some of the work that CSRs were doing with repeat customers; a report was established and distributed to a broad group on a weekly basis to monitor the global adoption of that feature; 3 or 4 years later, that feature was pretty much defunct…but I’ll be damned if we didn’t have someone still spending 15 minutes every Monday morning putting together that report and blasting it out to the masses! Relevancy is a slippery slope — it’s easy to think relevant means “directly links to the bottom line,” which it doesn’t necessarily need to do (see my last post).
3. Timely — Avinash has a great example of a company that had a query that took 3 months to run. That’s an extreme. He is also an anti-“real-time” guy, which I wholeheartedly support. Timeliness is indeed key. Somehow, I’d like to work Frequency in there, too, though. BI vendors often talk about having data that is real-time or near real-time…and then start pitching how you can check your dashboard “every morning.” This is misguided. The reason to have data near-real-time is so that whenever a user looks at it, it is as current as possible. Businesses are like boats — the bigger they get, the more room they need to turn. If you plan your Marketing campaigns on a 2-month horizon, then it doesn’t make a whole heckuva lot of sense to check your results every day! As a matter of fact, if you start making changes before you’ve let your last set of changes play out, you’re headed for a heap of trouble! But, Frequency is more a business usage of the metric than an attribute of the metric itself, so I’ll call this a side note to Timeliness.
4. Instantly Useful — I love this one as much as Avinash does. The challenge, in my experience, is that, when someone looks at data that is not instantly useful (read: actionable…but I suspect Avinash steered clear of that term due to its overuse), he almost never says, “I guess I shouldn’t be looking at that.” Rather, he says, “It’s not useful now…but it will be if we keep reporting it for the next few months,” or “It’s not useful now, but it’s important for me to see it.” That’s why I’m a major proponent of probing for actionability when establishing the metrics, rather than waiting until after they’ve been delivered and then seeing if they drive action. And, to be clear, “no action” is a valid action in my book, as long as it’s a conscious decision to take no action (as in, “we are hitting our target for this metric, so our ‘action’ is to maintain the status quo).

Good, good stuff that!