The need to define your Key Performance Indicators is an old, tired concept. Been there, done that, we get it.*

But how many of us think about the need to revisit those KPIs? Perhaps you went through an extensive definition exercise. . . three years ago. Since then, you have redesigned your website, optimized via A/B testing, introduced a mobile app, and created the ability to sign up via phone. Do those thoroughly discussed KPIs still apply?

On the one hand, your true “Single Performance Indicator” should stand the test of time, since it measures your overall business success – which ultimately doesn’t change much. An ecommerce or B2B company still wants to drive sales, a content site wants ad revenue, a non-profit wants donations.

However, for the majority of businesses, digital does not encompass their entire customer interaction. It is common for digital KPIs be a leading indicators of overall business success. For example, submitting a lead or form, signing up for a trial, viewing pricing, downloading a coupon or comparing products can be online KPIs. They may not involve money directly changing hands (yet), but they are a crucial first step. These digital KPIs are even more likely to change as you add new site or mobile features.

So how often should you revisit your digital KPIs?

This can depend on two things:

1. How fast your development is. If it takes you two years to redesign a website or launch an app, you can probably revisit your KPIs less frequently, since your action is generally slow moving. (If nothing has changed, then it’s unlikely your KPIs need updating.)

2. Your customer purchase cycle. If you have a short, frequent buying cycle, you may want to revisit your KPIs more often. If you are a B2B company with a two-year buying cycle, this may be less critical.

My recommendation is to conduct a review of your KPIs and leading indicators at minimum, once a year. If nothing else, a review where nothing changes at least assures both the business and the analytics team that their efforts are appropriately directed.

If you have a short purchase cycle, and/or typically iterate quickly as a company, this might be better done every six months, and/or while planning for big projects that will may materially change your KPIs. For example, let’s say your signup process was previously all online. Now, a new project is introducing the ability for customer service representatives to help customers sign up via phone. A project like that could have a huge impact on your existing KPIs, and a reassessment is critical.

Now, keep in mind this applies to your overall business KPIs. Individual campaigns should still begin with consideration of what KPIs will be used to measure their success, every time.

Analytics is continually evolving, and performance measurement must assess your business against up-to-date objectives. However, KPIs are not only important for reporting, but also for analysis and optimization, which ultimately seeks to drive performance. By revisiting your KPIs, you can ensure that both the business and your analysts are focused on the right things.

* And still, it’s not always done. But that’s a topic for another day.

I’ll admit it: I’m a Nate Silver fanboy. That fandom is rooted in my political junky-ism and dates back to the first iteration of fivethirtyeight.com back in 2008. Since then, Silver joined the New York Times, so fivethirtyeight.com migrated to be part of that media behemoth, and, more recently, Silver left the New York Times for ESPN — another media behemoth. This bouncing around has been driven by Silver’s passion for various places where data is abundant and underutilized: starting with online poker, then baseball analytics, and then a sharp turn to political polling (the original fivethirtyeight.com), which then went even more deeply into politics (the Times iteration of fivethirtyeight.com), which then went broadly into data across many subjects (his book), and which then stayed fairly broad…but with a return to some heavier sports (with the ESPN iteration of fivethirtyeight.com).

Silver talks a lot about what data can and cannot do and how it gets mis-used, and he often dives into details of statistical analysis that I really can’t quite follow. But, he also has a whole other aspect of what he (and his team) does really, really well that I haven’t seen him talking about much. These are twofold:

1. Picking the questions that are worth answering
2. Effectively visualizing that data

These are both key to his success, but they’re also key to any analyst’s ability to deliver value within their organizations.

Picking Questions Worth Answering

Silver originally picked questions that simply intrigued him (winning at online poker, better analyzing baseball players, predicting election outcomes), and those wound up getting him to questions that had mass appeal. Now, as a media site, the questions his team picks, I assume, have a heavy component of “will this drive traffic?” The questions have a pretty diverse range:

• In the wake of the Sandy Hook shootings, what happened with media coverage and public opinion about gun control? [Article]
• Will the recent moves to add calorie counts to fast food menus actually change consumer consumption behavior? [Article]
• Would lifting the ban that prevents gay men from donating blood meaningfully move the needle on blood donations? [Article]

These questions  are often driven by current events and, clearly, would be of interest to a sufficiently large number of potential readers.

“But I’m not trying to drive impressions with my analyses! I just want to drive my business forward!” you exclaim! “How does this relate to me?!”

I’ll claim that it does, but I’ll admit it’s a somewhat meta argument. The dream for most analysts is to find something that gets widely shared internally, because the work reveals something that is surprising and actionable. It’s sooooo easy to lose sight day-in and day-out of the need to be tackling questions that will be most likely to lead to dissemination and action. fivethirtyeight.com — any media site, really — has to focus on content that will be “popular” (in some definition of the word). As an analyst, shouldn’t we constantly be going beyond reacting to the questions that fall in our lap and seeking out meaningful questions to answer? 

For me, every time I read an article on fivethirtyeight.com and think, “Aw, man! That author is so lucky to have gotten to dig into that!” I try to remind myself that I do have some control over what I dig into with most of my clients, and I should constantly be seeking questions that would have broad and actionable appeal (and pushing them to identify those questions themselves).

Effectively Visualizing that Data

This second aspect of the content on fivethirtyeight.com is more tangible and directly applicable. It’s not that every article nails it, but most of the articles include visualized data, and most of those visualizations are very well thought through — neither picking a “standard” visualization, nor getting fancy for fanciness’s sake.

I’m a casual college football fan, at best, but it’s been interesting to watch Silver struggle with predicting who would be in the first “final four” with the change to the championship system that went into place this year. One of his approaches was to run simulations based on what clues he could find about how the selection committee would act, combined with predictions for the results of as-yet-unplayed games. This resulted in a chart like the one below.

Although the one below didn’t actually get the final four “right,” in that TCU dropped out and Ohio State was in…this was something that was almost impossible to accurately predict (between the wildcard of the selection committee’s process, and the fact that Ohio State surprised everyone by blowing out Wisconsin in the Big Ten championship game that occurred several days after he ran this simulation). But, the visualization works on two levels: 1) at a glance, it’s clear which teams his analysis show as being in contention for a final four spot, and 2) the use of the heatmap and dividing lines provides a second level of detail as to the skewing and variability that the model predicted for each team:

 Are you not a “sportsball” (<– Michele Kiss hat tip) fan? Let’s look at an example from politics!

When Jeb Bush took an offical pre-pre-pre-pre-“I’m running for U.S. President” step, Silver asked the question: “Is Jeb Bush Too Liberal To Win the Republican Nomination in 2016?”  To tackle this, he pulled third party data from three different sources that all used different techniques to quantify where various political figures fall on the liberal-conservative spectrum. The result? Another exceedingly well-presented visualization!

Again, the visualization works on two levels: 1) at a glance, it shows that Bush appears to skew to the left side of the conservative spectrum, but he’s not extremely so, and 2) the second layer of detail shows where current (potential) and past Republican candidates fall relative to each other, how consistent each of 2 or 3 different measurement systems aligned  when making that assessment (see Rand Paul!), and even how the times they have a’ changed as to the “average” for the party (for Congress):

The great visualizations aren’t limited to sports and politics, nor are they limited to Silver’s posts. One final example is, in one sense, “just” a simple histogram, but it’s a histogram that has had some real care put into by Mona Chalabi. She tried to answer the question: “How Common Is It For A Man To Be Shorter Than His Partner?” She was limited to secondary data (which was quite limiting!), and she noted at the outset that, for a range of reasons, the results weren’t all that surprising. But, in the histogram below, look at how much care was put into adding clear labels (“Woman taller.” “Man taller”), using color to emphasize the “answer to the original question,” and even the addition of a simple vertical line to represent “equal height.”

I absolutely love the level of care that fivethirtyeight.com puts into their visualizations. They clearly have a well-defined style guide when it comes to the palette, fonts, and font size. But, as with any good style guide, those constraints enable a high level of creativity to then determine what the truly best way to visualize the information is.

fivethirtyeight.com is my newest most favorite site. As I opened with, much of the underlying content is actually on topics I care about, but I’m going to justify my on-going consumption of that content by claiming that it is also a source of inspiration and motivation for improving my work as an analyst!

I was reading a post last week by one of the Big Names in web analytics…and it royally pissed me off. I started to comment and then thought, “Why pick a fight?” We’ve had more than enough of those for our little industry over the past few years. So I let it go.

Except I didn’t let it go.

I was still fuming about it later that day.

And the next day.

And now…almost a week later. Still fuming.

I went back to the post to see if any of the commenters had called out the blather for what it was (I’d initially read the post shortly after it was published, so there were no comments at the time). Several dozen comments…and they were all fawning over the content: “This is brilliant! I’m totally going to start doing what this recommends.”

Here’s the issue: what the post recommended, in my mind, was wrong. And, thus…

I am radical

There it is — in all of it digital pixel clarity from Merriam-by-gawd-Webster:

If a Big Name writes a post that I read as some of the worst possible advice ever gets glowingly praised by his acolytes (many of whom are my industry peers)…am I radical or an idiot? Is a radical just an idiot with delusions of grandeur (see Act Three from this recently rebroadcast This American Life episode)?

Sometimes, certainly. But, as I get older, spend more and more time in the analytics profession (I passed the decade mark several years ago), and continue to work with client after client that has approached their analytics work by trying to apply clichés that have been (mis)interpreted as best practices, I’m becoming increasingly entrenched in my belief that, while I may have some radical and contrarian views…these views are right.*

Some of my favorites (most hated) of these clichés:

• “Good dashboards don’t just show what happened. They show why it happened.” Wrong!
• “Dashboards are only useful if they include insights and recommendations.” Poppycock!!!
• “Good analysts dig into the data after a campaign and find insights from that data.” I hate this one because it’s insidious — good analysts do do exactly this, but, when an analyst or marketer makes this statement, they’re generally making it as a way to ignore what needs to happen before the analyst digs in in order to make this an efficient reality.
• “Google Analytics Intelligence Events and Adobe Analytics Anomaly Detection are the wave of the future — ‘the technology’ is finally telling analysts where they should start their analysis!” For the love of all things dimensional and metrical, please remove your tool-centric cranium from your rectum. NO!!!
• “If the marketer hasn’t articulated what questions they want to answer, the analyst should know the business well enough to come up with those questions — hypotheses — on their own and should dig into the data and answer them.” This is another insidious one — analysts wayyyy too willing to ignore the marketer’s experience and brain as a critical part of the team.
• “Designers shy away from analytics. They know it will stunt their ability to be creative.” I’m not a violent man, but I’ve wanted to punch more than one analyst in the face when I’ve heard this “statement of fact.”

Occasionally, I’ve taken some of these clichés head-on and ripped out a blog post, like this one about why I don’t include text-based commentary on dashboards. In other cases, I’ve bitten my tongue (I have permanent teethmarks on it to prove it. I’ve had to bite pretty hard).

But, these supposed truisms get my goat every time I come across them in a post or in my work. Just because a lot of people have said something, and it seems to make sense and be easy to comprehend, doesn’t make it true.

So, I’ll give myself the radical label — it’s easier to spell than contrarian.

That’s the easy part. What is the internet if not a forum for individuals to criticize and complain? (Well, it’s a place for people to post and view cat videos…but criticizing and complaining is easily in the Top 5.) That brings me to…

I am a pragmatist

I am a firm believer in the maxim:

“If you aren’t trying to change it, don’t complain about it.”

I can document that, in one way or another, I’ve spent well over half of my career trying to “change it.” I’ve tried out different approaches when I’ve seen “truisms” not work. I’ve refined how I approach different situations, and I’ve spent countless hours (well, in theory, they’re countable…but I haven’t always tracked my time at that level — poor data capture, I guess) developing and refining different ways to counter these industry myths.

At the core of that work is one word: pragmatism. I’ve never proposed an approach that isn’t workable in practice:

• I’ve distilled each “radical” approach to its essence — to the point where I’m terrified that what I’m stating is so clear and obvious that my audience won’t realize that it’s a radical departure from how they have actually been operating. I tried to summarize the essence of these ideas as aphorisms in a post last year (not a complete list, but a start).
• I’ve built tools and templates that take these pragmatic concepts and make them directly applicable. Many of those tools are posted here, but I also regularly post downloadable templates in individual blog posts.

And I still fail. But I’m working on it. It’s why I’m now a consultant — I want to spend as much time with as many different analysts and marketers who realize “something isn’t working with this analytics stuff” at as many companies as possible to try to “change it.”

* I am an analyst to the core and am wired with the fairly common trait therein of rampant insecurity. I’ve spent most of my professional life assuming that everyone else knows a lot more than me, and it’s only a matter of time until I’m found out and revealed as an utter fraud. It takes a lot for me to make the bold statement: “I am right.”

Photo credit: Flickr / Adrian Tombu

If I could give one piece of advice to an aspiring analyst, it would be this: Stop showing your ‘math’. A tendency towards ‘TMI deliverables’ is common, especially in newer analysts. However, while analysts typically do this in an attempt to demonstrate credibility (“See? I used all the right data and methods!”) they do so at the expense of actually being heard.

The Cardinal Rule: Less is More

Digital measurement is not ninth grade math class. So by default, analysts should refrain from providing too much detail about how they arrived at their conclusions and calculations.

What should you show?

• Your conclusions / findings

• Any assumptions you had to make along the way (where these impact the conclusions)

• Your recommendations

• An estimated revenue impact

• A very brief overview of your methodology (think: abstract in a science journal.) This should be enough for people to have necessary context, but not so much they could repeat your entire analysis step by step!

What shouldn’t you show?

• Detailed methodology

• Calculations or formulas used

• Data sources

• Props / vars / custom variables / events used

• That’s not to say that this stuff isn’t important to document. But don’t present it to stakeholders.

But Of Course: “It Depends”

Because there is never one rule that applies to all situations, there are additional considerations.

How much ‘math’ you show will also depend on:

1. The level of your audience

   • Your manager should get more detail. After all, if your work is wrong, they are ultimately responsible.

   • Executives will typically want far less detail.

2. The individual

   • Some individuals need to see more detail to be confident in relying upon your work. For example, your CFO may need to see more math than your Creative Director. Get to know your stakeholders over time, and take note of who may need a little extra background to be persuaded.

   • Note: We commonly hear the argument “But my stakeholders really want more details!” Keep in mind there is a difference between them hearing you out, and truly wanting it. To test this, try presenting without the minutiae (though, have it handy) and see whether they actually ask for it.

3. Your findings

   • Are you confirming or refuting existing beliefs?

   • If what you are presenting comes as a surprise, you should be prepared to give more detail as to how you got to those findings.

In Your Back Pocket

Keep additional information handy, both for those who might want it, as well as to remind yourself of the details later. A tab in the back of a spreadsheet, an appendix in a presentation or a definitions/details section in a dashboard can all be a handy reference if the need arises now, or later.

[Credit: Thanks to Tim Wilson, Christopher Berry, Peter O’Neill and Tristan Bailey for the discussion on the topic!]

I’ll admit it: I’m something of a late joiner to the attribution management bandwagon. Over the last year or so, though, I’ve come around, and I attribute that (pun totally intended) to some great clients and some vigorous discussions with both platform providers and practitioners.

What I’ve seen — and was a victim of myself — was confusion about what true and meaningful attribution actually is. While it’s widely understood (and pined for!) that attribution management is intended to get beyond the “last click” — factoring in the contribution of each of multiple consumer touchpoints that lead up to a conversion — there are fundamentally two different approaches to actually achieving that goal…and one of them has some pretty major flaws.

There seems to be limited awareness and very little acknowledgment or discussion of these two different approaches, which is a problem. Marketers know what they want — accurate and meaningful attribution of credit to each of their channels and initiatives — but the conversation gets murky in a hurry when it comes to how to best achieve that goal.

Basic Attribution Is Flawed

Basic attribution is the approach that web analytics vendors tend to promote as “attribution management.” It’s more about cross-session visitor tracking — tracking which channels drove a visitor to the web site over time, and then enabling the marketer or analyst to choose how they want to distribute the credit among those channels (e.g., 40% to the first touchpoint, 40% to the last touchpoint, and the remaining 20% distributed between all touches that occurred in between). This is attribution, but it’s attribution with a couple of non-trivial shortcomings:

1. In most cases, this attribution is limited to “clickthrough” traffic — it tends to ignore the impact of impressions, because impression data is not something that is readily available within many web analytics implementations
2. The marketer has to choose how to distribute credit among the tracked channels based on their own opinion and instincts. This is reminiscent of medieval medicine: “We believe this disease is caused by an excess of blood, so we’re going to bleed the patient. If he still dies, we either didn’t bleed him early enough, or we didn’t bleed him enough” is really not all that different from, “We believe that the last touchpoint contributes 3 times as much to the ultimate conversion as all prior touchpoints combined (a ‘last touch = 75%’ model).” In modern times, marketers don’t inadvertently kill consumers with misguided attribution, but the scenarios are similar in that they start with non-fact-based assumptions.

Both of these gaps are troubling. In some cases, various platforms — including web analytics platforms — tackle the first issue through integration with DSPs or other media-based data sources, but the second issue almost always remains.

Advanced Attribution

Advanced attribution, almost by definition, addresses the first issue above. Advanced attribution providers use various techniques for data capture beyond the data available when a visitor clicks through to the site. It’s the second issue, and how advanced attribution can address it, that gets really interesting. True advanced attribution removes “assumption” and “instinct” (i.e., “picking an attribution model”) as the starting point.

One approach to do this is, in a sense, multivariate testing in the absence of the ability to define a control group at the outset. As a simplistic example, think of a series of marketing touchpoints (impressions and clickthroughs) you are tracking at the user level: A, B, C, and D. If you have a group of users for whom you tracked their touches as “A –> B –> C –> D –> <conversion>,” and another group of users for whom you tracked their touches as “A –> B –> D –> <conversion>” then, without choosing how much proportional credit any individual step should get, you can assign (attribute) the value delivered by “C” in this sequence: it’s the incremental lift between the first sequence and the second.

Obviously, this has to be done for hundreds (or thousands) of touchpoint-series. But, even without fully understanding the math and modeling that goes into that, this is obviously a more robust approach to the problem.

And…It’s Not Just Technology

Shifting gears a bit, there’s another wrinkle when it comes to successfully implementing an advanced attribution management program: the technology is just one part of the equation. In this sense, attribution management is no different than web analytics, a data warehouse, or a CRM platform: if you overly rely on the technical implementation on its own to deliver results, you’re liable to stumble on multiple process, communication, and organizational hurdles along the way. With luck, you will be able to recover, but why not eliminate those hurdles altogether? It’s doable, but it requires flexing your soft skills throughout the course of the implementation: establishing realistic and clear measures of program success up front, ensuring your stakeholders understand what advanced attribution is (and isn’t), involving IT early and often to minimize last-minute technical surprises, and developing and rehearsing your process for converting results from the program into action.

I’ve been fortunate to get to partner with Adometry (now Google) to write a white paper on this very subject: “10 Tactics for Building an Effective Attribution Management Program.”

There are few sentence openers that set off analytics alarm bells for me more than:

“It would be interesting to see…”

That phrase gives me a 15-minute cardio workout without needing to get up from my chair. Really. (Well. Almost. I definitely have developed a physiological reaction to the phrase — elevated heart rate, burning sensation in my ears, etc.)

The curse of the analyst is how often we find ourselves producing results that fall into the dastardly void of “interesting but not actionable.” The real kicker is that, if we let ourselves be guided by “interesting”-based requests, and we repeatedly deliver analyses that scratch those curiosity itches, then, over time, we wind up with the worst possible feedback:

“The reports you deliver have a lot of interesting information…but they don’t have insights. They don’t make recommendations. They don’t tell me what I should do.”

In other words, the reports and analyses we’ve been delivering are exactly what was requested, but seldom produce actionable insights and, ultimately, lead to nothing more than a big, fat, depressing, “So, what?”

The root cause of this vicious and soul-sucking ritual goes back to the initial request: a lack of discipline regarding the intake of the request itself. The “So what?” test can be applied much more cheaply at the point of intake rather than waiting until a full-blown analysis has been conducted and presented!

In a perfect world, where egos do not have to be protected and where organizational pecking orders are not part of the identification and qualification of requests for analysis, it’s simple (the request below is essentially verbatim from an email, just with masked/bracketed details — the subsequent exchange is idealized and fictitious):

Executive: I woke this morning thinking about an interesting impact metric for our website.  To what extent does a blip in twitter result in a blip on our website.  For example, I suspect that something related to [cultural topic] was trending on [date].  Did we also see an increase in traffic on our website?  We could play around with the metric and use  our website analytics to see where people were going.

Analyst: That’s a pretty broad ask. Can you help me understand what we would do with that data?

Executive: We’d have a better understanding of, when a relevant topic spikes in social media, how people behave on our site.

Analyst: Okay. But, wouldn’t you expect that to change depending on the topic? And, I’m trying to envision what the results of such an analysis might look like where you would easily be able to act on the information.

Executive: Well…er…hmmm. That’s a good question. I guess it would…well…no…maybe not. Um. Wow. I really hadn’t thought this through. Now that you’re asking me to…I don’t think this would actually be all that useful. And, you would probably have to spend a lot of time to respond to the request. Thanks for probing a bit rather than just running off and spinning your wheels on my behalf!

We don’t live in a perfect world. Vague requests are going to get floated. Organizational hierarchies and relationship-building realities mean that, as analysts, we do have to chase “interesting” requests that lead nowhere. But, that doesn’t mean we shouldn’t recognize and strive to minimize how often that happens. Here’s how you can do that:

• Condition yourself to go to full alert whenever the word “interesting” is used (as well as “understand,” as in, “I want to understand…<some sort of behavior>;” and, heck, let’s throw in, from the example above: “play around!”)
• When you’re on full alert, probe for clarification as much as you can without being an ass — be delicate!
• Even if you are not able to probe very much, ask yourself the “So what?” question repeatedly — frame the specific questions that you might try to answer based on the request, envision a possible answer, and then apply the litmus test: “So what? If that was the answer, what would we do differently than we’re doing now?”

These tips are all pre-analysis, but they focus your analysis and the way the results get delivered.

The life of an analyst invariably includes responsibility for some set of recurring reports: daily, weekly, monthly, and even quarterly. I hate reports. Or, to be more precise, I have come to hate the term “report.” (I’ve also developed something of an aversion to the term “analysis,” but that’s a topic for another day.)

To make a bold claim: corporate cultures force employees to develop cognitive dissonance when it comes to recurring reports:

• We believe we need to get them, that they’re supposed to be lengthy, that emailing them as PowerPoint presentations makes sense, and that we’re supposed to use them to drive the business forward
• We often don’t actually look at them when they arrive in our inboxes, and, when we do, we simply scroll through each slide, look at it long enough to know what it’s saying, and then close the file and get on with our lives

It’s cognitive dissonance because we believe we need something…even though, in practice, it’s not something we get much value out of.

Occasionally, this cognitive dissonance actually bubbles up into our consciousness — unrecognized for what it is — and results in a conversation with the analyst or the analytics manager that is, essentially, a single statement:

“I get this report each month, and it has the data I asked for in it, but it doesn’t include any meaningful insights or recommendations that actually help me drive the business.”

I’ve seen or heard a statement to this effect more times than I can count. I’ve also seen the typical reaction: the report gets longer! Analysts are pleasers by nature. If we hear “the report isn’t working,” and that’s articulated as a, “it’s missing something” (insights, recommendations) critique, then the “obvious” way to fix it is to “add the missing stuff.”

In my mind, I look like this when this happens (apologies to any kiddos reading this who don’t get the Susan Powter reference):

Performance Measurement vs. Hypothesis Validation

I’ve been beating a fairly persistent drum over the past few years that is a pretty simple rhythm. But, it’s also an, apparently, somewhat radical departure from long-established corporate norms that have not evolved, even though the tools we have for analytics have advanced dramatically.

This tune I’ve been pounding out has an easy, if not particularly melodic, claim as its chorus: there are fundamentally two (and only two) ways that analytics can actually be used to drive a business forward: performance measurement and hypothesis validation:

There’s a temptation to look at the above image and make the leap that “performance measurement” is “reporting” and that “hypothesis validation” is “analysis.” I’d be fine with that leap…if that’s how most businesspeople actually defined the words. And they don’t.

Consider the following requests or statements made every day in businesses around the world:

• “Be sure to include insights and recommendations in the monthly report!”
• “The report must include analysis of what happened.”
• “Can you analyze the results of the recent campaign and tell me if it was successful?”
• “I need a weekly insights report for the web site.”

Ick. Ugh. And phooey! These all represent a conflation of performance measurement and hypothesis validation. I hate the fact that both “hypothesis” and “validation” are four-syllable, fancy-pants-sounding words, but it’s hard to argue that they’re not more descriptive and precise than “analysis.”

My Dreamy-Dream State of Information Delivery

Let me wave my magic wand to how I believe analytics stuff should be delivered. It’s pretty simple:

• The only recurring reports are: single-screen dashboards with KPIs (with targets) organized under clearly worded business goals. They are automated or semi-automated and are pushed out to stakeholders with minimal latency on whatever schedule makes sense. If it’s a weekly report that runs Sunday through Saturday, and the recipient actively uses email, it hits their inbox (embedded in the email body — NOT as an attachment) on Sunday morning at 12:01 AM. It’s got clear indicators for each KPI as to whether the metric is on track or not. It does not include any qualitative commentary, insights, or recommendations (see my last post).
• Additional information is delivered in an ad hoc fashion and is driven entirely by what hypotheses were prioritized for validation, when they were validated, and who actually cares about the results. The core “results” are delivered on no more than five slides or pages (and, ideally, fit on 1 or 2). When appropriate, there is a supplemental appendix, a separate detailed writeup that goes into the methodology and detailed results, and/or a reasonably cleanly formatted spreadsheet with the deeper data.

“OMG, Tim! Are you actually saying that the only regularly scheduled reporting we should be doing are 1-page dashboards? And, are you saying that we should never be presenting more than 5 slides of information when we present analysis results?”

Yup. That’s my dreamy dream world.

But, sadly, although common sense says that makes sense — both from saving analysts time and preventing narcolepsy-by-PowerPoint — that is a too-radical shift for most companies.

So…

A Recurring Report Structure that Deviously Moves in This Direction

Let’s get a little pragmatic. Harry Potter had less trouble taking out Voldemort than he would have had if he’d tried to drastically curtail the volume and length of corporate monthly reports.

Photo by Christian Guthier

So, let’s accept that monthly reports are here to stay for the foreseeable future. How can we make them better? Well, we have to combine performance measurement and hypothesis validation. So, first, I still say we do a low-latency, one-page, automated or quasi-automated distribution of the performance measurement dashboard. The intro to that email is pretty simple:

“Below is the performance measurement dashboard for X for <timeframe>. We are in the process of developing the full report. Let us know if you have any specific questions or comments based on the dashboard itself.”

That’s it. It’s in the stakeholders’ inboxes on the first day of the month. Everyone knows at a glance if anything has gone awry performance-wise. If something in that limited set of data is unexpected, the recipients are asked to quickly respond. In many cases, someone on the distribution list will already know what the root cause is, (“Oh. Yeah. We turned off paid search 3 weeks into the month. We forgot to let everyone know we’d done that.”) In other cases, a KPI’s miss of its target will be cause for immediate concern, and the root cause is not immediately known. That’s good! Better to have the alarm raised on Day 1 rather than on Day 9 when a “full report” is finally made available! The more people hypothesizing early about the root cause, the better!

So, then, what does the “full report” that goes out several days or a week later look like? Like this:

1. Title slide
2. Dashboard — the exact same one that was sent out on the first of the month
3. A slide listing all of the questions (hypotheses — in plain English!) that were tackled during the previous month and an indicator as to whether each question was definitively answered or not (some of these may have been spawned by the initial dashboard — trying to get to the root cause of a problem that manifested itself there), and an indicator if there is action that should be taken for each question. It’s a simple table.
4. One slide for each question that was definitively answered — the title of the slide is the question. Big, bold text underneath provides the answer. Big, bold text that summarizes the recommended action (“No action warranted” is an acceptable recommended action, as long as there is a different potential answer to the question that would have led to actual action). The body of the slide is a clean and clear set of context (including a chart or two as warranted) providing the essence of why the answer is what it is.
5. A single slide of “(Preliminary) questions to be addressed this month.” Some of the questions that were not definitively answered (from slide 3) may be included here if additional work will likely support answering them.

That’s it. It’s still pretty radical, but it can safely be labeled a “monthly report” because it has a defined structure and has multiple pages!

The last slide is “Preliminary” because, if the report is presented to stakeholders, this is the opportunity to look ahead from an analytics perspective. You have your audience’s attention (because you’ve delivered such useful information in an easy-to-consume way), and you want to now collaborate with them to make sure that you are spending your time as efficiently as possible. It quickly becomes clear that the last slide in this month’s report is the basis for the third slide in next month’s report (other questions will come in over the course of the month that will adjust the final list of questions answered), and it will help stakeholders learn that analysis isn’t merely an after-the-fact (after the end of the month) exercise — it starts by looking ahead to what the business wants to learn!

A note on one-time reports

Sometimes, a campaign runs for a short enough period of time that there is only a single “report” rather than a recurring weekly or monthly report. In these situations, the structure is still very similar. But, the “questions” in the last slide are limited to “questions to be answered through future campaigns.” Otherwise, the structure is identical.

Still too radical?

What do you think? Would this report structure work in your organization (it’s not a theoretical construct — I’ve used it with multiple clients and it’s a direction I try to subtly evolve any report where I inherit some other report structure)?

It’s something of a given that any efficient and meaningful analysis will be driven by a clear hypothesis or set of hypotheses. Yet…it’s rare for analysts or marketers to actually think or speak in terms of hypotheses. That’s a problem, in my mind. It leads to gross inefficiency on the part of the analyst (casting about semi-aimlessly in various analytics platforms), and it leads to often non-actionable results (because the results answer questions that can’t lead to action).

Having said that, the word “hypothesis” itself can be intimidating. And, the fact is that, for marketers, we not only need to be working with clearly articulated hypotheses, but we need to be sure that the results of validating those hypotheses will actually be actionable!

To avoid intimidating language while also ensuring actionability, I’ve started using a pretty simple two-sentence construct when it comes to approaching almost any analysis:

1. I believe… [some idea about the site or channel]
2. If I am right, we will… [take some specific action]

The first statement is nothing more than the articulation of a hypothesis. The second statement ensures that the hypothesis is sufficiently specific to be validated, and it ensures that there is the possibility of taking real action based on the results of the analysis.

I wrote up some additional thoughts on this approach in a recent article on Practical eCommerce, and it is a core part of the presentation I will be giving at eMetrics in Boston in early October.

I keep posting things elsewhere and forgetting to get a post here to reference them.

Last fall, I pitched a session topic to Jim Sterne for the eMetrics conference that occurred last week. At the time, I was just a few weeks into my job at Clearhead, and I figured that, by April 2013, I’d easily have a fully baked, deliverable-supporting process that I could use as the basis for the session.

You’re expecting this sentence — the one following that last paragraph — to say, “Boy…was I wrong!” The fact is…I was mostly right!

A handful of articles, posts, and content all came out of my effort to get spit and polish on the material in time for the session:

• The eMetrics session itself, as well as the various downloadable templates that accompanied it, are posted at clearhead.me/emetrics
• I did a high-level summary of the content and approach in a Practical eCommerce article that was published last week
• My unified theory of analytics (requests) was an operational umbrella for the ADAPT to Act and Learn thinking
• Thanks to Avinash, I sorta’ rediscovered Lean Analytics right as I was wrapping up the presentation

Lots of content. You be the judge if it’s good content. Or, if you’re reading this shortly after it got posted and you’re in central Ohio, come get an abbreviated version at this month’s Columbus Web Analytics Wednesday.

“We’re going to use a ‘test and learn’ approach” has become as common a buzzphrase as, “We’re going to be data-driven,” and “We’re going to derive actionable insights.” I’m not a fan of buzzphrases. Buzzphrases tend to originate as statements of an aspirational goal that then quickly morph to be treated as accepted reality. When someone like Eric Peterson steps up and delves into one of these buzzphrases, I do figurative backflips of joy.

The devil is in the details (which is not only a buzzphrase, but a full-on cliché…but it’s true!). And, the details come down to the right people with a valid process using capable tools. When it comes to “test and learn,” the gap between concept and actual implementation often seems to be a true chasm.

The concept: use a combination of A/B (and multivariate) testing and the analysis of historical data to test hypotheses. Based on the disproving or failure to disprove each hypothesis, take appropriate action to drive continuous improvement.

The actual implementation: HiPPOs, lack of clarity on what the KPIs are (without KPIs to optimize against, there can be no optimization), limited resources, over-focus on a specific technique or tool as “the answer,” inability to coordinate/align between marketers/designers/strategists/analysts, analyses resulting in “light gray or dark gray” conclusions rather than “black or white” ones, and so on.

On top of the challenges that have always existed, even in the “simple” world of a brand’s digital presence being primarily limited to their web site and the drivers of traffic to the site (SEO, SEM, banner ads, affiliate programs), we now operate in a world that includes social media. And, most of a brand’s social media activity cannot be A/B tested in a classical sense, so that tried-and-trued (but, alas, still too rare) technique is not available.

None of these challenges mean that “test and learn” is an unattainable ideal. But, it does mean that a strong process with a diligent steward (read: an analyst who is willing to expend some bandwidth as a project manager) is in order. For reasons we’ll cover at a later date, I’m working on codifying such a process, based on what has (and hasn’t) worked for me in past and current roles. Here we go!

Step 1: Develop a Structured, Living Learning List

Step 1 is key. When we talk about learning in a digital data context, we’re talking about a never-ending process. This isn’t “Algebra I,” where a syllabus can be developed once, locked down, and then pulled out semester after semester to each new set of incoming students. Rather, we’re talking about a list that will grow over time. Use Excel. Use MS Access. Use a spreadsheet in Google Drive. Or, get fancy, and use Sharepoint or Jive or any of a gazillion knowledge management platforms. It doesn’t really matter. But, having a centralized, living, taggable and trackable list of “learning possibilities” is critical. Otherwise, great ideas can be fleeting and temporal — lost to a tragedy of poor timing and imperfect human memory.

This list is a list of learning opportunities that any stakeholder (core or extended) proposed as being a useful target for testing and analysis. Here’s a start for what should be captured for each item on the list:

• A title for the learning opportunity
• The name of the person who submitted it
• The date it was submitted
• A description of the question being asked
• The potential business impact (High/Medium/Low) that answering the question would enable

Those are “core” pieces of information that the person submitting the opportunity needs to provide. In addition, the list needs to include some other fields to be populated over time:

• The status of the question (open, in work, cancelled/rejected, completed, etc.)
• The date the question was cancelled or completed
• What sort of testing or analysis would be required to answer the question (historical data analysis, secondary research, primary research, A/B or multivariate testing, in-market experimentation, etc.)
• The level of effort / time estimated to answer the question
• A summary of the results of the analysis (the “answer” to the question) and where the full analysis can be found

Once you’ve settled on where this list will live, who will maintain it, and exactly what fields it will contain, it’s time to move on to…

Step 2: Capture Ideas from Stakeholders

A fairly common delusion in business is that analysts have access to all of the data and have tools at their disposal that will crunch that data in a way such that insights will magically emerge. It doesn’t work that way. Analysis is an exercise in asking smart and valid business questions and then using the specifics of each question to frame and execute an analysis to get an answer.

Analysts are only ONE source for smart and valid business questions!

The reason we set up the list in Step 1 the way we did was so that we can capture more questions than we could possibly ever answer. That’s a fantastic situation in which to be, because it means you have a vast pool of learning opportunities to draw from, rather than scrambling around to find one or two things worth analyzing.

The idea is to make it very easy for any stakeholder who has an idea or a question to quickly and easily get it recorded and available for consideration for analysis: the developer who read a Mashable article that inspired a thought about the current web site, the designer who was torn between the treatment of the global navigation on the site, the marketer who knows an upcoming campaign will be a litmus test as to whether a particular channel is worth the company’s investment or not, etc.

This doesn’t mean this process is all about volume. The “input form” (the first bulleted list above) should force some basic consideration on the part of the submitter to qualify the idea. The fact is, it’s the people in the organization who are most interested in being data-informed, and the people who are most interested in moving the business forward, will be the people who engage in this quasi-crowdsourced learning process.

Step 3: Develop a Means of Prioritizing the Ideas

While Step 2 is intended to be as egalitarian as possible, the act of prioritizing the ideas is, by necessity, much less so. The HiPP  is the HiPP for a reason (I say HiPP here rather than HiPPO because we’re talking about the Highest Paid Person rather than her specific Opinions), and the person who is accountable for the budget that pays for the analyst deserves a greater say in where the analyst spends her time.

First, of course, there needs to be some set of agreed-to criteria for prioritizing the ideas. The specifics will vary, but these will likely include:

• The likely impact to the business if the question is successfully answered (including how quickly the organization will be able to act on the results and the cost to the organization to do so)
• The long-term applicability of the results of the exercise
• The expected time and cost to conduct the analysis

An assessment of these criteria should be captured and recorded in the list developed in Step 1. But, they are inherently subjective assessments, so it still comes back to people-driven decision making as to what gets tested and analyzed and when. There are several ways to tackle this — the ones listed below are all ones that I have used with success in one form or another over the years, but I’m sure there are others:

• Have a core team of stakeholders regularly review the list of questions and decide which ones to tackle and when (see Step 5)
• Set up a way for everyone who submitted ideas to also vote on ideas — a “thumbs-up” for an idea moves it up the list such that the more people who give it a boost, the more likely it is to be a question that gets tackled
• A modified combination of the above is to vary the voting weight of each person; I’ve been through exercises where everyone is given an amount of (fake) money that they get to “invest” in the questions they would like to see answered. They can invest in as few or as many questions as they like. With this approach, the key decision makers / budget owners can be given more money to invest

This is one of the trickier aspects of managing the program, because it requires that the right people are actively engaged with the process and available to participate on a recurring basis (see Step 5).

Step 4: Have a Clear Start/End for Each Analysis

Step 4 and Step 5 are all about process and rigor. Each accepted/approved learning opportunity should be treated as a mini-project and managed as such. In most cases, the analyst can also be the project manager. But, in some cases, the project management may be something that a professional project manager should take on. Key elements of project managing the analysis include:

• Identifying all of the people who will be involved or impacted in some way (a RASCI matrix is handy for this)
• Development of a work breakdown structure — a list of all of the steps that will be required to complete the test or analysis, as well as the sequence of that effort
• Development of a project schedule — when each task on the work breakdown structure will be tackled (in some cases, there will be a “wait and see” period that needs to be built into the schedule; with social media, especially, it’s common to need to explicitly change a tactic for a period of time — a week or two — and then evaluate the results of that change; in these cases, there are periods of time when there is no actual analysis “work” being performed on the project)
• Establishment and publication of key milestones
• A plan for communication — who will be updated, when, and how
• A commitment to regularly checking the project schedule against the actual work completed

“Egad!” you exclaim. “All of THAT is supposed to be done by the analyst?” Well, yes. It doesn’t have to be a huge deal. For many analyses, this can all be done in an Excel spreadsheet with a few recurring Outlook or Google Calendar reminders. In many cases, the test or analysis may be so small that the “schedule” is a single day. But, having the diligence to think through the what, the who, the how, and the when — and then managing the work to the results of that thinking — brings rigor to the work and credibility to the process. It prevents wheel-spinning, and, by feeding back to the list from Step 1, helps build an inventory of discrete, completed work over time that can then be used to assess the overall effectiveness of the analytics program and ensure that what has been learned in the past gets applied in the future.

Step 5: Develop a Fixed Cadence for Updates

As I noted at the beginning of Step 4, that step and this one are complementary. And, in some ways, they are in tension:

• Step 4 — recognize that each analysis is different and unique and has it’s own schedule; that schedule may be a single day, it may be a week, it may be several months. Treat it as such and manage each effort as a small project
• Step 5 — establish a fixed cadence for providing communication, updates, and assessment of the overall program

The fixed cadence may be daily (although I’ve never had a case where that is warranted, the Agile development methodology dictates daily stand-ups, so it may be that there are analytics programs where that is warranted), weekly, or even monthly. Having been at an agency for the last three years, monthly was often the most frequent cadence I could manage. This fixed cadence can include the delivery of recurring performance measurement results (KPI-driven dashboards) if those require an in-person review. But, the focus of these communications should be on the learning plan:

• What questions from the list have been answered since the last update
• What questions are currently being worked on and how (historical data analysis, A/B test, adjustment of digital tactics for a fixed period of time to measure results, etc.)
• What new questions have come in for consideration

If this cadence includes a formal meeting with the stakeholders, which is ideal, then a discussion that generates new questions, as well as the prioritization of new questions, can also be part of this meeting.

“Test and Learn” Is the Core of Analysis

In addition to laying out a practical process for effectively driving continuous learning, I hope I have also illustrated that “testing” is inextricably bound with “analysis” and vice versa. We can’t treat testing as being limited to A/B and multivariate testing and analysis as being limited to historical data. To truly learn in a way that delivers business value, the focus has to be on the business questions. Depending on the question, the best way to answer the question should be selected from a comprehensive arsenal at the analyst’s disposal, and the overall process has to be rigorously managed.

It’s amazing what an airplane flight can do when it comes to finishing up books that have been lingering. Especially when you pack the “fun” book in your checked baggage so you can’t cheat. Sucks when you arrive at the hotel to find that your “fun” book is, apparently, still sitting in the bedroom at home! Argh!

Part 2 will be a bit less harsh, as Ayres does eventually touch on some of my bigger issues. Sort of. I’m still not satisfied with his treatment, though.

Ayres does, at least once, throw in a pretty critical caveat: “As long as you have a large enough dataset, almost any decision can be crunched.” That’s a HUGE caveat, even in our increasingly data-capturing world. As a matter of fact, I like to talk about the data explosion. But, when I discuss it, it’s as a warning that you can’t just see “getting the raw data” as the biggest challenge in making data-driven decisions. My claim is that the bigger challenge is developing discipline about how you approach that data. While Ayres does periodically speak to the fact that there is real skill and creativity required to develop the hypotheses that you want to test, he does not put nearly enough emphasis on this point. And, not only that there needs to be diligence in developing the hypothesis, but also considerable rigor in determining how that hypothesis will be tested.

Ayres walks through a pretty fascinating example of super crunching gone awry in the case of John Lott, who did some super crunching that demonstrated that concealed handgun laws brought down the crime rate. According to Ayres, Lott made an error in the data prep for his analysis that, when corrected, did not show this at all. Ayres uses the example more to preach that even data-oriented people can still get caught up emotionally and refuse to face hard facts. While that is undoubtedly true, Ayres also misses the opportunity to speak in any real depth as to the amount of data prep work that needs to be done to normalize and cleanse data before actually running a regression. He does mention this…but it is very brief.

More good stuff: Ayres devotes a good chunk of a chapter to explaining (and illustrating) just how bad humans are at gauging the quality of their own intuition. Many of the points he makes here echo Daniel Gilbert’s points in Stumbling on Happiness. Points very well taken. But — and I’m sure Ayres would say that my next statements prove his point and that I’m just one more person in denial — things get stretched pretty far at times here. For instance, Ayres claims that the safety procedures that flight attendants follow as a hard script almost all of the time make them more effective than a less structured approach. Seeing as I was sitting on a flight, where I had easily tuned out the flight attendant’s script, I had a “Gimme a break!” moment. Everyone who travels at all has occasionally stumbled across a Southwest (and it even happened to me on United once) flight where the flight attendant has a little more fun. Everyone listens! And, they are clearly sticking to the content of the script, if not the specific language.

Oops. Language. I’ll nitpick just a little bit. Ayres uses “digitalize” a lot. Maybe he can’t help it — he’s a lawyer AND an academic, so why go with the shorter, common synomym — digitize — when a longer word will suffice. He also defines CDI as “consumer data integration” in the context of Acxiom Corporation’s services. While Google indicates this is one possible explanation of the acronym, even Acxiom seems to use the breakdown that I’m much more familiar with: customer data integration. Splitting hairs a little bit, I realize. But, it’s an indication that Ayres really isn’t all that familiar with CDI, which is a descriptor of a host of technologies that are trying to actually solve all of the complexity of normalizing data from disparate data source that Ayres barely acknowledges. Ayres also uses “commodification” a lot, and I was prepared to zing him there, too…but Google shows this as a more common word than it’s synomym, commodization. So, I learned something there!

Two more specific beef examples and I’ll wrap up.

Ayres devotes a good chunk of writing to various plans that use super crunching to predict the likelihood of recidivism for inmates who are being paroled. The statement that really got me was that, according to the data, an inmate who received a score of four or higher when his history was plugged into a certain model would have a 55% chance of committing another sex offense within 10 years. Ayres then jumps into example of one such inmate who received a score of four, was paroled anyway, and promptly disappeared. The 55% is what gets me, though. Ayres ignores that this is far from an overwhelming indication. It may very well be my liberal bias that I don’t think that it’s a slam dunk to keep everyone locked up knowing that 45% of these people would NOT commit another sex offense within ten years. Ayres completely glosses over this question, which seems to be an ethical one worth addressing. I actually made a note — Minority Report — when I read this. I was thinking of the Philip K. Dick-inspired Steven Spielberg movie that starred Tom Cruise. In the movie, citizens are arrested and based on “foreknowledge” — a vision by one of three “pre-cogs” (it’s a very small leap to turn these specially-endowed humans into a super crunching computer) that the person is going to commit a murder in the future. The movie is chilling in a 1984 kind of way. And, ultimately, condemns persecuting someone for something they have not yet done. Ayres sounds just a little too much like the films main antagonist — Director Lamar Burgess (played by Max von Sydow) — for my comfort. Interestingly, later in the book, Ayres refers to the same film…but not at all for the same reasons. Rather, he references the personalized ads that Tom Cruise is bombarded with at all times while walking down a street in the movie (which really isn’t that far of a stretch from today, given the proliferation of RFID technology).

Second specific beef has to do with NASA history. Ayres points to Gus Grissom as one of the astronauts who balked at the idea that the Mercury splashdown capsules would be designed so that that would not be able to be opened from the inside. Ayres then points out that, because the astronauts wan out, Gus Grissom was able to panic upon splashdown and blow the hatch open prematurely. Which he did. And almost drowned. First off, Ayres gives no acknowledgement to the fact that there are a lot of reasons to believe (as NASA does) that the hatch blew open due to an equipment malfunction — not because Grissom did anything inappropriate. Second, Gus Grissom was killed in a fire inside a training capsule for the Apollo 1 mission. I honestly don’t know enough of my history here to know if the training capsule could be opened from the inside, or, if not, if it would have made a difference. My guess is that the fire was so fast that it wouldn’t have mattered. But, on both accounts, Grissom seems like a spectacularly lousy example of Ayres’s point.

So, I’ll wrap up. The book has left a dirty taste in my mouth. Ayres delivered a book that he hopes will sell with the gee-whiz factor. My guess is that Bantam Books smelled “Jump on the Freakonomics Bandwagon” money, and, as bandwagon books, movies, and TV shows are prone to do, it underdelivers. The book dramatically downplays the challenges involved in super crunching — not just challenges in the “it’s hard” sense, but challenges in the “large, clean, relevant data sets” are not the same as “large amounts of raw data.” He actually includes examples of super crunching where, when the results came out, they were applied on a limited scale because the basic approach to the analyses were called into question. So, how about a chapter on “design of experiments?” He does have good points, and his assertion that companies could benefit from having a designated “devil’s advocate” to question — hard — any analysis is a great idea. There is definitely a shock of wheat here and there in the book. Unfortunately, Ayres spends most of his writing about the chaff.

I promised in an earlier post that I would read Ian Ayres new book, Super Crunchers, and it’s taken me longer to get to it than I’d planned. And, while it’s a fairly easy read, it’s not getting a nightly hit on my nightstand because it tends to get me worked up.

I’m fully aware that I got the book with preconceived notions that it might set me off. I’m doing my darnedest, though, to be objective in my observations about it. And I haven’t finished it, yet, but I’m going to lose track of the different points I want to make if I don’t start recording them.

My fundamental beef with the book is that it picks and pulls anecdotes that paint a picture of how easy it is and how common it is becoming to do super crunching. So far, it has done very little to articulate all the ways that super crunching does not or cannot work…either “yet” or “ever.” Much more on that to come. To me, it reads mostly like an effort to jump on the Freakonomics bandwagon. Freakonomics was original and interesting. Super Crunchers is a poor, poor follow-on.

Ayres does reference some great books in his opening pages. James Surowiecki’s The Wisdom of Crowds is a must-read. And I thoroughly enjoyed Moneyball (although I’m an admitted college baseball junkie). I’m ambivalent about Freakonomics, but generally feel positive about it. It may be that it just seems to play too much to the mass market, and I like to think I’m higher brow than that. 😉

Ayres views the world with a heavy, heavy academic bias. As such, he thinks data is cleaner than it is, people are easier to change than they are, and time machines exist. I’ll tackle the last one first. Ayres presents super crunching as an easy, two-party formula: 1) the ability to run regressions (just plug the date into the software and press “Go” is what is sounds like), and 2) A/B testing is a no-brainer (and just occurs in the world naturally if you look for it).

Here’s where the time machine comes in. On page 69, he expounds on how politicians are using super crunching. And then he extends how they could use it even more as follows:

“Want to know how negative radio ads will influence turnout of both your and your opponent’s voters? Run the ads at random in some cities and not in others.”

Stop and think about that one for a minute. Clearly, we’re talking about a statewide or federal election. You’re on a 2-, 4-, or 6-year election cycle. You run negative ads in some cities and not others. The election happens. You lose. You assess voter turnout. Great. Now what? The election is OVER!

Earlier in the same paragraph, Ayers started at a more reasonable level by discussing political scientists. Extend that to political consultants, and you’ve got some validity in super crunching. The RNC and DNC could probably learn a lot by exerting some influence and doing some randomized testing on this sort of thing with all of their candidates and then applying that knowledge in the next election (they will get pushback from their candidates, who are much more concerned about winning their current election than they are about providing data for the common good). But Ayers very explicitly dives down to a prescription for one candidate’s strategy in one campaign against one opponent. And that just doesn’t work.

Why does this irk me so much? Because the casual reader might not realize what a joke that statement is. And, thus, might walk away from the book with a vague notion that super crunching can be applied to every situation with ease!

A more general criticism is that Ayers picks mammoth, established companies that have massive data sets to work from for most of his examples. When I worked at a $600 million company, which was by no means mammoth, but still generated a lot of data, we frequently found that we didn’t have enough data to answer the most burning questions with any statistical validity. Why? Because: 1) the real world is very, very noisy, and it’s really hard to control for all of the likely independent variables (in Ayers election example, there was no mention of the candidates’ issues and how those may resonate differently in different cities regardless of what ads were run), 2) businesses operate in the fourth dimension (time) — if they have a six-month sales cycle and they want to run a test of how something early in the cycle influences events late in the sales cycle, they can be in a pickle when it comes to actually capturing enough data to run a legitimate regression, and 3) the data is NEVER as clean as folk in academia or who haven’t really sat down and understood the processes think it is.

I’ll jump ahead to Ayres’s discussion of evidence-based medicine. He does let in a sliver of “it ain’t always a slam dunk” when he discusses (pp. 90 and 91) a study that linked excessive caffeine consumption to increased risk of heart disease…without controlling for the fact that there was a correlation between caffeine consumption and the likeliness that the subject smoked. That reminded me of a huge commercial super crunching gaff that used A/B testing and regression heavily…but that bit one of the top consumer brands of all time in the butt: Coca-Cola.

Coke doesn’t show up in Ayres’s index, so I’m assuming I’m not going to stumble across the New Coke fiasco. Two different books have given reasons for that flop: in Stumbling on Happiness (I’m almost positive that was the book, but I read Blink at the same time and there was a lot of overlap in the concepts), Daniel Gilbert made the point that the test was flawed. Coke assumed that a single sip in a blind taste test would be a valid indicator. It turns out that your first sip of a soft drink gives you one experience, while the last 80% of the drink gives you another experience — after your taste buds / brain have adjusted to the “new” experience. In Waiting for Your Cat to Bark, the Eisenberg brothers attributed the failure more to the fact that the Coke flavor was sufficiently ingrained in the soul of Coke drinkers that, by golly, they didn’t want that flavor to change! Either way (and it was likely a combination of both) this was a super crunching disaster!

Back to evidence-based medicine. There is some absolutely fascinating stuff in this chapter. And there is no refuting that the medical profession is wildly behind the times when it comes to adopting information technology. And Ayres outlines some pretty interesting initiatives on that front that I genuinely hope succeed. But, again, there were some alarming statements in the chapter that illustrated Ayres’s academic naivete. He plugs in a quote from Dr. Joseph Britto around a feature under development in his (very cool) Isabel system that will enable doctors to enter their notes in a patient’s medical record through a “structured flexible set of input fields to capture essential data.” Britto is quoted as saying, “If you have structured fields, you then force a physician to go through them and therefore the data that you get are much richer that had you left him on his own to write case notes, where they tend to be brief.”

Whoa!!!!

TWO big beefs with this line of thought.

Beef No. 1: I HAVE gone through many situations where someone who wants to use the data (and, therefore, wants data that is comprehensive and complete) makes a massive logical leap that the answer is simply to make the system “force” the people who are inputting the data to provide more data and to provide it in a way that tees it up for valid, straightforward analysis. Life. Just. Don’t. Work. That. WAY! The only time I have seen this work is when immediate, direct value to the person inputting the data is realized by the change. “Do it this way, and you will feed a much larger aggregate data set that you will ultimately benefit from” simply does not work. “Required fields” don’t work. If you introduce a process that takes as little as 15 seconds longer than the old process, those required fields quickly become populated with a single character or the first value on the list (for structured inputs). Taking a top-down approach and auditing and enforcing better data input requires a Herculean effort, but it’s doable. What concerns me is that neither Britto nor Ayres acknowledge that it’s not as simple as a system deployment. The data analyst’s approach is going to be towards capturing everything about the patient, regardless of whether it seems relevant. That way, data mining may find things that no one thought were relevant but turned out to be. Unfortunately, 95% of that “comprehensive” data is not going to be relevant, and there’s no way to identify where that 95% is. Britto and Ayres want to put the burden of capturing all of that data on the doctors, which is simply not practical.

My current company is actually experimenting with a tool on the CRM side that acknowledges this challenge. ShadeTree Technology is a plug-in for Salesforce.com that, at the end of the day, is trying to drive more comprehensive, more consistent, cleaner CRM data. BUT, they’re taking an approach that the only way that will work is if the tool provides immediate, direct benefit to the people who are using it. We are early on in our implementation of the tool, but the fact that ShadeTree understands this concept is a key reason that we are implementing it.

Beef no. 2: I don’t have a medical background, but I have three kids. Our oldest wound up in the hospital for a week when he was six years old. He is a very bright and articulate kid. What landed him in the hospital was a pain that was somewhere in his hip/upper thigh area, deep under the skin. X-rays, bone scans, and CAT scans weren’t able to pinpoint the location of the issue. It took an MRI to find a small abscess. And it took a battery of tests to figure out that he’d gotten salmonella in his bloodstream that, ultimately caused the abscess. How is this relevant? Because we had a helluva time trying to understand exactly what his symptoms were. “Upper thigh” is different from “hip.” Even with a willing, articulate kid, identifying “pain in the bone” vs. “pain in the muscle” was damn near impossible. Humans are not numbers and data. “Pain” is subjective. “Small brown spots on the skin” has numerous subjective elements (another example from this chapter). Ayres wildly glosses over the challenges here, and that bothers me.

To be clear, I’m not saying that none of these initiatives will work. And, for all I know, Britto actually is much more sensitive to the issues than Ayres makes him sound. What sticks in my craw is that Ayres continually presents the world as a place where clean, comprehensive, easily analyzed data already exists or where it’s going to be relatively easy to collect it. That is painting an unrealistic picture. It may sell books, but it’s not all that actionable and, I claim, will actually hinder serious readers’ ability to actually improve their own usage of data.

More to come in Part 2…