A Big Question that social and digital media marketers grapple with constantly, whether they realize it or not:

Is “awareness” a valid objective for marketing activity?

I’ve gotten into more than a few heated debates that, at their core, center around this question. Some of those debates have been with myself (those are the ones where I most need a skilled moderator!).

The Arguments For/Against Awareness

Here’s the absolutist argument against awareness:

“There is no direct business value in driving ‘Awareness.’ It’s a hope and a prayer that increasing awareness of your brand/product will eventually lead to increased sales, but, if you’re not actually making that link with data, then you might as well admit that you’re trying to live in the Mad Men era of Marketing.”

Here’s the absolutist argument for awareness:

“While ‘the funnel’ has been completely blown up by the introduction of digital and the increasingly fragmented consumer experience, it’s impossible for a consumer to make a purchase of a consumer brand without being aware that the brand exists. Logically, then, if and until we know that 100% of our target consumers are aware that we exist (and even what we stand for — awareness is more than just ‘recognize the brand’ and, when I [the absolutist] say ‘awareness’ I mean that consumers have some knowledge of the brand, and that knowledge gives them a favorable impression!). But, between that fragmented experience and the fact that it’s totally reasonable to expect a time delay between achieving ‘awareness’ and a consumer actually making a purchase, we just have to accept that we won’t reasonably be able to tie directly to sales as easily as direct response activity can!”

Obviously, any time an argument gets framed with “absolutist” viewpoints, the blogger thinks the reality is somewhere in between the two extremes.

And I do.

But I’m much closer to the absolutist-for-awareness position. I wouldn’t possibly be considering pre-ordering a WhistleGPS if I wasn’t at least aware that the product exists. At the same time, I am only vaguely aware of when it crept into my consciousness as existing. Now, many of the impressions that led to my awareness are trackable, and, if and when I pre-order, those impressions (the digital ones, at least, but I think all of my exposure has been digital) can be linked to me as a purchaser. But, the conversion lag will be several months at that time — even when trackable, that’s not “real-time” conversion data that could have been used to optimize their sponsored posts or remarketing campaigns. So, whether I’m being included in a media mix model or an attribution management exercise, I’m posing some big challenges.

But That Doesn’t Mean I’m Happy with Awareness

The against-awareness absolutists have a valid point, in that “hope and a prayer” is really not a valid measurement approach. And, neither is “impressions,” which is what marketers often use as their KPI for awareness. Impressions is a readily available and easily understood measure, but it’s a measure of exposure rather than awareness.

IMPRESSIONS = EXPOSURE <> AWARENESS

So, the question for marketers is: “Is your goal to just increase brand exposure, or do you really care about increasing brand awareness?”

“Well, gee, Tim. You have to increase exposure of the brand — impressions! — in order to increase awareness. And, you can’t truly measure ‘awareness,’ can you?”

Oh, how I would kill to actually have that discussion. Because you can measure awareness in many cases. And, that can be extended to be both unaided or aided awareness, as well as brand affinity and even purchase intent!

I’m actually appalled at how often digital media agencies don’t more effectively measure the impact “awareness-driving” campaigns! It’s easy to resort to “impressions.” Is it laziness, or is it that they’re terrified that measuring awareness may be a much less compelling story than a “millions of impressions!!!” story?

Measuring Awareness

There is one more nuance here. We don’t actually want to measure awareness in absolute terms. Rather, we want to measure the increase or lift in awareness resulting from a particular campaign. And that is doable. Even macro-level — quarterly or annual — brand awareness surveys are more interested in if they have increased awareness since the prior study and, if so, by how much.

This is in not an endorsement of a specific product or service, but it would be disingenuous for me to describe one such methodology without crediting where I first saw and learned about it, which was through Vizu (the image below is from their home page):

This is for measuring the lift in awareness for a display ad campaign. The concept is fairly simple:

1. Track which users have been exposed to display ads and which ones haven’t.
2. Use a small portion of the ad buy to actually serve an in-banner survey to both groups to gauge awareness (or preference or intent or whatever attitudinal data you want).
3. Compare the “not exposed” group’s responses (your control) to the “exposed” group’s responses. The delta is the lift that the display campaign delivered.

This can — and generally needs to — measure the lift from multiple exposures to an ad. Repetition does matter. But, a technique like this can help you find the sweet spot for when you start reaching diminishing returns for incremental repeated impressions.

And, depending on the size of the media buy, a simple lift study like this can often be included as a value-add service. And it can be used to optimize the creative and placements against something much closer to “business impact” than clickthroughs or viewthroughs.

Vizu is actually part of Nielsen, which has other services for measuring awareness, and Dynamic Logic (part of Millward Brown) also offers solutions for measuring “brand” rather than simply measuring exposure.

My Advice? Be Precise.

At the end of the day, if you’re fine with measuring impressions then be clear that you really care more about exposure than actual awareness, affinity, or purchase intent. If you do care about true brand impact, then do some research and find a tool or service that enables you to measure that impact more appropriately.

WARNING: Gilligan contrarianism alert! The following post posits a thesis that runs contrary to popular opinion in the analytics community.

Many companies these days rely on some form of internal dashboard(s). That’s a good thing. Even better is when those companies have actually automated these dashboards – pulling data from multiple data sources, structuring it in a way that directly maps to business objectives, and delivering the information in a clean, easy-to-digest format. That’s nirvana.

Reality, often, is that the dashboards can only be partially automated. They wind up being something an analyst needs to at least lightly touch to bridge inevitable API gaps before delivering them on some sort of recurring schedule: through email, through an intranet, or even in person in a regularly scheduled meeting.

So, what is the purpose of these dashboards? Here’s where a lack of clarity — clearly communicated — becomes a slippery slope faster than Miley Cyrus can trigger a TV viewer’s gag reflex. Dashboards are, first and foremost, performance measurement tools. They are a mechanism for quickly (at a glance!) answering a single question:

“Are we achieving the goals we set out to achieve?”

They can provide some minimal context around performance, but everything beyond answering that question is a distant second purpose-wise.

It’s easy enough to wax sophomoric on this. It doesn’t change the fact, though, that one of the top complaints dashboard-delivering analysts hear is: “I get the [weekly/monthly/quarterly] dashboard from the analyst, but it doesn’t have recommendations on it. It’s just data!”

I get it. And, my response? When that complaint is leveled, it’s a failure on the part of the analyst to educate (communicate), and a failure of process — a failure to have mechanisms in place to deliver actionable analytical results in a timely and effective manner.

But…here…I’m just going to lay out the various reasons that dashboards are not the place to expect to deliver recommendations, because, in my experience, analysts hear that complaint and respond by trying to introduce recommendations to their dashboards. Why shouldn’t they? I can give four reasons!

Reason No. 1: Dashboards Can’t Wait

Another complaint analysts often hear is that dashboards aren’t delivered quickly enough at the end of the reporting period. Well, no one, as far as I know, has found a way to stop time. It marches on inexorably, with every second taking exactly one second, every minute having a duration of 60 seconds, and every hour having a duration of 60 minutes (crappy Adam Sandler movies — pardon the adjectival redundancy — notwithstanding).

Source: aussiegal

Given that, let’s step back and plot out a timeline for what it takes in an “insights and recommendations delivered with the dashboard” scenario for a dashboard that gets delivered monthly:

1. Pull data (can’t happen until the end of the previous month)
2. Consolidate data to get it into the dashboard
3. Review the data — look at KPIs that missed targets and supporting metrics that moved unexpectedly
4. Dig in to do analysis to try to figure out why those anomalies appeared
5. IF the root cause is determined, assess whether this is something that needs “fixing” and posit ways that it might be fixable
6. Summarize the results — the explanation for why those anomalies appeared and what might be done to remedy them going forward (if the root cause was something that requires a near-term change)
7. Add the results to the dashboard
8. Deliver the dashboard
9. [Recipient] Review the dashboard and the results
10. [Recipient] Decide whether to take action
11. [Recipient] If action will be taken, then take the action

Seems like a long list, right? I didn’t write it trying to split out separate steps and make it needlessly long. What’s interesting is that steps 1 and 2 can (and should!) be shortened through automation. Aside from systems that are delayed in making their data available, there is no reason that steps 1 and 2 can’t be done within hours (or a day) of the end of the reporting period.

Steps 3 through 7, though, are time-consuming. And, often, they require conversations and discussion — not to mention time to actually conduct analysis. Despite vendor-perpetuated myths that “the tool” can generate recommendations… tools really suck at doing so (outside of highly operationalized processes).

Here’s the other kicker, though: steps 9 through 11 take time, too! So, realistically, let’s say that steps 1 and 2 take a day, steps 3 through 8 take a week, steps 9 and 10 takes 3 days (because the recipient doesn’t drop everything to review the dashboard when it arrives), and then step 11 takes a week (because “action” actually requires marshalling resources and getting something done). That means — best case — we’re 2.5 weeks into the month before action gets taken.

So, what happens at the end of the month? The process repeats, but there was only 1.5 weeks of the change actually being in place… which could easily get dwarfed by the 2.5 weeks of the status quo!!!

Let’s look at how a “dashboard without insights” process can work:

1. Pull data (can’t happen until the end of the previous month)
2. Consolidate data to get it into the dashboard
3. Deliver the dashboard (possibly calling out any anomalies or missed targets)
4. [Recipient] Review the dashboard and hones in on anything that looks troubling that she cannot immediately explain (more on that in the next section)
5. The analyst and the recipient identify what, if any, trouble spots require deeper analysis and jointly develop actionable hypotheses to dig in
6. The analyst conducts a very focused analysis (or, in some cases, proposes an A/B test) and delivers the results.
7. [Recipient] If action is warranted, takes action

Time doesn’t stop for this process, either. But, it gets the information into the business’s hand inside of 2 days. The analyst doesn’t waste time discovering root causes that the business owner already knows (see the next section). The analysis that gets conducted is focused and actionable, and the business owner is already primed to take action, because she participated in determining what analyses made the most sense.

Reason No. 2: Analysts Aren’t Omniscient

I alluded to this twice in the prior paragraph. Let’s look at a generic and simplistic (but based on oft-observed real-world experience) example:

1. The analyst compiles the dashboard and sees that traffic is down
2. The analyst digs into the traffic sources and sees that paid search traffic is down dramatically
3. The analyst digs in further and sees that paid search traffic went to zero on the 14th of the month and stayed there
4. The analyst fires off an urgent email to the business that paid search traffic went to zero mid-month and that something must be wrong with the site’s SEM!
5. The business responds that SEM was halted mid-month due to budget adjustments, and they’ve been meaning to ask what impact that has had

What’s wrong with this picture? Steps 2 through 4 are largely wasted time and effort! There is very real analysis to be done… but it doesn’t come until step 5, when the business provides some context and is ready for a discussion.

This happens all the time. It’s one of the reasons that it is imperative that analysts build strong relationships with their marketing stakeholders, and one of the reasons that a sign of a strong analytics organization is one where members of the team are embedded – literally or virtually – in the teams they support.

But, even with a strong relationship, co-location with the supported team, regular attendance at the team’s recurring meetings, and a spot on the team’s email distribution list, analysts are seldom aware of every activity that might result in an explainable anomaly in the results delivered in a dashboard.

This gets to a data source that gets ignored all too often: the minds and memories of the marketing team. There is nothing at all wrong with an analyst making the statement: “Something unexpected happened here, and, after I did some cursory digging, I’m not sure why. Do you have any ideas as to what might have caused this?” There are three possible responses from the marketer who is asked this question:

• “I know exactly what’s going on. It’s almost certainly the result of X.”
• “I’m not sure what might have caused that, but it’s something that we should get to the bottom of. Can you do some more digging to see if you can figure it out?”
• “I’m not sure what might have caused that, but I don’t really care, either. It’s not important.”

These are quick answers to an easy question that can direct the analyst’s next steps. And, two of the three possible answers lead to a next step of moving onto a value-adding analysis — not pursuing a root cause that will lead to no action! Powerful stuff!

Reason No. 3: Insights Don’t have a Predictable and Consistent Length

I see it all the time: a standard dashboard format that, appropriately, has a consistent set of KPIs and supporting metrics carefully laid out in a very tightly designed structure. Somewhere in that design is a small box – at the top of the dashboard, at the bottom right of the dashboard, somewhere – that has room for a handful of bullet points or a short paragraph. This  area of the dashboard often has an ambitious heading: “Insights,” “Recommendations,” “Executive Summary.”

The idea – conceived either on a whiteboard with the initial design of the dashboard, or, more likely, added the first time the dashboard was produced – is that this is where the analysts real value will be manifested. THIS is where the analyst will place the Golden Nuggets of Wisdom that have been gleaned from the data.

Here’s the problem: some of these nuggets are a flake of dust, and some are full-on gold bars. Expecting insights to fit into a consistent, finite space week in and week out or month in and month out is naïve. Sometimes, the analyst has half a tweet’s worth of prose-worthy material to include, which makes for a largely empty box, leaving the analyst and the recipient to wonder if the analyst is slacking. At other times, the analyst has a handful of useful nuggets to impart…but then has to figure out how to distill a WordPress-sized set of information into a few tweet-sized statements.

Now, if you buy into my first two reasons as to why recommendations shouldn’t be included with the dashboard in the first place, then this whole section becomes moot. But, if not — if you or your stakeholders still insist that performance measurement include recommendations — then don’t constrain the space to include that information to a fixed box on the dashboard.

Reason No. 4: Insights Can’t Be Scheduled

A scene from The Marketer and the Analyst (it’s a gripping — if entirely fictitious — play):

Marketer: “This monthly dashboard is good. It’s showing me how we’re doing. But, it doesn’t include any insights based on the performance for the month. I need insights to take action!”

Analyst: “Well, what did you do differently this month from previous months?”

Marketer: “What do you mean?”

Analyst: “Did you make any changes to the site?”

Marketer: “Not really.”

Analyst: “Did you change your SEM investment or strategy?”

Marketer: “No.”

Analyst: “Did you launch any new campaigns?”

Marketer: “No.”

Analyst: “Were there any specific questions you were trying to answer about the site this month?”

Marketer: “No.”

Analyst: ???!

Raise your hand if this approximates an exchange you’ve had. It’s symptomatic of a completely ass-backward perception of analytics: that the data is a vast reserve of dirt and rock with various veins of golden insights threaded throughout. And, that the analyst merely needs to find one or more of those veins, tap into it, and then produce a monthly basket of new and valuable ingots from the effort.

The fact is, insights come from analyses, and analyses come from hypotheses. Some analyses are small and quick. Some are large and require gathering data – through an A/B or multivariate test, for instance, or through a new custom question on a site survey. Confusing “regularly scheduled performance measurement” with “hypothesis-driven analysis” has become the norm, and that is a mistake.

While it is absolutely fine to measure the volume and value of analyses completed, it is a recipe for failure to expect a fixed number of insights to be driven from and delivered with a scheduled dashboard.

A Final Word: Dashboards vs. Reports

Throughout this post, I’ve discussed “dashboards.” I’ve steered clear of the word “report,” because it’s a word that has become pretty ambiguous. Should a report include insights? It depends on how you define a report:

• If the report is the means by which, on a regularly scheduled basis, the performance of a [site/campaign/channel/initiative] is performing, then my answer is: “No.” Reasons 1, 2, and 4 explain why.
• If the report is the term used to deliver the results of a hypothesis-driven analysis (or set of hypothesis-driven analyses), then my answer is, “Perhaps.” But…why not call it “Analysis Results” to remove the ambiguity in what it is?
• If the report is intended to be a combination of both of the above, then you will likely be delivering a 25+ deck of rambling slides that — despite your adoration for the content within — is going to struggle to hold your audience’s attention and is going to do a poor job of both measuring performance and of delivering clearly actionable analysis results.

We live in a real-time world. Consumers — all marketers have come to accept — have short attention spans and consume content in bite-sized chunks. An effective analyst delivers information that is super-timely and is easily digestible.

So. Please. Don’t spend 3 weeks developing insights and recommendations to include on a 20-page document labeled “dashboard.”

Last week, I had the pleasure of presenting at a SEER Interactive conference titled “Marketing Analytics: Proving and Improving Online Performance.” The conference was at SEER’s main office, which is an old church (“old” as in “built in 1850” and “church” as in “yes…a church”) in the Northern Liberties part of Philadelphia. The space itself is, possibly, the most unique that I’ve presented in to date (photo courtesy of @mgcandelori — click to view the larger version…that’s real stained glass!):

As luck would have it, Michele Kiss attended the conference, which meant all of the speakers got a pretty nice set of 140-character notes on the highlights of what they’d said.

Reviewing the stream of tweets afterwards, I realized I’ve developed quite a list of aphorisms that I tend to employ time and again in analytics-oriented conversations. I’m sufficiently self-aware that I’ll often preface them with, “So, this is soapbox #23,” but, perhaps, not self-aware enough to not actually spout them!

The occasion of standing on an actual altar (SEER maintained much of the the space’s original layout) seemed like a good time to put together a partial catalog of my go-to one-liners. Enjoy!

Being data-driven requires People AND Process AND Technology

I beat this drum fairly often. It’s not enough to have a pristine and robust technology stack. Nor is it sufficient to have great data platforms and great analysts. I believe — firmly — that successful companies have to have a solid analytics process, too. Otherwise, those wildly-in-demand analysts sifting through exponentially growing volumes of data don’t have a prayer. Effective analytics has to be efficient analytics, and efficiency comes from a properly managed process for identifying what to test and analyze.

Identifying KPIs is nothing more than answering two question: 1) What are we trying to achieve? and 2) How will we know if we’ve done that?

I’ve got to credit former colleague Matt Coen for the clarity of these. I like to think I’ve done a little more than just brand them “the two magic questions,” but it’s possible that I haven’t! The point here is that business-speak is, possibly, more vile than Newspeak. “Goals” vs. “strategies” vs. “objectives” vs. “tactics” — these are all words that different people define in different ways. I actually have witnessed — on multiple occasions — debates between smart people as to whether something is an “objective” or a “strategy.”

As soon as we use the phrase “key performance indicator,” the acronym “KPI,” or the phrase “success measure,” we’re asking for trouble. So, whether I verbally articulate the two questions above, or whether I simply ask them of myself and try to answer them, I try to avoid business-speak:

1. What are we trying to achieve? Answer that question without data. It’s nothing more than the elevator pitch for an executive who, while making conversation while traveling from the ground floor to the 8th floor, asks, “What’s the purpose of <insert whatever you’re working on>?”
   
2. How will we know if we’ve done that? This question sometimes gets asked…but it skips the first question and invites spouting of a lengthy list of data points. As a plain English follow-on to the first question, though, it invites focus!

The K in KPI stands for “Key” — not for 1,000.

This one is a newer one for me, but I’ll be using it for a lonnnng time. All too often, “KPI” gets treated as a fancy-pants way to say “data” or “metrics” or “measures.” Sure, we feel like we’re business sophisticates when we can use fancy language…but that doesn’t mean that we should be using fancy language poorly! I covered this one in more detail in my last post…but I’m going to repeat the picture I used there, anyway, because it cracks me up:

A KPI is not a KPI if it doesn’t have a target.

“Visits” is not a KPI. Nor is “conversion rate.” Or “customer satisfaction.” A KPI is not a KPI without a target. Setting targets is an inexact science and is often an uncomfortable exercise. But…it’s not as hard as it often gets made out to be.

Human nature is to think, “If I set a target and I miss it…then I will be viewed as having FAILED!” In reality, that’s the wrong view of targets. If you work for a manager, a company, or a client where that is the de facto response…then you need to find a new job.

Targets set up a clear and objective way to: 1) ensure alignment on expectations at the outset of an effort, and 2) objectively determine whether you were able to meet those expectations. If you wildly miss a very-hard-to-set target, then you will have learned a lot more and will be better equipped to set expectations (targets) the next time.

This all leads into another of my favorites…

You’re never more objective about what you *might* accomplish than before you set out to achieve it.

“I have no idea and no expectations!” is almost always an unintentional lie. Somebody decided that time and energy would be spent on the campaign/channel/initiative/project. That means there was some expectation that it would be worthwhile to do so. And “worthwhile” means something. It’s really, really hard to, at the end of a 6-month redesign where lots of people pulled lots of long hours to hit the launch date, stand up and say, “This didn’t do as well as we’d hoped.” In the absence of targets, that never happens. The business owner or project manager or analyst automatically starts looking for ways to illustrate to the project team and to the budget owner that the effort paid off.

But, that’s short-sighted. For starters, without a target set up front, just about any reporting of success will carry with it a whiff of disingenuousness (“You’re telling me that’s good…but this is the first I’m hearing that we knew what ‘good’ would look like!”). And, the after-the-fact-looking-for-success means effort is spent looking backwards rather than minimal effort to look backwards so that real effort can go into looking forward: “Based on how we performed against our expectations (targets), what should we do next, and what do we expect to achieve?”

Any meaningful analysis is based on a hypothesis or set of hypotheses.

I’ve had the debate many times over the years as to whether there are cases where “data mining” means “just poking around in the data to see what patterns emerge.” In some cases, the person is truly misguided and believes that, with a sufficiently large data set and sufficiently powerful analytical tools, that is truly all that is needed: data + tools = patterns –> actionable insights. That’s just wrongheaded.

More often, though, the debate is an illustration that a lot of analysts don’t realize that, in reality, they and their stakeholders are actually testing hypotheses. Great analysts may subconsciously be doing that…but it’s happening. The more we recognize that that’s what we’re doing, the more focused and efficient we can be with our analyses!

Actionable hypotheses come from filling in the blanks on two questions: 1) I believe _______, and 2) If I’m right, I will ______.

Having railed against fancy business-speak…it’s really not all that cool of me to be floating the word “hypothesis” now, is it? In day-to-day practice…I don’t! Rather, I try to complete these two statements (in order) before diving into any hypothesis:

1. I believe [some idea]  — this actually is the hypothesis. A hypothesis is an assumption, an idea, a guess, a hunch, or a belief. Note that this isn’t “I know…” and it’s not even “I strongly believe…” It’s the lowest level of conviction possible, so we should be fine learning (quickly, and with data) when the belief is untrue!
   
2. If I am right, I will [take some action] — this isn’t actually part of the hypothesis. Rather, it’s a way to qualify the hypothesis by ensuring that it is sufficiently focused that, if the belief holds up, action could be taken. In my experience, taking one broad belief and breaking it down into multiple focused hypotheses leads to much more efficient and actionable analysis.

Like the two magic questions, I don’t necessarily force my clients to use this terminology. I’ll certainly introduce it when the opportunity arises, but, as an analyst, I always try to put requests into this structure. It helps not only focus the analysis (and, often, promote some probing and clarification before I dig into the time-intensive work of pulling and analyzing the data), but focus the output of the analysis in a way that makes it more actionable.

Do you have favorite analytics aphorisms?

I’d love to grow my list of meaningful analytics one-liners. Do you have any you use or have heard that you really like?

The bane of many analysts’ existence is that they find themselves in a world where the majority of their day is spent on the receiving end of a steady flow of vague, unfocused, and misguided requests:

“I don’t know what I don’t know, so can you just analyze the traffic to the site and summarize your insights?”

“Can I get a weekly report showing top pages?”

“I need a report from Google Analytics that tells me the gender breakdown for the site.”

“Can you break down all of our metrics by: new vs. returning visitors, weekend vs. weekday visitors, working hours vs. non-working hours visitors, and affiliate vs. display vs. paid search vs. organic search vs. email visitors? I think there might be something interesting there.”

“Can you do an analysis that tells me why the numbers I looked at were worse this month than last?”

“Can you pull some data to prove that we need to add cross-selling to our cart?”

“We rolled out a new campaign last week. Can you do some analysis to show the ROI we delivered with it?”

“What was traffic last month?”

“I need to get a weekly report with all of the data so I can do an analysis each week to find insights.”

The list goes on and on. And, in various ways, they’re all examples of well-intended requests that lead us down the Nefarious Path to Reporting Monkeydom. It’s not that the requests are inherently bad. The issue is that, while they are simple to state, they often lack context and lack focus as to what value fulfilling the request will deliver. That leads to the analyst spending time on requests that never should have been worked on at all, making risky assumptions as to the underlying need, and over-analyzing in an effort to cover all possible bases.

I’ve given this a lot of thought for a lot of years (I’m not exaggerating — see the first real post I wrote on this blog almost six years ago…and then look at the number of navel-gazing pingbacks to it in the comments). And, I’ve become increasingly convinced that there are two root causes for not-good requests being lobbed to the analytics team:

• A misperception that “getting the data” is the first step in any analysis — a belief that surprising and actionable insights will pretty much emerge automagically once the raw data is obtained.
• A lack of clarity on the different types and purposes of analytics requests — this is an education issue (and an education that has to be 80% “show” and 20% “tell”)

I think I’m getting close to some useful ways to address both of these issues in a consistent, process-driven way (meaning analysts spend more time applying their brainpower to delivering business value!).

Before You Say I’m Missing the Point Entirely…

The content in this post is, I hope, what this blog has apparently gotten a reputation for — it’s aimed at articulating ideas and thoughts that are directly applicable in practice. So, I’m not going to touch on any of the truths (which are true!) that are more philosophical than directly actionable:

• Analysts need to build strong partnerships with their business stakeholders
• Analysts have to focus on delivering business value rather than just delivering analysis
  
• Analysts have to stop “presenting data” and, instead “effectively communicate actionable data-informed stories.”

All of these are 100% true! But, that’s a focus on how the analyst should develop their own skills, and this post is more of a process-oriented one.

With that, I’ll move on to the three types of analytics requests.

Hypothesis Testing: High Value and SEXY!

Hands-down, testing and validation of hypotheses is the sexiest and, if done well, highest value way for an analyst to contribute to their organization. Any analysis — regardless of whether it uses A/B or multivariate testing, web analytics, voice of the customer data, or even secondary research — is most effective when it is framed as an effort to disprove or fail to disprove a specific hypothesis. This is actually a topic I’m going to go into a lot of detail (with templates and tools) on during one of the eMetrics San Francisco sessions I’m presenting in a couple of weeks.

The bitch when it comes to getting really good hypotheses is that “hypothesis” is not a word that marketers jump up and down with excitement over. Here’s how I’m starting to work around that: by asking business users to frame their testing and analysis requests in two parts:

Part 1: “I believe…[some idea]”

Part 2: “If I am right, we will…[take some action]”

This construct does a couple of things:

• It forces some clarity around the idea or question. Even if the requestor says, “Look. I really have NO IDEA if it’s ‘A’ or ‘B’!” you can respond with, “It doesn’t really matter. Pick one and articulate what you will do if that one is true. If you wouldn’t do anything different if that one is true, then pick the other one.”
• It forces a little bit of thought on the part of the requestor as to the actionability of the analysis.

And…it does this in plain, non-scary English.

So, great. It’s a hypothesis. But, how do you decide which hypotheses to tackle first? Prioritization is messy. It always is and it always will be. Rather than falling back on the simplistic theory of “effort and expected impact” for the analysis, how about tackling it with a bit more sophistication:

• What is the best approach to testing this hypothesis (web analytics, social media analysis, A/B testing, site survey data analysis, usability testing, …)? That will inform who in your organization would be best suited to conduct the analysis, and it will inform the level of effort required. 
• What is the likelihood that the hypothesis will be shown to be true? Frankly, if someone is on a fishing expedition and has a hypothesis that making the background of the home page flash in contrasting colors…common sense would say, “That’s a dumb idea. Maybe we don’t need to prove it if we have hypotheses that our experience says are probably better ones to validate.”
  
• What is the likelihood that we actually will take action if we validate the hypothesis? You’ve got a great hypothesis about shortening the length of your registration form…but the registration system is so ancient and fragile that any time a developer even tries to check the code out to work on it, the production code breaks. Or…political winds are blowing such that, even if you prove that always having an intrusive splash page pop up when someone comes to your home page is hurting the site…it’s not going to change.
• What will be the effort (time and resources) to validate the hypothesis? Now, you damn well better have nailed down a basic approach before answering this. But, if it’s going to take an hour to test the hypothesis, even if it’s a bit of a flier, it may be worth doing. If it’s going to take 40 hours, it might not be.
• What is the business value if this hypothesis gets validated (and acted upon)? This is the “impact” one, but I like “value” over “impact” because it’s a little looser.

I’ve had good results when taking criteria along these lines and building a simple scoring system — assigning High, Medium, Low, or Unknown for each one, and then plugging in some weighted scores for each value for each criteria. The formula won’t automatically prioritize the hypotheses, but it does give you a list that is sortable in a logical way, It, at least, reveals the “top candidates” and the “stinkers.”

Performance Measurement (think “Reporting”)

Analysts can provide a lot of value by setting up automated (or near-automated) performance measurement dashboards and reports. These are recurring (hypothesis testing is not — once you test a hypothesis, you don’t need to keep retesting it unless you make some change that makes sense to do so).

Any recurring report* should be goal- and KPI-oriented. KPIs and some basic contextual/supporting metrics should go on the dashboard, targets need to be set (and set up such that alerts are triggered when a KPI slips). Figuring out what should go on a well-designed dashboard comes down to answering two questions:

1. What are we trying to achieve? (What are our business goals for this thing we will be reporting on?)
2. How will we know that we’re doing that? (What are our KPIs?)

They need to get asked and answered in order, and that’s a messier exercise oftentimes than we’d like it to be. Analysts can play a strong role in getting these questions appropriately answered…but that’s a topic for another time.

Every other recurring report that is requested should be linkable back to a dashboard (“I have KPIs for my paid search performance, so I’d like to always get a list of the keywords and their individual performance so I have that as a quick reference if a KPI changes drastically.”)

Having said that, a lot of tools can be set up to automatically spit out all sorts of data on a recurring basis. I resist the temptation to say, “Hey…if it’s only going to take me 5 minutes to set it up, I shouldn’t waste my time trying to validate its value.” But, it can be hard to not appear obstructionist in those situations, so, sometimes, the fastest route is the best. Even if, deep down, you know you’re delivering something that will get looked at the first 2-3 times it goes out…and will never be viewed again.

Quick Data Requests — Very Risky Territory (but needed)

So, what’s left? That would be requests of the,. “What was traffic to the site last month?” ilk. There’s a gross misperception when it comes to “quick” requests that there is a strong correlation between the amount of time required to make the request and the amount of time required to fulfill the request. Whenever someone tells me they have a “quick question,” I playfully warn them that the length of the question tends to be inversely correlated to the time and effort required to provide an answer.

Here’s something I’ve only loosely tested when it comes to these sorts of requests. But, I’ve got evidence that I’m going to be embarking on a journey to formalize the intake and management of these in the very near future, so I’m going to go ahead and write them down here (please leave a comment with feedback!).

First, there is how the request should be structured — the information I try to grab as the request comes in:

• The basics — who is making the request and when the data is needed; you can even include a “priority” field…the rest of the request info should help vet out if that priority is accurate.
• A brief (255 characters or so) articulation of the request — if it can’t be articulated briefly, it probably falls into one of the other two categories above. OR…it’s actually a dozen “quick requests” trying to be lumped together into a single one. (Wag your finger. Say “Tsk, tsk!”
• An identification of what the request will be used for — there are basically three options, and, behind the scenes, those options are an indication as to the value and priority of the request:
  • General information — Low Value (“I’m curious,” “It would be be interesting — but not necessarily actionable — to know…”)
  • To aid with hypothesis development — Medium Value (“I have an idea about SEO-driven visitors who reach our shopping cart, but I want to know how many visits fall into that segment before I flesh it out.”)
  • To make a specific decision — High Value
• The timeframe to be included in the data — it’s funny how often requests come in that want some simple metric…but don’t say for when!
• The actual data details — this can be a longer field; ideally, it would be in “dimensions and metrics” terminology…but that’s a bit much to ask for many requestors to understand.
• Desired delivery format — a multi-select with several options:
  • Raw data in Excel
  • Visualized summary in Excel
  • Presentation-ready slides
  • Documentation on how to self-service similar data pulls in the future

The more options selected for the delivery format, obviously, the higher the effort required to fulfill the request.

All of this information can be collected with a pretty simple, clean, non-intimidating intake form. The goal isn’t to make it hard to make requests, but there is some value in forcing a little bit of thought rather than the requestor being able to simply dash off a quickly-written email and then wait for the analyst to fill in the many blanks in the request.

But that’s just the first step.

The next step is to actually assess the request. This is the sort of thing, generally, an analyst needs to do, and it covers two main areas:

• Is the request clear? If not, then some follow-up with the requestor is required (ideally, a system that allows this to happen as comments or a discussion linked to the original request is ideal — Jira, Sharepoint, Lotus Notes, etc.)
• What will the effort be to pull the data? This can be a simple High/Medium/Low with hours ranges assigned as they make sense to each classification.

At that point, there is still some level of traffic management. SLAs based on the priority and effort, perhaps, and a part of the organization oriented to cranking out those requests as efficiently as possible.

The key here is to be pretty clear that these are not analysis requests. Generally speaking, it’s a request for data for a valid reason, but, in order to conduct an analysis, a hypothesis is required, and that doesn’t fit in this bucket.

So, THEN…Your Analytics Program Investment

If the analytics and optimization organization is framed across these three main types of services, then conscious investment decisions can be made:

• What is the maximum % of the analytics program cost that should be devoted to Quick Data Requests? Hopefully, not much (20-25%?).
• How much to performance measurement? Also, hopefully, not much — this may require some investment in automation tools, but once smart analysts are involved in defining and designing the main dashboards and reports, that is work that should be automated. Analysts are too scarce for them to be doing weekly or monthly data exports and formatting.
• How much investment will be made in hypothesis testing? This is the highest value

With a process in place to capture all three types of efforts in a discrete and trackable way enables reporting back out on the value delivered by the organization:

• Hypothesis testing — reporting is the number of hypotheses tested and the business value delivered from what was learned
• Performance measurement — reporting is the level of investment; this needs to be done…and it needs to be done efficiently
• Quick data requests — reporting is output-based: number of requests received, average turnaround time. In a way, this reporting is highlighting that this work is “just pulling data” — accountability for that data delivering business value really falls to the requestors. Of course, you have to gently communicate that or you won’t look like much of a team player, now, will you?

Over time, shifting an organization to think it terms of actionable and testable hypotheses is the goal — more hypotheses, fewer quick data requests!

And, of course, this approach sets up the potentially to truly close the loop and follow through on any analysis/report/request delivered through a Digital Insight Management program (and, possibly, platform — like Sweetspot, which I haven’t used, personally, but which I love the concept of).

What Do You Think?

Does this make sense? It’s not exactly my opus, but, as I’ve hastily banged it out this evening, I realize that it includes many of the ways that I’ve had the most success in my analytics career, and it includes many of the structures that have helped me head off the many ways I’ve screwed up and had failures in my analytics career.

I’d love your thoughts!

*Of course, there are always valid exceptions.

Several weeks ago, Stéphane Hamel wrote a post that got me all re-smitten with his thought process. In the post, he postulated that there are three heads of online analytics. He covered three different skillsets needed to effectively conduct online analytics: business acumen, technical (tools) knowledge, and analysis. And, he made the claim that no one person will ever excel at all three, which led to his case for building out teams of “analysts” who have complementary strengths.

I’ve had several unrelated experiences with different clients and internal teams of late that have led me to try to capture, in a similar fashion, the three-legged stool of an online analytics program. Just as others have started tacking on additional components to Stéphane’s three skillsets, I’m sure my three-legged stool will quickly become a traditional chair…then some sort of six-legged oddity. But, I’d be thrilled if I could consistently communicate the basics to my non-analyst co-workers and clients:

I hold to a pretty strict distinction between “measurement and reporting” and “analysis,” and I firmly believe there is value in “reporting,” as long as that reporting is appropriately set up and applied.

Just as I believe that reporting should generally occur either as a one-time event (campaign wrap-up, for instance) or at regular intervals, I firmly believe that testing and analysis should not be forced into a recurring schedule. It’s fine (desirable) to be always conducting analysis, but the world of “present the results of your analysis — and your insights and recommendations therein — once/month on the first Wednesday of the month” is utterly asinine. Yet…it’s a mindset with which a depressing majority of companies operate.

Reporting Done Poorly…Which Is an Unfortunately Ubiquitous Habit

I’ve been client side. I’ve been agency side. I’ve done a decent amount of reading on human nature as it relates to organizational change. My sad conclusion:

The business world has conditioned itself to confuse “cumbersome decks of data” with “reporting done well.”

It happens again and again. And again. And…again! It goes like this:

1. Someone asks for some data in a report
2. Someone else pulls the data
3. The data raises some additional questions, so the first person asks for more data.
4. The analyst pulls more data
5. The initial requestor finds this data useful, so he/she requests that the same data be pulled on a recurring schedule
6. The analyst starts pulling and compiling the data on a regular schedule
7. The requestor starts sharing the report with colleagues. The colleagues see that the report certainly should be useful, but they’re not quite sure that it’s telling them anything they can act on. They assume that it’s because there is not enough data, so they ask the analyst to add in yet more data to the report
8. The report begins to grow.
9. The recipients now have a very large report to flip through, and, frankly, they don’t have time month in and month out to go through it. They assume their colleagues are, though, so they keep their mouths shut so as to not advertise that the report isn’t actually helping them make decisions. Occasionally, they leaf through it until they see something that spikes or dips, and they casually comment on it. It shows that they’re reading the report!
10. No one tells the analyst that the report has grown too cumbersome, because they all assume that the report must be driving action somewhere. After all, it takes two weeks of every month to produce, and no one else is speaking up that it is too much to manage or act on!
11. The analyst (now a team of analysts) and the recipients gradually move on to other jobs at other companies. At this point, they’re conditioned that part of their job is to produce or receive cumbersome piles of data on a regular basis. Over time, it actually seems odd to not be receiving a large report. So, if someone steps up and asks the naked emperor question: “How are you using this report to actually make decisions and drive the business?”…well…that’s a threatening question indeed!

In the services industry, there is the concept of a “facilitated good.” If you’re selling brainpower and thought, the theory goes, and you’re billing out smart people at a hefty rate, then you damn well better leave behind a thick binder of something to demonstrate that all of that knowledge and consultation was more than mere ephemera!

And, on the client side, if the last 6 consultancies and agencies that you worked with all diligently delivered 40-slide PowerPoint decks or 80-page reports, then, by golly, you’re going to look askance at the consultant who shows up and aims for actionable concision!

Nonetheless, I will continue my quixotic quest to bring sanity to the world. So, onto the three legs of my analytics stool…

First, Plan (Dammit!!!)

Get a room full of experienced analysts together and ask them where any good analytics program or initiative starts, and you’ll get a unanimous response that it starts: 1) at the beginning of the initiative, and 2) with some form of rigorous planning.

The most critical question to answer during analytics planning is: “How are we going to know if we’re successful?” Of course, you can’t answer that question if you haven’t also answered the question: “What are we trying to accomplish?” Those are the two questions that I wrote about in this Getting to Great KPIs post.

Of course, there are other components of analytics planning:

• Where will the data come from that we’ll use?
• What other metrics — beyond the KPIs — will we need to capture?
• What additional data considerations need to be factored into the effort to ensure that we are positioned for effective analysis and optimization down the road?
• What (if any) special tagging, tracking, or monitoring do we need to put into place (and who/how will that happen)?
• What are the known limitations of the data?
• What are our assumptions about the effort?
• …and more

In my experience both agency-side and client-side, this step regularly gets skipped like it’s a smooth round, rock in the hand of an adolescent male standing on the shore of a lake on a windless day.

An offshoot of the planning is the actual tagging/tracking/monitoring configuration…but I consider that an extension of the planning, as it may or may not be required, depending on the nature of the initiative.

Next, Measure and Report

Yup. Measurement’s important. That’s how you know if you’re performing at, above, or below your KPIs:

Here’s where I start to get into debates, both inside the analytics industry and outside. I strongly believe that it is perfectly acceptable to deliver reports without accompanying insights and analysis. Ideally, reports are automated. If they’re not automated, they’re produced damn quickly and efficiently.

Dashboards — the most popular form of reports — have a pretty simple purpose: provide an at-a-glance view of what has happened since the last update, and ensure that, at a glance, any anomalies jump out. More often than not, there won’t be anomalies, so there is nothing that needs to be analyzed based on the report! That’s okay!

I was discussing this concept with a co-worker recently, and, in response to my claim that reports should simply get delivered with minimal latency and, at best, a note that says, “Hey, I noticed this apparent anomaly that might be important. I’m going to look into it, but if you (recipient) have any ideas as to what might be going on, I’d love to get your thoughts,” she responded:

I think this makes sense, but wouldn’t we provide some analysis as to the “why” on the monthly reports?

I immediately went to the “dashboard in your car” analogy (I know — it breaks down on a lot of fronts, but it works here) with my response:

You don’t look at your fuel gauge when you get in the car every day and ask, “Why is the needle pointing where it is?” You take a quick look, make sure it’s not pegged on empty, and then go about your day.

That’s measurement. It may spawn analysis, but, often, it does not. And that’s to be expected!

Which Brings Us to Testing and Analysis

Analysis requires (or, at least, is much more likely to yield value in an efficient manner) having conducted some solid planning and having KPI-centric measurement in place. But, the timing of analysis shouldn’t be forced into a fixed schedule.

The bottom part of the figure above gets to the crux of the biscuit when it comes to timing: sometimes, the best way to answer a business question is through analyzing historical data. Sometimes, the best way to answer a question is through go-forward testing. Sometimes, it’s a combination of the two (develop a theory based on the historical data, but then test it by making a change in the future and monitoring the results). Sometimes the analysis can be conducted very quickly. Other times, the analysis requires a large chunk of analyst time and may take days or weeks to complete.

Facilitating the collaboration with the various stakeholders, managing the analysis projects (multiple analyses in flight at once — starting and concluding asynchronously based on each effort’s unique nature), can absolutely fall under the purview of the analyst (again referencing Stéphane’s post, this should be an analyst with a strong “head” for business acumen).

In Conclusion…(I promise!)

There is a fundamental flaw in any approach to using data that attempts to bundle scheduled reporting with analysis. It forces efforts to find “actionable insights” in a context where there may very well be none. And, it perpetuates an assumption that it’s simply a matter of pointing an analyst at data and waiting for him/her to find insights and make recommendations.

I’ve certainly run into business users who flee from any effort to engage directly when it comes to analytics. They hide behind their inboxes lobbing notes like, “You’re the analyst. YOU tell me what my business problem is and make recommendations from your analysis!” I’m sure some of these users had one too many (and one is “too many”) interactions with an analyst who wanted to explain the difference between a page view and a visit, or who wanted to collaboratively sift through a 50-page deck of charts and tables. That’s not good, and that analyst should be flogged (unless he/she is less than two years out of college and can claim to have not known any better). But, using data to effectively inform decisions is a collaborative effort. It needs to start early (planning), it needs to have clear, concise performance measurement (KPI-driven dashboards), and it needs to have flexibility to drive the timing and approach of analyses that deliver meaningful results.