A good friend of mine from my youth (and still today) read my first real post on this blog, and it really resonated with him — he’s commented offline about it a couple of times over the past couple of months. He pinged me today asking for some resources that would elaborate on the subject. My kneejerk reaction was: there ain’t none — it’s not a distinction people make.

But, I did a quick Google search, anyway. Immediately, I turned up a post by a highly credible person — Jim Novo. Jim’s been absent from the Yahoo! webanalytics forum of late, but that’s where I first read him, and he’s got experience and considerable smarts.

Turns out, he had a post from February of this year (2007) on the exact subject of Reporting vs. Analysis. The first thing I noticed was that he referenced a post from Avinash Kaushik that touches on the subject as well. This really is a small world, apparently (see my last entry in this blog…where I referenced an Avinash post!).

When I skimmed Jim’s article, my initial reaction was that he was indeed using “reporting” and “analysis” in a different way than I define them. But, on a closer read, I don’t think that’s the case. He (and, by extension, Eric Peterson) really is calling a report something that you monitor to see if you’re doing what you want (maintaining the status quo or driving improvements), whereas analysis is about trying ot understand what’s going on. Interesting stuff. I feel validated. 😉

Rack that up as one of those popular, throwaway cliches, stated with a ho-hum air as if to say, “It’s so factual and irrefutable that I can’t believe I’m wasting my body’s energy pumping carbon dioxide converted from oxygen into the atmosphere to say it.”

Drives me nuts.

My personal fantasy? Anyone who makes this statement is banned from accessing or using any data for a year.

Why? Because, as stated this way, it fairly directly implies that any sort of data analysis is just a way to drive someone’s agenda or spin the results of an initiative. And, data can absolutely be used to do this. But it doesn’t have to be.

While I may sound like a broken record, I hope that I more sound like a well-produced album, with a selection of tunes that, while all on a similar theme, approach that theme from various angles.

So, two ways to come at the, “whatever you want it to” comment.

Situation 1
Someone makes this statement when discussing data they have looked at or the results of an analysis that was undertaken, directly or indirectly, at their behest. I had this happen today, and the fellow’s initial beef was that the analysis that we had done did not aggressively and vividly back up his own strongly held beliefs about a certain business situation. What he was looking for was an analysis that simply supported his assertion as to the current state of affairs. What temperature does blood boil at? When he dropped the, “You can make data…” comment, it was an intellectual slap in the face of sorts (not that he saw it that way — this is not a fellow who is particularly self-aware as to the impact of his words, and he was by no means trying to insult…um…my chosen career). I pushed back (rather calmly; please hold while I pat myself on the back) that, if he already knew what he wanted the data to say, and if he was just going to push for multiple iterations on the analysis until it said that, then what’s the point? Analysis should be all about answering questions. In this sort of situation, the only question is, “Can I dip my advocacy brush into a bucket of data and paint the picture that already exists in my mind’s eye?”

Situation 2
This is the case when someone presents data, and someone that is seeing it presented doesn’t buy into what it supposedly supports. Ironically, the person who points out the spinnability of data is the same person who would spin it for his/her own purposes (see Situation 1). This is, quite simply, sad. It’s a waste of a company’s money to pay for someone to take this on. More often than not, though, what happens in this situation is that the user/presenter of the data simply didn’t know how to effectively use the data. It’s sooooo tempting to start with the data. If you do that, you have an infinite number of ways to pivot it, plot it, and print it. And, after making several dozen charts, and realizing you’ve got a real snoozer in the works if you present all of them, you narrow down to 2 or 3 that seem relevant. And, human nature says, the more they seem to show something positive, the more of a relevancy boost they’ll get. The way to avoid this is to have the pre-data discipline to articulate what you’re trying to get at. Publicize those (or, at least, write them down for yourself — it will keep you on the right track!). Doggonit! Seems like half of the songs in this blog-album wind up referring back to one of my first posts. It’s a “how to avoid Situation 2” prescription.

Yes, you can make the data say whatever you want. But, that’s an awfully jaded view of the world. And, with some proper up-front discipline, you won’t be wasting your time trying to make it say what you want, and you won’t be providing much of an opportunity for the people who are receiving the analysis or report to have that cliche pop into their heads.

I’ve been offline for a while, primarily because I’ve been in the process of relocating my family from Austin, TX to Dublin, OH (Columbus, OH, basically). The last part of that relocation was me driving the entire trip in one shot with our two labs. My wife and our three kids had left two weeks earlier, taken a meandering trip up, then handled the closing on our Dublin house and the movers showing up and getting everything moved in.

Things did work out well for me there.

But, to the data geek front:

I decided that I’d go for a one-day trip, with a planned “out” to get a hotel if things got unsafe fatigue-wise. According to Google, the door-to-door trip was 1,272 milies and would take about 19 hours and 43 minutes. I brought along the driving directions, which I followed to a T, and logged actual elapsed times and mileage at various waypoints along the journey.

	Miles
Waypoint	Google	Actual	% Difference
Depart Dripping Springs, TX	0.0	0.0	–
Exit I-35E onto I-30 E in Dallas, TX	215.5	203.0	-5.8%
Exit I-30 onto I-440 E in Little Rock, AR	531.5	505.4	-4.9%
Exit I-440 onto I-40 E in Little Rock, AR	541.5	515.0	-4.9%
Stay on I-40 E in Memphis, TN	681.4	649.0	-4.8%
Stay on I-65 N as leaving Nashville, TN	880.5	840.0	-4.6%
Take I-71 N in Louisville, KY	1056.8	1008.0	-4.6%
Stay on I-71 N in Cincinnati, OH	1128.4	1095.2	-2.9%
Arrive Dublin, OH	1272.5	1215.3	-4.5%

Not a bad variance, actually. And, who knows? Maybe the odometer on my truck is off. The fact is, I didn’t particularly care how far I drove — the critical factor was how long I drove. So, how did Google do on the timing front? Same waypoints, but cumulative elapsed times: Google’s estimate versus the digital clock in my truck:

	Cum. Hours
Waypoint	Google	Actual	% Difference
Depart Dripping Springs, TX	0.0	0.0	–
Exit I-35E onto I-30 E in Dallas, TX	3.5	3.1	-9.6%
Exit I-30 onto I-440 E in Little Rock, AR	8.2	8.0	-1.8%
Exit I-440 onto I-40 E in Little Rock, AR	8.3	8.2	-2.0%
Stay on I-40 E in Memphis, TN	10.4	10.1	-2.9%
Stay on I-65 N as leaving Nashville, TN	13.3	13.2	-1.1%
Take I-71 N in Louisville, KY	16.1	15.9	-1.4%
Stay on I-71 N in Cincinnati, OH	17.2	17.2	0.0%
Arrive Dublin, OH	19.6	19.5	-0.8%

Holy. COW! I arrived 10 minutes earlier than Google predicted after driving over 1,200 miles!

Now, to be a good data analyst, I’m going to have to assume that, if I made this same drive 100 times, I wouldn’t hit it within 10 minutes all that often. There are simply too many variables. What would the standard deviation of my total trip times be, though, I wonder? At a minimum, we’ll have to wait until gas prices come down considerably and until I slip into some sort of mild dementia to find that out. 19 hours of driving…solo…in one day was fairly brutal, and not the safest of things to do!

I heard this again today. It’s a mini-mantra in my current company, and I couldn’t disagree more.

There’s a fairly famous Albert Einstein quote: “Not everything that can be counted counts, and not everything that counts can be counted.” (It’s also sometimes quoted as: “Everything that counts cannot be counted and everything that can be counted does not count.” Same difference, and I have no idea which, if either, is precisely correct). Supposedly, this hung in his office. It’s hung in my office — prominently — for several years.

All too often, it seems like we shy away from setting objectives if we can’t think of a way to easily measure them. Then, in practice, we try to achieve those objectives, anyway, because we just know it’s the right thing to do. I’m a firm, firm, firm believer in having a clear (and clearly articulated) vision, then developing a strategy for achieving that vision. Let that strategy be the guiding principle — not the measurability of your day-to-day actions.

Could I be more vague?

I love that Ben and Jerry’s has held firm to their quirky, environmentally conscientious vision for the entire life of their company. Sure, they measure the return on specific flavors, and they measure the effectiveness of their marketing campaigns. Do they try to tightly measure the effectiveness of their brand? I don’t know…but I doubt it. Their brand is driven by who they are and who they will always be. That counts, but is very, very hard to measure. Yet, it is at the core of their operations year in and year out.

At a more tactical level, we’ve been doing a lot of work around building nurturing programs. There absolutely has to be a core belief that these programs directly add value for the prospect who is being nurtured. If they do that well, then they will drive more business to the company that is doing the nurturing. It’s darn near impossible to measure that value-add to the prospect, so there can be a drift, over time, of just focussing on the business that the nurturing is driving. Proceed with caution! If the unmeasurable — what can’t be counted — aspects of the program do not remain core to every decision, then what can be counted may start to suffer, as “nurturing” starts evolving into “spam.”

I spent all week in partner training for Eloqua, which is one of the premier marketing automation companies. My company has used them for ourselves as well as for our clients for several years with great results. We’ve recently deepened our partnership so that we can actually set up and manage instances of the system for our clients, which is a pretty exciting proposition.

Over the course of the week, we spent 3-4 hours on various aspects of reporting in the tool. I’d done minimal poking around in the various Eloqua reports prior to the training (my focus to this point has been more on learning the guts of Salesforce.com and our internal systems and how they apply to our processes), and, to be honest, I’d been pretty frustrated.

One of the most annoying aspects of their reporting was that the majority of the reports I looked at were based on selecting a given “time span:” last day, last 2 days, last week, last month, etc. I was usually looking for trend data — how many visits to our Web site by week over the past few months, how many e-mail opens or clickthroughs, etc. Occasionally, I’d find myself buried in the interface at a point where I could specify a time range, but it always seemed like the available reports were pretty limited.

After the training, I have a better understanding of their (wildly confusing) user interface…so I think I can get to many more time range reports now.

More importantly, I had an epiphany this afternoon.

First and foremost, products like Eloqua and Salesforce.com (and Oracle applications, and Siebel, and SAP, and you-get-the-idea) are process automation/optimization tools. This wasn’t the epiphany. It’s just a fact. But, while it’s true that processes generate data, it’s wildly naive to think that a perfectly efficient, perfectly functioning process produces perfectly accessible data. As a matter of fact, there seems to be almost a negative correlation between these two areas. But that’s really a topic for another post.

In an earlier post, I wrote about the difference between metrics and analysis. I spent a couple of paragraphs on what I call “operational reporting” (I didn’t coin the term…but I use it!). The definition of an operational report is that it is part of a defined process: an invoice is part of a billing process, for example. Eloqua’s reporting is really oriented towards operational reporting. For instance, “I sent out an invite to a webinar yesterday, and I need the detail of who opened it, who clicked through on it, who registered for the event, and who didn’t.” It makes sense that “time span” reporting would be used here.

I, however, had been looking more at possible metrics reports and data for analysis. Time range reports become much more useful — even more useful if time increments can be a dimension in the report. I don’t know for sure, but I suspect time range reporting was not initially available in Eloqua. As phenomenal as the tool is…it’s got some definite shortcomings on the data-for-metrics and data-for-analysis front, IMHO.

But, at least I now have a framework to work within when I’m trying to get at that data in a meaningful way.

In my mind, all too often, we erroneously equate “reporting” with “analysis.” This can lead to a lot of cycles of spinning confusedly through reams of data or, worse, the belief that we “took action from the data” just because we converted a spreadsheet into a chart.

A former colleague of mine, Shane Stephens, and I sat down a few years ago and decided that there are really three different ways to use data:

Operational Reporting

This is when data is being reported at a high frequency and, often, at a very granular level with a discretely defined role in a given process. A daily report of all bookings from the prior day for a given salesperson’s territory is one example (it’s only a good example if his/her process includes reviewing that list each day and following up with any customers that he needs to check in on once they have placed an order). A call center report that breaks down wait times by different controllable factors is another example — used for adjusting staffing throughout the week, for instance. We even included an invoice-“printing” system as an operational report — it’s a highly detailed, highly structured report that gets sent to a customer letting him/her know what payment is due.

That’s all I’ll write about operational reporting — in a lot of ways, it’s pretty simple. Trouble arises, though, when someone starts to repurpose such a report: “I get a daily detail report of bookings for my region, so I’m just going to combine all of those into a spreadsheet to see what my bookings to date for the quarter are.” Or, same compilation, but, “…so I can analyze sales in my territory.” This winds up being darn cumbersome and can create all sorts of issues with data interpretation and application. Maybe I’ll come back to that later.

Metrics Reporting

Metrics reporting typically has aggregated data: total bookings for a territory, total bookings for the company, lead-to-sales conversion rate, etc. Key Performance Indicators (KPIs) are always metrics, but all metrics aren’t necessarily KPIs. Metrics are very different from operational reports. And, they’re a lot easier to turn into vortexes of wasted energy.

There are at least two ways that I’ve seen people get into a death spiral with metrics:

1. Confusing metrics with analysis. They’re wildly different…which should become evident by the end of this post.
2. Starting with the data when determining metrics instead of starting with objectives

I’ll tackle the latter first. The “easy” way to get data quickly is to start out by asking what data is easily available, and then choosing your metrics from that list. This is just wrong, wrong, WRONG! It’s tempting to do, and even experienced analysts who know what a slippery slope this is can easily fall into the trap. But, it’s still WRONG! (not that I have strong opinions here…)

In the long run, the right place to start when determining metrics is with what you’re trying to accomplish in business terms rather than data terms: “We’re trying to improve the effectiveness of our direct marketing efforts,” “We’re trying to grow the company,” “We’re trying to make the company more profitable,” “We’re trying to improve the user experience on our Web site.” A couple of these teeter on the edge of sounding like being in “data terms.” For instance, isn’t “grow the company” the same as “increase revenue?” Maybe. Maybe not.

The next step is to tighten up the definitions of what your objectives are. Still, stay away from thinking about the data. Think about how you would explain to your spouse, a friend, or a peer in another department what it is you are really trying to accomplish.

Random aside: A lot of business articles and books claim that, to establish good metrics, you have to start at the very top level of the company and then drill down to more detailed/granular metrics. That’s one of the fundamental premises for the Balanced Scorecard, I think. I like a lot about the balanced scorecard approach…but not this piece of it. It may work for some companies, but, in my experience, it’s just too much to try to start at the highest level and then drill all the way down to get any metrics. Rather, if the top levels of an organization have clearly articulated the company’s vision, strategy, and high level tactics, I’m all for empowering individual departments to figure out what they should be achieving and then starting there with the metrics. This does mean there needs to be some validation of metrics once they’re settled upon in order to ensure alignment. But, I’ve never seen a department (or even a project — project metrics should follow the same approach) that doesn’t get 85% of the way there by just knowing the company and understanding their role and then working through their own proposed metrics.

Back to the main point on metrics. Once you have really, really clear, tight objectives, you can sit back and brainstorm on how to measure your progress towards them. What you’ll find is that there are some objectives that can be measured very easily with a single metric. But, with other objectives, there will not be a perfect metric. In those cases, you can shoot for one or more proxies for the objective. This is actually a good sign — it means you’ve got some clear objectives that are hard to measure. That’s a damn sight better than having clear metrics with objectives that are hard to articulate!

You’re not quite done at this point with metrics. It’s absolutely critical that you set targets for each metric. It’s really tempting to want to “just measure it for a while, because we don’t even have a baseline.” Resist the temptation! If you can get some set of historical data within the next day or two, fine. Wait. I won’t be that persnickety. But, if not, set a target anyway! Come up with a number that is so high/good that you don’t need any historical data to know you’d be thrilled to hit it. Come up with the opposite — a number that would be so low/bad that you would know there’s a problem. Start working from both directions to see how much you can close the gap before being in “no idea”-land. Then, split the difference. Sure, you may be WAY off, but it’s going to be a much more useful discussion once you have actual data to put against it.

The last step is a validation step, really. For each metric, ask yourself what you would do if you missed the target. Are there actions that you (or your department) can and would take if you missed the target? Or, would you simply go and tell another department that they have a problem (Oopsy! That means it’s a metric for them — not for you; it’s their call as to whether they should use it!). Would you cross your fingers and wait for another month and hope the number looks better then? If that’s the case, you’re admitting that you either don’t know how to actually impact the number or you can’t impact the number. It’s not a valid metric.

Enough on that (starting to think I bit off more than I should’ve with my first real entry here).

Moving on to…

Analysis

Analysis is very different from metrics reporting. While metrics reporting is all about measuring the performance of a person, a department, a process, a project, or a company…and knowing what corrective action to take if there is a performance issue…analysis is about trying to figure out what’s going on with something.

The best way to approach analysis is to start with a hypothesis. If you don’t have a clear hypothesis, you’ll find yourself going in even worse circles than if you started with the data when identifying your metrics. Put simply:

1. Start with a clear hypothesis
2. Ask yourself what action you will take if the hypothesis is disproven or not disproven. If there are not clearly different actions…then you’re wasting your time. It might be a fun analysis, but it’s not going to be particulary worthwhile (contrary to popular belief, data mining, which is one form of analysis, is not simply a case of, “dump all the data into a fancy tool and see what it spits back out that you can use” — you need hypotheses for data mining!)
3. Develop an approach that would enable you to disprove the hypothesis with as little data as possible
4. Get that data…and only that data
5. Perform the analysis

It’s tempting to pull extra data just so it’s there. And, that’s okay, as long as you don’t expand the scope of the data-pulling dramatically. Generally, just remember that it is a lot easier to sequence together a series of small analyses (if we disprove hypothesis X then we will test sub-hypothesis Y) than trying to do it all at once in one fell swoop.

That’s all for now!