QA: It's for Analysts, Too (and I'm not talking about tagging)

There is not an analyst on the planet with more than a couple of weeks of experience who has not delivered an analysis that is flawed due to a mistake he made in pulling or analyzing the data. I’m not talking about messy or incomplete data. I’m talking about that sinking feeling when, following your delivery of analysis results, someone-somewhere-somehow points out that you made a mistake.

Now, it’s been a while since I experienced that feeling for something I had produced. <Hold on for a second while I find a piece of wood to knock on… Okay. I’m back.> I think that’s because it’s an ugly enough feeling that I’ve developed techniques to minimize the chance that I experience it!

As a blogger…I now feel compelled to write those down.

I get it. There is a strong urge to skip QA’ing your analysis!

No one truly enjoys quality assurance work. Just look at the number of bugs that QA teams find that would have easily been caught in proper unit testing by the developer. Or, for that matter, look at the number of typos that occur in blog posts (proofreading is a form of QA).

Analysis QA isn’t sexy or exciting work (although it can be mildly stimulating), and, when under the gun to “get an answer,” it can be tempting to hasten to the finish by skipping past a step of QA, but it’s not a wise step to skip.<

I mean it.  Skipping Analysis QA is bad, bad, BAD!

9 times out of 10, QA’ing my own analysis yields “nothing” – the data I pulled and the way I crunched it holds up to a second level of scrutiny. But, that’s a “nothing” in quotes because “9 times everything checked out” is the wrong perspective. That one time in ten when I catch something pays for itself and the other nine analyses many times over.

You see, there are two costs of pushing out the results of an analysis that have errors in them:

1. It can lead to a bad business decision. And, once an analysis is presented or delivered, it is almost impossible to truly “take it back.” Especially if that (flawed) analysis represents something wonderful and exciting, or if it makes a strong case for a particular viewpoint, it will not go away. It will sit in inboxes, on shared drives, and in printouts just waiting to be erroneously presented as a truth days and weeks after the error was discovered and the analysis was retracted.
2. It undermines the credibility of the analyst (or, even worse, the entire analytics team). It takes 20 pristine analyses* that hold up to rigorous scrutiny to recover the trust lost when a single erroneous analysis is delivered. This is fair! If the marketer makes a decision  (or advocates for a decision) based on bad data from the analyst, they wind up taking bullets on your behalf.

Analysis QA is important!

With that lengthy preamble, below are my four strategies for QA’ing my own analysis work before it goes out the door.

1. Plausibility Check

Like it or not, most analyses don’t turn up wildly surprising and dramatic insights. When they do – or, when they appear to – my immediate reaction is one of deep suspicion.

My favorite anecdote on this front goes back almost a decade, when a product marcom who had been digging into SEO and making tweaks to his product line’s main landing page, popped his head into my cubicle one day and asked me if I’d seen “what he’d done.” He’d been making minor — and appropriate — updates to his product line’s main landing page to try to improve the SEO. When he looked at a traffic report for the page, he saw a sudden and dramatic increase in visits starting one day in the middle of the prior month. He immediately took a printout of the traffic chart and told everyone he could find — including the VP of marketing — that he’d achieved a massive and dramatic success by updating some meta data and page copy!

Of course…he hadn’t.

I dug into the data and pretty quickly found that a Gomez (uptime/load time monitoring software) user agent was the source of the increased traffic. It turned out that Gomez was pitching my company’s web admins, and they’d turned on a couple of monitors to have data to show to the people in the company to whom they were pitching. (The way their monitors worked, each check of the site recorded a new visit, and none of those monitors were filtered out as bots…until I discovered the issue and updated our bots configuration.)
In other words, “Doh!!!”

That’s a dramatic example, but, to adjust the “if it seems too good to be true…” axiom:

If the data looks too surprising or too counter intuitive to be true…it probably is!

Considering the plausibility of the results is not, in and of itself, actual QA, but it’s a way to get the hairs on your back standing up to help you focus on the other QA strategies!

2. Proofread

Proofreading is tedious in writing, and it’s not much less tedious in analytics. But, it’s valuable!

Here’s how I proofread my analyses for QA purposes:

• I pull up each query and segment in the tool I created it in and literally walk back through what’s included.
• I re-pull the data using those queries/segments and do a spot-check comparison with wherever I wound up putting the data to do the analysis
• I actually proofread the analysis report – no need to have poor grammar, typos, or inadvertently backwards labeling.

That’s really all there is to it for proofreading. It takes some conscious thought and focus, but it’s worth the effort.

3. Triangulation

This is one of my favorite – and most reliable – techniques. When it comes to digital data and the increasing flexibility of digital analytics platforms, there are almost always multiple ways to come at any given analysis. Some examples:

• In Google Analytics, you looked at the Ecommerce tab in an events report to check the Ecommerce conversion rate for visits that fired a specific event. To check the data, build a quick segment for visits based on that event and check the overall Ecommerce conversion rate for that segment. It should be pretty close!
• In SiteCatalyst, you have a prop and an eVar populated with the same value, and you are looking at products ordered by subrelating the eVar with Products and using Orders as the metric. For a few of the eVar values, build a Visit-container-based segment using the prop value and then look at the Products report. The numbers should be pretty close.
• If you’ve used the eCommerce conversion rate for a certain timeframe in your analysis, pull the visits by day and the orders by day for that timeframe, add them both up, and divide to see if you get the same conversion rate.
• Use flow visualization (Google Analytics) or pathing (SiteCatalyst) to compare results that you see in a funnel or fallout report – they won’t match, but you should be able to easily explain why when the steps when they differ.
• Pull up a clickmap to see what it reports when you’ve got a specific link tracked as an event (GA) or a custom link (SiteCatalyst).
• If you have a specific internal link tracked as an event or custom link, compare the totals for that event to the value from the Previous Page report for the page it links to.

You get the idea. These are all web analytics examples, but the same approach applies for other types of digital analysis as well (if your Twitter analytics platform says there were 247 tweets yesterday that included a certain keyword, go to search.twitter.com, search for the term, and see how many tweets you get back).

Quite often, the initial triangulation will turn up wildly different results. That will force you to stop and think about why, which, most of the time, will result in you realizing why that wasn’t the primary way you chose to access the data. The more ass-backwards of a triangulation that you can come up with to get to a similar result, the more confidence you will have that your data is solid (and, when a business user decides to pull the data themselves to check your work and gets wildly different results, you may already be armed to explain exactly why…because that was your triangulation technique!).

4. Phone a friend

Granted, for this one, you have to tap into other resources. But, a fresh set of eyes is invaluable (there’s a reason that development teams generally split developers out from the QA team, and there’s a reason that even professional writers have an editor review their work).

When phoning a friend, you actually can request any or all of the three prior tips:

• Ask them if the results you are seeing pass the “sniff test” – do they seem plausible?
• Ask them to look at the actual segment or query definitions you used – get them to proofread your work.
• Ask them to spot-check your work by trying to recreate the results – this may or may not be triangulation (even if they approach the question exactly as you did, they’re still checking your work).

To be clear, you’re not asking that they completely replicate your analysis. Rather, you’re handing them a proverbial napkin and asking them to quickly and messily put a pen to that napkin to see if anything emerges that calls your analysis into question.

This Is Not As Time-Consuming As It Sounds

I positively cringe when someone excitedly tells me that they “just looked at the data and saw something really interesting!”

• If it’s a business user, I shake my head and gently probe for details (“Really? That’s interesting. Let me see if I’m seeing the same thing. How is it that you got this data?…”)
• If it’s an analyst, I say a silent prayer that they really have found something really interesting that holds up as interesting under deeper scrutiny. The more surprising and powerful the result, the stronger I push for a deep breath and a second look.

So, obviously, there is a lot of judgment involved when it comes to determining the extent of QA to perform. The more complex the project, and the more surprising the results, the more time it’s worth investing in QA. The more you get used to doing QA, the earlier in the analysis you will be thinking about it (and doing it), and the less incremental time it takes.

And it’s worth it.

Photos courtesy of, in order, hobvias sudoneighm, Terry Whalebone, Nate Steiner, and Bùi Linh Ngân.

*Yup. I totally made that number up…but it feels about right based on my own experience.