Web Analytics Platforms Are Fundamentally Broken

Farris Khan, Analytics Lead at ProQuest and Chevy Volt ponderer extraordinaire, tweeted the question that we bandy about over cocktails in hotel bars the world over during any analytics gathering:

His tweet came on the heels of the latest Beyond Web Analytics podcast (Episode 48), in which hosts Rudi Shumpert and Adam Greco chatted with Jenn Kunz about “implementation tips.” Although not intended as such, the podcast was skewed heavily (95%) towards Adobe/Omniture Sitecatalyst implementations. As the dominant enterprise web analytics package these days, that meant it was chock full of useful information, but I found myself getting irritated with Omniture just from listening to the discussion.

My immediate reply to Farris’s tweet, having recently listened to the podcast, reflected that irritation:

Sitecatalyst throws its “making it much harder than it should be” talent on the implementation side of things, and I say that as someone who genuinely likes the platform (I’m not  a homer for any web analytics platform — I’ve been equally tickled pink and wildly frustrated with Google Analytics, Sitecatalyst, and Webtrends in different situations). I’m also not criticizing Sitecatalyst because I “just don’t understand the tool. ” I no longer get confused by the distinction between eVars, sProps, and events. I’ve (appropriately) used the Products variable for something totally separate from product information. I’ve used scView for an event that has nothing to do with a shopping cart. I’ve set up SAINT classifications. I’ve developed specs for dynamically triggering effectively named custom links. I’ve never done a stint as an Adobiture employee as an implementation engineer, but I get around the tool pretty well.

Given that I’ve got some experience there, I’ve also worked with a range of clients who have Sitecatalyst employed on their sites. As such, I’ve rolled my eyes and gnashed my teeth at the utter botched-ness of multiple clients’ implementations, and, yes, I’ve caught myself making the same type of critical statements that were rattled off during the podcast about companies’ implementations:

• Failure to put adequate up front planning into their Sitecatalyst implementation
• Failure to sufficiently document the implementation
• Failure to maintain the implementation going forward on an on-going basis
• Failure to invest in the people to actually maintain the implementation and use the data (Avinash has been fretting about this issue publicly for over 5 years)

In the case of the podcast, though, I wasn’t participating in the conversations — I was simply listening to others’ talk. The problem, though, was that I heard myself chiming in. I jumped right  on the “it’s the client’s fault” train, nodding my head as the panel described eroded and underutilized implementations. But, then a funny thing happened. As  I stepped back and listened to what “I” would have been saying, I got a bit unsettled. I realized I’d been seduced by the vendor. Through my own geeky pride at having cracked the nut of their inner machinations, I’d crossed over to vendor-land and started unfairly blaming the customer for technology shortcomings:

If the overwhelming majority of companies that use a given platform use it poorly…shouldn’t we shine a critical light on the platform rather than blaming the users?

I love digital analytics. I enjoy figuring out new platforms, and it’s fun to develop implement something elegantly and then let the usable data come pouring in that I can feed into reports and use for analysis. But:

• I’ve been doing this for a decade — hands-on experience with a half-dozen different tools
• It’s what I’m most interested in doing with my career — it beats out strategy development, creative concepting, campaign ideation, and any and every other possible marketing role
• I’m a sharp and motivated guy

In short…I’m uniquely suited to the space. I’m neither the only person who is really wired to do this stuff nor even in the 90th percentile of people who fit that bill. But the number of people who are truly equipped to drive a stellar Sitecatalyst implementation are, best case, in the low thousands, and, worst case, in the low hundreds. At the same time, demand for these skills is exploding. Training and evangelization is not going to close the gap! The Analysis Exchange is a fantastic concept, but that’s not going to close the gap, either.

There is simply too much breadth of knowledge and thought required to effectively work in the world of digital analytics for a tool to have a steep learning curve with undue complexity for implementation and maintenance. The Physics of the Internet means there are a relatively finite number of types of user actions that can be captured. Sitecatalyst has set up a paradigm that requires so much client-side configuration/planning/customization/maintenance/incantations/prayer that the majority of implementations are doomed to take longer than expected (much longer than promised by the sales team) and then further doomed to be inadequately maintained.

The signals that Adobe is slowly taking steps to merge the distinction between eVars and sProps is an indication that they realize that there are cases where the backend architecture needlessly drives implementation complexity. But, just as the iPhone shattered the expectations we had for smartphones, and the iPad ushered in an era of tablet computing that will garner mass adoption, Adobe has a very real risk of Sitecatalyst becoming the Blackberry of web analytics. Sitecatalyst 15, for all of the excitement Adobe has tried to gin up, is a laundry list of incremental fixes to functional shortcomings that the industry has simply complained about for years (or, in the case of the the introduction of segmentation, a diluted attempt to provide “me, too” functionality based on what a competitor provides).

The vendors have to take some responsibility for simplifying things. The fact that I can pull Visits for an eVar and Visits for an sProp and get two completely different numbers (or do the same thing for instances and page views) is a shortcoming of the tool. We’ve got to get out of the mode of simply accepting that this will happen, that a deep and nuanced understanding of the platform is required to understand the difference, and then gnashing our teeth when more marketers don’t have the interest and/or time to develop that deep understanding of the minutia of the tool.

<pause>

Although I’ve focused on Sitecatalyst here, that doesn’t mean other platforms are beyond reproach:

• Webtrends — Why do I have to employ black magic to get my analysis and report limits set such that I don’t miss data? Why do I have to employ Gestapo-like processes to prevent profile explosion (and confusion)? Why do I have to fall back on weeks-long reprocessing of the logs when someone comes up with a clever hypothesis that needs to be tested?
• Google Analytics — Why can’t I do any sort of real pathing? Why do I start bumping up against sampled data that makes me leery…just when I’m about to get to something really cool I want to hang my hat on? Why is cross-domain and cross-subdomain tracking such a nightmare to really get to perform as I want it to?

My point here is that the first platform that gets a Jobs-like visionary in place who is prepared to totally destroy the current paradigm is going to have a real shot at dominating over the long haul. There are scads of upstarts in the space, but most of them are focused on excelling at one functional niche or another. Is there the possibility of a tool (or one of the current big players) really dramatically lowering the implementation/maintenance complexity bar (while also, of course, handling the proliferation of digital channels well beyond the traditional web site) so that the skills we need to develop can be the ones required to use the data rather than capture it?

Such a paradigm shift is sorely needed.

Update: Eric Peterson started a thread on Google+ spawned by this post, and the lengthy discussion that ensued is worth checking out.