New Data on the Strategic Use of Web Analytics

Recently Google published the results of a Forrester Research study they had commissioned (PDF) to help the broader market understand the use and adoption of free web analytics solution.  Google should be applauded for commissioning Forrester to conduct this work, especially given the quality of the research and the level of insights provided.  Without a doubt, free solutions like Google Analytics and Yahoo Web Analytics are having an impact on our industry and driving change in ways few of us ever imagined.

I really did enjoy the Forrester report, primarily because the author (John Lovett) managed to surface totally new data.  When he first told me that over half of Enterprise businesses were using free solutions I have to admit I didn’t believe him.  In a way I still don’t, but perhaps that’s only because I work with a slightly different sample than he presents.  Regardless, John’s report paints a picture of an increasingly challenging market for companies selling web analytics and a new sophistication among end users.

Speaking of sophistication, there are a few points in the report that I question, and since I have pretty good luck getting feedback from readers on big picture stories I figured I’d bring them up here in the blog.  Before I do I want to emphasize that I am not questioning Forrester or John’s work—I am merely trying to explore some data that I find contrary to my own experience in this public forum.  To this end I pose a handful of questions that I would love to discuss either openly in comments or via email.

The first point I question is the observation in Figure 3 that 70% of companies report having a “well-defined analytics strategy.”  Two years ago my own research found that fewer than 10% of companies worldwide had a well-defined strategy for web analytics.  Last year Econsultancy reported that only 18% of the companies in their sample had a strategy for analytics.  To jump from these low numbers to the majority of Enterprises just doesn’t square with my general experience in the industry.

.

Remember, the implication of this data point is that 70% of all companies having more than 1,000 employees have a “well-defined analytics strategy.”  According to a 2004 report from the U.S. Census Bureau there were just over 12,000 companies in the U.S. with more than 1,000 employees.  Without assuming any growth between 2004 and 2009, Forrester’s 70% figure would result in over 8,500 companies in the U.S. that have a “well-defined” strategy for web analytics. Does that sound right to you?

Consider that the combined customer count for Omniture, WebTrends, Coremetrics, and Unica combined in the U.S. doesn’t even add up to 8,500 companies.  Even if you use the more conservative 13% who “strongly agree” with Forrester’s statement you end up with over 1,500 U.S. companies.  I may suffer from sample bias, but personally I can barely think of 150 companies that I would identify as having any strategy for web analytics, much less a “well-defined” one.

Most companies I talk to have the beginnings of an over-arching strategy—they’ve realized the need for people and are beginning to reduce their general reliance on click-stream data alone.  But given that I think about this topic from time to time, I think a “well-defined” strategy for web analytics takes into account multiple integrated technologies, appropriate staffing, and well thought-out business and knowledge processes for putting their technology and staff to work.  What does the phrase “well-defined strategy” imply to you?

Similarly, if 60% of companies truly believed that “investments in Web analytics people are more valuable than investments in Web analytics technology” there would be THOUSANDS of practitioners employed in the U.S. alone.  But again, every conference, every meeting, every conference call, and every other data point suggests that the need for people in web analytics is still an emerging need.  Hell, Emetrics in San Jose earlier this year barely drew 200 actual practitioners by my count.  How many web analytics practitioners do you think there are in the United States?

Same problem with the rest of the responses to Figure 3 on web analytics as a “technology we cannot do without” (75%) and the significance of the role web analytics plays in driving decisions (71%).  Perhaps I’m talking to entirely the wrong people, perhaps I’m interpreting these data wrong, and perhaps I’ve gone flat-out crazy, but these responses just don’t match my personal understanding and experience in the web analytics industry.

This issue of data that simply does not make sense, while not universally manifest in the report, manifests elsewhere as well. For example, Figure 8 reports on the percentage of application used segmented by fee and free tools:

.

When I look at these responses and see that 63 percent of respondents using fee-based tools and 50 percent of respondents using free tools claim to be effectively using more than half the available functionality, again I find myself scratching my head. As this data appears to speak to the general sophistication of use of analytics I went back and looked at Dennis Mortensen’s quantitative study of how IndexTools was being used around the world.

Dennis reports that fewer than 10% of his customers were using even the most basic “advanced” features in web analytics (report customization) and that fewer that 4% of his customers (on average) are making any “advanced” use of the IndexTools application. While this dataset is somewhat biased towards European companies who I believe, on average, to be somewhat behind their U.S. counterparts it does provide an objective view in how web analytics are used that seems to directly contradict the self-reported responses in Forrester’s figure 8.

Clearly there is a gap between the responses John collected and the current state of the web analytics market.  Since John is a very smart guy I know part of his rebuttal will include the observation that he surveyed people directly responsible for web analytics (see Forrester’s methodology) and that people in general have a tendency towards positivism. Trust me, my son is the most handsome little boy ever born and my daughter’s beauty is only matched by that of Aphrodite … same for your kids, right?

Given the difficulty associated with gathering truly objective data regarding the use of web analytics, this type of self-reported data is usually what we have to go on.  While Omniture, WebTrends, Coremetrics, and Unica all have the fundamental capability to report data similar to that provided by Mr. Mortensen, it may not be in their best interests to expose underwhelming adoption and unsophisticated use (if that is what the analysis uncovered.)  Ultimately we’re forced to accept these self-reported responses and  then reconcile them against our own views, which is why I’m asking my readers what they think about the data Forrester is reporting!

Regarding these self-reported attitudinal responses on how web analytics is used strategically, perhaps the truth is found in the companies who “strongly agree” with John’s statements.  If we apply this lens, as opposed to the more optimistic view, we get the following:

• 17% of companies recognize that web analytics is a technology they cannot live without;
• Web analytics plays a significant role in driving decisions at 12% of companies;
• 13% of companies have a well-defined web analytics strategy;
• 9% of companies recognize that investments in people are more valuable than investments in technology

These numbers start to make a lot more sense to me.  Likely the truth, as with so much in our industry, lies somewhere in between, but I would love to hear what you think about these adjusted numbers.  Do the lower numbers make more sense to you, or do you agree with John’s more optimistic assessment?

Unfortunately if the lower numbers are correct the implication is that despite the incredibly hard work that companies, consultants, and industry thought-leaders around the world have done for years we still have an incredibly long way to go before web analytics is recognized as the valuable business practice that you all know it can be!

Regardless I want to state that I do not disagree at all with the fundamental thesis in this report, that “free” is creating a whole new level of interest in web analytics and that, given proper consideration, free is an excellent alternative to paid solutions.  Lacking clear strategy and resources, too many companies have wasted too much money on paid solutions for free to not be compelling.  Thanks to the dedication of the Google and Yahoo teams, the world now has access to great applications that are in some regards more compelling than fee-based alternatives.

While I may not have said this a few years ago, today I honestly do believe that “free” is a viable and appropriate alternative to fee-based solutions. While not appropriate in every situation, it is irresponsible to suggest that any company not willing to fully engage in web analytics should pay for ongoing services and support. Given advances from Google and the availability of Yahoo Web Analytics, any motivated company large or small now has access to a wealth of data that can be translated into information, insights, and recommendations.

Conversely I agree with John (and Jim, and almost ever thought leader I respect) who states that you need to “prioritize your business needs and culture for analytics first and then evaluate the tools.”  This goes back to the fundamental value proposition at Analytics Demystified: It’s not the tools you use but how you use them. If you’re not invested in developing and executing a clearly defined strategy for digital measurement, you may as well be grepping your log files.

I would love your feedback on this post, either directly in comments or via email. Thanks again to the folks at Google for making this awesome research freely available and to John Lovett for shedding light on this incredibly important aspect of our sector.  Remember: we are analysts—our jobs are to ask hard questions and then ask even harder ones!