Congratulations to the WAA Standards Committee!
I wanted to say congratulations to Jason Burby, Angie Brown, and everyone on the Web Analytics Association’s Standards Committee for publishing their standards document last week. Given the number of web analytics terms they defined (26) and the somewhat slow process the Association has for getting documents approved, this effort is a huge milestone for the organization, one that Jason and Angie deserve great praise for indeed!
If you haven’t already downloaded and read the definitions, check them out here (PDF download).
While the PDF document says that the final product is “Web Analytics Definitions – Version 4.0” this is clearly a “Web Analytics 1.0” document. The committee relegated all of the really wonderful Web 2.0 stuff like AJAX, RSS, XML, and the such to the same confusing obscurity they exist in today with the comment “certain technologies including (but not limited to) Flash, AJAX, media files, downloads, documents, and PDFs do not follow the typical page paradigm but may be definable as pages in specific tools.”
Given the last year’s push towards measuring Web 2.0 the right way and some great, insightful work from folks like Ian Houston and Judah Phillips it is kind of a shame that this document doesn’t address event-based measurement architecture more directly. The group does define “event” but only does so under the header of “Conversion Metrics” stating that an event is “any logged or recorded action that has a specific date and time assigned to it by either the browser or server.
Sounds like the definition of a Web 2.0 event to me, but I’m not sure why this is relegated to conversion metrics.
Regardless, this is great and valuable and useful work on the part of these hard-working volunteers. But the definition of standards raises one particularly important question: Given the definition of standards, what the hell do web analytics practitioners do with them?
The Fundamental Problem
The fundamental problem with these definitions (and any standard definitions IMHO) is that without an enforcement mechanism they are unlikely to provide any real benefit to the folks in the trenches. As long as smart folks like Eric Enge at Stone Temple Consulting continue to uncover as much as a 154% difference in the measured number of visitors and a 161% difference in the measured number of page views between concurrently deployed solutions, the average web analytics end user should not be comforted by the existence of standards.
Put another way, it is not the definition of standards that makes a difference, it is the adherence to standards by technology vendors that will provide the portability of skills, knowledge, and solutions so desired by many in our industry. Jason Burby sagely points this out in his Clickz article on his volunteer work when he says:
“Companies often switch metrics tools and subsequently change the terms they use to discuss analytics. One tool will call something one name, while another tool calls it by a different name or applies different meanings to a very similar name. When people switch tools and bring data with them, they don’t get an apples-to-apples comparisons. As a result, companies lose the important year-over-year view.
Though the new standards won’t instantly take care of that issue, they provide a step in the right direction.”
The Barrier to the Adoption of Standards
The problem as I see it is this: For many web analytics vendors, the way they calculate some of the critical metrics in web analytics is the “secret sauce” in their solution. Consider the WAA’s definition of unique visitors which states that unique visitors are:
“The number of inferred individual people (filtered for spiders and robots), with a designated reporting timeframe, with activity consisting of one or more visits to a site. Each individual is counted only once in the unique visitor measure for the reporting period.”
This is perfectly reasonable, but the definition goes on to say that “a unique visitor count is always associated with a time period (most often a day, week, or month), and it is a non-additive metric.”
Do you wonder what the folks at Visual Sciences who have spent millions to perfect their “data wheels” technology that effectively removes the “time period” requirement would say to this? One of the major value propositions at Visual Sciences (at least during my brief tenure) was that time was irrelevant — if you wanted the number of unique visitors for the football season, you dragged your mouse across the calendar; if you wanted the number of unique visitors for a few hours during the day, you dragged your mouse; if you wanted the number of unique visitors to your site since recording began, you dragged your mouse.
You can make the case that this example more or less removes the time dependence associated with the WAA definition. But should all the vendors who don’t have this capability (anywhere you are forced to use metrics like “Daily Unique Visitors”) spend the R&D money necessary to eliminate the dependence on time? Or should Visual back this functionality out of their application?
When you start to think about these kinds of things, much less issues associated with data sampling and data roll-off that occurs for a litany of reasons, you can start to understand why I made this somewhat snide comment in a MediaShift article awhile back:
“A friend of mine described it as the most beautiful fantasy…but it would never happen,” consultant Peterson said. “Omniture has a $1 billion market cap, and I don’t see Omniture tearing apart their technology to calculate unique visitors and page views differently because all their competitors have decided there’s a different way to do it. It’s hard to imagine. Not impossible. Fantasies sometimes come true.”
Ironically the cost isn’t the main problem: The impact on existing customers who would be forced to learn new definitions and suffer from potentially dramatic changes in data collection and reporting is the main problem. Do you want to be the person who has to tell a Fortune 500 customer that because you’re adopting more standard definitions that their page view count will suddenly drop by 35% month-over-month?
I had to do that once. Trust me here, it wasn’t a fun conversation to have.
An Idea in the Absence of a Solution
Given that I think that the WAA has produced some incredibly valuable work, despite some potential barriers to the work’s adoption, I do have an idea that I would love to see the Association follow-up on, one that would add a tremendous amount of value to this already great work.
I would love to see the Standards Committee create a matrix of standards compliance for each of the vendors in the marketplace today. Basically a checklist that details on a term-by-term basis which vendors are currently using the WAA definitions that would let companies looking for a solution to include that criteria in their assessment. Something that would let everyone quickly determine:
- How standards compliant a given solution is (and which solution today is “most compliant”)
- Which standard definitions are calculated out-of-box in each solution (for example, “Original Referrer” and “Bounce Rate”)
- Which currently available solutions dramatically differ from the norm in their use of standard terms
Something like this would probably have to be backed up with some documentation or examples as proof points, just for reference. And yeah, this is kind of a lot of work, but if you think about it all you really need is for one WAA member per solution to poke around in their documentation and then someone (Jason and Angie maybe) to collate the results and write it up. I would be happy to contribute the matrix assessment for the web analytics solution I’m using now if that would up!
Who knows, maybe we’d discover that all the vendors are already standards compliant and there really isn’t a problem with definitions!
What Do You Think?
I’d love to hear what all of you think about the new standards and my concerns about how they’ll be used (or not used.) Am I missing something? Were you disappointed to not see something that spoke more clearly to your concerns about Web 2.0 technology? Or are you just pleased that the WAA published these definitions and see them as a small-but-important first step?
OK, we have finally entered the Epistemological Age of Web Analytics; it was a necessary step we needed to reach at some point. Why am I not surprised that it’s Eric who is brigning us there?
Your comments are getting straight to the heart of the matter: what is our analytical framework? On which paradigm is it based? We are in need of naming the reality we observe, we need concepts, we need definitions, and the WAA document is a great step forward for Homo Marketingus.
Agreeing on those terms, on the reality we examine, is essential as a field. It is also essential for its adoption, because it is supposed to profoundly change how we do marketing.
The problem is that, so far, the brand of microscope has not only shown the microbe, it has defined its very existence. Sure, it’s been a well known facts for years that the instrumentation of science impacts how well you can do the science. But scientists thrive to neutralize that effect. That is why they try so hard to agree on things… ’till the next debate.
We have given too much importance to vendors; they told us what the reality was. It was normal during the Awareness Age we are slowly leaving, and I very much commend all those men and women who for years developed products, and educated us all.
But it is now time for us to tell them what this is all about. The dynamic has to be put upside down. Vendors need to make products that will deliver what we need, not what they figured out.
Your idea of standard compliance is great. It will make many people cry their heart out, but it will be a fantastic way of finally shifting the balance of power.
Great post! One of the major contributions lately.
Jacques: It will certainly be interesting to see how the vendors react to these new standards, I could not agree more.
The cynic in me insists that most will claim compliance because some of the definitions the WAA provided are kind of ambiguous — plus there is no discussion about sampling and other data accuracy issues that are probably a bigger factor in the “un-comparability” of any two solutions. But the optimist in me agrees with you that this is a huge step forward!
Incidentally, it is the optimist in me that thinks we should build a compliance matrix. Are you willing to help with that project if asked?
Thanks again for your very thoughtful comments!
You bet I’d be willing!
“You can make the case that this example more or less removes the time dependence associated with the WAA definition.”
Hmm; I don’t think that’s right. All it means is that Visual Sciences lets you choose your own time period. (And, without looking at the underlying data, there’s no way that “uniqueness” can be compared across time periods.) In any case, that’s a very minor part of your post.
I glanced through the PDF and didn’t see anything that would help resolve differences between 2 or more tools. Those depend on settings such as inactivity timeout, session timeout and other factors that were covered so well in the Stone Temple document.
Jacques: Awesome! I haven’t heard from anyone in the WAA yet I will keep you posted.
Scott: Okay, but what about this. Let’s say for the sake of argument that the WAA’s requirement of time dependence is actually a bad idea. Let’s say that the definition should have said “unique visitors should be calculable over whatever period of time necessary, granular to the level of the minute” which is more in line with practical usage of this metric.
Would it be reasonable for every vendor not having this capability to re-tool their entire application from the ground up to accommodate this requirement? I can assure you, more vendors than not require time-dependence in their use of unique visitors, but few would be likely to bear the expense of re-tooling (IMHO).
Anyway, thanks for your comment!
Eric/Scott: Interesting points that start to touch on the core problems for the industry as a whole.
On one hand we have the mysterious Secret Sauce. I don’t have a problem with Secret Sauce per-se, but if you say 1+1=2 and I say it =3 because of my secret sauce, then who is “right”?
That’s why we have standards, in all industries, to enable everyone to be raised up to the same highest *standard*.
The danger in viewing standards is from the sole perspective of vendors. Standards are there to enforce OUR views, as the general populace, ON vendors, not the other way around. I grotesquely simplify, but hopefully you get the vibe.
The related question thus becomes, if we/WAA can’t even get the vendors, who we’re friendly with, to agree to come to common terms, then what hope have we got of convincing the rest of the planet that these standards, and by implication the WAA itself, should be taken seriously? And hence convince everyone globally to absorb the pain of a transition.
Where we don’t have a true standard is typified by the Stone Temple report you mention. Quite frankly the results from that report *appals* me. Granted it’s not even remotely a scientific methodology, but it does highlight a massive problem that we as an industry have.
Flip it another way, although I do recognise that arguing via analogy, and via strawpersons, is plain rude. 🙂
Imagine if TCP/IP had multiple vendors each having their “secret sauce” to have things work that little bit different and better for their clients.
The Internet we know and struggle with would not even remotely exist. It’d be the good old days of thousands of individual fiefdoms as typified by the then Compuserve and co. from the 80’s and early 90’s. And we would not be having this conversation. Your gain no doubt. 😉
So who *are* we trying to help here? Or are we just creating feel good documents?
The cold hard brutality is that if we are serious about standards, then vendors who don’t meet the grade don’t get the tick. No ifs, buts or maybes about it. The business risk for the vendors is that of having or not having the ‘tick’. Does that make a difference to their bottom line? Their call. Those who can, will, those who can’t won’t. Survival of the fittest and best at it’s finest.
FWIW, and recognising that critiquing is far easier, have noted a number of issues with the WAA document:
http://www.stedee.id.au/2007/08-27/web_analytics_association_standard_metrics_definitions
And part of that is no longer true. Finally joined the WAA last night. 🙂
Cheers!
– Steve
I agree with Steve whe he says:”Standards are there to enforce OUR views, as the general populace, ON vendors, not the other way around”.
At the end, be it the imperial or metric system, we got to agree on how long a foot or a meter is. But is it even possible, when we got the dirtiest data around, as Jim Novo would say?
I mean, why did it take Visual Sciences millions of dollars to get to a consistent visitor count?
If we want to impose our standards to vendors, we obviously need to come up with clear ones, and *feasible* ones I’d say…
Steve: I think we’re a ** long ** way from the vendors changing their “sauce” just because the WAA (or any organization) has said “hey, you should do it this way!” That was the point I made in the MediaPost article — that this is still a relatively nascent industry and there is no enforcement mechanism.
But you make a good point, to be sure.
I guess my response would be that the WAA standards are a good starting point and that the compliance matrix I propose is a good half-way point between “standards enforcing OUR views” and blowing off the WAA as ineffective because it lacks any enforcement mechanism.
Jacques: I think you misinterpreted my comment about Visual Sciences. The money the company spent developing their technology went into many things I’m sure. A new approach to counting de-duplicated unique visitors is just one of them, but I don’t think that speaks to consistency — it speaks to accuracy and flexibility (okay, accuracy in as much as this data can be accurate.)
That’s what I meant.
I’m not sure what you mean by “feasible” in your comment. I think that anything is possible, don’t you, given resources and time … it’s more a question of the practicality of a company making a massive change to their technology just because the “standards” say they should.
Maybe I’m not understanding you.
Thanks to you both for your comments!
Eric, your right when you say “given resources and time”, that’s the beauty of software. What I had in mind was the usual stuff we all come across: cookie deletion, multiple computers, etc. and inherent difficulties of the web data. For example, we could ask that 1 visitor = 1 person as loudly as we could, but it would be extremely hard to deliver (ok, a bit of an extreme example here).
[…] I join with Eric Peterson in congratulating the WAA. Eric also offers some good explanations that are definitely worth a read. Click here to see for yourself. […]
“…changing their sauce”
I do agree. But it helps to have goals.
World Domination here we come! 😉
More seriously, if the vendors at least see where we’re all trying to head, then perhaps the move will simply happen organically, no enforcement needed.
I could see the low end tools use standards compliance as a marketing plug – that happens all over.
Gradually that bubbles up into the big end of town such that no-one can avoid it.
Again with the analogies, that’s exactly how IP succeeded. vs Decnet, IPX, NetBEUI, SNA, X.25 etc etc etc. And to a certain extent we’re seeing the same thing (again, and again) with HTML.
And as you pointed out in the posting – we haven’t touched on the web2.0 stuff. So plenty of room for growth and continuous improvement.
Going back to an older posting as it touches here – the cookie deletion stuff. If several high end products using the *same* collection technology can’t even get close to agreeing on the critical numbers? Are we perhaps barking up the wrong trees (forest?) by worrying about cookie deletion rates, or Panel accuracy, or (my personal bugbear) Javascript vs Logs, or or or. 🙂
THIS is where we appear to really need a standard, but I suspect we’re a long way from even sitting down to draw it up. Need to crawl before we can run. 🙂
BTW Jacques? It’s “metre” not “meter”.
Boom. Boom. QED. Etc. 🙂
Cheers!
Hi Eric,
I’ve been too busy to respond since reading this post earlier in the week, and am limited for time again today, but I wanted to say “right on” and “good thinking.”
The real power of these standards will only come if consumers (i.e. users who buy web analytics products) demand adherence from vendors.
That’s why I recommend in my blog that the WAA start auditing and certifying vendors for compliance (like the IAB) within their interfaces and documentation (not core code).
In the meantime, the WAA should create and publish a compliance score (from 0-100%) for ranking vendor products. The score could be based off the matrix you suggest.
I’d be glad to help with the matrix too. 😉
Judah
Very late into this discussion (wretched holiday – cannot get on top of RSS reader with the web analytics group being the biggest offender)
So Eric you’ll probably be the only one reading this.
😉
I think your matrix idea is the right thing at the right time. It will get the ball rolling and gradually catch vendors’ attention.
I agree with Steve that it is probably the small vendors that will adopt first using the matrix as a marketing tool.
Given time (and development cycles) vendors will adapt and adopt but only if we (clients, practitioners, WAA) keep the pressure.
I will be more than glad to help with the European vendors (especially the ones present in the UK). So keep me informed (you have my email address).
Michael
A bit heavy on the moaning. Rome wasn’t built in a day, and neither are standards.
Just because you can change the time window in Visual Sciences doesn’t mean the time dimension of the metric goes away–it’s just more dynamic than in other packages.
Three years ago, when Jason Burby and I were appointed Co-Chairs of the WAA Standards Committee (Angie took over my place when I joined Burton Group and had to resign so I wouldn’t be using my industry analyst position to influence standards), we talked through these very issues and I was very much for not trying to boil the ocean.
We wanted to start off with the foundation–the metrics–and then build on that. In fact, that “build from the ground up” vision was where I came up with the count and ratio framework. To quote from an e-mail I sent to Bryan Eisenberg on September 9, 2004, “There is a method to the madness. The reason for introducing the counts and ratios framework was to imply a methodology of moving from defining the numerator and denominator used in a ratio, and then the ratio itself. Too many times, I have seen people rhapsodize over a KPI ratio that is built using undefined counts — they get all consumed by the formula calculation, and never think to ask, do we trust the numbers this ratio is built on?”
This incremental strategy was the same for gaining adoption of the metrics. The vision was to put a stake in the ground on the metrics and then start certifying the vendors’ compliance. We couldn’t measure compliance until we had the definitions.
Furthermore, the idea was not to remove each vendor’s “secret sauce,” but rather (for example) to put an asterisk next to every metric that was WAA compliant. A vendor could mix the WAA metrics in with their own, or have a report view that was WAA only. How they implemented the UI would be up to them, but a client would at some point be able to look at a WAA-compliant metric from Omniture, look at the same WAA-compliant metric in WebTrends, and see the same number.
The standards problem here is similar to what companies are discovering in taxonomies at the moment–that it’s a mistake to homogenize everything, because then you take away the value that departments and disciplines get from using their own unique terms. At the same time, there must be some kind of universally understood language, because otherwise no one can communicate with anyone outside their department. Therefore, you need some kind of yin and yang thing, where basic terms are universally understood but where there is tolerance of unique terms as well.
I also expected that the certification process would end up refining the terms as well, as vendors would point out issues that would require the definitions to be tweaked.
So perhaps the process isn’t going as quickly as you wish, but it’s certainly in line with the longterm goal we started out with.
Guy: Thanks for your feedback and your historical perspective on the standards document in general. I agree with you about needing to put that asterisk on each vendor metric that was (or was not) compliant — which is why I proposed the standards matrix idea.
In terms of time lines … I suppose this is just the risk you run with an almost-all volunteer organization. To me three years to produce the current document seems like kind of a long time but hopefully things will speed up now that the document is live!
Thanks again!
Hello,
Fascinating discussion, this. I enjoyed the references to non-enforcement. It brought back memories of Gort and Klaatu (the reason a robot race was created was to enforce laws without bias). I also greatly appreciated Jacques’ comments. Very astute and well reasoned (my opinion).
I’m intrigued that this discussion is taking place (and I worked with the Standards Committee for a bit). The challenge is (to me) inherently applied mathematics. This is the solution I keep hearing applied to other challenges (although I’m not sure people involved in the discussion recognize it as such). “Events” are an example of this. Understand the outcome in as minute a detail as possible, then the methods (the secret sauces) that produce that outcome are irrelevant because it is the outcome that matters. Example (from an inherently Joseph perspective and not an attempt to pollute the waters): define a page-view as any event that causes the visitor to recognize new information has been presented because only then will the visitor (re)view the page. Consciousness is largely a reactive process because our ability to conceive of a future is based greatly on our ability to understand our past.
Anything causing the event is irrelevant (at this level). Only the visitor refocusing their attention is relevant because until that happens whatever happens on the screen is not recognized by the visitor as part of their experience.
Joseph: Phew, I bet you were a lot of fun on the standards committee, weren’t you? Jason and Angie are trying to get consensus on simple definitions and you’re talking about consciousness and cognition.
I appreciate the comment and am glad to know you’re reading the blog!
[…] of the major things that bloggers (Eric Peterson, Marshall Sponder, Avinash Kaushik, et al.) and pundits pointed out that these three women echoed […]
[…] WAA Standard Definitions. When the definitions were first published back in August of last year I offered hearty congratulations but also commented that standards without any kind of transparency or enforcement are unlikely to […]
[…] you have to love the IAB for putting this out there. Unlike the Web Analytics Association’s Standards which I believe are an excellent start but are a little soft in areas, the IAB is basically telling […]