WAA Standards Update: Thursday, November 6th

ANOTHER UPDATE: WebTrends just posted their own WAA Standard’s compatibility matrix at the new WebTrends blog. I have to say that when I first suggested that we needed a vendor-by-vendor assessment of Standards compliance back in August of last year I didn’t think it would take this long for the information to materialize. That said, despite the wait I’m glad to see most of the vendors stepping up!  I suppose I expect to have to update this post again when Omniture and Nedstat publish their information.

UPDATE: Coremetrics just posted a press release that describes their application’s WAA Standards compatibility at their web site. With Coremetrics participating in the creation of a global standards compliance matrix the only remaining vendors that need to provide this type of documentation are Omniture, WebTrends, and Nedstat.

One of the key challenges we face as an industry is without a doubt the lack of standard terms and definitions applied across different vendor solutions. John Lovett from Forrester led a huddle for us at the X Change conference this past August and the subject just keeps coming up.

The fine folks at the Web Analytics Association are actively trying to do something about this situation and have recently published a draft update of the WAA Standard Definitions.  When the definitions were first published back in August of last year I offered hearty congratulations but also commented that standards without any kind of transparency or enforcement are unlikely to be applied in any meaningful way.

Since that post three companies have publicly offered up any kind of documentation regarding their level of standards compliance — IndexTools (now Yahoo Web Analytics), Google Analytics (via Justin Cutroni at EpikOne), and most recently Unica (thanks Akin!) — but I am hopeful that this is about to change.  I am hopeful because of a conversation I had at Emetrics with Angie Brown, the head of the Standards Committee.

Angie and the Standards Committee will be presenting the updated definitions on Thursday, November 6th at noon Eastern / 9:00 AM Pacific.  I think this call is open to everyone (Angie, correct me if I’m wrong) but to help build awareness for this call and the Standards Committee in general I was lucky enough to interview Angie.  My questions (in bold) and Angie’s answers follow.

Tell me a little bit about the recent update to the standards document?

We took the feedback we received from last year — emails, blog postings, rumors, etc. — and revisited each of our original 26 definitions. This review caused us to drop one term (the distinction between “single page visits” and “single page view visits”, too confusing and not useful), we added four terms (frequency, recency, conversion rate, and impressions), and we spent the majority of the last year comparing our definitions with the tools we use in our everyday jobs.

As a result of the review, we’ve included a list of “ask your vendor” questions where we’ve been able to identify different methodologies used by different tools.

Who are some of the key people working on this project with you?

There are so many wonderful contributors on the Standards Committee. Most meetings have 8-12 participants, and they change over time based on employment status, workload, travel schedules, etc. But we do have a core group of people who joined way back in 2006 — Judith Pascual (my co-chair extraordinaire), Aseem Patel, Anna Long, and Bob Russotti — who are still active in the committee today. We have many newer members that contributed tons to this document as well, but having some continuity as we move from version to version to version has been critical.

We are also fortunate to have had outstanding guidance, enthusiasm, and representation on the WAA Board while we worked on this document, first from Avinash Kaushik and now from our current Director, Robbin Steif.

Which of the web analytics vendors are participating in the project?

Both Coremetrics and Unica are regular participants: Coremetrics since early 2006, and Unica since early 2007. A representative from Omniture came to our last meeting, and we’re looking forward to more participation from him in the future. I had a chance to talk with some of our other WA vendors at eMetrics last week, and hope we can get them interested.

Can you describe some of the ways the definitions/this work have been adopted so far?

I have heard from several vendor contacts that people are asking about compliance with the WAA definitions when sending out web analytics RFPs. We’ve also heard from people who have distributed the definitions throughout their company so everyone has a common terminology when discussing analytics.

Even more gratifying is the response from the international web analytics community, especially in Europe. Their WA market is much more fragmented than it is here in the US, with a larger variety of tools, so having common terminology is even more meaningful. One of our vendor members has a customer who translated the document into French on their own so they could share it with their coworkers! A high priority for next year is to work with the International Committee to get our document translated into other languages.

Awhile back I congratulated you on the work but complained that there needed to be teeth behind the document.  Since that time only IndexTools (who’s CTO is now a WAW Board member) and Google Analytics (via Justin Cutroni from Epik) have produced any kind of documentation describing their application’s compliance with WAA definitions.  Why do you think the other vendors are slow to respond?

I applaud both Dennis and Justin for taking that initiative without any prodding, and I think it says volumes about both men and the companies they represent. To be fair, I should mention that both Unica and Coremetrics also presented comparisons to the committee as part of their regular participation, although neither has published their results publicly. (NOTE: Unica has since published their comparison via Akin Arikan’s Multichannel Marketing Metrics blog.)

I’m merely speculating on why there hasn’t been more public response, but I suppose it’s because we haven’t asked them point blank to do so. However, this will change. Part of the feedback we’re soliciting from the industry this year will include a letter to each of our WAA member vendors asking them for comparisons, and we intend to publish the responses through the committee. The only reason we haven’t done this yet is because we’re putting the finishing touches on some scenarios (more detail later) that we also want our vendors to address.

Do you support the idea I proposed of a “Percent Standards Compliant” rating for every web analytics application/vendor?  If so, why?  If not, why not?

For now, the committee’s focus is on transparency over compliance. When the “ask your vendor” questions are answered, and the scenarios addressed, we will be able to include methodologies in the definitions in such a way that an overall percent compliance will be more meaningful. I’m not opposed to the idea, nor would I try to discourage anyone else from doing so; I just think it’s premature. Here’s why:

Our current definitions are broad, and we know that vendors use different calculations. In fact, that’s the whole reason we created the “ask your vendor” questions. There are also cases where the metrics described are not exposed in “standard” reports, but can be easily configured by the analyst in some tools. Or situations where a concept exists, but is called something else or simply isn’t exposed in the tool. Because of that broadness, I don’t think percentages calculated today will be different enough from vendor to vendor to provide much differentiation. Or worse yet, “compliance” for specific metrics will be open to interpretation.

For example, look at both Dennis’ and Justin’s responses to our “Return Visitor” definition. Dennis says IndexTools does not comply, while Justin says Google Analytics does. Because they’ve given good explanations (yay transparency!), it’s clear that both tools treat this concept in a similar manner: neither exposes the “return visitor” in the tool, but rather the visits made by returning visitors. If that’s the prevailing way this concept is exposed in the other tools, then it’s not so much a matter of compliance/non-compliance (an argument can be made either way, and indeed both interpreted compliance differently), but perhaps we on the Standards Committee should be defining Return Visits instead of Return Visitors.

So while I want the vendors to publicly tell the web analytics community how their tools stack up to our definitions, it will be as much about informing our Standards as about compliance.

What’s next for the standards document and standards committee?

1) Finish scenarios (more info below) and submit to vendors
2) Keep soliciting industry and vendor feedback on our 9/22 document (download it here)
3) Once feedback has been incorporated, work with the WAA International Committee to translate the final document into other languages

One of the more interesting things we’re doing right now is creating scenarios to find out how situations are handled in different tools. Not only will this provide critical information for practitioners who are transitioning from one tool to another (yes, we’ll publish the replies), but it will form the basis for more specific methodology-based standards.

For example, if the only activity a visitor performs on a site is a non-page event, do our tools count this activity as a visit, and what are the properties of this visit (entry/exit/duration)? Another example concerns average time on site: does the tool include bounced visits, and if so what value is used for these visits (zero? an estimate based on non-bounced views?)?

These are just two of the many situations where tools make different calculations that ultimately affect the numbers you use in your analysis, and why running multiple tools on the same site can lead an analyst to drink. But before we address how we should calculate, it will be beneficial to know the different ways we do calculate.

We’ll talk more about the scenarios during our upcoming WAA Webcast on Thursday, November 6, “Web Analytics Standards Update.” You can sign up on the WAA Site.

How can my readers help the WAA and the standards committee out?

Your readers can help us by giving us feedback. If the document is useful, please tell us. If not, what can we do better? Feedback can be broad or it can be specific to a certain metric. Ultimately, standards need to help the web analytics practitioner, and if we’re not on the right track we need to know.

Download the latest document from our WAA Standards Committee Page. If you attended eMetrics Stockholm or DC, you received a paper copy in your bag. We’ve started a blog post soliciting public comments on the WAA site. Other ways to provide feedback are by email (standards at webanalyticsassociation dot org), participating in our webcast, or starting a discussion about a specific term on the Web Analytics Forum.

There are so many bloggers in our space right now that if feedback consists solely of a post on one’s own blog or a comment on someone else’s, we may not find it so please drop us a line or post a comment on the WAA Blog to let us know.

Anything I forgot to ask you?

I want to expand a bit on the issue of standards with “teeth.” Above, I mentioned that we are more interested in transparency than compliance right now, but I don’t want to mislead anyone and make them think compliance isn’t important. It is, and compliance is our ultimate goal.

However, our industry is still young. There are any number of industry reports that will tell us most of the major tools provide similar basic functionality. A competent analyst can get valuable business insights out of any of them. And compliance to standards most certainly carries a cost, not just to vendors who will have to rewrite scripts and database queries, but to practitioners who will have to explain to their stakeholders why a number they’ve been reporting for years as 11.5 is suddenly 9.2.

Not having that happen any more is one of the best arguments for standards, but in order to get there we’re all going to have to get through the transition. The Committee wants to make sure that the standards are well-vetted, reasonable, and that they ultimately benefit the practitioners of our trade and confer a competitive advantage on vendors who comply. After all, if nobody complies, then why bother with standards?

As I discussed above with my Return Visitors vs. Return Visits example, if we find that the industry is already used to seeing a concept expressed a certain way, we’re not interested in forcing the entire industry to change their mind if both metrics are equally valuable. In that case, we want to standardize on the metric that people already find useful. We can’t do that without your feedback. And we can’t be sensitive to changes we’re asking from vendors if they don’t tell us how they do things right now. We have tried to figure it out for ourselves, but honestly? Most vendor documentation sucks.

Again, you can register to attend the WAA Standard’s Committee webcast on Thursday, November 6th by going to the Web Analytics Association web site.  Thanks again to Angie for taking the time to answer my questions!