Unique Visitors ONLY Come in One Size

Back in January I published a note about the proposed IAB Audience Reach Measurement Guidelines that generated a fair amount of interest. At the time I applauded the IAB for providing guidance regarding the definition of a “unique user” or “unique visitor” while noting some concerns about how the proposed definition would actually manifest. In summary, the new IAB definition of “unique visitor” needed to have some basis in underlying data that is based on secondary research that can be directly tied to “a person.”  Now that the IAB Audience Reach Measurement Guidelines have been officially published we can use the IAB’s own words:

“… in order to report a Unique User, the measurement organization must utilitze in its identification and attribution processes underlying data that is, at least in reasonable proportion, attributed directly to a person” and “In no instance may a census measurement organization report Unique Users purely through algorithms or modeling that is not at least partially traceable to information obtained directly from people, as opposed to browsers, computers, or any other non-human element.” (Section 1.2.4)

The last little bit references, I believe, the IAB’s distinction of four types of unique “countables” — Unique Cookies (Section 1.2.1), Unique Browsers (1.2.2), Unique Devices (1.2.3) and Unique Users or Unique Visitors (1.2.4).  The term “measurement organization” was a little, well, mystifying as was evidenced in my January post, and sadly the final document does little to clarify this term other than to say the “document is principally applicable to Internet Publishers, Ad-serving organizations, Syndicated Measurement Organizations and auditors” on the IAB web site.

This definition is important since in my last post the real conundrum appeared to be that if “measurement organization” included Omniture, WebTrends, Google, Coremetrics, etc. then the IAB was essentially saying that the vendors needed to change the way they reported Unique Visitors, at least for their clients who would be subject to the perview of the IAB and MRC.  What’s more, George Ivey from MRC never got back to my repeated requests for information, despite two members of the IAB working group (Josh Chasin from comScore and Pete Black from BPA Worldwide) openly disagreeing in their interpretation of the definition …

Well, a few weeks back I got a call from Joe Laszlo, an old co-worker of mine at JupiterResearch who is now the IAB’s Director for Analytics, the guy basically responsible for the document.  I always liked Joe and it was nice to hear from him again.  And Joe did clarify for me what a “measurement organization” is … he just didn’t directly clarify the impact on web analytics vendors.

According to Joe (and he will surely correct me publicly if I am misinterpreting our conversation) the “measurement organizations” that should be guided by this new definition of “Unique Users” are publishing organizations who are outwardly reporting their metrics for consideration by advertisers in the open market. Companies like AOL, Weather.com, ESPN, etc.  This is, I think, much more clear than the sentence a few paragraphs up that includes “Syndicated Measurement Organizations and auditors” and puts at least this part of the document in context: Essentially when using numbers coming from census-based systems, the IAB and MRC want publishers to start reporting Unique Visitor counts that have some basis in reality.

Pretty hard to disagree with Joe and the IAB on that point. We all pretty much agree that cookie-based visitor counting is messed up, and I think we can even agree that the degree to which these counts are “messed up” is a function of the target audience, the duration under examination, and the type of site.  For example, we expect cookie-based counts on sites that attract highly technical users on a daily basis to be much more impacted over a 90-day measurement period than, say, sites that attract largely non-technical users on a monthly basis over the same 90-day period.

So I’ll make one really bold statement right now, the kind that I have a tendency to regret but hey, it’s Monday and I’m feeling pretty good about the coming week:

The IAB are to be applauded for taking such a bold stand on the subject of counting and reporting unique visitors based on what we traditionally consider “web analytic” data.

I said as much in my last post … right after I said that the likelihood of the web analytics vendors following these recommendations was about the same as everyone waking up tomorrow to realize that the financial meltdown was a bad dream and the Dow is still over 14,000 (zero). The team of folks that the IAB brought together, which I understand included both Omniture and WebTrends, should be congratulated for taking a firm stand on one of the most dogged issues plaguing our collective industries (web analytics, online advertising, online publishing, syndicated research, etc.) for at least the past five years.

It is about time that we all agreed that “Unique Visitor” reports coming from census-based technologies frequently have no basis in reality. Further, we should all admit that cookie deletion, cookie blocking, multiple computers, multiple devices, etc. have enough potential to distort the numbers as to render the resulting numbers useless when used to quantify the number of human beings visiting a site or property.

Yes, before you grieve on me with your “but they are probably directionally correct” response I agree with you, they probably are, but fundamentally I believe that advertising buyers are at least as interested in the raw numbers as they are the direction they are moving. I say “probably are” because if you’re not taking the IAB’s advice and reconciling census-based data with data derived directly from people, well, you’re never sure if that change in direction is because your audience is changing, technology is changing, or there is a real and substantial increase or decline.

I mentioned above that my conversation with Joe didn’t really clarify the impact on web analytics vendors under the IAB’s new definition. Since I spent a fair amount of time thinking about the IAB guideline’s impact in this regard, I will make another bigger and bolder statement:

Starting immediately, I think the web analytics vendors and any company reporting a cookie-based count that is not in compliance with the IAB’s definition of “Unique Visitor” should stop calling said metric “Unique Visitors (or Users)” and correctly rename the metric “Unique Cookies”.

Yep, I am 100% in favor of using the IAB’s new terminology and being semantically precise whenever possible. The “Unique Visitor” counts in the popular web analytics applications are always actually counting cookies and so we should just go ahead and say that explicitly by calling them “Unique Cookies”. This change would actually give the web analytics vendors a neat opportunity … to battle to be the first to have a real “Unique Visitor” count that is based, as the IAB has suggested, on underlying data that is, at least in reasonable proportion, attributed directly to a person.

How could they do this? Let me count the ways:

1. Develop a standard practice around the use of log-in and registered user data
2. Work with third-party partners who are focused on gathering more qualitative data (for example, Voice of Customer vendors like ForeSee Results)
3. Work with third-party partners who are estimating cookie-deletion rates, or at least have the potential to (for example, Quantcast)
4. Work with third-party partners who can actually calculate cookie-deletion and multiple-machine use rates with some accuracy (for example, comScore, Google, Yahoo!)

I’m sure there are a few ways I am not thinking of, but these are the big four that have been talked about since 2005. While I expect to get some grief from paying clients about this statement, and I fully expect my suggestion to be widely ignored by the vendor community (no offense taken), I think this change would be a big step towards the recognition that there is only ONE DEFINITION of a “Unique Visitor” and this definition is only tangentially related to the number of cookies being passed around.

Like Soylent Green(TM), “Unique Visitors” are PEOPLE and our industry will go a long way towards maturation when we collectively agree on this fundamental truth.  It is not to say that Unique Cookies is not a valuable count — hell, in the absence of a strategy for reconciling cookies against people-based data unique cookies are all we have. But I do not believe that after nearly 15 years we are doing the online measurement community any justice by plugging our ears and signing “LA LA LA LA I CANNOT HEAR YOU GO AWAY!!!!!”

Which brings me to my last point …

I was really, really bummed out to read Jodi McDermott’s MediaPost article titled “Unique Visitors Come in Two Shapes and Sizes.” I was bummed because I have always liked Jodi since we worked together at Visual Sciences, because I think she is a brilliant member of our community, and because I knew I was going to end up writing these words … Jodi’s thesis is wrong and does the web analytics community a dis-service in attempting to defend a mistake by asking to water down a good definition just because it isn’t “hers” (in quotes since Jodi is a member of a larger committee charged with defining standards within the WAA.)

From Jodi’s article (which I recommend you read, especially the comments, and the emphasis is mine):

“Bravo to the IAB for forcing the issue with audience measurement companies to standardize the way that they report uniques, but from a Web analyst’s perspective — and as a member of the WAA Standards committee — I wish they would have not allowed the term “unique visitors” to be redefined in such a way as to allow for multiple definitions in the space. Web analysts and media planners today have a hard enough time trying to figure out which data source to use and which standard to apply when performing their job — but that issue is now compounded even more by multiple definitions of unique visitors. In defense of the IAB, its membership is comprised of some heavy-hitter companies who are not about to change that “tab” in their reporting UI that says “Unique Visitors” on it.  But in defense of  WAA individual and company members, which include vendors such as Omniture and WebTrends (who were both listed as “Project Participants” on the IAB document, interestingly enough), neither are we. The term will live on in both places.”

I think what Jodi has missed here is that the IAB has actually given the world a useful and more accurate definition of “Unique Visitors” than any used in the web analytics industry today. More importantly, given the relative weight, clout, and respect enjoyed by the IAB in the wider world, I don’t think their definition allows for “multiple definitions” … I rather think that over time the IAB expects their member companies, especially those who want to have their numbers audited and publicly used, will consider the IAB definition the definition of “Unique Visitors” and properly consider the term we web analysts widely use today to be “Unique Cookies.”

I’m not sure what Jodi means by “heavy-hitter companies who are not about to change their “tab”” since I’m aware of very few companies today that have implemented the IAB recommendation for practical and ongoing use. But I was incredulous when I read the statement regarding using the IAB’s new definition, “in defense of the WAA individual and company members, which include vendors such as Omniture and WebTrends, neither are we. The term will live on in both places.”

Seriously? Rather than start calling our cookie counts “Unique Cookies” and having a rational conversation with our bosses to explain that the technology we use is limited in its ability to discern real people, you prefer to throw down the gauntlet with the IAB and say “screw your definition?” Despite the criticism that has been both wrongly and rightly heaped on the WAA’s “standard” definitions, despite the considerable group that crafted the IAB’s definitions, and considering the fact that the WAA’s definition is wrong, you want to pick a fight?

Two wrongs never make a right, and you’re wrong twice here. Sorry.

I am not on the WAA Standards Committee, I am not on the WAA Board of Directors, and my dues with the WAA are about to lapse so I have no basis for representing the organization. Perhaps reading more into Jodi’s post given my knowledge of her passionate work in the WAA, but I would strongly encourage the current Board of Directors to examine Jodi’s statements in the context of the IAB relationship and the “bigger picture” at play.  Because while Jodi may speak for the WAA Standards Committee and by extension the entire WAA, she certainly does not speak for me.

I will gladly use the term “Unique Cookies” when I am talking about a cookie-based count and reserve the term “Unique Visitors” for those situations where I have some basis for doing so. More importantly I will encourge my clients and vendor friends to consider doing same. The IAB has given the entire measurement community a reason to take a huge leap forward and gain clarity around one of our most important metrics. To turn our back on this opportuntity because it will necessitate change, require additional explanation, or because “we like our definition better” is wrong, wrong, wrong.

Harrrrumph.

I suspect like previous posts on the subject this will generate some conversation. As usual I do not pretend to have all the answers and I welcome your feedback. I am, unfortunately, traveling all day Monday and will have limited ability to approve and respond to comments but I promise to do so as quickly as possible.