Thoughts on the proposed IAB Guidelines
UPDATE ON JANUARY 19, 2009: Peter Black from BPA Worldwide who was also on the IAB working group with Josh Chasin, wrote in and disagrees with Chasin’s characterization of who the “Unique User” language is targeting. I have email into the IAB and MRC’s George Ivie to clarify the situation. Watch this blog!
UPDATE ON JANUARY 18, 2009: Josh Chasin from comScore, who was a member of the IAB working group that defined the guidelines described in this post, wrote in to point out that I misinterpreted the IAB’s intent. While their web site clearly says …
“The IAB believes that all companies involved in audience measurement should be audited for their processes. These audits are intended to establish the source of any measurement discrepancies and to find potential solutions.
All measurement companies that report audience metrics have a material impact on interactive marketing and decision-making. Therefore, transparency into these methodologies is critical to maintaining advertisers’ confidence in interactive, particularly now, as marketers allocate more budget to the platform.”
… according to Chasin the IAB is excluding web analytics vendors from “all companies involved in audience measurement” and the type of companies that have a material impact on interactive marketing and decision-making. Since this doesn’t sound right to me at all I will warn the reader that some of the questions I raise in the following post may, in fact, be totally irrelevant (at least in the context of the IAB Proposed Measurement Guidelines.
If nothing else, with two days left in the open comment period, the IAB may want to use my confusion as an example and clarify the target for the recommendations made in the document.
Reader beware!
As long as we’re talking about web analytics standards I figured I would take the opportunity to offer up a few thoughts on the Interactive Advertising Bureau’s Audience Reach Measurement Guidelines that are open for public comment until January 20th. If you haven’t had a chance to read these proposed guidelines you should, especially if you have an interest in how we collectively communicate about data.
At 34 pages the document is certainly a slog to read–and I say this knowing full well that I have a tendency to write 50 page white papers! Since you’re all bright folks I’m just going to address some of the proposed language that stood out to me. And, as always, if you have any thoughts or positions on the proposed guidelines I’d love to hear from you!
Starting in Section 1.2 the IAB clarifies the relationship between “Unique Cookies”, “Unique Browsers”, “Unique Devices” and “Unique Users / Visitors”. The discussion about “Unique Devices” is interesting because this is a clear indication of the impact that mobile devices like the iPhone are having on audience measurement. Things start to get really interesting, however, in Section 1.2.4 where the IAB says (emphasis mine):
“However, in order to report a Unique User, the measurement organization must utilize in its identification and attribution processes underlying data that is, at least in a reasonable proportion,, attributed directly to a person. For instance, data collected from registrants is one possible source that can be utilized in creating a Unique Users measure by a census-based measurement organization, if registrants represent a reasonable proportion of the total user-base and when appropriate scientific projection methods are used for non-registrrants (sic). In no instance may a census measurement organization report Unique Users purely through algorithms or modeling that is not at least partially traceable to information obtained directly from people, as opposed to browsers, computers, or any other non-human element.“
Did you get that? Keep in mind that while at JupiterResearch I was among the first to publicize the decline in accuracy of visitor counting due to cookie deletion. In fact the report was subtitled “Addressing the Decline in Accuracy of Cookie-Based Measurement.” At the time people called me crazy and Seth Godin even accused me of living in an echo chamber (I have since forgiven Seth.)
Now, three years later, the IAB is expressly telling measurement vendors to stop reporting a metric called “Unique Visitors” or “Unique Users” unless they have a research-based strategy for determining the correct proportion of cookies to “real people” and have applied that calculation in a transparent way.
Whoa.
Think about this for a minute. Every one of the fine census measurement packages (nee web analytics) out there is reporting a Unique Visitor number, but I’ll go out on a limb here and propose that none of them are even vaguely adhering to the IAB proposed definition of a “Unique Visitor.” I’ll go a step further and postulate that, at least in the base offerings, these vendors don’t currently have the technical capability required to report an estimated/algorithmically derived “Unique Visitor” count based on scientific projection methods.
If I’m wrong about this I suspect I’ll hear about it, but I don’t think I’m wrong when it comes to the base offerings like SiteCatalyst, WebTrends Web Analyics, Coremetrics 2009, etc. And yes, I’m aware that end-users can use higher-end products like Discover on Premise and the data warehousing tools to apply a correction factor to UV counts, but that is not what the IAB is saying. This guideline is saying that correcting for cookie, browser, and device-related over-counting of unique visitors is the responsibility of the measurement vendor.
Again, whoa.
And as if that’s not a radical enough move, the document goes on to state in Section 2.2 that the vendors need to actually break out these correction factors across three components: first-cookie acceptance, deletion, and browser denial (again, emphasis mine):
“Cookie deletion rates, calibration methods and sources or estimation methods used to account for first-use, deletion, and non-accepting cookie groups should be disclosed by the audience measurement organization. The audience measurement organization should disclose census-based unique cookie counts and the estimated unique activity from first-use, deletion and non-accepting cookie groups separately and in aggregate. If the measurement organization relies on a unified model that makes reporting among these separate groups impossible, it may report these counts in aggregate only, but should be prepared to demonstrate in an audit the ability of its unified model to address each type of cookie completely.”
The IAB goes on in Section 2.4 to start to push web analytics into what is an uncomfortable position for some people, the use of algorithms and data models, to better report on unique visitors:
“As noted above, Publishers and Ad-servers will generally need to rely on algorithms (data models) to estimate the number of users attributable to the counts of Unique Cookies they develop. The underlying basis for this algorithm should be a study of actual users (i.e, people). Ideally, such a study would be based on direct contact and/or observation of people using the browser at the time of accessing web-site content or ads with the unique cookie, as well as observation of the number of browsers in use by these users. Additionally, inferences will need to be made about advertising activity of users with non-cookied browsers, so these types of users should also be contacted and observed. Also, the activity of users who access content from multiple locations (home, work, school) on different browsers should be factored into these algorithms.”
Finally, the IAB is telling the vendors they need to report the results of their research to their customers, essentially exposing flaws in their technology for all to see:
“The resulting study should be representative of, and projectable to, the users of the web-site or property, and periodically re-performed to reflect gradual changes in audience. Known weaknesses in the projection processes should be disclosed to users of Audience Reach Measurements.“
If you’re keeping track, the IAB is telling the vendors A) to completely change their definition of “Unique Visitors”, B) to start to actively research sources of inaccuracy on behalf of their customers, and C) pro-actively report known weaknesses in their system to their customers. Anyone want to place any bets on when the vendor community will adopt these recommendations? I’m going to be a little snarky here and put my money on “never in a million years.”
Seriously you have to love the IAB for putting this out there. Unlike the Web Analytics Association’s Standards which I believe are an excellent start but are a little soft in areas, the IAB is basically telling the measurement vendor community that they are doing the entire world a disservice by reporting unique visitor counts that are complete bollocks and they need to stop doing that post-haste! Okay, maybe I’m over-reading the document but the scope of changes required for any vendor to become IAB-compliant is dramatic, both technically and psychologically.
I’m not sure if Brandt Dainow had seen the IAB proposal when he besmirtched the fine work of the Web Analytics Association’s Standards Committee, but if you compare the two proposal documents (the WAA’s proposal can be found here in PDF form) you will detect a noticable difference. Personally I’m glad that my good friend Judah Phillips bridged the gap between the IAB and WAA and I find myself wondering, at least a little bit, whether the IAB+WAA relationship should be even deeper.
This all brings me to an excellent point that Bryan Robertson made on my last post on standards regarding how standards are defined and moved into common use. Bryan’s thesis is based on the W3C’s move from HTML 1.0 to XHTML and his point is that this transition to the XHTML standard came about because of A) a powerful standards body, B) a vocal community, and C) passionate thought leaders. Regarding a powerful standards body, Bryan specifically make a point that other folks have made, usually behind closed doors:
“Is the WAA powerful enough at this point in time, or do we need to continue to build momentum before the standards can be more bold? For example, is the WAA hand wringing too much over the polite “we’ll share with you if you share with us” arrangement with the IAB over standards definitions? Is the WAA in a tough position in trying to bring practitioners and vendors together at the same table?”
Bonus points to Bryan for willing to be direct on the conflict of interests arising from having two masters, vendor and practitioner. Again, I have nothing but profound respect for Angie and all of the other members of the WAA Standards Committee, but since I do know that vendors participated in the definition process I wonder a little bit how much impact they really had.
Anyway, I’m doing all the talking here and it’s a beautiful day so I will ask what you all think — either about the IAB proposal, Brandt Dainow’s assertion, Bryan’s thesis about the strength of the WAA, or anything else that strikes your fancy. Do you think the IAB standard for “Unique Users” has a snowball’s chance of being widely implemented? Do you think Brandt Dainow makes a good point (even if he does it in a lousy way)? Do you think the WAA may be better off working more closely with the IAB on Standards, given the IAB’s relative might?
My site host assures me that my comments table will not crash again so I look forward to hearing from you all.
