Analysis, Social Media

If the Data Looks too Amazing to Be True…

I’ve hauled out this same anecdote off and on for the past decade:

Back in the early aughts [I’m not Canadian, but I know a few of ’em], I was the business owner of the web analytics tool for a high tech B2B company. We were running Netgenesis (remember Netgenesis? I still have nightmares), which was a log file analysis tool that generated 100 or so reports each month and published them as static HTML pages. It took a week for all of the reports to process and publish, but, once published, they were available to anyone in the company via a web interface. One of the product marcoms walked past my cubicle one day early in the month, then stopped, backed up, and stuck his head in: “Did you see what happened to traffic to <the most visited page on our site other than the home page> last month?” I indicated I had not. We pulled up the appropriate report, and he pointed to a step function in the traffic that had occurred mid-month — traffic had jumped 3X and stayed there for the remainder of the month.

“I made a couple of changes to the meta data on the page earlier in the month. This really shows how critical SEO is! I shared it with the weekly product marketing meeting [which the VP of Marketing attended most weeks].”

I got a sinking feeling in my stomach, told him I wanted to look into it a little bit, and sent him on his way. I then pulled up the ad hoc analysis tool and started doing some digging and quickly discovered that a pretty suspicious-looking user-agent seemed to be driving an enormous amount of traffic. It turned out that Gomez was trying to sell into the company and had just set up their agent to ping that page so they could get some ‘real’ data for an upcoming sales demo. Since it was a logfile-based tool, and since the Gomez user agent wasn’t one that we were filtering out, that traffic looked like normal, human-based traffic. When the traffic from that user-agent was filtered out, the actual overall visits to the page had not shown any perceptible change. I explained this to the product marcom, and he then had to do some backtracking on his claims of a wild SEO success (which he had continued to make in the course of the few hours since we’d first chatted and I’d cautioned him that I was skeptical of the data). The moral of the story: If the data looks too dramatic to be true, it probably is!

This anecdote is an example of The Myth of the Step Function (planned to be covered in more detail in Chapter 10 of the book I’ll likely never get around to writing) — the unrealistic expectation that analytics can regularly deliver deep and powerful insights that lead to immediate and drastic business impact. And, the corollary to that myth is the irrational acceptance of data that shows such a step function.

Any time I do training or a presentation on measurement and analytics, I touch on this topic. In an agency environment, I want our client managers and strategists to be comfortable with web analytics and social media analytics data. I even want them to be comfortable exploring the data on their own, when it makes sense. But, (or, really, it’s more like “BUT“), I implore them that, if they see anything that really surprises them, to seek out an analyst to review the data before sharing it with the client. More often than not, the “surprise” will be a case of one of two things:

  • A misunderstanding of the data
  • A data integrity issue

All of this is to say, I know this stuff. I have had multiple experiences where someone has jumped to a wholly erroneous conclusion when looking at data that they did not understand or that was simply bad data. I’d even go so far as to say it’s one of my Top Five Pieces of Personal Data Wisdom!

And yet…

When I did a quick and simple data pull from an online listening tool last week, I had only the slightest of pauses before jumping to a conclusion that was patently erroneous.

Maybe it’s good to get burned every so often. And, I’m much happier to be burned by a frivolous data analysis shared with the web analytics community than to be burned by a data analysis for a paying client. It’s tedious to do data checks — it’s right up there with proof-reading blog posts! — and it’s human nature to want to race to the top of the roof and start hollering when a truly unexpected result (or a more-dramatically-than-expected affirming result) comes out of an analysis.

For me, though, this was a good reminder that taking a breath, slowing down, and validating the data is an unskippable step.

Analytics Strategy

The Privacy Apogee

The biggest topic that you will grapple with in 2011 is consumer privacy. We are at the most liberal and lenient point of consumer privacy in the history of time. It’s primarily because digital data is spewed by consumers with each click, like, Tweet, share, and update with reckless abandon. Consumers are barely aware of the digital footprints they’re creating and we don’t know how to handle it. There are no rules here.

Consumers are racing to new digital medium at breakneck speeds to be early adopters of the next best thing and are literally addicted to digital. Our obsession is so ravenous that almost half of smartphone users will wake up in the middle of the night to check for digital updates. It’s not their fault really, in fact I include myself in this frantic race to get the newest browser, the latest app, or to connect with nearly anyone who asks. Heck, I downloaded the Owner’s Manual to a Hyundai on my iPad within seconds of watching a TV commercial just because I could. I have no idea what data Hyundai now has on me and if or when I’ll start receiving ads or emails containing must-have offers for a car that I probably won’t ever buy (although it looks sweet!). My point is that we’re on the precipice of a substantive change in the way that consumer data is collected and utilized. If we (and by “we” I mean we digital measurers, organizations and institutions) don’t get our acts together in the first quarter of Q1 then we will have regulation forced upon us.

In my opinion, the number one most critical component for even getting off the ground with privacy protection is education. We must educate consumers, organizations, developers and governments to have a meaningful conversation about privacy. If we fall short of that, ignorance about how data is collected, how it’s used, and who uses it, will continue to be vilified by consumers and media sources that don’t know What they Know.

To that end, I’m working on a concept that I’m calling the Privacy Apogee.

Those of you who are up to speed on your celestial mechanics will know that an apogee reflects the furthest point of orbit from earth. What I seek to explore is the farthest point of ethical data collection from a consumer. My working diagram above depicts your average consumer at the epicenter of privacy and the way we track his digital activities using technology that extends from innocuous to invasive. My plan is to flesh out this concept with current tracking capabilities and potential consumer benefits. Moreover, I intend to create a blueprint for accountability. Ultimately the goal is to produce an infographic that conveys several things:

For consumers

      – The Privacy Apogee will illustrate data tracking capabilities that exist today and highlight some of the benefits of opting-in to these tracking practices.

For developers – It will offer guidance on what methods of data to collect and how to communicate data collection, storage and utilization practices in clear language.

For organizations – The Privacy Apogee will illustrate just how far – is too far – by showing what’s technically possible and what’s morally ethical.

In creating this work, I hope to educate and inform the masses by offering a public service that will open some eyes to the critical imperative for self-regulation before we have governmental mandates forced upon us. The Privacy Apogee will illustrate current technological capabilities for tracking consumers’ digital actions and offer both positive and negative repercussions of those actions.

So back to you Captain Blackbeak…I’m listening and this is what I’m doing to create change. It’s a change in perception. A change in education. And a change in direction for our industry. But like you, I cannot do this alone and need the support and mindshare of our industry. With the help of my partner Eric and the industry #measure pros out there my goal is to crowd source this idea to ensure that I’ve fully considered the technology capabilities and the benefits of tracking practices, So I need your help. The Web Analyst’s Code of Ethics is one part of this, but I’ll be working to define the pros and cons of data collection and the methods by which we accomplish our task. Stay tuned for more, as this is just the beginning…

But in the meantime, what do you think?

Analytics Strategy, Social Media

Is It Just Me, or Are There a Lot of #measure Tweets These Days?

<Standard “good golly I haven’t been blogging with my planned weekly frequency / been busy / try to get back on track in 2011” disclaimer omitted>

Update: This update almost warrants deleting this entire post…but I’m going to leave it up, anyway. See Michele Hinojosa’s link in the comment for a link to an Archivist archive of #measure tweets that goes back to May 2010 and doesn’t show anything like the spike the data below shows, and also shows an average monthly tweet volume of roughly 3X what the November spike below shows. Kevin Hillstrom also created a Twapper Keeper archive back in early November 2010, and the count of tweets in that archive to date looks to be in line with what the Archivist archive is showing. So…wholly invalid data and conclusion below!!!

Corry Prohens’s holiday e-greeting email included a list of hist “best of” for web analytics for 2010, and he really nailed it. That just further validates what all web analysts know: Corry is, indeed “Recruiter Man” for our profession. He’s planning to turn the email into a blog post, so, I’ll sit back and wait for that. But, I did suggest that the #measure hashtag probably deserved some sort of shout out (I actually dubbed #measure my “web analytics superhero-sans-cape” in my interview as part of Emer Kirrane‘s “silly series”).

That got me to thinking: how much, really, has the #measure community grown since it’s formal rollout in late July 2009 via an Eric Peterson blog post?

10 minutes in my handy-dandy online listening platform, and I had a nice plot of messages by month:

Yowza! My immediate speculation is that the jump that started in October was directly related to the Washington, D.C. eMetrics conference in the first week of October — the in-person discussions of social media, combined with the continuing adoption of smartphones, combined with the live tweeting that occurred at the conference itself (non-Twitter users at the conference picking up on how Twitter was being effectively used by their peers). That’s certainly a testable hypothesis…but it’s not one I’m going to test right now (add a comment if you’ve got a competing hypothesis or two — maybe I will dive a little deeper if we get some nice competing theories to try out; this will definitely — the horror! — fall in the “interesting but not actionable” category, so, shhhh!!!, don’t point your business users to this post!).

It’s also possible that the data is not totally valid — gotta love the messiness of social media! I’d love to have someone else do a quick “conversation volume” analysis of #measure tweets to see if similar results crop up. Unfortunately, Twitter doesn’t make that sort of historical data available, I shut off my #measure RSS feed archive a few months ago, and, apparently, no one (myself included) ever set up a TwapperKeeper archive for it. So, I can’t immediately think of an alternative source to use to check the data.

Thoughts? Observations? Harsh criticisms? Comment spammers (I know I can always count on you to chime in, you automated, Akismet-busting robots, you!)?


Analytics Strategy

Santa Puts Aprimo Under the Tree!

2011 is shaping up to be the year of big marketing. And luckily for us measurers, smart marketing is founded in data and measurement. With IBM’s recent acquisition rampage and now Teradata’s plans to buy Aprimo, there is unprecedented choice for integrated enterprise marketing solutions. Teradata announced today it’s intentions to buy the Enterprise Marketing Management leader for $525M with a closing date anticipated for sometime in Q1 2011. It’s a smart move in my opinion because the days of big data management and the ability to harness the consumer data firehose for elevated marketing are upon us.

On the executive briefing this morning, I pointed a question by asking if this acquisition was a response to IBM’s recent buying spree and the answer was a definitive no. Bill Godfrey, Aprimo’s Chief Executive Officer, quickly pointed out that Aprimo’s technology set covers 8 categories and that only one competes directly with the IBM/Unica offering. He iterated, “This is not a copy-cat move” with mild umbrage. Mr. Godfrey went on to eloquently explain that the merger pursues an independent strategy that brings a unified platform covering a very broad end-to-end spectrum of functionality. While the story sounded familiar, it’s a good one. It leverages the database storage and business analytics capabilities of Teradata and layers the marketing management and operations proficiency of Aprimo on top. This enterprise-ready integrated solution fuels a marketers’ paradise where insights are churned from data, which pumps intelligent life into automated marketing. All this happens within a closed-loop system that improves over time. Sounds rosy doesn’t it? To paraphrase Teradata’s CMO Darryl McDonald, “The combined solution will help accelerate revenue generating campaigns and leverage data for strategic insights and quick response.

Keep in mind that this isn’t entirely new territory for Teradata who has been offering marketing products to its customers for some time. With IWI (Integrated Web Intelligence) and TRM (Teradata Relationship Manager), it’s already servicing digital data integration and intelligent marketing to it’s customers. Yet, it will be interesting to see how many existing clients and new organizations adopt this complete functionality. My hunch is that this stack is not for the feint of heart nor the bootstrapped organization. It will work best with deeply integrated datasets, stored within big iron and activated using some complex Marketing Resource Management capabilities. All things that both Teradata and Aprimo excel at. But fair warning: Mom & Pop shops need not apply. However, if you’re a large enterprise looking to accelerate your marketing prowess, then this may be the solution you’ve had on your wish list all these years.

While integrating these technologies may take a while, and the promise of an end-to-end solution is no trivial pledge, I’m bullish on the deal. This is a step forward for marketers because it has the potential to deliver the ERP system they never had. It still doesn’t cover everything, but the combined solution sure does handle some critical moving parts.

Congrats to everyone at Aprimo for building an attractive offering and to Teradata for recognizing it. And Happy Holidays to all!

Adobe Analytics, General

Tracking Form Errors (Part 3)

(Estimated Time to Read this Post = 4 Minutes)

In this series of blog posts, I have been talking about how to see what types of Form Errors your website visitors are receiving so you can improve conversion. So far, we have learned how to see how many Form Errors your website is getting, which fields are causing those and how many Form Errors you get per Form and Visit. As my regular readers know, I like to go beyond the basics, so now we are going to kick it up a notch and get into some real fun stuff. Fasten your seat belts!

Which Fields on Which Forms?
In my first post of this series I shared a simplistic way to learn which form fields caused errors using a List sProp. However, correlating this to specific forms was a bit trickier. Here I will show how to do this, even if you don’t have Discover. The trick here is to set a Form Errors eVar that stores all of the fields which had an error when the above Form Errors Success Event is set. Since eVars have a longer character length, this should be possible for most forms that aren’t too long (which they shouldn’t be anyway!). I like to do this by concatenating the field values into one long string with a separator between each field. Here is an example of the report you want to have:

This report will look a bit like the one I described in the previous post, but as you will see, it is much more powerful since it is in the conversion area and can take advantage of Conversion Subrelations. Besides being able to see which combination of field errors are troubling users, you can open your Form ID reports, find a specific form and then break it down by this new Form Error eVar to see the specific form fields causing problems by form as shown here:

Using this report, we can see that for the first form shown above, 66% of the times visitors get a Form Error, they had eight form field errors (or left them blank). This data, when coupled with observational data using a tool like ClickTale can be invaluable in driving increased form conversions!

What % of Required Form Fields Have Errors?

While the above report, which shows Form Field Errors by Form, is powerful, one question it doesn’t answer is: How many of the required fields on my forms are not being filled out by users? The answer to this question can help you figure out which fields should/shouldn’t be required. So to answer this question, what you want to do is to look at each form that loads on your website and calculate how many fields the user received an error for and then divide that number by the total number of required form fields. For example, if you have a form with eight required fields, and the current user received two errors on that form, the calculation would be 2/8 or 25%. You should then pass this 25% value to an eVar when you are setting the Form Errors Success Event. Once you do this for all forms, you will have a report that looks like the one shown here. Using this report we can see that the highest number of Form Errors are cases where users are getting errors on every field (which is most likely people leaving all fields blank). Maybe our users don’t realize that these fields are required and we can do some testing to create a better experience or reduce the number of required fields?

If we want to see which forms are the ones that have the highest 100% Form Field Error Rate, all we need to do is break the above report down by Form ID:

Finally, if you are doing a good job of grouping your website forms using SAINT Classifications, you can see some super-cool reports. In the following report, I have grouped all of my website forms into high-level buckets of Demo and Free Trial. Then I broke this report down by the percentage of required fields that result in Form Errors.

You can see here that most website visitors on Demo forms are getting errors for 100% of the fields (probably leaving them blank!), while for the Free Trial, the largest percentage of required fields with errors is 10%. Interesting data indeed!

Final Thoughts
In this post, we have covered some advanced ways to see which fields produce errors on each form, see this by form and seen how to know which forms have the highest total required field error rates. These reports can provide an enormous amount of insight into what is happening on your forms with respect to errors and once you understand your visitor’s form behavior, you can apply these learnings to all forms on your site. In my next post, I will cover a tangentially related item (related to Forms, but not as much about Form Errors) that I think is super-cool.

Between this post and the last post, hopefully you have some food for thought when it comes to tracking how your website forms are doing so you improve your conversion rates…

General

Commerce Department and WAA Code of Ethics

Thanks to Tim Evans I was alerted to a report about the Commerce Department weighing in on privacy issues online.  Suffice to say I agree with the direction Commerce is giving the Obama administration.  Specifically the idea that, according to CNN’s Money, “the government ‘enlist the expertise and knowledge of the private sector’ to create ‘voluntary codes of conduct that promote informed consent and safeguard personal information.'”

More or less exactly what John Lovett and I proposed back in September of this year.

I have started reaching out to the media on this point — that we in the digital measurement community are already taking matters into our own hands and stepping up — but we could use your help! Please, if you know anyone in the press, send a link to this blog post along to them and help spread the word that as a community we can take responsibility for our own actions and we are willing to do what is right for consumers around the globe.

This issue affects all of us in the digital measurement sector — vendors, consultants, and practitioners alike. Please help us create awareness about our efforts.

Here are links to the relevant background materials:

Here is the link to the near-final draft of the Web Analysts Code of Ethics:

  • Last chance to shape the Web Analysts Code of Ethics (WAA Blog)

The Standards sub-committee for the Code of Ethics met yesterday and as I publish this blog post John Lovett is presenting the final version to the Web Analytics Association Board of Directors.  We expect the Code to be available to sign at the WAA web site in the coming weeks.

Adobe Analytics

Phew…!

Phew. It’s been crazy weeks for me lately. At the moment, we just put up the tree, kids are all quiet and I’m drinking a glass of red. It’s one of the rare moments these days that I have in solace…and it’s gone…the littlest one is squirmy with hiccups.

Okay, I’m back. Made a bottle and made the hand off to Mommy. I haven’t blogged in a long while and there so much to say but I just haven’t had time. So here’s the johnlovett highlight reel for Fall 2010:

  • We welcomed a new baby into our home. And that makes three. Three boys that is. I always thought the jump from one kid to two was really no problem. But, I can tell you that increasing the number of kids another 33% 50% is a big jump indeed. [The 33% designates the percentage of quantitative reasoning skills I’ve lost in the past month.] Our house is busier than ever with an 18-month old climbing the walls and an eldest brother at five running the show. Everybody is happy and healthy so I’m immensely grateful for the lack of sleep and craziness.
  • I’m writing a book for Wiley on Social Media Metrics. And it’s one of the hardest things I’ve ever done. I’ve got the story in my head and know what I want to write, yet cranking out 40 page chapters every other week is really tough. I’m nearly half way through my manuscript and I love the way its coming together. Although, if you’ve got a social analytics story of smashing success, miserable failure or sheer brilliance, I’d love to talk with you. I could always use more.
  • My business is off-the-charts busy. Looking back on twelve months since joining Demystified and I couldn’t be happier. It’s been a great year and the work I’m doing is motivating me to maintain work-a-holic proportions. Since Labor Day I spent 8 weeks on the road visiting clients, working on changing our industry and speaking at events from coast to coast with a business trip to Italy as a big November finale. I made it home with four days to spare before the baby was born. Whew.
  • And I’m happier than I’ve ever been. Who knew that chaos could be so rewarding? I always knew this was the case, but I love my job and I truly love the #measure industry. As measurers of digital medium, our roles are about to become indispensable. We’re on the precipice of a big data explosion and we’ll have the skills to float to the top. Big data is going to rush like a flood over enterprises and marketers alike and we measurers will be ready to slice and dice our way to sensibility. I like our chances.

More to follow on all these topics as I’m working three concurrent projects, writing two white papers and working through book chapters at present… Oh, and it’s my turn to change diapers, so I’m out.

Talk to y’all soon.
John

Adobe Analytics

Tracking Form Errors (Part 2)

In my last post, I started the process of identifying which form fields were producing the most errors. In this post, I will cover some related topics that will allow you to quantify how often you are getting Form Errors and how effective, in general, your forms are at converting website visitors.

How Many Form Errors Are You Producing?
While the solution I identified in my last post showed which form fields had more errors than others, in the web analytics space, we like hard, concrete numbers! Therefore, I would recommend that you set a Success Event each time website visitors encounter at least one form error (assuming you do validation when the Form Submit button is clicked). By setting a Success Event, you will have a nice chart that shows you the overall trend of Form Errors as shown here:

If you are passing a Name or ID for each form you have on your website, you can also use this Success Event to see which forms are getting the most number of errors like this:

In addition, you can set an Alert for the overall Form Error metric or for a specific Form Name/ID:

 

How Is Each Form Doing?
While knowing how many Errors a form gets is cool, as is often the case, we in the web analytics field care more about ratios! In the report above, it is alarming to see that the first form had 85 Form Errors but how do we know if that is good or bad? If we create a Calculated Metric to compare Form Errors to Form Views, we can see how many Form Errors visitors had in relation to each time the same Form was viewed. Based upon the data below, we can see a wide range of Form Error percentages depending upon the form:


Some of these percentages are quite high and represent amazing opportunities to do testing to see if they can be improved! In addition, when you create a calculated metric, besides just seeing it in an eVar report like the one above, you can also see it as a standalone metric. This means that you can see the overall trend of Form Errors per Form View (or Visit) to see if we are getting better or worse over time. This might make a great KPI metric for the team focused on Forms and Form Completions:

Final Thoughts
In my last post I covered a simple way to see which fields are causing problems for your visitors. In this post, I showed you how to quantify your Form Errors, see how much of an issue you may have and even see which Forms have the most Errors. In my next post I will show you some advanced ways to see which fields are causing errors and how to break this down by Form. Stay tuned!

Between this post and the last post, hopefully you have some food for thought when it comes to tracking how your website forms are doing so you improve your conversion rates…

Analysis, Reporting

Reporting: You Can't Analyze or Optimize without It

Three separate observations from three separate co-workers over the past two weeks all resonated with me when it comes to the fundamentals of effective analytics:

  • As we discussed an internal “Analytics 101” class  — the bulk of the class focusses on the ins and outs of establishing clear objectives and valid KPIs — a senior executive observed: “The class may be mislabeled. The subject is really more about effective client service delivery — the students may see this as ‘something analysts do,’ when it’s really a a key component to doing great work by making sure we are 100% aligned with our clients as to what it is we’re trying to achieve.”
  • A note added by another co-worker to the latest updated to the material for that very course said: “If you don’t set targets for success up front, someone else will set them for you after the fact.”
  • Finally, a third co-worker, while working on a client project and grappling with extremely fuzzy objectives, observed: “If you’ve got really loose objectives, you actually have subjectives, and those are damn tough to measure.”

SEO search engine optimization indiaIt struck me that these comments were three sides to the same coin, and it got me to thinking about how often I find myself talking about performance measurement as a critical fundamental building block for conducting meaningful analysis.

“Reporting” is starting to be a dirty word in our industry, which is unfortunate. Reporting in and of itself is extremely valuable, and even necessary, if it is done right.

Before singing the praises of reporting, let’s review some common reporting approaches that give the practice a bad name:

  • Being a “report monkey” (or “reporting squirrel” if you’re an Avinash devotee) — just taking data requests willy-nilly, pulling the numbers, and returning them to the requestor
  • Providing “all the data” — exercises of listing out every possible permutation/slicing of a data set, and then providing a many-worksheeted spreadsheet to end users so that they can “get any data they want”
  • Believing that, if a report costs nothing to generate, then there is no harm in sending it — automation is a double-edged sword, because it can make it very easy to just set up a bad report and have it hit users’ inboxes again and again without adding value (while destroying the analyst’s credibility as a value-adding member of the organization)

None of these, though, are reasons to simply toss reporting aside altogether. My claim?

If you don’t have a useful performance measurement report, you have stacked the deck against yourself when it comes to delivering useful analyses.

Let’s walk through a logic model:

  1. Optimization and analysis are ways to test, learn, and drive better results in the future than you drove in the past
  2. In order to compare the past to the future (an A/B test is a “past vs. future” because the incumbent test represents the “past” and both the incumbent and the challenger represent “potential futures”), you have to be able to quantify “better results”
  3. Quantifying “better results” mean establishing clear and meaningful measures for those results
  4. In order for measures to be meaningful, they have to be linked to meaningful objectives
  5. If you have meaningful objectives and meaningful measures, then you have established a framework for meaningfully monitoring performance over time
  6. In order for the organization to align and stay aligned, it’s incredibly helpful to actually report performance over time using that framework, quod erat demonstrandum (or, Q.E.D., if you want to use the common abbreviation — how in the hell the actual Latin words, including the correct spelling, were not only something I picked up in high school geometry in Sour Lake, TX, but that has actually stuck with me for over two decades is just one of those mysteries of the brain…)

So, let’s not just bash reporting out of hand, okay? Entirely too many marketing organizations, initiatives, and campaigns lack truly crystallized objectives. Without clear objectives, there really can’t be effective measurement. Without effective measurement, there cannot be meaningful analysis. Effective measurement, at it’s best, is a succinct, well-structured, well visualized report.

Photo: Greymatterindia

Adobe Analytics, General

Tracking Form Errors (Part 1)

Almost all websites have forms. Whether you are a B2B/Lead generation site, an eCommerce site, a travel site, etc… you most likely have forms. More importantly, you have people who don’t fill out your forms correctly and get some sort of error message. While error messages are a fact of life, in the web analytics/optimization world these are painful since you work so hard to get people to your site, to read your content and then agree to give you personal information. That is a lot of time and money spent only to have someone potentially abandon because they have problems with your forms. This represents your “low hanging fruit” so to speak – people who have already decided they like you and want to give you their information! In this series of posts, I am going to share some techniques for seeing how much of a problem your website has with form errors and in the next few posts I will cover some more advanced things you can do to diagnose these form error issues.


Which Fields Produce the Most Errors?
The first step in diagnosing form error issues is understanding which form fields are causing issues. Unfortunately, since a user might receive more than one error message, you have to pass in multiple values to a SiteCatalyst variable. This can be done using the Products variable, but since that is often already being used for more important purposes, I will suggest that you use a List Traffic Variable (sProp) to capture these values. Unfortunately, List sProps are not well documented and have some specific limitations (see Knowledge Base ID# 2305). All you need to know is that List sProps allow you to pass in delimited values and when you view them in the sProp report, these values will be split out. Let’s look at an example. Here we see a form in which a user has attempted to submit the form without filling out some required fields. What we want to do is capture which fields this user messed up (could mean incorrect value or leaving blank) so we can see which ones are messed up the most often. In this case, we see that the form errors are related to Job Title, E-mail Address, Phone #, Company Name and the MSA checkbox.

So in this case we can use a List sProp to capture the fields giving us errors. Here is how it would look in the JavaScript Debugger:

Unfortunately, List sProps are still constrained to the 100 character limit so if you have long forms you are out of luck or you can select the most important form fields to capture. Once you have captured the fields, you can open the sProp report and you will see something that looks like this:

In this case, we can see that we are getting the greatest number of errors on the Phone Number form field on the US website (I have added the site since forms exist in multiple sites). I could also filter this sProp report for just US or Japan form fields by using a text search of “us:” or “jp:” as needed. This report should help steer you in the right direction when it comes to fixing basic form field issues.

Correlating Form Field Errors to Forms
Once you have seen which form field errors, the next logical question is to see which forms had which errors. Unfortunately, one of the limitations of List sProps is that they cannot be used in Traffic Data Correlations. Therefore, if you want to breakdown form field errors by Form, you will need to use the Discover product as shown here:

If you don’t have access to Discover and seeing this type of breakdown is important to you, you may want to consider using the Products variable instead of a List sProp since the Products variable comes with full Subrelations by default (though this implementation will be significantly more difficult). I will also be covering a different way to approach this in my next post so stay tuned!

Final Thoughts
If you are not currently tracking form field errors, hopefully this will give you some ideas on how you can start the process of seeing where you are tripping up your visitors. Keep in mind that this post is just a start and that the next few posts will go into more advanced stuff you can do and how you can identify your biggest opportunities for improving conversion.

Analytics Strategy, Conferences/Community, General

FTC "Do Not Track?" Bring it on …

As the hubub around consumer privacy continues I was gently prodded by a friend to pipe up in the conversation.  While my feelings about how we have ended up in this position are pretty clear, and while my partner John and I have proposed what we believe is a step in the right direction regarding online privacy and the digital measurement community, it seems that some type of ban or limitation on online tracking is becoming inevitable.

Without getting political or debating the reality of what we can and cannot know about online visitors I have a single word response to the FTC:

Whatever.

Before you accuse me of changing my stripes or going completely nuts consider this: If the FTC is able to somehow pull off the creation of a universal opt-out mechanism, and if the browser developers support this mechanism despite clear and compelling reasons not to, and if consumers actually widely adopt the mechanism — all pretty big “ifs” in my humble opinion — then I believe the digital measurement industry will do what I have already described as inevitable:

We will hold a revolution!

Since my tenure at JupiterResearch back in 2005 I have been telling anyone who would listen to stop worrying about counting every visitor, visit, and page view and instead start thinking about statistically relevant samples, confidence intervals, and the algorithmic use of data to conduct analysis.  Yes, you need to work to ensure data quality — of course you do — but you don’t have to do it at the expense of your sanity, your reputation, or your job …

See, it turns out in our community it doesn’t really matter whether we are able to measure 100% of the population, 90% of the population, or even 80% of the population — what matters is that we are able to analyze our visitor populations and that are able to draw reasonable conclusions from that analysis.  Oh, we have to be empowered to conduct analysis as well, but that’s a whole other problem …

Statistical analysis of the data … trust me, it’s going to be all the rage in a few years. I’m not saying this simply because I have a white paper describing the third generation of digital measurement tools that will empower this type of analysis … although I would encourage you to download and read “The Coming Revolution in Web Analytics” (freely available thanks to the generous folks at SAS!)

I’m saying this because every day I see the writing on the wall.  Data volumes are increasing, data sources are increasing, and demands for insights are increasing, all while professional journalists, politicians, and political appointees are supposedly protecting our “God-given right to surf the Internet in peace” without any regard to the businesses, employees, and investors who depend to a greater or lesser degree on web-collected data to provide a service, pay their bills, and make a profit …

Okay, sorry, that was editorializing.  My bad.

Still, rather than wring our hands and gripe about how much the credit card companies know (which is a silly argument given that credit card companies provide tangible value in exchange for the data they collect … it’s called “money”) I believe it is time to do three things:

  1. Suck it up.
  2. Hold yourself to a higher standard.
  3. Buy “Statistics in Plain English” and start reading.

The good news is that we have access to lots and lots of great statistical analysis of sampled data today — we just might not realize it.  Consider:

Have I mentioned Excel, Tableau, and R?  Hopefully by now you get the gist … statistics is already all around us all the time, perhaps just not exactly where we expect it or, in the context of lower rates of data collection, where we will ultimately need it to be.

Perhaps the most encouraging evidence that we will be able to make this shift is the increasing attention the digital world is getting from traditional business intelligence market leaders like Teradata, FICO, IBM, and SAS.  I, for one, am more or less convinced that the gap between “web analytics” and “Analytics” is about to be closed even further … and here’s one guy that seems to agree with me.

We don’t need to thumb our noses at the privacy people — quite the opposite, and to this end John and I will be sitting down with a representative from the Center for Democracy and Privacy and Adobe’s Chief Privacy Officer MeMe Rasmussen at the next Emetrics in San Francisco! We also don’t need to stick our head’s back in the sand and hope this issue will simply go away — it won’t, trust me.

We need to prepare.

Prepare by committing yourself to not being that scary data miner that consumers are supposedly so afraid of; prepare by improving your data quality to the extent that you are able; and prepare by starting to communicate to leadership that it really doesn’t matter if you can count every visitor, every visit, and every page view — what matters is your ability to analyze data using the tools at your disposal to deliver value back to the business.

If you’re not sure how to do that, call us.

Viva la revolution!

DISCLOSURE: I mentioned and linked to lots of vendors in this post which I normally do not do. Some are clients of Analytics Demystified, others are not. If you have concerns about why we linked to one company and not another please don’t hesitate to email me directly.