Answers to questions about Visitor Engagement

I have had a ton of great feedback about the white paper Joseph Carrabis and I wrote on Analytics Demystified’s measure of Visitor Engagement.  Some folks have raised very good questions and I wanted to provide some of the answers to those questions here to better socialize the knowledge.

The first question came from Jonny Longden who asked:

“I was wondering if you could clear something up for me regarding your visitor engagement white paper? It is to do with the way the equation is stated:

Σ(Ci + Di + Ri + Li + Bi + Fi + Ii)

Apologies if this is me being unintelligent (I am not a mathematician), but the way I read this is that the result of the equation is the sum of the 7 different index values. However, in your examples on page 32 the VE appears to the be the average of those 7 values. Am I missing something?”

An excellent question and one that several other people asked.  When I wrote the equation I was looking for the simplest possible way to represent the relationship between the indices.  I overdid that and Jonny’s question points that out.  Technically, what I should have written is something like this:

VISITOR ( SESSION ( SUM(Ci + Di + Ri + Li + Bi + Fi + Ii) / 7 ))

Indicating that for every visitor in the set, the sum of indicies based on each visitor’s session needs to be divided by the total number of indices, which in this equation is seven.  The reason I recommend dividing by seven is that the resulting number will be between 0.00 and 1.00 every time, yielding a nice, clean number that can be translated to a percentage for communication’s sake.

My next question came from Nikolay Gradinarov who asked:

“Have you considered weighing the different indexes that are part of the Visitor Engagement calculation?”

Yes, I have considered weighting the different indices, as does nearly everyone who looks at the calculation.  The reason I don’t apply any weighting is that I don’t have any way to know what weighting to apply.  Put another way, since I don’t have another measure of engagement, I don’t have any basis for using weighting to correct components of the equation; thusly, at least to me, applying differential weighting to any of the indices seems contrived and likely to increase the complexity of explaining the calculation more than anything.

That said, those of you using the calculation are free to apply whatever weighting you like.  For example, if you didn’t have a good way to calculate the Feeback Index and wanted to exclude it from the calculation of Visitor Engagement, you would simply “zero weight” that index.  Or, if the HIPPO said that “duration is at least three times as important as anything else as a measure of engagement” then you could multiply the Duration Index value by three.

Keep in mind, relative to the last question, doing so will change the mathematics.  If you’re three weighting one index then you’ll either need to divide by 9 (i.e., six “1 weighted” indices plus a three weighted index) to get a value between 0.00 and 1.00 or understand that for some sets you’ll have a number greater than 1.00.

The final question I wanted to treat here comes from Elizabeth Robillard who asked, and I paraphrase since she sent quite the document, whether it was better to calculate the Recency Index using all of the sessions in the set or just the two most recent sessions.  Elizabeth’s point was that when she applied my Recency Calculation using all sessions and the two most recent sessions to a variety of made up situations she was confused by what the data was telling her regarding engagement.

The short answer is that Elizabeth (and any of you) can use whatever sessions you’d like when making the Recency Index calculation.  Again, if you choose to base your calculation on only the two most recent sessions then you’re functionally “zero weighting” the other sessions in the set, which is another assumption but one you’re free to make.

But the main point I would make here is that Elizabeth appears to be trying to examine a single component index and make a statement about the level of Visitor Engagement.  Tempting, I know, but not how the Visitor Engagement calculation is designed to be used.  The reason I have spent so much time evangelizing for/thinking about/understanding the mathematics behind/debating/etc. is that I believe that all of the component indices are required to understand the nuances of Visitor Engagement.

To better understand why I believe this, re-read the section on “Why a New Measure of Engagement” on pages 10 to 15.  Trying to determine the level of engagement of a visitor by looking only at the Recency Index is just like trying to make the same determination using nothing more then raw or average session duration data — I do not believe that a single measure or metric has the resolving power to determine the level of Attention that a visitor is paying to your web site (or whatever object you’re trying to measure.)

So you could have a very low recency between the two most recent sessions, yielding a Recency Index of 100% based on Elizabeth’s suggestion, but a 0% score for Click-Depth, Duration, Loyalty, Interaction, Brand, and Feedback … which would show a visitor who has been to your site twice recently but is otherwise paying no Attention.  Similarly, you could have someone who hasn’t been to the site in the last 30 days, yielding a Recency score of around 3% (1/30) but who had high scores for Click-Depth, Duration, Interaction, Brand, and Feedback and thusly appears to be paying Attention.

You can play these scenarios out all day long — trust me, I have done this again and again — but at the end of the day in my humble opinion no one index, metric, or measurement will provide you the same level of insight that these seven indices combined provides on a visitor-by-visitor basis.

Hopefully the answers to these questions are helpful to other’s of you reading the white paper.  Again, you can download the document freely from my web site and as always I welcome your feedback.