Twitter Performance Measurement with (a Heavy Reliance on) Twitalyzer

My Analyzing Twitter — Practical Analysis post a few weeks ago wound up sparking a handful of fantastic and informative conversations (“conversations” in the new media use of the term: blog comments, e-mails, and Twitter exchanges in addition to one actual telephone discussion). That’s sort of the point of social media, right? The fact that I can now use these discussions as an example of why social media has real value isn’t going to convince people who view it just as a way to tell the world the minutia of your life, because they would point out that gazing at one’s navel to better understand navel-gazing…is still just navel-gazing. So, yeah, if a brand knows that 145 million consumers have signed up for Twitter and knows that they are welcome to leverage it as a marketing channel, but just don’t fundamentally believe that it’s a channel to at least consider using, then neither anecdotes nor good-but-not-perfect data is going to convince them.

Many brands, though, are convinced that Twitter is a channel they should use and are willing to put some level of resources towards it. But, the question still remains: “How do we most effectively measure the results of our investment?” Everything in Twitter occurs at a micro level — 140 characters at a time. A single promotion with a direct response purchase CTA can be measured, certainly, but that’s an overly myopic perspective. So, what is a brand to do? For starters, it’s important to recognize there are (at least) three fundamentally different types of “measurement” of Twitter:

• Performance measurement — measuring progress towards specific objectives of the Twitter investment
• Analysis and optimization — identifying opportunities to improve performance in the channel
• Listening (and responding) — this is an area where social media has really started blurring the line between traditional outbound marketing, PR, consumer research, and even a brand’s web site; with Twitter, there is the opportunity to gather data (tweets) in near real-time and then respond and engage to selected tweets…and whose job is that?

The kicker is that all three of these types of “measurement” can use the same underlying data set and, in many cases, the same basic tools (with traditional web analytics, both performance measurement and analysis often use the same web analytics platform, and plenty of marketers don’t understand the difference between the two…but I’m going to maintain some  self-discipline and avoid pursuing that tangent here!).

This post is devoted to Twitter performance measurement, with a heavy, heavy dose of  Twitalyzer as a recommended key component of that approach. Have I done an exhaustive assessment of all of the self-proclaimed Twitter analytics tools on the market? No. I’ll leave that to Forrester analysts. I’ve gone deep with one online listening platform and have done a cursory survey of a mid-sized list of tools and found them generally lacking in either the flexibility or the specificity I needed (I will touch on at least one other tool in a future post that I think complements Twitalyzer well, but I need to do some more digging there first). Twitalyzer was (and continues to be) designed and developed by a couple of guys with serious web analytics chops — Eric Peterson and Jeff Katz. They’ve built the tool with that mindset — the need for it to have flexibility, to trend data, to track measures against pre-established targets, and to calculate metrics that are reasonably intuitive to understand. They’ve also established a business model where there is “unlimited use” at whichever plan level you sign up for — there is no fixed number of reports that can be run each month, because, generally, you want to see a report’s results and iterate on the setup a few times before you get it tuned to what you really need. So, there’s all of that going for it before you actually dive into the capabilities.

One more time: this is not a comprehensive post of everything you can do with Twitalyzer. That would be like trying to write a post about all the things you can do with Google Analytics, which is more of a book than a post. For a comprehensive Twitalyzer guide, you can read the 55-page Twitalyzer handbook.

Metrics vs Measures

The Twitalyzer documentation makes a clear distinction between “metrics” and “measures,” and the distinction has nothing to do with whether the type of data is useful or not. Measures are simply straight-up data points that you could largely get by simply looking at your account at any point in time — following count, follower count, number of lists the user is included on, number of tweets, number of replies, number of retweets, etc. Metrics, on the other hand, are calculated based on several measures and include things like influence, clout, velocity, and impact. Obviously, metrics have some level of subjectivity in the definition, but there are a number of them available, and everywhere a metric is used, you are one click away from an explanation of what goes into calculating it. The first trick is choosing which measures and metrics tie the most closely to your objectives for being on Twitter (“increase brand awareness” is a a very different objective from “increase customer loyalty by deepening consumer engagement”). The second trick is ensuring that the necessary stakeholders in the Twitter effort buy into them as valid indicators of performance.

For both metrics and measures, Twitalyzer provides trended data…as best they can. Twitalyzer is like most web analytics packages in that historical data is not magically available when you first start using the tool. Now, the reason for that being the case is very different for Twitalyzer than it is for web analytics tools. Basically, Twitter does not allow unlimited queries of unlimited size into unlimited date ranges. So, Twitalyzer doesn’t pull all of its measures and calculate all of the metrics for a user unless someone asks the tool to. The tool can be “asked” in two ways:

• Someone twitalyzes a username (you get more data if it’s an account that you can log into, but Twitalyzer pulls a decent level of data even for “unauthenticated” accounts)
• All of the tracked users in a paid account get analyzed at least once a day

When Twitalyzer assesses an account, the tool looks at the last 7 days of data. So, as I understand it, if you’re a paid user, then any “trend” data you look at is, essentially, showing a rolling 7-day average for the account (if you’re not a paid user, you could still go to the site each day and twitalyze your username and get the same result…but if you really want to do that, then suck it up and pay $10/month — it’ll be considerably cheaper if you have even the most basic understanding of the concept of opportunity costs). This makes sense, in that it reasonably smooths out the data.

Useful Measures

There isn’t any real magic to the measures, but the consistent capture of them with a paid account is handy. And, what’s nice about measures is that anyone who is using Twitter sees most of the measures any time they go to their page, so they are clearly understood. Some measures that you should consider (picking and choosing — selectivity is key!) include:

• Followers — this is an easy one, but it’s the simplest indication as to whether consumers are interested in interacting with your brand through Twitter; and if your follower count ever starts declining, you’ve got a very, very sick canary in your Twitter coal mine — consumers who, at one time, did want to interact with you are actively deciding they no longer want to do so; that’s bad
• Lists — the number of lists the user is a member of is another measure I like, because each list membership is an occasion where a reasonably sophisticated Twitter user has decided that he/she has stopped to think about his relationship with your brand, has categorized that relationship, AND has the ability to then share that category with other users.
• Replies/References — if other Twitter users are aware of your presence and are actually referencing it (“@<username>”), that’s generally a good thing (although, clearly, if that upticks dramatically and those references are very negative, then that’s not a good thing)
• Retweets — people are paying attention to what you’re saying through Twitter, and they’re interested in it enough to pass the information along

Twitalyzer actually measures unique references and unique retweets (e.g., if another user references the tracked account 3 times, that is 3 references but only 1 unique reference — think visits vs. page views in web analytics), but, as best as I can tell, doesn’t make those measures directly available for reporting. Instead, they get used in some of the calculated metrics.

A few other measures to consider that you won’t necessarily get from Twitalyzer include:

• Referrals to your site — there are two flavors of this, and you should consider both: referrals from twitter.com to your site (are Twitter users sharing links to your site overall?), and clickthroughs on specific links you posted (which you can track through campaign tracking, manually through a URL shortener service like bit.ly or goo.gl, or through Twitalyzer)
• Conversions from referrals — this is the next step beyond simply referrals to your site and is more the “meaningful conversion” (not necessarily a purchase, but it could be) of those referrals once they arrive on your site
• Volume and sentiment of discussions about your brand/products — Twitalyzer does this to a certain extent, but it does it best when the brand and the username are the same, and I’m inclined to look to online listening platforms as a more robust way to measure this for now

Calculated Metrics

Now, the calculated metrics are where things really get interesting. Each calculated metric is pretty clearly defined (and, thankfully, there is ‘nary a Greek character in any of the definitions, which makes them, I believe, easier for most marketers to swallow and digest). This isn’t an exhaustive list of the available metrics, but the ones I’m most drawn to as potential performance measurement metrics are:

• Impact — this combines the number of followers the user has, how often the user tweets, the number of unique references to the user, and the frequency with which the user is uniquely retweeted and uniquely retweets others’ tweets; this metric gets calculated for other Twitter users as well and can really help focus a brand’s listening and responding…but that’s a subject for another post
• Influence — a measure of the likelihood that a tweet by the user will be referenced or retweeted
• Engagement — a lot of brands still simply “shout their message” out to the Twitterverse and never (or seldom) reference or reply to other users; Twitalyzer calculates engagement as a ratio of how often the brand references other user compared to how often other users reference the brand; so, this is a performance measure that is highly influenced by the basic approach to Twitter a brand takes, and many brands have an engagement metric value of 0%. It’s an easy metric to change…as long as a brand wants to do so
• Effective Reach — this combines the user’s influence score and follower count with the influence score and follower count of each user who retweeted the user’s tweet to “determine a likely and realistic representation of any user’s reach in Twitter at any given time.” Very slick.

There are are a number of other calculated metrics, but these are the ones I’m most jazzed about from a performance measurement standpoint. (I’m totally on the fence both with Twitalyzer’s Clout metric and Klout‘s Klout score, which Twitalyzer pulls into their interface — there’s a nice bit of musing on the Klout score in an AdAge article from 30-Sep-2010, but the jury is still out for me.)

Setting Goals

Okay, so the next nifty aspect of Twitalyzer when it comes to performance measurement is that you can set goals for specific metrics:

Once a goal is set, it then gets included on trend charts when viewing a specific metric. “But…what goal should I set for myself? What’s ‘normal?’ What’s ‘good?'” I know those questions will come, and the answer isn’t really any better than it is for people who want to know what the “industry benchmark for an email clickthrough rate” is. It’s a big fat “it depends!” But, assessing what your purpose for using Twitter is, and then translating that into clear objectives, and then determining which metrics make the most sense, it’s pretty easy to identify where you want to “get better.” Set a goal higher than where you are now, and then track progress (Twitalyzer also includes a “recommendations” area that makes specific notes about ways you can alter your Twitter behavior to improve the scores — the metrics are specifically designed so that the way to “game” the metrics…is by being a better Twitter citizen, which means you’re not really gaming the system).

I’d love to have the ability to set goals for any measure in the tool, but, in practice, I don’t expect to do any regular performance reporting directly from Twitalyzer’s interface for several reasons:

• There are measures that I’ll want to include from other sources
• The current version of the tool doesn’t have the flexibility I need to put together a single page dashboard with just the measures and metrics I care about for any given account — the interface is one of the cleanest and easiest to use that I’ve seen on any tool, but, as I’ve written about before, I have a high bar for what I’d need the interface to do in order for the tool itself to actually be my ultimate dashboard

Overall, though, goal-setting = good, and I appreciate Eric’s self-admitted attempt to continue to steer the world of marketing performance measurement to a place where marketers not only establish the right metrics, but they set targets for them as well, even if they have to set the targets based on some level of gut instinct. You are never more objective about what it is you can accomplish than you are before you try to accomplish it!

But, Remember, That’s not All!

So, this post has turned into something of a Twitalyzer lovefest. Here’s the kicker: the features covered in this post are the least interesting/exciting aspects of the tool. Hopefully, I’ll manage to knock out another post or two on actually doing analysis with the tool and how I can easily see it being integrated into a daily process for driving a brand’s Twitter investment. Twitalyzer is focussed on Twitter and getting the most relevant information for the channel directly out of the API, unlike online listening platforms that cover all digital/social channels and, in many cases, are based on text mining of massive volumes of data (which, as I understand it, is generally purchased from one of a small handful of web content aggregators). It’s been designed by marketing analysts — not by social media, PR, or market research people.  It’s pretty cool and does a lot considering how young it is (and the 4.0 beta is apparently just around the corner). Like any digital analytics tool, it’s going to have a hard time keeping up with the rapid evolution of the channel itself, but it’s one helluva start!