R and Adobe Analytics: Did the Metric Move Significantly? Part 2 of 3

In my last post, I laid out that I had been working on a bit of R code to answer three different questions in a way that was repeatable and extensible. This post covers the second question:

Did any of my key metrics change significantly over the past week (overall)?

One of the banes of the analyst’s existence, I think, is that business users rush to judge (any) “up” as “good” and (any) “down” as “bad.” This ignores the fact that, even in a strictly controlled manufacturing environment, it is an extreme rarity for any metric to stay perfectly flat from day to day or week to week.

So, how do we determine if a metric moved enough to know whether it warrants any deeper investigation as to the “why” it moved (up or down)? In the absence of an actual change to the site or promotions or environmental factors, most of the time (I contend), metrics don’t move enough in a short time period to actually matter. They move due to noise.

But, how do we say with some degree of certainty that, while visits (or any metric) were up over the previous week, they were or were not up enough to matter? If a metric increases 20%, it likely is not from noise. If it’s up 0.1%, it likely is just ordinary fluctuation (it’s essentially flat). But, where between 0.1% and 20% does it actually matter?

This is a question that has bothered me for years, and I’ve come at answering it from many different directions — most of them probably better than not making any attempt at all, but also likely an abomination in the eyes of a statistician.

My latest effort uses an approach that is illustrated in the visualization below:

In this case, something went a bit squirrely with conversion rate, and it warrants digging in farther.

Let’s dive in to the approach and rationale for this visualization as an at-a-glance way to determine whether the metric moved enough to matter.

Anomaly Detection = Forecasting the Past

The chart above uses Adobe’s anomaly detection algorithm. I’m pretty sure I could largely recreate the algorithm directly using R. As a matter of fact, that’s exactly what is outlined on the time-series page on dartistics.com. And, eventually, I’m going to give that a shot, as that would make it more easily repurposable across Google Analytics (and other time-series data platforms). And it will help me plug a couple of small holes in Adobe’s approach (although Adobe may plug those holes on their own, for all I know, if I read between the lines in some of their documentation).

But, let’s back up and talk about what I mean by “forecasting the past.” It’s one of those concepts that made me figuratively fall out of my chair when it clicked and, yet, I’ve struggled to explain it. A picture is worth a thousand words (and is less likely to put you to sleep), so let’s go with the equivalent of 6,000 words.

Typically, we think of forecasting as being “from now to the future:”

But, what if, instead, we’re actually not looking to the future, but are at today and are looking at the past? Let’s say our data looks like this:

Hmmm. My metric dropped in the last period. But, did it drop enough for me to care? It didn’t drop as much as it’s dropped in the past, but it’s definitely down? Is it down enough for me to freak out? Or, was that more likely a simple blip — the stars of “noise” aligning such that we dropped a bit? That’s where “forecasting the past” comes in.

Let’s start by chopping off the most recent data and pretend that the entirety of the data we have stops a few periods before today:

Now, from the last data we have (in this pretend world), let’s forecast what we’d expect to see from that point to now (we’ll get into how we’re doing that forecast in a bit — that’s key!):

This is a forecast, so we know it’s not going to be perfect. So, let’s make sure we calculated a prediction interval, and let’s add upper and lower bounds around that forecast value to represent that prediction interval:

Now, let’s add our actuals back into the chart:

Voila! What does this say? the next-to-last reporting period was below our forecast, but it was still inside our prediction interval. The most recent period, thought, was actually outside the prediction interval, which means it moved “enough” to likely be more than just noise. We should dig further.

Make sense? That’s  what I call “forecasting the past.” There may be a better term for this concept, but I’m not sure what it is! Leave a comment if I’m just being muddle-brained on that front.

Anomaly Detection in Adobe Analytics…Is This

Analysis Workspace has anomaly detection as an option in its visualizations and, given the explanation above, how they’re detecting “anomalies” may start to make more sense:

Now, in the case of Analysis Workspace, the forecast is created for the entire period that is selected, and then any anomalies that are detected are highlighted with a larger circle.

But, if you set up an Intelligent Alert, you’re actually doing the same thing as their Analysis Workspace anomaly visualization, with two tweaks:

• Intelligent Alerts only look at the most recent time period — this makes sense, as you don’t want to be alerted about changes that occurred weeks or months ago!
• Intelligent Alerts give you some control over how wide the prediction interval band is — in Analysis Workspace, it’s the 95% prediction interval that is represented; when setting up an alert, though, you can specify whether you want the band to be 90% (narrower), 95%, or 99% (wider)

Are you with me so far? What I’ve built in R is more like an Intelligent Alert than it is like the Analysis Workspace  representation. Or, really, it’s something of a hybrid. We’ll get to that in a bit.

Yeah…But ‘Splain Where the Forecast Came From!

The forecast methodology used is actually what’s called Holt-Winters. Adobe provides a bit more detail in their documentation. I started to get a little excited when I found this, because I’d come across Holt-Winters when working with some Google Analytics data with Mark Edmondson of IIH Nordic. It’s what Mark used in this forecasting example on dartistics.com. When I see the same thing cropping up from multiple different smart sources, I have a tendency to think there’s something there.

But, that doesn’t really explain how Holt-Winters works. At a super-high level, part of what Holt-Winters does is break down a time-series of data into a few components:

• Seasonality — this can be the weekly cycle of “high during the week, low on the weekends,” monthly seasonality, both, or something else
• Trend — with seasonality removed, how the data is trending (think rolling average, although that’s a bit of an oversimplification)
• Base Level — the component that, if you add in the trend and seasonality to it will get you to the actual value

By breaking up the historical data, you get the ability to forecast with much more precision than simply dropping a trendline. This is worth digging into more to get a deeper understanding (IMHO), and it turns out there is a fantastic post by John Foreman that does just that: “Projecting Meth Demand Using Exponential Smoothing.” It’s tongue-in-cheek, but it’s worth downloading the spreadsheet at the beginning of the post and and walking through the forecasting exercise step-by-step. (Hat tip to Jules Stuifbergen for pointing me to that post!)

I don’t think the approach in Foreman’s post is exactly what Adobe has implemented, but it absolutely hits the key pieces. Analysis Workspace anomaly detection also factors in holidays (somehow, and not always very well, but it’s a tall order), which the Adobe Analytics API doesn’t yet do. And, Foreman winds up having Excel do some crunching with Solver to figure out the best weighting, while Adobe applies three different variations of Holt-Winters and then uses the one that fits the historical data the best.

I’m not equipped to pass any sort of judgment as to whether either approach is definitively “better.” Since Foreman’s post was purely pedagogical, and Adobe has some extremely sharp folks focused on digital analytics data, I’m inclined to think that Adobe’s approach is a great one.

Yet…You Still Built Something in R?!

Still reading? Good on ya’!

Yes. I wasn’t getting quite what I wanted from Adobe, so I got a lot from Adobe…but then tweaked it to be exactly what I wanted using R. The limitations I ran into with Analysis Workspace and Intelligent Alerts were:

• I don’t care about anomalies on weekends (in this case — in my R script, it can be set to include weekends or not)
• I only care about the most recent week…but I want to use the data up through the prior week for that; as I read Adobe’s documentation, their forecast is always based on the 35 days preceding the reporting period
• I do want to see a historical trend, though; I just want much of that data to be included in the data used to build the forecast
• I want to extend this anomaly detection to an entirely different type of visualization…which is the third and final part in this series
• Ultimately, I want to be able to apply this same approach to Google Analytics and other time-series data

Let’s take another look at what the script posted on Github generates:

Given the simplistic explanation provided earlier in this post, is this visual starting to make more sense? The nuances are:

• The only “forecasted past” is the last week (this can be configured to be any period)
• The data used to pull that forecast is the 35 days immediately preceding the period of interest — this is done by making two API calls: 1 to pull the period of interest, and another to pull “actuals only” data; the script then stitches the results together to show one continuous line of actuals
• Anomalies are identified as “good” (above the 95% prediction interval) or “bad” (below the 95% prediction interval)

I had to play around a bit with time periods and metrics to show a period with anomalies, which is good! Most of the time, for most metrics, I wouldn’t expect to see anomalies.

There is an entirely separate weekly report — not shown here — that shows the total for each metric for the week, as well as a weekly line chart, how the metric changed week-over-week, and how it compared to the same week in the prior year. That’s the report that gets broadly disseminated. But, as an analyst, I have this separate report — the one I’ve described in this post — that I can quickly flip through to see if any metrics had anomalies on one or more days for the week.

Currently, the chart takes up a lot of real estate. Once the analysts (myself included) get comfortable with what the anomalies are, I expect to have a streamlined version that only lists the metrics that had an anomaly, and then provides a bit more detail.

Which may start to sound a lot like Adobe Analytics Intelligent Alerts! Except, so far, when Adobe’s alerts are triggered, it’s hard for me to actually get to a deeper view get more context. That may be coming, but, for now, I’ve got a base that I understand and can extend to other data sources and for other uses.

For details on how the script is structured and how to set it up for your own use, see the last post.

In the next post, I’ll take this “anomaly counting” concept and apply it to the heatmap concept that drills down into two dimensions. Sound intriguing? I hope so!

The Rest of the Series

If you’re feeling ambitious and want to go back or ahead and dive into the rest of the series:

• Part 1 of 3: Two Dimensions, Many Metrics
• Part 3 of 3: Did the Metric Move Significantly (Along a Combination of Dimensions)?