Optimization Test Techniques – Part II of II

Landing Page Test/Landing Page Campaign

I’ve combined both the Landing Page Test and the Landing Page Campaign types here because they provide the same core test functionality. These types of test techniques are very different compared to an A/B test in that visitors can switch branches or experiences of a test. This technique is highly effective when your test strategy requires that visitors be able to change branches of a test versus the A/B approach where visitors are forced to maintain membership in a particular test branch.

A great example of such a test is around SEM reinforcement. Lets say you have two different ad campaigns taking place on Google where one ad campaign is promoting a particular product and the other campaign is promoting discount messaging. If you have a test running that is basically quantifying the value of message reinforcement associated with source, then you would have an A, B, C Campaign. Experience A would be the default content or what is currently running on the landing page. Experience B would be targeted to the first Google SEM messaging on product messaging and Experience C would be targeted to the second Google SEM ad around discount messaging.

To effectively run this type of scenario you would want to leverage the Landing Page Campaign in the event visitors happen to click through on both of the SEM ads. Using an A/B campaign with this type of test would reinforce the messaging of the first ad clicked on the landing page even if the visitor clicked the second ad because with the A/B campaign, you are stuck with the first Experience assigned for the life of the campaign. The Landing Page campaign would recognize the ad clicked on and switch you to the corresponding campaign experience even if that means switching branches.

The SEM reinforcement example is just one way the Landing Page technique can help add additional strategy. I often find this technique to also be helpful when targeting campaigns to particular behaviors on the site or when offline data is incorporated into your online optimization efforts.

The key difference between the Landing Page Test and The Landing Page Campaign is that a Landing Page Test is used for Multivariate testing where the strategy involves having the visitors change experiences within a Multivariate test design.

Monitoring Campaign

The monitoring campaign is an incredibly helpful asset for any organization that leverages Test&Target.

The monitoring campaign is typically used to collect data or to run concurrently with other campaigns to track results. The monitoring campaign is not typically used to display content, although it can if needed.

A great use case for using a monitoring campaign would be to set a baseline for conversion rates or revenue metrics like total sales, revenue per visitor, or average order value. I often recommend to customers that if they have the mboxes on the site but the alternative content isn’t ready, to start a monitoring campaign to not only see some metrics but also to familiarize yourself with the Test&Target platform.

The Monitoring Campaign was not designed to replace an organization’s analytics but many organizations soon find themselves using the Monitoring Campaign to provide data on certain behaviors defined in T&T or to even augment analytics with pathing reports. Another interesting use of Monitoring campaigns that I have seen helpful to organizations is using it to deploy tags to the site independent of T&T. Before tag management solutions became so popular, the mbox was a nice and easy way to get code to the page without having to bother IT if an mbox was already in place. Nowadays, T&T has a plugin functionality that can handle that without having to setup a Monitoring Campaign.

Multivariate or MVT Campaigns

Multivariate testing is a somewhat political topic in the testing world. There are many schools of thought when it comes to multivariate testing and there is also much debate about whether it is as beneficial as A/B testing. I will leave those topics for future blog posts but simply outline how the T&T platform approaches MVT testing here.

The default or productized MVT approach in the Test&Target platform is the Taguchi approach. This approach is a partial factorial approach in that not all possible combinations of elements and alternatives will be incorporated into the test design – only a portion of all possible will be. The key benefit T&T advocates here is that less time is needed to get results because less experiences require less traffic to the test.

Here is an example of the Taguchi approach: lets say you have 3 elements and 2 alternatives for each element. The elements are the Call to Action, the Color, and the Text. If you had two different iterations of each of these elements that would represent a 3X2 MVT design. If you mixed and matched each element and each alternative, all possible combinations would come to 3ˆ2 = 8. By applying the Taguchi approach, only 4 out of the 8 possible combinations will be tested. The 4 experiences that are tested are not a random four but rather the 4 according to the Taguchi model. T&T helps you with this as you go about creating your test within the platform. Here is an example of a test design created by T&T with a 3X2 MVT:

The Taguchi approach becomes especially handy when you have more then 3 elements. In the above example, testing 8 experiences versus the 4 wouldn’t present as much of a challenge as testing 7 elements with 2 alternatives. A 7X2 MVT with all possible combinations would require testing 128 experiences (2ˆ7) versus the Taguchi approach where only 8 experiences would be needed.

The reporting that comes along with a T&T Multivariate test is very similar to what you can expect from any other type of test technique with one exception. For MVT tests, T&T provides what is called an Element Contribution Report. This report is helpful for a number of things.

First, when you test only a subset of all possible test combinations you are only getting data on those tested experiences. This report presents to you what the “Predicted Best Experience” is based on data collected thus far. With a 7X2 MVT Taguchi test design, you are only testing 8 out of 128 possible experiences – this report tells you which experience would have been the best given the odds of you having it in your test design would be only 8/128.

Secondly, this report is helpful to understand how each element impacts the given success event, hence the name of the report being Element Contribution Report. This data is incredibly helpful because you can use it for other tests designs. For example, I have seen many Taguchi MVT Element Contribution reports infer that a certain Message Approach as a test alternative was incredibly impactful with high statistical confidence. Those organizations can now take that Message concept and incorporate it into A/B tests or even in offline Marketing efforts. This report essentially helps identify themes that can be incorporated into other Marketing efforts.

Here is an example of an Element Contribution Reprot where you can see each element and which alternative of that element was more successful, making up what would be the best test experience even if it wasn’t part of the test design. Additionally, you can see that the most influential element was the Submit Button :

Just because T&T’s default approach to MVT is the Taguchi approach, that doesn’t mean that you are limited to running partial factorial multivariate tests with this platform. I have worked with many clients over the years including one right now that is using T&T for full factorial MVT tests. To do this, you simply have to create your test design offline and set it up as an A/B test within T&T. The post test data is then analyzed offline as well to quantify interaction effects.

Optimizing Campaign

This type of campaign technique is surprisingly unique to the Test&Target platform given that it can be very helpful to any Optimization team within an Organization.

[Correction: SiteSpect also offers this automated optimizing test functionality.]

The Optimizing Campaign is not your typical test type. It isn’t designed for the type of learnings that you might be used to with running other types of tests where you are comparing different experiences across different visitor sets. It is designed to allow the testing platform automatically deliver the right experience at the right time.

Imagine if you will, five different pieces of content for testing. This content can be home page hero content, navigational elements, calls to action, email content….anything really that you wish to have tested as part of a test design. Typically you would approach this with either an A/B test technique or a Multivariate test so as to see which version versus the other led to increases in given success events. The Optimizing campaign test technique is designed to not maintain an equal distribution of test content to give you this data but rather it will automatically shift traffic to the better performing experience of all the possible experiences. If Experience C was consistently outperforming the other Experiences, the Optimizing Campaign will automatically shift more and more of that that visitor traffic to that test experience.

T&T takes the Optimizing Campaign to another level with how it leverages segments in this automation. If you include segments in this campaign setup, the Optimizing Campaign will provide its automation to those segments by automatically providing the most effective experience for that segment. Additionally, the reporting associated with this campaign type provides a report called “Insights” that shows what segments impacted what test offers and whether that impact was positive or negative. This is incredibly powerful because the tool is doing the discovery for you and you are then enabled to create a new campaign right from this report targeted to that discovered segment.

Here is a screen shot of an Insights report from T&T:

I find that the Optimizing Campaign test technique is most effective for tests that are being run in email campaigns. Lets say you have an email blast that is going out to 200,000 email subscribers and you were running an A/B/C test of content within that email. The Optimizing Campaign has the potential to learn what experience within that test design was the most successful from the first sets of visitors that opened that email. If, for example, the first 2,000 visitors reacted much more favorably to Experience B, the Optimizing Campaign would shift more and more visitors to get Experience B if they haven’t opened the email yet. This approach allows organizations to immediately capitalize on test learnings for short marketing cycles like those in email campaigns.