3 Mistakes Marketers Make When A/B Testing for Subscription Content

I talk a lot about the importance of A/B split testing for optimizing paywalls and subscription sales, but after fielding some questions from readers,

I talk a lot about the importance of A/B split testing for optimizing paywalls and subscription sales, but after fielding some questions from readers, it became apparent that there are some definite bad practices going on out there.The top three mistakes made by marketers when trying to run A/B tests for subscription content are as follows:1. Not understanding the importance of significance. Business, unlike science and math, has more room for gut feelings and best guesses. Therefore, many marketers without a background in statistics fail to understand the importance of statistical significance when conducting A/B tests. Some even go so far as to say that if they don’t have enough traffic to get significant results, they’ll just choose the test page that converts more.This is a bad decision. It’s like saying that if you flip a coin 100 times, and it comes out heads 80 times, then it make sense to always bet on heads when flipping a coin. But everyone knows that the next 100 flips might result in 30 heads and 70 tails.So don’t put stock in any test results that aren’t statistically significant.2. Running a test for too long. When we spoke to Justin Rondeau of WhichTestWon, he cautioned against running a test “until you reach significance.” Basically, your null hypothesis in testing is that there’s no difference in conversion rate for test page A and test page B. If you run a test until you reach significance, you’re assuming there is a difference.In addition, running a test for any longer than five or six weeks will cause interpretation problems due to monthly/quarterly/seasonal differences in consumer behavior. Insider members can get a link to a handy online test duration calculator in our How-To on The Basics of A/B Testing for Subscription Content.3. Testing for click-throughs only. When you test, the first thing you need to do is know your intention and clarify your goal. Do you want to increase revenue? Or get more email sign-ups? Or maybe the best thing for your subscription sales is to get more people signed up for a free demo.Whatever it is, identifying your goal is the single most critical part of testing. Without a clear goal and metric to measure the success of the changes your testing will inform, you’ll just be testing for click-throughs and other second tier stats.Chris Oyolokor, VP of Marketing at Idea Incubator, showed us last year how monthly subscriptions can sometimes earn a site more revenues than higher-priced annual subscriptions. A less savvy marketer may have relied more on click-through rates than revenues to assess the impact of test results and a strategic change in marketing.So know your goal. Then use your tests to inform how you can get there.

Up Next

Register Now For Email Subscription News Updates!

Search this site

You May Be Interested in: