Perhaps you’ve been told you should be “testing” to discover how to optimize your site for conversions and subscription sales. But do you have enough traffic to test? And how do you know what to test? Or if your tests are scientifically sound? In this brief How-To, we asked testing expert Justin Rondeau of WhichTestWon to explain the basics of A/B split testing so that you can test with confidence and interpret your results wisely. Plus, discover the most popular testing tools, and alternative data-driven tactics for sites without enough traffic to test.

What Do I Need to Run A/B Tests?

For those of you unfamiliar with A/B split testing, it’s basically a method by which you can test whether one page converts statistically significant more visitors than another. (We’ll explain the importance of statistical significance in a bit.)

To run an A/B test you’ll need:

• Data on your site analytics before testing
• A minimum of 200 conversions a month
• Someone to design your test pages
• A clearly defined conversion goal
• Basic understanding of math and statistics

Testing expert Justin Rondeau of WhichTestWon says that if you want to test, each “test” page will need about 100 conversions — that’s 200 people clicking through to fulfill an order, sign up for an email newsletter, download a free whitepaper, or whatever action you want your site visitors to take. These 200 conversions should occur in no longer than a five to six week period since running a test longer than that is unsound. Rondeau says if you have about 5,000 site visitors a month, you should be safe, but look at your analytics and conversion rate to be sure.

The Research & Page Identification Phase

Rondeau states that running a test is fairly easy; it’s the research and analysis phases that take longer.

Before beginning a test, you need to spend significant time analyzing your website analytics in order to determine the best pages and elements to test.

“First, identify the pages of most opportunity,” Rondeau says. These are not necessarily the worst-performing pages, but the ones you can get the most bang for your buck in terms or either conversion or moving the needle for your organization.

Subscriptions sites usually test for these main needle-moving conversion goals by prospects:

• becoming a paying subscriber
• signing up for a free e-newsletter
• downloading a white paper
• signing up for a trial or free demo
• completing a member profile
• opting for an annual over a monthly plan

Rondeau says that 93% of marketers use their analytics to identify pages related to conversion goals, but he also mentions augmenting that research that with surveys, feedback from your customer service department, and in-page analytics (are prospects clicking above or below the fold, bounce rates, time of page, etc.). Also, check to see if your value proposition/customer benefit language is consumable in the amount of time most users spend on the site.

The following tools can also inform your research:

• Crazy Egg: a heat mapping tool that uses an algorithm to deduce where people are likely looking when they arrive on your page
• UserTesting.com: When you spend too much time in the bubble of your organization, it can be hard to put yourself in your users’ shoes. This site gives you feedback on how people outside your organization are experiencing your site.
• Exit surveys: Exit surveys are pop-ups that appear as a visitor tries to exit a site. Because they cause friction, exit surveys will never provide huge amounts of data, but the information you do get will be vital and helpful.

Once you identify which pages you want to test and have a clearly defined conversion goal, you’ll want to isolate which element you’re going to test. This is critically important. If you change two elements on a page and run a simple A/B test, you won’t know whether it was element A or element B that caused an increase or decrease in conversions.

An exception can be made if you’re doing a “proof of concept” test where you’re positioning your message and the look/feel of your site in two very different ways. But these tests should be followed by more stringent one-element tests.

Running an A/B Test

After deciding what to test, running a test is fairly easy:

1. Have your Web designed design the test page. Your control page should be your existing page.
2. Determine the duration of your test. This is likely based on the number of conversions you’ll need to reach significance. Rondeau recommends using Visual Website Optimizer’s on-site calculator. Rondeau also recommends rounding up from the recommended testing period to the nearest “week” interval. For example, if the calculator says you should run a test for 18 days, round up to 21 days, or three weeks. That way, you control for any variation due to day of week fluctuations in activity.
   
   

   

3. Log into your testing tool. See the sidebar for options both free and paid.
4. Split your traffic 50/50. You can decide, according to your defined goal, whether you want to split all traffic or subsets of traffic, such as new visitors or returning visitors. (Note: you should not split your traffic as new visitors vs. returning visitors; this is not a controlled test. Instead, you should have half your new visitors go to the Page A, half to Page B. Returning visitors would not be included in this test example.)
5. End your test according to your estimates in step #2. You want results that are statistically significant at the .05 to .01 level, with a 95% or 99% confidence interval, but Do NOT run a test “until you get significance.”

Why shouldn’t you run a test until you get significance? Because it’s scientifically unsound. The basic assumption, or null hypothesis, in testing is that there is no change in conversions between your test pages. When you get significance, your test says there is. But it’s easier to get significance as your test sample size get exponentially bigger. However, you also increase your chance of a false positive. It’s okay if you don’t get significance; this is valuable data to know as well, as it helps you determine which levers to pull for optimization.

Interpreting Results

Rule #1: Do not assume Page B leads to more conversions if there is not a statistically significant difference in conversion rates between Page A and Page B.

Here’s why. We all know that if you flip a coin, you have a 50% chance of getting heads and 50% chance of getting tails.

But, if you flip a coin 100 times, it’s not like you’re going to get 50 heads and 50 tails. More likely, you’ll get 80 heads and 20 tails. This does not mean that you are 80% more likely to get heads when you flip a coin. Because the next time, you could get 30 heads and 70 tails.

If your test is not statistically significant, then all you have is data on conversions. The data could change in a week. Without significance, Page A and Page B are performing equally well, and you’re allowed to choose which one you like best.

But, if you do have statistical significance, then you have a few more questions to ask yourself:

1. How big was the lift in conversions?
2. What does that lift mean for me in terms of dollars? For example, a 1% lift in conversions on a $1 download for a site that gets 10,000 visitors a month is $100. Is that much money worth the amount of development time it would take to implement the change? At the same time, a 1% lift for a site with millions of visitors and a $100 product could mean thousands of dollars in revenue.
   In addition, perhaps a higher conversion rate leads to lower retention, as Chris Oyolokor explains in our On-Demand Video on Price Testing. So, do the math. Then make the decision.

Editor’s Note: Conversion optimization managers are often eager to test and excited about any lift in results. They don’t always extrapolate the results to understand what that means for your business. Keep this in mind when talking with conversion optimization managers. Don’t rain on their parade when they’re excited to test, but don’t blindly follow their lead. Better yet, communicate with them early on in the testing process to identify elements that would increase your bottom line. Otherwise, you can end up with an “optimized” site that’s in the red.

Other Data-Driven Tactics

So what do you do if you don’t have the traffic to test?

As we explained in The Research & Page Identification Phase section of this article, there are other methods for getting data about how your customers experience your site that can help you make informed decisions about changes. Google Analytics will allow you to set and track goals, which are a way of measuring your site’s effectiveness. Make sure to track your exit rate and bounce rate.

Check out Crazy Egg and UserTesting.com. If the heat maps show visitors are looking at your search bar and not your conversion button, re-design your page.

Also, log all your customer service data, from compliments to complaints. And value whatever data you get from exit surveys. Plus, if you have the time and resources, consider surveying your users by phone or email.

And don’t underestimate your ability to make an informed decision. It’s great to get a gut check through data, but sometimes you have to operate and forge ahead without a fully completed photo. (Editor’s Note: Sometime your gut and data may be telling you to NOT make a change. Listen then, too.) Just be clear about what metrics you’re using to measure success (revenues, profits, registered users, etc.) and measure the effectiveness of your decisions by that metric consistently.