Increasing conversions and subscription sales requires a watchful eye toward performance metrics on your website. Yet those seeking to improve their site often wonder where to start.
In this How-To we will cover:
- What is Multivariate Testing?
- What you need for multivariate tests
- Running the tests
- Vendors to consider
What is Multivariate Testing?
Multivariate testing finds what content or creative variation of a webpage produces the best improvement in defined goals – whether it’s conversions, form completions, registrations, or any other webpage metric. The testing process measures the effectiveness of different elements or multiple variables (hence the name!), as they are presented to website visitors, seeking to discover through data which combination of site components leads to statistically significant changes. The only limit to the amount of variables and combinations in multivariate testing is the time period it takes for your website to obtain a statistically valid sample of visitors.
That’s why multivariate testing is only for sites with a high amount of traffic. For low-traffic sites, we recommend reading up on The Basics of A/B Testing for Subscription Content. A/B testing measures the better of only two content variations and is much simpler to implement and run than Multivariate Testing.
What you need for Multivariate Tests
1. Traffic
First you’ll need to consider if you have the proper amount of web traffic for multivariate testing to be statistically significant. The more variables that are tested means that more traffic will be needed, and sites with smaller amounts of traffic won’t have the right amount of visitors for testing to be effective.
If your site has low-traffic, it might be beneficial to run A/B tests to achieve your goals. Starting small with A/B testing the most important component of your site, and then work down the list from there.
Testing expert Justin Rondeau says that if you want to run an A/B test, each “test” page will need to achieve about 100 conversions in no longer than a five to six week period since running a test longer than that is unsound. Rondeau says that 5,000 site visitors is a safe threshold to run A/B tests, but look at your analytics and conversion rate to be sure.
Multivariate testing requires much more traffic than an A/B test requires. We recommend using an online calculator to ascertain the overall time period that testing will require.
2. Testing Elements
Next, you’ll need to identify the pages and page elements you’ll want to test. This will most likely be the most time-consuming step in the process.
Subscription sites typically test the pages that correspond to the following conversion goals:
- becoming a paying subscriber
- signing up for a free e-newsletter
- downloading a white paper
- signing up for a trial or free demo
- completing a member profile
- opting for an annual over a monthly plan
As stated previously in our A/B Testing How-to, Justin Rondeau says that 93% of marketers use their analytics to identify pages related to conversion goals, but that research should be augmented with visitor surveys, feedback from your customer service department, and in-page analytics (prospects clicking above or below the fold, bounce rates, time on page, etc.).
The following tools can further inform your research:
- Crazy Egg: a heat mapping tool that uses an algorithm to deduce where people are likely looking when they arrive on your page
- UserTesting.com: gives feedback on how people outside of your organization are experiencing your site
- Exit Surveys: pop-ups that appear as a visitor tries to exit a site provide data that can vital and helpful
3. Identifying On-page Variables
Once you’ve identified the page or pages you want to test, the next step is critical. Since the factors of web design are essentially limitless, there are many combinations of variables to test, including:
- headlines
- marketing copy
- buttons
- colors
- form fields
- page position
The combinations of variables you can test are, in theory, endless. We recommend thinking hard about which elements on which pages you are interested in testing, and then working down the list.
4. Understanding Testing Goals or KPI’s
Once pages and variables to test are identified, how will you know if your tests are effective? Keep in mind the overall goal of your website and company. Subscription sites may want to consider which combination of variables lead to the most favorable changes in:
- New subscription registrations
- Subscription renewals
- Clicks on buttons or links
- Form completions
- Frequency of visits
- Time spent on site
Running the Tests
Running the test(s) is dependent on your team’s technical ability and what tool you may be using.
- Have your web design team create the test pages. Control pages should be your existing pages.
- Determine the duration of your test. This is based on the number of conversions you’ll need to reach statistical significance. Rondeau has recommended using Visual Website Optimizer’s on-site calculator to determine the time period needed for your results to be statistically significant. Rondeau recommends rounding up from the recommended testing period to the nearest “week” interval. For example, if the calculator says you should run a test for 18 days, round up to 21 days, or three weeks. That way, you control for any variation due to day of week fluctuations in activity. NOTE: Other tools have similar capability to understand how long a test will take depending on traffic and variables being tested.
- Within your testing tool, split your traffic among the variations you wish to test. Some tools will do this for you automatically.
- Run your tests according to your timeframe estimates in step #2.
If your site gets a moderate amount of traffic, tests may have to be run over a longer period of time to establish a good sample size that’s large enough to be considered statistically significant.
Vendors
Identifying page variables, creating different versions of the page, and monitoring their performance over a statistically significant period of time can be tricky – especially without a staff to create the pages for you and monitor the results. Luckily there are a myriad of vendors available to do the work for you.
Your overall level of experience with multivariate testing will no doubt factor into what vendor you choose. Some vendors capture the results for you but don’t provide suggestions on what to do next, so we recommend due diligence when choosing a vendor. To get you started, here’s a sample list of (some, not all) the many multivariate testing vendors available online:
Google Analytics
Free Trial: No
Cost: Free
The analytics platform offered by Google is free to use and monitors various website traffic metrics. It offers multivariate testing through Google Content Experiments, where users are able to create up to ten different landing pages for optimization. Users seeking to create and run content experiments need to sign up for a Google Analytics account and tracking code.
Optimizely
Free Trial: Yes
Cost: $19-$399/month
Optimizely offers an easy-to-use visual editor, advanced web and mobile developer tools, and real-time results and reporting.
Conversion Multiplier
Free Trial: No
Cost: $2,500-$7,500/month
Conversion Multiplier is able to test thousands of webpage combinations at a time with analysis algorithms, with an eye towards which combination best maximizes profits for your site.
Convert
Free Trial: Yes
Cost: From $9/month
Convert offers a visual editor that allows for easy creation of different pages to test as well as Google Analytics integration.
Monetate
Free Trial: Yes
Cost: Flat fee based on previous month’s traffic
Monetate has a marketer-friendly UI that allows you to create unlimited tests and direct traffic to them.
SiteSpect
Free Trial: No
Cost: Based on the type of technology selected
SiteSpect offers testing for clients’ entire website and leverages tag-free testing for both behavioral analytics and site traffic for cost-effective analytics.
