What It Takes To Build A Million $ Experiment

6 key principles to include in subscription conversion testing to yield better results.

A few years ago, we created an experiment that unlocked 4-million-dollar incremental revenue over a year for one of our subscription clients.

Since then, we’ve replicated this winning formula multiple times driving incremental growth for numerous clients. I wanted to share the six key principles we apply to get there.

1) Address the optimizers’ guilty pleasure

Optimizers have a natural tendency to build tests to try something new, as opposed to fixing a deep problem users are facing. These novelty tests can vary (changing button colors, incentive prominence, removing a form field at checkout, etc.) and they have become the Optimizers’ guilty pleasure. Why? They can be inspired by a ton of case studies or best practices and don’t require a high level of creativity to answer the ‘‘what if” questions that haunt optimizer’s minds.

The danger is that Novelty Tests become a subconscious excuse to avoid experiments that require a higher cognitive effort.

When working with The Times of London we developed a profound understanding of the problems users were trying to solve, before they would even consider purchasing a subscription – their aspirations, motivations and anxieties. Using these insights, we strengthened the value proposition of membership and re-engineered the entire online journey, increasing eCommerce conversion +333%. These types of effortful experiments are crucial as they are often a catalyst for long-term, positive transformations within an organization.

2) Focus on the problem, not the solution

Great experiments start with the problem, not the solution. To identify user problems, you need to develop an empathetic mindset. 

George Orwell is an inspiring model of empathy. In 1920, to discover what life was like for those living on the social margins, he dressed as a tramp with shabby shoes and coat, and lived on the streets of East London. “I wanted to submerge myself, to get right down among the oppressed”, he said.

Subscription marketers need to use the products themselves but also interview and watch people using their subscription in real-life, on the train, in the supermarket queue, etc. Users are people, no-one wants to be traffic. And people often have complex situations that we, optimizers, have to profoundly understand in order to design experiences that supports their various needs.

3) Craft an emotional story

By nature, no one is attached to a product or a company, we become attached to the stories behind them. Stories add perceived economic value to a product because they spark emotions, which are at the core of any (purchase) decision. In one of his studies, neuroscientist Antonio Damasio’s proved that people who had damaged the part of the brain that controls emotion were unable to make decisions.

The good news is that every company has a story to tell, whether it is brand heritage, founder bio, product story or history. For instance, with a magazine or newspaper brand, it is impactful to showcase to subscribers the research, hard work and sometimes risks taken by journalists to write an article.

4) Trigger an ‘Aha’ moment

Marketing is not just about meeting user intent anymore, it’s about finding ways to surpass it in an unexpected way. Generating an ”Aha” moment requires creating a positive disconnect between prospect expectations and the experience delivered to increase engagement. Humans have an innate desire to resolve uncertainty, marketers can benefit from this behavior by incorporating a Pandora Effect to their campaigns – an irresistible feeling of curiosity.

This tactic is particularly powerful when applied to digital performance marketing. To promote their subscription service Netflix nailed this by announcing the latest season of House of Cards with powerful GIFs starring Kevin Spacey’s menacing stare, and an unpredictable headline that holds attention and stimulates our brains.

5) Ensure a high level of contrast

Optimizely has studied thousands of test variations. Over half of all variations are written with 5 or fewer lines of code. Complex variations (with over 50 lines of code) are 2.1 to 2.2 X more likely to win. That said, please, PLEASE, don’t task your developer to build every test variation with a minimum of 50 lines of code. My point is to highlight the correlation between the level of changes made to an experience, code effort and success rate for the experiment.

Although testing one variable at the time has benefits, million $ experiments often require that you change several variables at once and accept that you won’t know how every change you made to the experience influence the test results. (By the way, this can be alleviated by leveraging user interaction and voice of customer research for each variation created.)

6) Apply learnings company-wide

If you want to make millions out of one experiment, you need to turn this experiment into a gift that keeps on giving. If you break down the test goals and let tests run long enough, you get granular insights that will allow you to build a roadmap to maximize revenue and optimize the entire customer experience – online, offline, across all platforms and channels.

If you discover a new winning storyline on your website, for instance, you can leverage elements of it to optimize your email campaigns and ad creative. One vital area for subscription brands to focus on is Customer Service. Testing the team’s script and the angle taken to sell your subscription product or retain customers over the phone or on live chat is essential. Too often departments operate in silos. It is crucial to have a continuous insight loop between marketing communication, customer service script and customer service feedback.

Summary

Optimizing is not just about launching as many tests as possible. It’s about finding the balance between test pace and depth. Experiments that require high cognitive effort tend to fall at the bottom of the optimization roadmap, as they are perceived to slow down optimization momentum. Remember, the investment is worth it. Those tests are the only ones that can truly initiate sustained changes within your organization.

You may not get it right the first time or unlock $$$$ with your first test but it is crucial to persist. Once you get the recipe for the secret sauce, you can fuel your entire subscription sales funnel with the lifeblood of a few tests.

Up Next

Register Now For Email Subscription News Updates!

Search this site

You May Be Interested in:

Log In

Join Subscription Insider!

Get unlimited access to info, strategy, how-to content, trends, training webinars, and 10 years of archives on growing a profitable subscription business. We cover the unique aspects of running a subscription business including compliance, payments, marketing, retention, market strategy and even choosing the right tech.

Already a Subscription Insider member? 

Access these premium-exclusive features

Monthly
(Normally $57)

Perfect To Try A Membership!
$ 35
  •  

Annually
(Normally $395)

$16.25 Per Month, Paid Annually
$ 195
  •  
POPULAR

Team
(10 Members)

Normally Five Members
$ 997
  •  

Interested in a team license? For up to 5 team members, order here.
Need more seats? Please contact us here.