Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions

Can I Test More Than One Variable
at a Time?
Statisticians answer some of the most commonly asked A/B
testing questions

We’re sharing on Twitter!
#Webclinic

Confusion around online testing
• Designing and managing
experiments is listed as the
most significant challenge

The MarketingExperiments Library
Business Software Suite
#1 On-Demand. 6459+ World Clients
Award-Winning Solution. Free Trial
www.XXXXXXXXXX.com/Business
54%Conversion
21%Conversion
97%Conversion
109%Conversion
31%Conversion
124%Conversion
201%Conversion
29%Conversion
15%Conversion
100%Conversion
21%Conversion137%Conversion
63%Conversion
Over 15 years of research, spanning more
than 10,000 pages/paths and 1 billion emails

Today’s Team
Bob Kemper
Senior Director, Sciences
As Senior Director of Sciences, Bob manages enterprise experimentation and
discovery processes for the corporation and for the individual operating companies.
He serves on the Curriculum and Alumni boards for the Executive MBA program of
the Davis College of Business at Jacksonville University, and on the board of the
Jacksonville chapter of the National Association of Business Economists (NABE).

Today’s Team
Benjamin Filip
Manager, Data Sciences
Ben participates in high-level marketing analytics and optimization including profit
analysis, data mining, customer profile analysis and Web metrics analysis to
determine visitor behaviors in online marketing funnels.

Today’s Team
Paul Cheney
Editorial Analyst
Paul works directly with Flint McGlaughlin to study, manage and catalogue the
MECLABS library of case studies and experiments. Essentially, he spends most of his
day scouring over reports and data from the experiments that come through the lab.

Today, we are going to answer some of the most
common questions marketers ask about online testing.

QUESTION #1: Can I test more than
one variable at a time?

How many variables were changed?
Treatment
Logo
Logo
Logo
Control
Logo
increase in conversion
Through a more professional design and a clearer process,
the treatment generated 58% more conversions.
58%

Logo
Control
Logo
Treatment
By clarifying the value, reducing friction and mitigating
anxiety, the treatment produced 262% more leads.
262%

Control Treatment
By clarifying the value proposition and simplifying the
form, the treatment generated 189% more leads.
189%

When to test more than one variable
Radical Redesigns: The objective is to
challenge the control enough to generate a
significant difference
Focused variable clusters: The objective is to
test the highest performing variables and
increase channel specificity
Single variable testing: The objective is to
determine relative impact by isolating
variables.
Optimal Testing Cycle

What you need to understand
1. We should design our tests so they are useful.
2. A variable can be anything the test designer defines.
3. First, define variables that test the category to “get in the zone.”
4. From there, focus on relative impact of individual page elements.

QUESTION #2: What is multivariate
testing and when should I use it?

Definition: Multivariate testing
Multivariate Testing
A number of approaches for the simultaneous testing of multiple factors (variables), each having
two or more possible levels (values) to determine which composite treatment (combination of
factor values) yields the best overall performance.
Value 1 of 4
Value 1 of 4
Val
ue
1
of
4
Value 1 of 4
Value 1 of 4
Value 1 of 4
Val
ue
1 of
4
Value 1 of 4
Value 1 of 4
Value 1 of 4
Valu
e 1
of 4
Value 1 of 4
Value 1 of 4
Value 1 of 4
Valu
e 1
of 4
Value 1 of 4
Value 1 of 4
Value 1 of 4
Value
1 of 4
Value 1 of 4
Value 1 of 4
Value 1 of 4
Value
1 of 4
Value 1 of 4
Value 1 of 4
Value 1 of 4
Value
1 of 4
Value 1 of 4
Value 1 of 2
Value 1 of 2
Value
1 of 2
Variable
Variable
Variable
• Each combination is
presented to a portion of
arriving traffic.
• The best combination is
detected.
• Other insights about the
relative ‘importance’ and
‘variable interaction’ are
also possible.

1. Multivariate testing (in marketing) is simply an automated way to run many A/B
tests at once.
2. Consider using multivariate testing when:
• You have a large amount of traffic.
• You can interpret complicated test results.
• You have the time to run a longer test.
• You have already tested your way into the best offer category.

QUESTION #3: How do I test if I
have low traffic?

1. Testing with lower traffic levels will always require a trade-off between time,
budget or certainty.
2. With that in mind, there are a few potential ways you can solve the low traffic
problem:
• Test in the channel.
• Lower your desired level of confidence.
• Design more radical treatments.

QUESTION #4: How do I determine
what to test?

How do you determine what to test?
Using a clear optimization and
testing strategy is the least
popular method for determining
what to test.

Where to test in a path?
The Channel
(Ocnn)
The Presentation
(Oprn)
The Product Offer
(Opr)
Opr > Oprn > Ocnn
Wherein:
Opr = Optimization of Product Offer
Oprn = Optimization of Presentation
Ocnn = Optimization of Channel

What to test on a page?
Wherein:
“C” = Probability of conversion
“m” = Motivation of user
“v” = Force of the value proposition
“i” = Incentive (additional) to take action
“f” = Friction elements present
“a” = Anxiety elements present
C = 4m + 3v + 2(i - f) - 2a ©

Background: A website that sells retail and wholesale collector items.
Goal: To increase conversion rate.
Primary Research Question: Which version of the second step in the conversion funnel will produce the
highest conversion rate?
Approach: A/B variable cluster split test that focused on reducing anxiety through credibility indicators,
copy and re-organization of existing page elements.
Experiment ID: (Protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP1305
Research Notes:

Experiment: Background

• When we analyzed the metrics, we realized
there were leaks throughout the checkout
process. The credit card submission page
stood out as a low cost opportunity for
immediate return.
• When we analyzed the metrics even further,
we saw this step also had the highest lost
revenue per cart (more than double any other
step).
• From this, we hypothesized optimizing this
step would have the highest potential return
on our efforts.
Fallout Report: New Customers
Experiment: Background

What might be causing the fallout?
• It is unclear why the credit card is
required when payment method is
different.
• The complexity of the Purchase
Agreement Terms causes confusion and
concern.
• There is no indication that the credit
card information is secure.
Control
Experiment: Control
C = 4m + 3v + 2(i - f) - 2a ©

How we addressed the issues:
• Third-party security indicators have
been added.
• Clearer explanation of why a credit
card is required and that it will not be
charged.
• “Satisfaction Guaranteed” promise is
emphasized.
Treatment
Experiment: Treatment
C = 4m + 3v + 2(i - f) - 2a ©

Design Conversion Rate
Control 82.33%
Treatment 86.04%
Relative Difference 4.51%
5% Increase in Total Conversion
The new credit card page increased conversion by 4.51%
 What you need to understand: While it might seem like a small increase, choosing this specific step
in the sales funnel to test resulted in a projected $500,000+ increase in revenue per year. This
underscores the potential impact of a properly identified research question.
Experiment: Results

1. Knowing where to test requires knowing your data.
• As a general rule, you can use: Opr > Oprn > Ocnn
2. Knowing what to test requires a systematic methodology.
• As a general rule, you can use: C = 4m + 3v + 2(i-f) - 2a

QUESTION #5: How do I know my
test is valid?

How do I know my test is valid?
• Only 15% of marketers
monitor test validity.

1. Use a standardized statistical method for determining validity.
2. There are four main validity threats marketers should be conscious of when testing:
a. Sample Distortion
b. Selection Effect
c. History Effect
d. Instrumentation Effect
3. Beware of only relying on a testing tool for validity.

QUESTION #6: When should I stop
a test?

When to stop a testLevelofConfidence
Samples Collected
Stop Test Here
95%
Required Samples

1. It is safe to stop a test when your data meet two criteria:
a) Sufficient sample size
b) Desired level of confidence

QUESTION #7: What is the best
testing tool?

5 Factors to Consider
1. Budget
2. Testing objectives
3. Flexibility
4. Ease of use
5. In-house expertise

Consider 5 factors when evaluating testing platforms:
a) Budget
b) Testing objectives
c) Flexibility
d) Ease of use
e) In-house expertise

SAVE $100
Discount Code: 481-OC-3007
MECLABS.com/OnlineTesting

MECLABS Research Partnership Opportunities
MECLABS conducts rigorous experiments in the new science of optimization.
We apply our discoveries to help leaders optimize the financial performance
of their sales and marketing programs.
Learn more about how you may
be a fit for a MECLABS research
partnership:
• Select Research Partnership
Opportunities on the post-
webinar survey
• Contact us directly
info@MECLABS.com
1-877-635-0565
x

Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions

Similar to Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions (20)

More from MarketingExperiments

More from MarketingExperiments (20)

Recently uploaded

Recently uploaded (20)

Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions