Data-Driven UI/UX Design with A/B Testing

Data-Driven UI/UX Design
TopDev Event, Aug 2017
Jack Nguyen, Country Manager VN & TH

Rules for today?
No rules
Engaged & Ask questions
Quiz & Discussion

Truly understand the value
What I hope
Clear approach to get started
Practical do & don’t

Obama raised $60 Million by
Running a Simple Experiment.

Obama raised $60 Million by Running a Simple Experiment.
Challenge: Raise money if no one has heard of you and you only have a
website?
Idea: Every visitor to the site is an opportunity to generate lead, convert
them into volunteer for donation

Optimize the sign up pages to get more leads

Experiment different variations of CTA buttons

Experience different variations of Media
Campaign staff’s favorite

… And the winner is?
E-mail Sign up Rate
11.6%
+40.6%
Vs. Original 8.26%
10% of email subscribers convert into volunteers
21$ donation per volunteers on average
+2,880,000 e-mails
+60 millions in donation

A scientific method to measure how a new idea
(version B/variation) performs against an
existing implementation (version A/control)

Where does this belong to?
● Conversion Rate Optimization (CRO)
● A/B Test: Best 1 for All
● Segmentation: Best 1 for Some
● Personalization: Best 1 for 1
Our focus for today!

HiPPO
Highest Paid Person’s Opinion
Highest Paid Person’s in the Office
Higher-paid or higher-rank employees have the biggest say when a decision has to be made.
Describe an organization's reliance on human instinct rather than data in the decision-making process.

“My design is much more beautiful and intuitive compared to what we
have now! It’s so obvious, don’t think we need to run an A/B test”
“My wife (of the boss) told me that this banner is confusing. Let’s just
remove it!”
“Let’s put our team focus on launching this feature. This is gonna be
huge!”

“I just need to redesign my website”

Why is it important for your business?

Increasing Marketing Cost
How much longer can you survive this battle?

3x Increase in CPC in 2016 vs 2008

Low Conversion Rate
How much money are you leaving on the table?

Opportunity for you to gain competitive
advantage!

Data is generated when users browse through
pages and do their tasks

When there is enough data to make a decision,
evaluate the results and make a decision
(A or B)

4 Steps Framework of A/B Testing
Step 1
Data
Figure out what
to improve
Step 2
Hypothesis
Make an
educated guess
Step 3
Experiment
Test
your guess
Step 4
Act
Decide what’s
next

Step 1: Data
Figure out what to improve

Data helps you identify the most painful
problems people have (NOT the one bother
you the most)
Find the problems worth solving!

Quantitative Data
Tells you where to test

What kind of quantitative data you need?
● User’s page browsing behaviour
● User’s feature interaction behaviour
● User’s marketing source information
● User’s historical purchasing behaviour
● ...

Find pages with HIGH traffic and HIGH bounce rate
● Landing pages in top 10-20
with bounce rate much higher
than site average
● This is where you are burning
your marketing dollars without
caring about the full user’
journey

Find pages with HIGH traffic but LOW page value
● These are the pages with highest traffic
going through
● But does not contribute much to
conversion action
● How to make it work for you?

Find the MOST painful areas on the buying journey
● Identify the steps with highest
drop-off rate to optimize
● Understand where users go to
next

Find the MOST interacted features with LOW Conversion Rate

Qualitative Data
Give you an idea of what should be tested

What kind of qualitative data you need?
● User testing (video of how users browse your site)
● Survey data (for detailed feedbacks)
● Heat mapping (quick glance at elements interaction)
● Customer service
● ...

Use exit surveys to learn why people leave
Apply this survey on the funnel step with the highest drop-off

Use post-purchase survey to understand why people buy
Know what drive conversions to know your selling points, customer’s perception to drive
conversions for others

Get feedback on the goal of discovery pages

Different A/B Test type
● Element testing
● Multivariate testing (MVT)
● Page testing
● Flow testing

This is what
you should
have by
now!
… Now
What?

Prioritize your testing roadmap for maximum results

1. Population Size
3 Key Criterias
2. Potential Performance Improvement
3. Estimated Technical Difficulty to Set up
You can assign a score for each of this and relevant weightages

Estimate potential value of a Test

Step 2: Hypothesis
Make an educated guess

What’s the aim of having a hypothesis?
The hypothesis should:
● Capture the essence of the change you propose to make and what you
think the effect will be
● A clear understanding and a plan that addresses what you will learn by
testing the hypothesis

Ask yourself these questions when crafting hypothesis
● If you fail, what did you learn that you will apply to future designs?
● If you succeed, what did you learn that you will apply to future
designs?
● How much work are you willing to put into your testing in order to
get this learning?

Structure of a good hypothesis

Step 3: Experiment
Test your educated guess

LIFT Model
Look at marketing touch points through the user perspective
to design test variations

Landing page Influence Function for Tests
Widerfunnel concept

Key KPIs
Without this, NEVER start your experiments!

2 Types of KPIs
Primary metrics: The key metrics that will be directly impacted if the
hypothesis hold true.
Example: If you A/B test a clearer CTA message, then: KPI = CTR on this
CTA
Secondary metrics: The indirect but important metrics that would be resulted if
the hypothesis hold true.
Example: If you A/B test a clearer CTA message, then: KPI = CR (as an users
would be more likely to checkout due to clearer messaging)

Good vs Bad KPIs
Bad KPIs:
● No. of pageviews
● Time spend on pages
● No. of sign-ups
● No. of transactions
● ...
Good KPIs:
● Share of page views with
at least 75% scroll
● Share of page views with
at least 10 mins. on page
● Sign-up Rate
● Conversion Rate
● ...
2 common problems:
● Vanity metrics
● Not an effectiveness metrics

Step 4: Act
Results evaluation & Decide what’s next

How to know if the
result is real?

What is statistically significance?
To be safer!
Less likely to
be wrong!

Confidence Interval (CI)
P = Conversion Rate
Z will be calculated based on significance level
The more confidence you want to be that the
difference is real → The bigger statistical
significance → The bigger z will be
Hence, if you want to have a more confident
range of the actual CR, the bigger sample size
will be required

Launch or keep testing?
1.3%
1.6%
1.7%
2.5%
Control
Var A
Var B
Var C
Conversion Rate
Confidence Interval

If you not only want to be certain of the change but also of the magnitude of the change
Minimum Detectable Effect (MDE)
The minimum relative change in conversion rate you would like to be able to detect - MDE
You’ll need less traffic to detect big changes and more traffic to detect small changes
To demonstrate, let’s use an example with a 20% baseline conversion rate and a 5% MDE.
Based on these values, your experiment will be able to detect 80% of the time when a variation
underlying conversion rate is actually 19% or 21% (20%, +/- 5% × 20%)

The smaller the improvement
impacts that you want to
detect (MDE), the bigger the
required sample size to be
really sure that you can detect
that small change when it
happens - MORE
OPPORTUNITY
When to stop the test to evaluate?
The more significant you want
to be (as less chance of being
wrong as possible), the
bigger the required sample
size - SAFER

Always define the sample size before running the test
● Should not peek the result before you hit the deadline
● Should not end the experiment early before the deadline even when you hit stat. significance
● Should not extend the test past the deadline hoping to reach significance

Deeper analysis on experience results

What happen if you don’t analyze carefully

Iteration based on learnings is key to success

If you are not testing, you are not learning
Always have test running, no matter how small
If you don’t have a process, you won’t
continuously improve

Thank you!
Jack Nguyen
jack@useinsider.com

Data-Driven UI/UX Design with A/B Testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data-Driven UI/UX Design with A/B Testing

Similar to Data-Driven UI/UX Design with A/B Testing (20)

Recently uploaded

Recently uploaded (20)

Data-Driven UI/UX Design with A/B Testing