尊敬的 微信汇率:1円 ≈ 0.046078 元 支付宝汇率:1円 ≈ 0.046168元 [退出登录]
SlideShare a Scribd company logo
A/B Testing
By: Kapil Saxena
What Is A/B Testing?
 Develop

two versions of a page
 Randomly show different versions to users
 Track how users perform
 Evaluate (that's where statistics comes in)
 Use the better version
Why A/B Test?
A

typical website converts 2% of visitors
into customers
 People can't explain why they left
 Small changes can make a big difference
 How about +40%?
 Google believes it works, see Content
Experiments in Google Analytics
What Can You A/B Test?
 Removing

form fields
 Adding relevant form fields
 Marketing landing pages
 Different explanations
 Having interstitial pages
 Email content
 Any casual decisions you care about
A/B Tests Do Not Substitute For
 Talking

to users
 Usability tests
 Acceptance tests
 Unit tests
 Thinking
The G-Test
A

method for comparing 2 data sets
 It was invented by Karl Pearson in 1900
 It is a close relative of the chi-square test
 It is our main method for evaluating A/B
tests
 There are alternatives
Limitations Of The G-Test
 Only

answers yes/no questions (but you
pick the question)
 Only handles 2 versions (there is a
workaround)
 Requires independence in samples
 Does not do confidence intervals
What To Measure
 Start

your A/B test
 Divide your users into groups A and B
 Decide whether each person did what
you want
 Reduce your results to 4 numbers: ($a_yes,
$a_no, $b_yes, $b_no)
G-Test Evaluation
 Select

a yes/no question about users
 Divide users in A and B into yes/no
 Perform complicated G-test calculation
to calculate $p
 Our confidence is 1-$p
 Make decision if our confidence is near
100% and we have enough samples
 Enough samples is at least 10 yes and no
results in each test
The Conversion Funnel
Your Conversion Funnel







Every company has one or more conversion
funnels
You should know yours, and be actively trying
to improve each step
Each step can be tracked with some metric
Most A/B tests concentrate on one step in the
funnel
Expect to run multiple A/B tests against each
Standardize these metrics
Examples Of Metrics
 Sessions,

sessions with registration
 People who searched, who viewed detail
page, contacted, leased
 People who saved favorites, started a
cart, completed purchase
 People who saw at least 3 pages, clicked
on an ad
 Anything measurable and important to
your business
Too Many Metrics?
 You

may have many metrics
 High confidence on one may be chance
 Believe if it was the metric you tried to
change
 Believe if very high confidence
 Believe if metrics agree
 Conflicting metrics require business
decision
Is That It?
 You

now know enough to run a successful
A/B test!
 If you do everything right
 If you do it wrong you won't know
 You'll just get random answers
 And believe them
Compare Apples To Apples
 Traffic

behaves differently at different

times
 Friday night ≠ Monday morning
 First week in month ≠ last week in month
 Last week's visitors have done more than
this week's
 Do not try to compare people from
different times
Be Careful When Changing
The Mix
A

and B can receive unequal traffic
 But do not change the mix they get
 Wrong Change(90/10) A vs B to (80/20)
 You are implicitly comparing old A's with
new B's
 Right Change(10/10/80) A vs B vs
Untested to (20/20/60)
 This comes up repeatedly
What Is Wrong With This?
 Suppose

you are A/B testing a change to
your product page
 You log hits on your product page
 You log clicks on Buy Now
 You plug those numbers into the A/B
calculator
 Is this OK?
Beware Hidden Correlations!
 Correlations

increase variability, and
therefore $g_test
 Some people look at many product
pages
 Their buying behaviour is correlated on
those pages
 This increases the size of chance
fluctuations
 Leading to wrong results
Guarantee Independence
 Whatever

granularity (session, person,
event) you make A/B decisons on...
 Needs to be what you test on
 In this case measure people who hit your
product page
 Measure people who clicked on Buy Now
 Those are the right statistics to use
 This comes up repeatedly
Wrong Metric








At Rent.com we changed the title of our
lease report email
The new email had improved opens and
clicks
That was because it interested people who
were still looking for a place to live
That email needed to interest people who
had already found a place to live
We looked at the wrong metric, and it cost us
millions
This mistake is fairly rare
That's It!
 Those

are the big mistakes that I've seen
 You now know how to do an A/B test
 ...and should have good odds of getting it
right
 Of course there is more to know
 But this is the core
Thanks!!!

More Related Content

What's hot

Google Analytics Overview
Google Analytics OverviewGoogle Analytics Overview
Google Analytics Overview
Anvil Media, Inc.
 
Lean Analytics for Startups and Enterprises
Lean Analytics for Startups and EnterprisesLean Analytics for Startups and Enterprises
Lean Analytics for Startups and Enterprises
Lean Analytics
 
Sales Segmentation & Qualification for B2B SaaS Companies
Sales Segmentation & Qualification for B2B SaaS CompaniesSales Segmentation & Qualification for B2B SaaS Companies
Sales Segmentation & Qualification for B2B SaaS Companies
Guillaume Lerouge
 
Guide to-google-analytics google 4
Guide to-google-analytics google 4Guide to-google-analytics google 4
Guide to-google-analytics google 4
Nizam Uddin
 
An Introduction to the Google Analytics 360 Suite
An Introduction to the Google Analytics 360 SuiteAn Introduction to the Google Analytics 360 Suite
An Introduction to the Google Analytics 360 Suite
Search Laboratory
 
Inbound Marketing: A Love Story
Inbound Marketing: A Love StoryInbound Marketing: A Love Story
Inbound Marketing: A Love Story
HubSpot
 
Growth marketing
Growth marketingGrowth marketing
Growth marketing
Onur Polat
 
Lean startup metrics
Lean startup metricsLean startup metrics
Lean startup metrics
Stuart Eccles
 
Measurefest - GA4 From Migration to Measurement - The Key To Success.pptx
Measurefest - GA4 From Migration to Measurement - The Key To Success.pptxMeasurefest - GA4 From Migration to Measurement - The Key To Success.pptx
Measurefest - GA4 From Migration to Measurement - The Key To Success.pptx
Sam Thomas
 
Customer segmentation
Customer segmentationCustomer segmentation
Customer segmentation
Sathya Narayanan
 
Google analytics version 4 in details
Google analytics version 4 in detailsGoogle analytics version 4 in details
Google analytics version 4 in details
Omkar Nandi
 
One Further - Spektrix and Google Analytics 4
One Further - Spektrix and Google Analytics 4One Further - Spektrix and Google Analytics 4
One Further - Spektrix and Google Analytics 4
One Further
 
Google Analytics 4: A Quick Start Guide
 Google Analytics 4: A Quick Start Guide Google Analytics 4: A Quick Start Guide
Google Analytics 4: A Quick Start Guide
Higher Education Marketing
 
🎙GA4 Chances & Differences Guide
🎙GA4 Chances & Differences Guide🎙GA4 Chances & Differences Guide
🎙GA4 Chances & Differences Guide
Şahin Seçil
 
Ultimate Guide to Funnel Optimization
Ultimate Guide to Funnel OptimizationUltimate Guide to Funnel Optimization
Ultimate Guide to Funnel Optimization
Sean Johnson
 
Introduction to Google Analytics
Introduction to Google AnalyticsIntroduction to Google Analytics
Introduction to Google Analytics
Cemal Buyukgokcesu
 
How Best-In-Class B2B Organizations Operationalize ABM
How Best-In-Class B2B Organizations Operationalize ABMHow Best-In-Class B2B Organizations Operationalize ABM
How Best-In-Class B2B Organizations Operationalize ABM
G3 Communications
 
Whatsapp business ppt
Whatsapp business pptWhatsapp business ppt
Whatsapp business ppt
digital omnia
 
29 Growth Hacking Quick Wins
29 Growth Hacking Quick Wins29 Growth Hacking Quick Wins
29 Growth Hacking Quick Wins
Mattan Griffel
 
Introduction to web analytics
Introduction to web analyticsIntroduction to web analytics
Introduction to web analytics
Rajeev Pillai
 

What's hot (20)

Google Analytics Overview
Google Analytics OverviewGoogle Analytics Overview
Google Analytics Overview
 
Lean Analytics for Startups and Enterprises
Lean Analytics for Startups and EnterprisesLean Analytics for Startups and Enterprises
Lean Analytics for Startups and Enterprises
 
Sales Segmentation & Qualification for B2B SaaS Companies
Sales Segmentation & Qualification for B2B SaaS CompaniesSales Segmentation & Qualification for B2B SaaS Companies
Sales Segmentation & Qualification for B2B SaaS Companies
 
Guide to-google-analytics google 4
Guide to-google-analytics google 4Guide to-google-analytics google 4
Guide to-google-analytics google 4
 
An Introduction to the Google Analytics 360 Suite
An Introduction to the Google Analytics 360 SuiteAn Introduction to the Google Analytics 360 Suite
An Introduction to the Google Analytics 360 Suite
 
Inbound Marketing: A Love Story
Inbound Marketing: A Love StoryInbound Marketing: A Love Story
Inbound Marketing: A Love Story
 
Growth marketing
Growth marketingGrowth marketing
Growth marketing
 
Lean startup metrics
Lean startup metricsLean startup metrics
Lean startup metrics
 
Measurefest - GA4 From Migration to Measurement - The Key To Success.pptx
Measurefest - GA4 From Migration to Measurement - The Key To Success.pptxMeasurefest - GA4 From Migration to Measurement - The Key To Success.pptx
Measurefest - GA4 From Migration to Measurement - The Key To Success.pptx
 
Customer segmentation
Customer segmentationCustomer segmentation
Customer segmentation
 
Google analytics version 4 in details
Google analytics version 4 in detailsGoogle analytics version 4 in details
Google analytics version 4 in details
 
One Further - Spektrix and Google Analytics 4
One Further - Spektrix and Google Analytics 4One Further - Spektrix and Google Analytics 4
One Further - Spektrix and Google Analytics 4
 
Google Analytics 4: A Quick Start Guide
 Google Analytics 4: A Quick Start Guide Google Analytics 4: A Quick Start Guide
Google Analytics 4: A Quick Start Guide
 
🎙GA4 Chances & Differences Guide
🎙GA4 Chances & Differences Guide🎙GA4 Chances & Differences Guide
🎙GA4 Chances & Differences Guide
 
Ultimate Guide to Funnel Optimization
Ultimate Guide to Funnel OptimizationUltimate Guide to Funnel Optimization
Ultimate Guide to Funnel Optimization
 
Introduction to Google Analytics
Introduction to Google AnalyticsIntroduction to Google Analytics
Introduction to Google Analytics
 
How Best-In-Class B2B Organizations Operationalize ABM
How Best-In-Class B2B Organizations Operationalize ABMHow Best-In-Class B2B Organizations Operationalize ABM
How Best-In-Class B2B Organizations Operationalize ABM
 
Whatsapp business ppt
Whatsapp business pptWhatsapp business ppt
Whatsapp business ppt
 
29 Growth Hacking Quick Wins
29 Growth Hacking Quick Wins29 Growth Hacking Quick Wins
29 Growth Hacking Quick Wins
 
Introduction to web analytics
Introduction to web analyticsIntroduction to web analytics
Introduction to web analytics
 

Viewers also liked

Building Agility for Automated Mobile Apps Testing
Building Agility for Automated Mobile Apps TestingBuilding Agility for Automated Mobile Apps Testing
Building Agility for Automated Mobile Apps Testing
Kapil Saxena
 
Master agile development and testing
Master agile development and testingMaster agile development and testing
Master agile development and testing
vmglover
 
Happy Hour - An Introduction to A/B Testing
Happy Hour - An Introduction to A/B TestingHappy Hour - An Introduction to A/B Testing
Happy Hour - An Introduction to A/B Testing
UnderstandingE
 
Mobile automated testing_erank
Mobile automated testing_erankMobile automated testing_erank
Mobile automated testing_erank
Perfecto Mobile
 
Webinar key considerations in mobile web automation
Webinar key considerations in mobile web automationWebinar key considerations in mobile web automation
Webinar key considerations in mobile web automation
Perfecto Mobile
 
Mobile App Testing Strategy
Mobile App Testing StrategyMobile App Testing Strategy
Mobile App Testing Strategy
Software Assurance LLC
 
Smarter Test Automation for Web & Mobile Apps
Smarter Test Automation for Web & Mobile AppsSmarter Test Automation for Web & Mobile Apps
Smarter Test Automation for Web & Mobile Apps
Keao Caindec
 
Webinar learn how to test any mobile app style from within eclipse using real...
Webinar learn how to test any mobile app style from within eclipse using real...Webinar learn how to test any mobile app style from within eclipse using real...
Webinar learn how to test any mobile app style from within eclipse using real...
Perfecto Mobile
 
Mdc2010 Automated Mobile Testing
Mdc2010 Automated Mobile TestingMdc2010 Automated Mobile Testing
Mdc2010 Automated Mobile Testing
momobangalore
 
mohit anand
 mohit anand mohit anand
mohit anand
BabuDevanandam
 
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala HariprasadDesigning Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Agile Testing Alliance
 
Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...
Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...
Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...
Perfecto Mobile
 
Mobile automation using selenium cucumber & appium
Mobile automation using selenium cucumber & appiumMobile automation using selenium cucumber & appium
Mobile automation using selenium cucumber & appium
Selenium Cucumber
 
Appium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriverAppium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriver
Andrii Dzynia
 

Viewers also liked (14)

Building Agility for Automated Mobile Apps Testing
Building Agility for Automated Mobile Apps TestingBuilding Agility for Automated Mobile Apps Testing
Building Agility for Automated Mobile Apps Testing
 
Master agile development and testing
Master agile development and testingMaster agile development and testing
Master agile development and testing
 
Happy Hour - An Introduction to A/B Testing
Happy Hour - An Introduction to A/B TestingHappy Hour - An Introduction to A/B Testing
Happy Hour - An Introduction to A/B Testing
 
Mobile automated testing_erank
Mobile automated testing_erankMobile automated testing_erank
Mobile automated testing_erank
 
Webinar key considerations in mobile web automation
Webinar key considerations in mobile web automationWebinar key considerations in mobile web automation
Webinar key considerations in mobile web automation
 
Mobile App Testing Strategy
Mobile App Testing StrategyMobile App Testing Strategy
Mobile App Testing Strategy
 
Smarter Test Automation for Web & Mobile Apps
Smarter Test Automation for Web & Mobile AppsSmarter Test Automation for Web & Mobile Apps
Smarter Test Automation for Web & Mobile Apps
 
Webinar learn how to test any mobile app style from within eclipse using real...
Webinar learn how to test any mobile app style from within eclipse using real...Webinar learn how to test any mobile app style from within eclipse using real...
Webinar learn how to test any mobile app style from within eclipse using real...
 
Mdc2010 Automated Mobile Testing
Mdc2010 Automated Mobile TestingMdc2010 Automated Mobile Testing
Mdc2010 Automated Mobile Testing
 
mohit anand
 mohit anand mohit anand
mohit anand
 
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala HariprasadDesigning Agile Test Strategy for Mobile Apps By Parimala Hariprasad
Designing Agile Test Strategy for Mobile Apps By Parimala Hariprasad
 
Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...
Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...
Cross Platform Mobile Test Automation using Selenium WebDriver by Perfecto Mo...
 
Mobile automation using selenium cucumber & appium
Mobile automation using selenium cucumber & appiumMobile automation using selenium cucumber & appium
Mobile automation using selenium cucumber & appium
 
Appium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriverAppium Mobile Test Automation like WebDriver
Appium Mobile Test Automation like WebDriver
 

Similar to A/B testing

Milion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B TestingMilion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B Testing
Azhar Bandeali
 
Introduction to Conversion Optimization - Nelio A/B Testing for WordPress
Introduction to Conversion Optimization - Nelio A/B Testing for WordPressIntroduction to Conversion Optimization - Nelio A/B Testing for WordPress
Introduction to Conversion Optimization - Nelio A/B Testing for WordPress
Nelio Software
 
Your Guide To Conversation Rate Optimization
Your Guide To Conversation Rate OptimizationYour Guide To Conversation Rate Optimization
Your Guide To Conversation Rate Optimization
UserEngage
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
Jack Nguyen (Hung Tien)
 
Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics Wednesday #10Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics
 
Alistaire croll lean analytics - montreal lean startup circle - september 2018
Alistaire croll   lean analytics - montreal lean startup circle - september 2018Alistaire croll   lean analytics - montreal lean startup circle - september 2018
Alistaire croll lean analytics - montreal lean startup circle - september 2018
Lean Startup Circle Montreal
 
The Power of A/B Testing
The Power of A/B TestingThe Power of A/B Testing
The Power of A/B Testing
Alexandre Pallota
 
A/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsA/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate Skills
Pirate Skills
 
Validation and hypothesis based product management by Abdallah Al-Khalidi
Validation and hypothesis based  product management by Abdallah Al-KhalidiValidation and hypothesis based  product management by Abdallah Al-Khalidi
Validation and hypothesis based product management by Abdallah Al-Khalidi
Abdallah Al-Khalidi
 
iSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doingiSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doing
Bridgeline Digital
 
Understand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesUnderstand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakes
TheFamily
 
Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009
Kayden Kelly
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experiments
Deven
 
Conversion Rate Optimisation - A fundamental part of your digital marketing s...
Conversion Rate Optimisation - A fundamental part of your digital marketing s...Conversion Rate Optimisation - A fundamental part of your digital marketing s...
Conversion Rate Optimisation - A fundamental part of your digital marketing s...
Visualsoft
 
Experiment Driven Development
Experiment Driven DevelopmentExperiment Driven Development
Experiment Driven Development
Mike Gehard
 
Christi olson-huzzah-searchand social
Christi olson-huzzah-searchand socialChristi olson-huzzah-searchand social
Christi olson-huzzah-searchand social
Jenn Mathews Marketing Consulting
 
Web Analytics: A Measurement System of Website
Web Analytics: A Measurement System of WebsiteWeb Analytics: A Measurement System of Website
Web Analytics: A Measurement System of Website
ITDogadjaji.com
 
Killer Marketing Bonus
Killer Marketing BonusKiller Marketing Bonus
Killer Marketing Bonus
Hiten Shah
 
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Traction Conf
 
2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite
Tim Stewart
 

Similar to A/B testing (20)

Milion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B TestingMilion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B Testing
 
Introduction to Conversion Optimization - Nelio A/B Testing for WordPress
Introduction to Conversion Optimization - Nelio A/B Testing for WordPressIntroduction to Conversion Optimization - Nelio A/B Testing for WordPress
Introduction to Conversion Optimization - Nelio A/B Testing for WordPress
 
Your Guide To Conversation Rate Optimization
Your Guide To Conversation Rate OptimizationYour Guide To Conversation Rate Optimization
Your Guide To Conversation Rate Optimization
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
 
Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics Wednesday #10Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics Wednesday #10
 
Alistaire croll lean analytics - montreal lean startup circle - september 2018
Alistaire croll   lean analytics - montreal lean startup circle - september 2018Alistaire croll   lean analytics - montreal lean startup circle - september 2018
Alistaire croll lean analytics - montreal lean startup circle - september 2018
 
The Power of A/B Testing
The Power of A/B TestingThe Power of A/B Testing
The Power of A/B Testing
 
A/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsA/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate Skills
 
Validation and hypothesis based product management by Abdallah Al-Khalidi
Validation and hypothesis based  product management by Abdallah Al-KhalidiValidation and hypothesis based  product management by Abdallah Al-Khalidi
Validation and hypothesis based product management by Abdallah Al-Khalidi
 
iSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doingiSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doing
 
Understand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakesUnderstand A/B Testing in 9 use cases & 7 mistakes
Understand A/B Testing in 9 use cases & 7 mistakes
 
Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experiments
 
Conversion Rate Optimisation - A fundamental part of your digital marketing s...
Conversion Rate Optimisation - A fundamental part of your digital marketing s...Conversion Rate Optimisation - A fundamental part of your digital marketing s...
Conversion Rate Optimisation - A fundamental part of your digital marketing s...
 
Experiment Driven Development
Experiment Driven DevelopmentExperiment Driven Development
Experiment Driven Development
 
Christi olson-huzzah-searchand social
Christi olson-huzzah-searchand socialChristi olson-huzzah-searchand social
Christi olson-huzzah-searchand social
 
Web Analytics: A Measurement System of Website
Web Analytics: A Measurement System of WebsiteWeb Analytics: A Measurement System of Website
Web Analytics: A Measurement System of Website
 
Killer Marketing Bonus
Killer Marketing BonusKiller Marketing Bonus
Killer Marketing Bonus
 
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
 
2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite2017 06-test withintelligence-conversionelite
2017 06-test withintelligence-conversionelite
 

Recently uploaded

Multivendor cloud production with VSF TR-11 - there and back again
Multivendor cloud production with VSF TR-11 - there and back againMultivendor cloud production with VSF TR-11 - there and back again
Multivendor cloud production with VSF TR-11 - there and back again
Kieran Kunhya
 
Leveraging AI for Software Developer Productivity.pptx
Leveraging AI for Software Developer Productivity.pptxLeveraging AI for Software Developer Productivity.pptx
Leveraging AI for Software Developer Productivity.pptx
petabridge
 
Database Management Myths for Developers
Database Management Myths for DevelopersDatabase Management Myths for Developers
Database Management Myths for Developers
John Sterrett
 
Product Listing Optimization Presentation - Gay De La Cruz.pdf
Product Listing Optimization Presentation - Gay De La Cruz.pdfProduct Listing Optimization Presentation - Gay De La Cruz.pdf
Product Listing Optimization Presentation - Gay De La Cruz.pdf
gaydlc2513
 
Kubernetes Cloud Native Indonesia Meetup - June 2024
Kubernetes Cloud Native Indonesia Meetup - June 2024Kubernetes Cloud Native Indonesia Meetup - June 2024
Kubernetes Cloud Native Indonesia Meetup - June 2024
Prasta Maha
 
From NCSA to the National Research Platform
From NCSA to the National Research PlatformFrom NCSA to the National Research Platform
From NCSA to the National Research Platform
Larry Smarr
 
APJC Introduction to ThousandEyes Webinar
APJC Introduction to ThousandEyes WebinarAPJC Introduction to ThousandEyes Webinar
APJC Introduction to ThousandEyes Webinar
ThousandEyes
 
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - MydbopsMySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
Mydbops
 
Supplier Sourcing Presentation - Gay De La Cruz.pdf
Supplier Sourcing Presentation - Gay De La Cruz.pdfSupplier Sourcing Presentation - Gay De La Cruz.pdf
Supplier Sourcing Presentation - Gay De La Cruz.pdf
gaydlc2513
 
DynamoDB to ScyllaDB: Technical Comparison and the Path to Success
DynamoDB to ScyllaDB: Technical Comparison and the Path to SuccessDynamoDB to ScyllaDB: Technical Comparison and the Path to Success
DynamoDB to ScyllaDB: Technical Comparison and the Path to Success
ScyllaDB
 
Introduction to ThousandEyes AMER Webinar
Introduction  to ThousandEyes AMER WebinarIntroduction  to ThousandEyes AMER Webinar
Introduction to ThousandEyes AMER Webinar
ThousandEyes
 
Ubuntu Server CLI cheat sheet 2024 v6.pdf
Ubuntu Server CLI cheat sheet 2024 v6.pdfUbuntu Server CLI cheat sheet 2024 v6.pdf
Ubuntu Server CLI cheat sheet 2024 v6.pdf
TechOnDemandSolution
 
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdfLee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
leebarnesutopia
 
Automation Student Developers Session 3: Introduction to UI Automation
Automation Student Developers Session 3: Introduction to UI AutomationAutomation Student Developers Session 3: Introduction to UI Automation
Automation Student Developers Session 3: Introduction to UI Automation
UiPathCommunity
 
ThousandEyes New Product Features and Release Highlights: June 2024
ThousandEyes New Product Features and Release Highlights: June 2024ThousandEyes New Product Features and Release Highlights: June 2024
ThousandEyes New Product Features and Release Highlights: June 2024
ThousandEyes
 
An Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise IntegrationAn Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise Integration
Safe Software
 
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
dipikamodels1
 
Elasticity vs. State? Exploring Kafka Streams Cassandra State Store
Elasticity vs. State? Exploring Kafka Streams Cassandra State StoreElasticity vs. State? Exploring Kafka Streams Cassandra State Store
Elasticity vs. State? Exploring Kafka Streams Cassandra State Store
ScyllaDB
 
Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0
Neeraj Kumar Singh
 
Fuxnet [EN] .pdf
Fuxnet [EN]                                   .pdfFuxnet [EN]                                   .pdf
Fuxnet [EN] .pdf
Overkill Security
 

Recently uploaded (20)

Multivendor cloud production with VSF TR-11 - there and back again
Multivendor cloud production with VSF TR-11 - there and back againMultivendor cloud production with VSF TR-11 - there and back again
Multivendor cloud production with VSF TR-11 - there and back again
 
Leveraging AI for Software Developer Productivity.pptx
Leveraging AI for Software Developer Productivity.pptxLeveraging AI for Software Developer Productivity.pptx
Leveraging AI for Software Developer Productivity.pptx
 
Database Management Myths for Developers
Database Management Myths for DevelopersDatabase Management Myths for Developers
Database Management Myths for Developers
 
Product Listing Optimization Presentation - Gay De La Cruz.pdf
Product Listing Optimization Presentation - Gay De La Cruz.pdfProduct Listing Optimization Presentation - Gay De La Cruz.pdf
Product Listing Optimization Presentation - Gay De La Cruz.pdf
 
Kubernetes Cloud Native Indonesia Meetup - June 2024
Kubernetes Cloud Native Indonesia Meetup - June 2024Kubernetes Cloud Native Indonesia Meetup - June 2024
Kubernetes Cloud Native Indonesia Meetup - June 2024
 
From NCSA to the National Research Platform
From NCSA to the National Research PlatformFrom NCSA to the National Research Platform
From NCSA to the National Research Platform
 
APJC Introduction to ThousandEyes Webinar
APJC Introduction to ThousandEyes WebinarAPJC Introduction to ThousandEyes Webinar
APJC Introduction to ThousandEyes Webinar
 
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - MydbopsMySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
 
Supplier Sourcing Presentation - Gay De La Cruz.pdf
Supplier Sourcing Presentation - Gay De La Cruz.pdfSupplier Sourcing Presentation - Gay De La Cruz.pdf
Supplier Sourcing Presentation - Gay De La Cruz.pdf
 
DynamoDB to ScyllaDB: Technical Comparison and the Path to Success
DynamoDB to ScyllaDB: Technical Comparison and the Path to SuccessDynamoDB to ScyllaDB: Technical Comparison and the Path to Success
DynamoDB to ScyllaDB: Technical Comparison and the Path to Success
 
Introduction to ThousandEyes AMER Webinar
Introduction  to ThousandEyes AMER WebinarIntroduction  to ThousandEyes AMER Webinar
Introduction to ThousandEyes AMER Webinar
 
Ubuntu Server CLI cheat sheet 2024 v6.pdf
Ubuntu Server CLI cheat sheet 2024 v6.pdfUbuntu Server CLI cheat sheet 2024 v6.pdf
Ubuntu Server CLI cheat sheet 2024 v6.pdf
 
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdfLee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
 
Automation Student Developers Session 3: Introduction to UI Automation
Automation Student Developers Session 3: Introduction to UI AutomationAutomation Student Developers Session 3: Introduction to UI Automation
Automation Student Developers Session 3: Introduction to UI Automation
 
ThousandEyes New Product Features and Release Highlights: June 2024
ThousandEyes New Product Features and Release Highlights: June 2024ThousandEyes New Product Features and Release Highlights: June 2024
ThousandEyes New Product Features and Release Highlights: June 2024
 
An Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise IntegrationAn Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise Integration
 
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
 
Elasticity vs. State? Exploring Kafka Streams Cassandra State Store
Elasticity vs. State? Exploring Kafka Streams Cassandra State StoreElasticity vs. State? Exploring Kafka Streams Cassandra State Store
Elasticity vs. State? Exploring Kafka Streams Cassandra State Store
 
Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0
 
Fuxnet [EN] .pdf
Fuxnet [EN]                                   .pdfFuxnet [EN]                                   .pdf
Fuxnet [EN] .pdf
 

A/B testing

  • 2. What Is A/B Testing?  Develop two versions of a page  Randomly show different versions to users  Track how users perform  Evaluate (that's where statistics comes in)  Use the better version
  • 3. Why A/B Test? A typical website converts 2% of visitors into customers  People can't explain why they left  Small changes can make a big difference  How about +40%?  Google believes it works, see Content Experiments in Google Analytics
  • 4. What Can You A/B Test?  Removing form fields  Adding relevant form fields  Marketing landing pages  Different explanations  Having interstitial pages  Email content  Any casual decisions you care about
  • 5. A/B Tests Do Not Substitute For  Talking to users  Usability tests  Acceptance tests  Unit tests  Thinking
  • 6. The G-Test A method for comparing 2 data sets  It was invented by Karl Pearson in 1900  It is a close relative of the chi-square test  It is our main method for evaluating A/B tests  There are alternatives
  • 7. Limitations Of The G-Test  Only answers yes/no questions (but you pick the question)  Only handles 2 versions (there is a workaround)  Requires independence in samples  Does not do confidence intervals
  • 8. What To Measure  Start your A/B test  Divide your users into groups A and B  Decide whether each person did what you want  Reduce your results to 4 numbers: ($a_yes, $a_no, $b_yes, $b_no)
  • 9. G-Test Evaluation  Select a yes/no question about users  Divide users in A and B into yes/no  Perform complicated G-test calculation to calculate $p  Our confidence is 1-$p  Make decision if our confidence is near 100% and we have enough samples  Enough samples is at least 10 yes and no results in each test
  • 11. Your Conversion Funnel       Every company has one or more conversion funnels You should know yours, and be actively trying to improve each step Each step can be tracked with some metric Most A/B tests concentrate on one step in the funnel Expect to run multiple A/B tests against each Standardize these metrics
  • 12. Examples Of Metrics  Sessions, sessions with registration  People who searched, who viewed detail page, contacted, leased  People who saved favorites, started a cart, completed purchase  People who saw at least 3 pages, clicked on an ad  Anything measurable and important to your business
  • 13. Too Many Metrics?  You may have many metrics  High confidence on one may be chance  Believe if it was the metric you tried to change  Believe if very high confidence  Believe if metrics agree  Conflicting metrics require business decision
  • 14. Is That It?  You now know enough to run a successful A/B test!  If you do everything right  If you do it wrong you won't know  You'll just get random answers  And believe them
  • 15. Compare Apples To Apples  Traffic behaves differently at different times  Friday night ≠ Monday morning  First week in month ≠ last week in month  Last week's visitors have done more than this week's  Do not try to compare people from different times
  • 16. Be Careful When Changing The Mix A and B can receive unequal traffic  But do not change the mix they get  Wrong Change(90/10) A vs B to (80/20)  You are implicitly comparing old A's with new B's  Right Change(10/10/80) A vs B vs Untested to (20/20/60)  This comes up repeatedly
  • 17. What Is Wrong With This?  Suppose you are A/B testing a change to your product page  You log hits on your product page  You log clicks on Buy Now  You plug those numbers into the A/B calculator  Is this OK?
  • 18. Beware Hidden Correlations!  Correlations increase variability, and therefore $g_test  Some people look at many product pages  Their buying behaviour is correlated on those pages  This increases the size of chance fluctuations  Leading to wrong results
  • 19. Guarantee Independence  Whatever granularity (session, person, event) you make A/B decisons on...  Needs to be what you test on  In this case measure people who hit your product page  Measure people who clicked on Buy Now  Those are the right statistics to use  This comes up repeatedly
  • 20. Wrong Metric       At Rent.com we changed the title of our lease report email The new email had improved opens and clicks That was because it interested people who were still looking for a place to live That email needed to interest people who had already found a place to live We looked at the wrong metric, and it cost us millions This mistake is fairly rare
  • 21. That's It!  Those are the big mistakes that I've seen  You now know how to do an A/B test  ...and should have good odds of getting it right  Of course there is more to know  But this is the core
  翻译: