尊敬的 微信汇率：1円 ≈ 0.046078 元 支付宝汇率：1円 ≈ 0.046168元 [退出登录]
SlideShare a Scribd company logo

Why everything is an A/B Test at Pinterest

•Download as PPTX, PDF•
•
Krishna Gade
Krishna Gade

This document discusses how Pinterest uses A/B testing for all aspects of their product and business. It summarizes key metrics used to measure user acquisition, activation, engagement and churn. It also provides examples of how Pinterest ran A/B tests for new features, copy testing, SEO changes and infrastructure updates to analyze impact on user behavior and site traffic. The document emphasizes the importance of testing all changes through experiments to understand their effects.

Data & Analytics
1 of 48
Krishna Gade Why everything is an A/B test at Pinterest
Confidentia l 2
What is Pinterest? A visual bookmarking tool Discover an inspiring idea Save it to a board Go do it
Pinterest is a data product
Data Flow App events Kafka Secor Skyline Pinball Redshift Pinalytics Features Qubole (Hadoop) Singer
Confidentia l Metrics 7
Confidentia l User Lifecycle New Users Active Users Dormant Users Acquisition Activation Churn Churn Resurrection
Confidentia l Terminology ● MAU – Monthly Active User ● WAU – Weekly Active User ● DAU – Daily Active User ● WAR – Weekly Active Repinner ● WAC – Weekly Active Clickthrougher ● WARC – Weekly Active Repinner or Clickthrougher ● 1d7 – User who comes back at-least once in 7 days after signup ● 1rc7 - User who repins/clicks at-least once in 7 days after signup
Confidentia l MAUs ● Current progress to quarterly goal ● MAU forecast ● MAUs by app/gender/country ● MAU Accounting
Confidentia l MAU Accounting
Confidentia l Acquisition Metrics ● Total Signups ● Signups by App ● Signups by Referrer ● Invites Sent ● Unique Invite Senders ● Invite Signups
Confidentia l Activation Metrics Activation Rate = 1rc7/signups ● Activation Rate by App/Gender/Referrer ● Overall signups to 1rc7 ● Signup engagement funnel ● Signups to WAU 35 days after signup
Confidentia l Engagement Metrics ● *AU ratios ● # of Emails sent ● # of Emails opened ● # of Emails clicked ● # of Notifications sent ● # of Notifications clicked
Confidentia l Resurrection Metrics ● # of Users who were dormant for 28+ days but came back, split by platform, referrer etc
Confidentia l Takeaways ● Emphasize on activation (process of converting a new user to MAU) ● Activation critical to long-term growth ● Deeply understand how different segments of users interact
Confidentia l Automated Metrics 17
Confidentia l
Confidentia l Chart creation
Confidentia l Reuse existing query result
Confidentia l Metrics creation
Confidentia l A/B Experimentation 22
Confidentia l Categories of Experiments ● A/B testing new features ● Copy testing ● SEO changes ● Infrastructure updates
Confidentia l A/B Testing Framework
Confidentia l
Confidentia l
Confidentia l
Confidentia l A/B Testing Dashboard
Confidentia l
Confidentia l p-value (via chi-squared test) Statistical tests for validation
Confidentia l Impact of Badging
Confidentia l Impact of Badging ● 7% lift in DAU ● 2.5% lift over long term ● Effective at increasing engagement levels
Confidentia l Copy Testing Definition: Testing different variations of text we display to the user. ● A/B Testing copy was a tedious process ● Even when we did this we were optimizing only for English
Confidentia l New Approach ● Ability to run a copy test w/o changing a line of code. ● Support for translating text dynamically ● Use Multi-armed bandits to converge to showing best performing text
Confidentia l Multi Armed Bandits
Confidentia l SEO 36 50+ Billion Pins categorized by people into more than 1 Billion Boards
Confidentia l 37 What makes SEO so challenging? What is the Impact of the new SEO feature? Will this new change affect SEO? Uhhh... Dunno... Uhhh... Ask Google.
Confidentia l KEEP CALM and RUN AN SEO EXPERIMENT
Confidentia l What is an SEO experiment? 39 Regular AB Experiment SEO Experiment Metrics User behavior Search engine referred traffic Unique id for hashing userid / sessionid page url
Confidentia l What is an SEO experiment? Launch ● Measure the gap ● Before vs After launch
Confidentia l Dashboard 41 DIFF = enabled - control Traffic Gain
Confidentia l 42 Decision Making 1 2 3 4 SHIP! SHIP! SHUT DOWN! VALIDATE
Confidentia l 43 Experiment Driven SEO What is the Impact of the new SEO feature? We got XX% of traffic growth from it! Can we use more resources to make bigger impact? No, the experiment result is flat, so let’s ship it. SEO experiment SEO experiment Is my UI change affecting SEO negatively?
Confidentia l Why you should be A/B testing even your infrastructure
Confidentia l Infrastructure Changes ● 2013: A new web framework ● 2014: Pyapns ● 2015: User service
Confidentia l Experiment results for replacing PyAPNs
Confidentia l Conclusions ● Figure out what success means ● Define your metrics ● Automate metric computation ● A/B test everything!
Confidentia l Thank You!

Recommended

A/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at SpotifyA/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at Spotify
Danielle Jabin
 
A/B testing at Spotify
A/B testing at SpotifyA/B testing at Spotify
A/B testing at Spotify
Ali Sarrafi
 
A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation
WrangleConf
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online products
Ashish Dua
 
Prioritization Method for Every Case by fmr Atlassian Principal PM
Prioritization Method for Every Case by fmr Atlassian Principal PMPrioritization Method for Every Case by fmr Atlassian Principal PM
Prioritization Method for Every Case by fmr Atlassian Principal PM
Product School
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
Dan Chuparkoff
 
The Power of A/B Testing
The Power of A/B TestingThe Power of A/B Testing
The Power of A/B Testing
Alexandre Pallota
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
Ashish Dua
 

Recommended

A/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at SpotifyA/B Testing Pitfalls and Lessons Learned at Spotify
A/B Testing Pitfalls and Lessons Learned at Spotify
Danielle Jabin
 
A/B testing at Spotify
A/B testing at SpotifyA/B testing at Spotify
A/B testing at Spotify
Ali Sarrafi
 
A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation A/B Testing at Pinterest: Building a Culture of Experimentation
A/B Testing at Pinterest: Building a Culture of Experimentation
WrangleConf
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online products
Ashish Dua
 
Prioritization Method for Every Case by fmr Atlassian Principal PM
Prioritization Method for Every Case by fmr Atlassian Principal PMPrioritization Method for Every Case by fmr Atlassian Principal PM
Prioritization Method for Every Case by fmr Atlassian Principal PM
Product School
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
Dan Chuparkoff
 
The Power of A/B Testing
The Power of A/B TestingThe Power of A/B Testing
The Power of A/B Testing
Alexandre Pallota
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
Ashish Dua
 
A/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerA/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product Manager
Product School
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
Janessa Lantz
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scale
courseratalks
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
Product School
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
Ramkumar Ravichandran
 
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PMControlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Product School
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework Design
Patrick McKenzie
 
Experimentation Platform at Netflix
Experimentation Platform at NetflixExperimentation Platform at Netflix
Experimentation Platform at Netflix
Steve Urban
 
Best Practices for Benchmarking the Website User Experience featuring Measuri...
Best Practices for Benchmarking the Website User Experience featuring Measuri...Best Practices for Benchmarking the Website User Experience featuring Measuri...
Best Practices for Benchmarking the Website User Experience featuring Measuri...
UserZoom
 
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsNetflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Chris Saint-Amant
 
Ab testing
Ab testingAb testing
Ab testing
Fahad Zahid
 
Lean Analytics for Startups and Enterprises
Lean Analytics for Startups and EnterprisesLean Analytics for Startups and Enterprises
Lean Analytics for Startups and Enterprises
Lean Analytics
 
UX and Scrum
UX and ScrumUX and Scrum
UX and Scrum
Roman Pichler
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
Zack Notes
 
Customer to Product Idea Iteration by Amazon's Product Manager
Customer to Product Idea Iteration by Amazon's Product ManagerCustomer to Product Idea Iteration by Amazon's Product Manager
Customer to Product Idea Iteration by Amazon's Product Manager
Product School
 
First 90 days as a Product Manager
First 90 days as a Product ManagerFirst 90 days as a Product Manager
First 90 days as a Product Manager
Product School
 
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Dealmaker Media
 
A Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market FitA Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market Fit
Lean Startup Co.
 
10 Guidelines for A/B Testing
10 Guidelines for A/B Testing10 Guidelines for A/B Testing
10 Guidelines for A/B Testing
Emily Robinson
 
21 Actionable Growth Hacking Tactics
21 Actionable Growth Hacking Tactics21 Actionable Growth Hacking Tactics
21 Actionable Growth Hacking Tactics
Jon Yongfook
 
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
DevOpsDays Tel Aviv
 

More Related Content

What's hot

A/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerA/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product Manager
Product School
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
Janessa Lantz
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scale
courseratalks
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
Product School
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
Ramkumar Ravichandran
 
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PMControlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Product School
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework Design
Patrick McKenzie
 
Experimentation Platform at Netflix
Experimentation Platform at NetflixExperimentation Platform at Netflix
Experimentation Platform at Netflix
Steve Urban
 
Best Practices for Benchmarking the Website User Experience featuring Measuri...
Best Practices for Benchmarking the Website User Experience featuring Measuri...Best Practices for Benchmarking the Website User Experience featuring Measuri...
Best Practices for Benchmarking the Website User Experience featuring Measuri...
UserZoom
 
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsNetflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Chris Saint-Amant
 
Ab testing
Ab testingAb testing
Ab testing
Fahad Zahid
 
Lean Analytics for Startups and Enterprises
Lean Analytics for Startups and EnterprisesLean Analytics for Startups and Enterprises
Lean Analytics for Startups and Enterprises
Lean Analytics
 
UX and Scrum
UX and ScrumUX and Scrum
UX and Scrum
Roman Pichler
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
Zack Notes
 
Customer to Product Idea Iteration by Amazon's Product Manager
Customer to Product Idea Iteration by Amazon's Product ManagerCustomer to Product Idea Iteration by Amazon's Product Manager
Customer to Product Idea Iteration by Amazon's Product Manager
Product School
 
First 90 days as a Product Manager
First 90 days as a Product ManagerFirst 90 days as a Product Manager
First 90 days as a Product Manager
Product School
 
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Dealmaker Media
 
A Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market FitA Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market Fit
Lean Startup Co.
 
10 Guidelines for A/B Testing
10 Guidelines for A/B Testing10 Guidelines for A/B Testing
10 Guidelines for A/B Testing
Emily Robinson
 
21 Actionable Growth Hacking Tactics
21 Actionable Growth Hacking Tactics21 Actionable Growth Hacking Tactics
21 Actionable Growth Hacking Tactics
Jon Yongfook
 

What's hot (20)

A/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product ManagerA/B Testing with Yammer's Product Manager
A/B Testing with Yammer's Product Manager
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
 
Talks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet ScaleTalks@Coursera - A/B Testing @ Internet Scale
Talks@Coursera - A/B Testing @ Internet Scale
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
 
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PMControlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
Controlled Experimentation aka A/B Testing for PMs by Tinder Sr PM
 
A/B Testing Framework Design
A/B Testing Framework DesignA/B Testing Framework Design
A/B Testing Framework Design
 
Experimentation Platform at Netflix
Experimentation Platform at NetflixExperimentation Platform at Netflix
Experimentation Platform at Netflix
 
Best Practices for Benchmarking the Website User Experience featuring Measuri...
Best Practices for Benchmarking the Website User Experience featuring Measuri...Best Practices for Benchmarking the Website User Experience featuring Measuri...
Best Practices for Benchmarking the Website User Experience featuring Measuri...
 
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.jsNetflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
 
Ab testing
Ab testingAb testing
Ab testing
 
Lean Analytics for Startups and Enterprises
Lean Analytics for Startups and EnterprisesLean Analytics for Startups and Enterprises
Lean Analytics for Startups and Enterprises
 
UX and Scrum
UX and ScrumUX and Scrum
UX and Scrum
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
 
Customer to Product Idea Iteration by Amazon's Product Manager
Customer to Product Idea Iteration by Amazon's Product ManagerCustomer to Product Idea Iteration by Amazon's Product Manager
Customer to Product Idea Iteration by Amazon's Product Manager
 
First 90 days as a Product Manager
First 90 days as a Product ManagerFirst 90 days as a Product Manager
First 90 days as a Product Manager
 
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
Startup Metrics for Pirates: AARRR! (Startonomics SF 2008)
 
A Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market FitA Playbook for Achieving Product-Market Fit
A Playbook for Achieving Product-Market Fit
 
10 Guidelines for A/B Testing
10 Guidelines for A/B Testing10 Guidelines for A/B Testing
10 Guidelines for A/B Testing
 
21 Actionable Growth Hacking Tactics
21 Actionable Growth Hacking Tactics21 Actionable Growth Hacking Tactics
21 Actionable Growth Hacking Tactics
 

Similar to Why everything is an A/B Test at Pinterest

Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
DevOpsDays Tel Aviv
 
Software Release Orchestration and the Enterprise
Software Release Orchestration and the EnterpriseSoftware Release Orchestration and the Enterprise
Software Release Orchestration and the Enterprise
XebiaLabs
 
Building turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet videoBuilding turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet video
Nir Yungster
 
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
South Tyrol Free Software Conference
 
Video Recommendation Engines as a Service
Video Recommendation Engines as a ServiceVideo Recommendation Engines as a Service
Video Recommendation Engines as a Service
Kamil Sindi
 
Making the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTestMaking the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTest
QASymphony
 
Developing Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CDDeveloping Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CD
Paul Hepworth
 
Monktoberfest Fast Delivery
Monktoberfest Fast DeliveryMonktoberfest Fast Delivery
Monktoberfest Fast Delivery
Adrian Cockcroft
 
Continuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teamsContinuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teams
Laurent PY
 
The Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CHThe Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CH
Matteo Bonifazi
 
Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...
Adjust
 
Continuous Delivery at Snyk
Continuous Delivery at SnykContinuous Delivery at Snyk
Continuous Delivery at Snyk
Anton Drukh
 
Data-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to QualityData-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to Quality
TechWell
 
Freedom and Responsibility
Freedom and ResponsibilityFreedom and Responsibility
Freedom and Responsibility
Mike Ruangutai
 
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous DeliveryEnabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
VMware Tanzu
 
Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product
Product School
 
How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...
TAUS - The Language Data Network
 
From four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a teamFrom four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a team
Rich Allen
 
Developing and deploying the Netflix API service
Developing and deploying the Netflix API serviceDeveloping and deploying the Netflix API service
Developing and deploying the Netflix API service
Sangeeta Narayanan
 

Similar to Why everything is an A/B Test at Pinterest (20)

Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" StageStartupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
Startupfest 2015: KRISHNA GADE (Pinterest) - "How To" Stage
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
 
Software Release Orchestration and the Enterprise
Software Release Orchestration and the EnterpriseSoftware Release Orchestration and the Enterprise
Software Release Orchestration and the Enterprise
 
Building turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet videoBuilding turn-key recommendations for 5% of internet video
Building turn-key recommendations for 5% of internet video
 
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
SFScon21 - Paolo d’Incau - Going to production in a few months – How we did it!
 
Video Recommendation Engines as a Service
Video Recommendation Engines as a ServiceVideo Recommendation Engines as a Service
Video Recommendation Engines as a Service
 
Making the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTestMaking the Switch from HP Quality Center to qTest
Making the Switch from HP Quality Center to qTest
 
Developing Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CDDeveloping Quality Products Quickly through a Culture of CI/CD
Developing Quality Products Quickly through a Culture of CI/CD
 
Monktoberfest Fast Delivery
Monktoberfest Fast DeliveryMonktoberfest Fast Delivery
Monktoberfest Fast Delivery
 
Continuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teamsContinuous testing for Agile and DevOps teams
Continuous testing for Agile and DevOps teams
 
The Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CHThe Firebase tier for your mobile app - DevFest CH
The Firebase tier for your mobile app - DevFest CH
 
Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...Mobile video ads: How to acquire high quality new users and measure marketing...
Mobile video ads: How to acquire high quality new users and measure marketing...
 
Continuous Delivery at Snyk
Continuous Delivery at SnykContinuous Delivery at Snyk
Continuous Delivery at Snyk
 
Data-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to QualityData-Driven Software Testing: The New, Lean Approach to Quality
Data-Driven Software Testing: The New, Lean Approach to Quality
 
Freedom and Responsibility
Freedom and ResponsibilityFreedom and Responsibility
Freedom and Responsibility
 
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous DeliveryEnabling Business Outcomes Through Extreme Programming and Continuous Delivery
Enabling Business Outcomes Through Extreme Programming and Continuous Delivery
 
Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product Intro to Data Analytics with Oscar's Director of Product
Intro to Data Analytics with Oscar's Director of Product
 
How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...How to make sure the right quality is delivered by my translation vendor? (Se...
How to make sure the right quality is delivered by my translation vendor? (Se...
 
From four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a teamFrom four to forty in four years - lessons from growing a team
From four to forty in four years - lessons from growing a team
 
Developing and deploying the Netflix API service
Developing and deploying the Netflix API serviceDeveloping and deploying the Netflix API service
Developing and deploying the Netflix API service
 

Recently uploaded

Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Do People Really Know Their Fertility Intentions? Correspondence between Sel...Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Xiao Xu
 
Salesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - CanariasSalesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - Canarias
davidpietrzykowski1
 
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your DoorAhmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Russian Escorts in Delhi 9711199171 with low rate Book online
 
Royal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cash
Royal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cashRoyal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cash
Royal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cash
Ak47
 
Bangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts ServiceBangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts Service
nhero3888
 
AI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOW
AI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOWAI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOW
AI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOW
arash10gamer
 
A review of I_O behavior on Oracle database in ASM
A review of I_O behavior on Oracle database in ASMA review of I_O behavior on Oracle database in ASM
A review of I_O behavior on Oracle database in ASM
Alireza Kamrani
 
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
mona lisa $A12
 
Bangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In Bangalore
Bangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In BangaloreBangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In Bangalore
Bangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In Bangalore
yashusingh54876
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call GirlCall Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
sapna sharmap11
 
Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...
Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...
Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...
ThinkInnovation
 
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
mparmparousiskostas
 
Call Girls Lucknow 8923113531 Independent Call Girl Service in Lucknow
Call Girls Lucknow 8923113531 Independent Call Girl Service in LucknowCall Girls Lucknow 8923113531 Independent Call Girl Service in Lucknow
Call Girls Lucknow 8923113531 Independent Call Girl Service in Lucknow
hiju9823
 
❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...
❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...
❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...
jasodak99
 
PCI-DSS-Data Security Standard v4.0.1.pdf
PCI-DSS-Data Security Standard v4.0.1.pdfPCI-DSS-Data Security Standard v4.0.1.pdf
PCI-DSS-Data Security Standard v4.0.1.pdf
incitbe
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
prijesh mathew
 
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls HyderabadHyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
binna singh$A17
 
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
PsychoTech Services
 
IBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTXIBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTX
EbtsamRashed
 
_Lufthansa Airlines MIA Terminal (1).pdf
_Lufthansa Airlines MIA Terminal (1).pdf_Lufthansa Airlines MIA Terminal (1).pdf
_Lufthansa Airlines MIA Terminal (1).pdf
rc76967005
 

Recently uploaded (20)

Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Do People Really Know Their Fertility Intentions? Correspondence between Sel...Do People Really Know Their Fertility Intentions? Correspondence between Sel...
Do People Really Know Their Fertility Intentions? Correspondence between Sel...
 
Salesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - CanariasSalesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - Canarias
 
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your DoorAhmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
Ahmedabad Call Girls 7339748667 With Free Home Delivery At Your Door
 
Royal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cash
Royal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cashRoyal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cash
Royal-Class Call Girls Thane🌹9967824496🌹369+ call girls @₹6K-18K/full night cash
 
Bangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts ServiceBangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts Service
 
AI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOW
AI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOWAI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOW
AI WITH THE HELP OF NAGALAND CAN WIN. DOWNLOAD NOW
 
A review of I_O behavior on Oracle database in ASM
A review of I_O behavior on Oracle database in ASMA review of I_O behavior on Oracle database in ASM
A review of I_O behavior on Oracle database in ASM
 
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
Delhi Call Girls Karol Bagh 👉 9711199012 👈 unlimited short high profile full ...
 
Bangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In Bangalore
Bangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In BangaloreBangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In Bangalore
Bangalore Call Girls ♠ 9079923931 ♠ Beautiful Call Girls In Bangalore
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call GirlCall Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
 
Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...
Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...
Difference in Differences - Does Strict Speed Limit Restrictions Reduce Road ...
 
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
Optimizing Feldera: Integrating Advanced UDFs and Enhanced SQL Functionality ...
 
Call Girls Lucknow 8923113531 Independent Call Girl Service in Lucknow
Call Girls Lucknow 8923113531 Independent Call Girl Service in LucknowCall Girls Lucknow 8923113531 Independent Call Girl Service in Lucknow
Call Girls Lucknow 8923113531 Independent Call Girl Service in Lucknow
 
❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...
❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...
❣VIP Call Girls Chennai 💯Call Us 🔝 7737669865 🔝💃Independent Chennai Escorts S...
 
PCI-DSS-Data Security Standard v4.0.1.pdf
PCI-DSS-Data Security Standard v4.0.1.pdfPCI-DSS-Data Security Standard v4.0.1.pdf
PCI-DSS-Data Security Standard v4.0.1.pdf
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
 
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls HyderabadHyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
Hyderabad Call Girls Service 🔥 9352988975 🔥 High Profile Call Girls Hyderabad
 
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
Essential Skills for Family Assessment - Marital and Family Therapy and Couns...
 
IBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTXIBM watsonx.data - Seller Enablement Deck.PPTX
IBM watsonx.data - Seller Enablement Deck.PPTX
 
_Lufthansa Airlines MIA Terminal (1).pdf
_Lufthansa Airlines MIA Terminal (1).pdf_Lufthansa Airlines MIA Terminal (1).pdf
_Lufthansa Airlines MIA Terminal (1).pdf
 

Why everything is an A/B Test at Pinterest

  翻译：