SlideShare a Scribd company logo
1 of 45
Download to read offline
A/B Testing in Android
Nir Hartmann
Drippler
Droidcon Tel-Aviv 2014
Why do we need A/B Testing?
• Tests takes the guesswork out
• Enables data-backed decisions
• Enhances engagement and retention
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
What is an A/B Test ?
• Case study – onboarding screen
Define the test
• Hypothesis – The layout with the Google+ button at
the left will increase the number of total registered
users.
• Goal – A registered user (the user can skip
registration).
• View event
- Login fragment onCreate().
- Login activity onCreate().
• Variables – Facebook button position (left or right).
• Participants – New users.
Amazon A/B Testing SDK
• Very customizable, you can do just about anything as
long as you know what it is you want to do
• It’s free
• Drippler created an open source library that simplify
the process
Setup the A/B test
• Setup identifier
– https://developer.amazon.com/al/index.html
Setup the A/B test
• Create a project
Setup the A/B test
• Create the test
Dive into the code
• https://github.com/Drippler/ABTester
public class MyApplication extends Application {
@Override
public void onCreate() {
super.onCreate();
ABTester.init(getApplicationContext(),
"my_public_key", "my_private_key");
}
}
private void initLoginActivityTest() {
try {
ABTester.syncPreFetch(
TimeUnit.SECONDS.toMillis(15),
new ABTest("Login page test", false, "Facebook is
first”) );
} catch (TimeoutException e) {
// Couldn't reach amazon servers
}
}
Fetch the test
Login Fragment
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setRetainInstance(true);
ABTester.recordEvent(
"Login fragment shown",
false);
}
Login Fragment
@Override
public View onCreateView(LayoutInflater inflater,
ViewGroup container, Bundle savedInstanceState) {
boolean shouldShowFacebookFirst =
ABTester.getBoolean("Login page test", "Facebook is
first", false);
if (shouldShowFacebookFirst)
return inflater.inflate(R.layout.facebook_first, null);
else
return inflater.inflate(R.layout.google_first, null);
}
Report goal event
public void onUserLoggedIn() {
ABTester.recordEvent("Sign in", false);
}
@Override
protected void onPause() {
super.onPause();
/* Submit the events that were previously stored locally.
* asynchronously
* call it in the onPause() method of an activity */
ABTester.submitEvents();
}
Analyze the results
Variation Views Conversions Conversion
rate
Change
Google+
first
1064 320 30.08%
Facebook
first
1043 250 23.97% -20.30%
Dice experiment
Goal: maximize the amount of 3’s we get in a 200
dice roll
Dice experiment
Hypothesis: wearing a hat will increase the
chance to roll a 3
Analyze the results
Variation Views Conversions Conversion
rate
Change
Hat off 200 31 15.50%
Hat on 200 38 19.00% +22.58%
Conversion is never a single number.
Confidence level
• Measure the reliability of an estimate
– The confidence levels help us understand if the
results are different merely by chance or by
reason
• 95% confidence level is considered good
Analyze the results
Variation Views Conversions Conversion
rate
Change
Google
First
1064 320 30.08% ±
2.32%
Facebook
First
1043 250 23.97% ±
2.18%
-20.30%
• Confidence level of 99%
Analyze the results
https://developer.amazon.com/public/apis/manage/ab-
testing/doc/math-behind-ab-testing
Choose the best variation
• Launch
– Choose the winning variation
– Control the percentage of customers that receive
a new feature
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png
Segmentation
Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button’s click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – All users
Create the test
Rate us DialogFragment
public class RateUsDialog extends DialogFragment {
public static void show(FragmentManager fm,
int color) {
RateUsDialog rateUs = new RateUsDialog();
Bundle extras = new Bundle();
extras.putInt(“color”, color);
rateUs.setArguments(extras);
rateUs.show(fm, “my tag”);
ABTester.recordEvent("Rate us dialog shown", false);
}
Rate us DialogFragment
@Override
public Dialog onCreateDialog(Bundle
savedInstanceState) {
int color = getArguments().getInt("color");
return createColoredDialog(color);
}
Rate us DialogFragment
private Dialog createColoredDialog(int color) {
...
.setPositiveButton("Rate", new OnClickListener() {
@Override
public void onClick(DialogInterface dialog, int
which) {
ABTester.recordEvent("Rate button click", false);
}
});
return myDialog;
Rate us test
1) Asynchronously prefetching SplashActivity
Default timeout is 60 seconds, and can be overriden by
using preFetch(long timeout, ABTest... Test)
ABTester.preFetch(
new ABTest("Rate us test", false, "Rate button color")
);
Rate us test
2) Show the dialog
String fetchedColor = ABTester.getString(
"Rate us test", "Rate button color", "#F5F5F5");
int color = Color.parseColor(fetchedColor);
RateUsDialog.show(getFragmentManager(), color);
3) Submitting the results onPause()
ABTester.submitEvents();
Analyze the results
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
865 234
27.05%
± 1.51%
Variation
A (green)
904 250
27.65%
± 1.49%
-0.2%
Variation
B (red) 830 230
27.71%
± 1.55% +2.4% 51%
What can I do with these results?
Segmentation
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
432 92 21.30% ± 1.97%
Variation A
(green)
464 165 35.56% ± 2.22% +66.9% 98.7%
Variation B
(red)
420 120 28.57% ± 2.20%
+34.1%
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
433 142 32.79% ± 2.26% +22.2% 97.1%
Variation A
(green)
440 85 19.32% ± 1.88% -27.9%
Variation B
(red)
410 110 26.83% ± 2.19% -18.8%
Under 40
Over 40
Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – All users
Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – Age specific tests
Setup the A/B test
• Create a segment
Create the segment
Assign the segment
• In your code before the fetch
– ABTester.addDimension(”age", myAge);
• Dimension will be remembered forever
• ABTester library will automatically add a
“percentile” dimension
– ABTester.addDimension(“percentile”,
new Random().nextInt(100));
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://unbounce.com/a-b-testing/shocking-results/
Multiple Experiments
Multiple Experiments
Multiple Experiments
• Serial tests
– Run the tests one after the other, without the
need to redistribute your app
– More accurate but takes more time
ABTester.preFetch(
new ABTest("Rate us test", false, "Rate button
color", "Rate actionbar icon")
);
Multiple Experiments
• Parallel tests
– Run the tests together, increasing the ‘noise’ for
dependent tests
– Faster
ABTester.preFetch(
new ABTest("Rate us button", false,
"Rate button color”)
new ABTest("Rate us actionbar", false,
"Rate button icon”)
);
Not a replacement for common sense
Thanks, any questions ?
Don’t overthink it, use Drippler’s A/B test library
https://github.com/Drippler/ABTester
Nir Hartmann, nhartmann@drippler.com
Drippler
Droidcon Tel-Aviv 2014

More Related Content

What's hot

Pairwise testing
Pairwise testingPairwise testing
Pairwise testingKanoah
 
Gap assessment Continuous Testing
Gap assessment   Continuous TestingGap assessment   Continuous Testing
Gap assessment Continuous TestingMarc Hornbeek
 
Agile Tester - Crash Slides
Agile Tester - Crash SlidesAgile Tester - Crash Slides
Agile Tester - Crash SlidesSamer Desouky
 
Comparison of automation and manual testing pixel values technolabs
Comparison of automation and manual testing pixel values technolabsComparison of automation and manual testing pixel values technolabs
Comparison of automation and manual testing pixel values technolabsPixel Values Technolabs
 
Machine learning in software testing
Machine learning in software testingMachine learning in software testing
Machine learning in software testingThoughtworks
 
[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams
[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams
[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile TeamsHo Chi Minh City Software Testing Club
 
Test Case Prioritization Techniques
Test Case Prioritization TechniquesTest Case Prioritization Techniques
Test Case Prioritization TechniquesKanoah
 
Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...
Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...
Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...Ho Chi Minh City Software Testing Club
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely
 
Artificial intelligence in qa
Artificial intelligence in qaArtificial intelligence in qa
Artificial intelligence in qaTaras Lytvyn
 
Continuous delivery test strategies
Continuous delivery test strategiesContinuous delivery test strategies
Continuous delivery test strategiesHylke Stapersma
 

What's hot (20)

Pairwise testing
Pairwise testingPairwise testing
Pairwise testing
 
Agile Testing - Not Just Tester’s Story _ Dang Thanh Long
Agile Testing - Not Just Tester’s Story _ Dang Thanh LongAgile Testing - Not Just Tester’s Story _ Dang Thanh Long
Agile Testing - Not Just Tester’s Story _ Dang Thanh Long
 
Gap assessment Continuous Testing
Gap assessment   Continuous TestingGap assessment   Continuous Testing
Gap assessment Continuous Testing
 
Agile Tester - Crash Slides
Agile Tester - Crash SlidesAgile Tester - Crash Slides
Agile Tester - Crash Slides
 
Comparison of automation and manual testing pixel values technolabs
Comparison of automation and manual testing pixel values technolabsComparison of automation and manual testing pixel values technolabs
Comparison of automation and manual testing pixel values technolabs
 
[HCMC STC Jan 2015] Practical Experiences In Test Automation
[HCMC STC Jan 2015] Practical Experiences In Test Automation[HCMC STC Jan 2015] Practical Experiences In Test Automation
[HCMC STC Jan 2015] Practical Experiences In Test Automation
 
Test Design with Action-based Testing Methodology - Ngo Hoang Minh
Test Design with Action-based Testing Methodology - Ngo Hoang MinhTest Design with Action-based Testing Methodology - Ngo Hoang Minh
Test Design with Action-based Testing Methodology - Ngo Hoang Minh
 
QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)
 
Machine learning in software testing
Machine learning in software testingMachine learning in software testing
Machine learning in software testing
 
[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams
[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams
[HCMC STC Jan 2015] How To Work Effectively As a Tester in Agile Teams
 
Test Case Prioritization Techniques
Test Case Prioritization TechniquesTest Case Prioritization Techniques
Test Case Prioritization Techniques
 
ISTQB Foundation Agile Tester 2014 Training, Agile SW Development
ISTQB Foundation Agile Tester 2014 Training, Agile SW DevelopmentISTQB Foundation Agile Tester 2014 Training, Agile SW Development
ISTQB Foundation Agile Tester 2014 Training, Agile SW Development
 
Methodology: IT test
Methodology: IT testMethodology: IT test
Methodology: IT test
 
Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...
Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...
Deliver Fast, Break Nothing Via Effective Building Developer and Tester Colla...
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 
Artificial intelligence in qa
Artificial intelligence in qaArtificial intelligence in qa
Artificial intelligence in qa
 
Continuous delivery test strategies
Continuous delivery test strategiesContinuous delivery test strategies
Continuous delivery test strategies
 
Mobile Video Games Testing Principles - Benjamin Poirrier
Mobile Video Games Testing Principles - Benjamin PoirrierMobile Video Games Testing Principles - Benjamin Poirrier
Mobile Video Games Testing Principles - Benjamin Poirrier
 
QTest
QTest QTest
QTest
 
Agile testing
Agile  testingAgile  testing
Agile testing
 

Similar to Drippler's A/B test library

Transforming One Small Step At A Time: Optimisation & Testing
Transforming One Small Step At A Time: Optimisation & TestingTransforming One Small Step At A Time: Optimisation & Testing
Transforming One Small Step At A Time: Optimisation & TestingBuilding Blocks
 
Data Insights Talk
Data Insights TalkData Insights Talk
Data Insights TalkMetail
 
Supercharge Your Testing Program
Supercharge Your Testing ProgramSupercharge Your Testing Program
Supercharge Your Testing ProgramOptimizely
 
Data-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PMData-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PMProduct School
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online productsAshish Dua
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101Ashish Dua
 
How Experimentation Plays a Role in PM by Expedia Sr. PM
How Experimentation Plays a Role in PM by Expedia Sr. PMHow Experimentation Plays a Role in PM by Expedia Sr. PM
How Experimentation Plays a Role in PM by Expedia Sr. PMProduct School
 
Data Insight Leaders Summit Barcelona 2017
Data Insight Leaders Summit Barcelona 2017Data Insight Leaders Summit Barcelona 2017
Data Insight Leaders Summit Barcelona 2017Harvinder Atwal
 
Data-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of ProductData-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of ProductProduct School
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsRamkumar Ravichandran
 
Webinar: Common Mistakes in A/B Testing
Webinar: Common Mistakes in A/B TestingWebinar: Common Mistakes in A/B Testing
Webinar: Common Mistakes in A/B TestingOptimizely
 
Taming The HiPPO
Taming The HiPPOTaming The HiPPO
Taming The HiPPOGoogle A/NZ
 
How to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B TestHow to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B TestKissmetrics on SlideShare
 
Anton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBAnton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBIevgenii Katsan
 
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Can I Test More Than One Variable at a  Time? Statisticians answer some of th...Can I Test More Than One Variable at a  Time? Statisticians answer some of th...
Can I Test More Than One Variable at a Time? Statisticians answer some of th...MarketingExperiments
 
Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...
Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...
Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...MarketingExperiments
 
Six sigma for beginner
Six sigma for beginnerSix sigma for beginner
Six sigma for beginnerYusar Cahyadi
 
Build a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization StrategyBuild a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization StrategySavage Marketing
 
The guide to A/B testing
The guide to A/B testingThe guide to A/B testing
The guide to A/B testingSarah Dentes
 

Similar to Drippler's A/B test library (20)

Transforming One Small Step At A Time: Optimisation & Testing
Transforming One Small Step At A Time: Optimisation & TestingTransforming One Small Step At A Time: Optimisation & Testing
Transforming One Small Step At A Time: Optimisation & Testing
 
Data Insights Talk
Data Insights TalkData Insights Talk
Data Insights Talk
 
Supercharge Your Testing Program
Supercharge Your Testing ProgramSupercharge Your Testing Program
Supercharge Your Testing Program
 
The Finishing Line
The Finishing LineThe Finishing Line
The Finishing Line
 
Data-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PMData-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PM
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online products
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
 
How Experimentation Plays a Role in PM by Expedia Sr. PM
How Experimentation Plays a Role in PM by Expedia Sr. PMHow Experimentation Plays a Role in PM by Expedia Sr. PM
How Experimentation Plays a Role in PM by Expedia Sr. PM
 
Data Insight Leaders Summit Barcelona 2017
Data Insight Leaders Summit Barcelona 2017Data Insight Leaders Summit Barcelona 2017
Data Insight Leaders Summit Barcelona 2017
 
Data-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of ProductData-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of Product
 
A/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'tsA/B Testing Best Practices - Do's and Don'ts
A/B Testing Best Practices - Do's and Don'ts
 
Webinar: Common Mistakes in A/B Testing
Webinar: Common Mistakes in A/B TestingWebinar: Common Mistakes in A/B Testing
Webinar: Common Mistakes in A/B Testing
 
Taming The HiPPO
Taming The HiPPOTaming The HiPPO
Taming The HiPPO
 
How to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B TestHow to Successfully Run Your First Website A/B Test
How to Successfully Run Your First Website A/B Test
 
Anton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQBAnton Muzhailo - Practical Test Process Improvement using ISTQB
Anton Muzhailo - Practical Test Process Improvement using ISTQB
 
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Can I Test More Than One Variable at a  Time? Statisticians answer some of th...Can I Test More Than One Variable at a  Time? Statisticians answer some of th...
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
 
Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...
Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...
Maximizing Subscription Revenue: How 3 businesses increased their subscriptio...
 
Six sigma for beginner
Six sigma for beginnerSix sigma for beginner
Six sigma for beginner
 
Build a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization StrategyBuild a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization Strategy
 
The guide to A/B testing
The guide to A/B testingThe guide to A/B testing
The guide to A/B testing
 

Recently uploaded

Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 

Recently uploaded (20)

Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 

Drippler's A/B test library

  • 1. A/B Testing in Android Nir Hartmann Drippler Droidcon Tel-Aviv 2014
  • 2. Why do we need A/B Testing? • Tests takes the guesswork out • Enables data-backed decisions • Enhances engagement and retention
  • 3. Road map • What is an A/B test ? • Segmentation • Multiple Experiments
  • 4. What is an A/B Test ? • Case study – onboarding screen
  • 5. Define the test • Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users. • Goal – A registered user (the user can skip registration). • View event - Login fragment onCreate(). - Login activity onCreate(). • Variables – Facebook button position (left or right). • Participants – New users.
  • 6. Amazon A/B Testing SDK • Very customizable, you can do just about anything as long as you know what it is you want to do • It’s free • Drippler created an open source library that simplify the process
  • 7. Setup the A/B test • Setup identifier – https://developer.amazon.com/al/index.html
  • 8. Setup the A/B test • Create a project
  • 9. Setup the A/B test • Create the test
  • 10. Dive into the code • https://github.com/Drippler/ABTester public class MyApplication extends Application { @Override public void onCreate() { super.onCreate(); ABTester.init(getApplicationContext(), "my_public_key", "my_private_key"); } }
  • 11. private void initLoginActivityTest() { try { ABTester.syncPreFetch( TimeUnit.SECONDS.toMillis(15), new ABTest("Login page test", false, "Facebook is first”) ); } catch (TimeoutException e) { // Couldn't reach amazon servers } } Fetch the test
  • 12. Login Fragment @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setRetainInstance(true); ABTester.recordEvent( "Login fragment shown", false); }
  • 13. Login Fragment @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) { boolean shouldShowFacebookFirst = ABTester.getBoolean("Login page test", "Facebook is first", false); if (shouldShowFacebookFirst) return inflater.inflate(R.layout.facebook_first, null); else return inflater.inflate(R.layout.google_first, null); }
  • 14. Report goal event public void onUserLoggedIn() { ABTester.recordEvent("Sign in", false); } @Override protected void onPause() { super.onPause(); /* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ ABTester.submitEvents(); }
  • 15. Analyze the results Variation Views Conversions Conversion rate Change Google+ first 1064 320 30.08% Facebook first 1043 250 23.97% -20.30%
  • 16. Dice experiment Goal: maximize the amount of 3’s we get in a 200 dice roll
  • 17. Dice experiment Hypothesis: wearing a hat will increase the chance to roll a 3
  • 18. Analyze the results Variation Views Conversions Conversion rate Change Hat off 200 31 15.50% Hat on 200 38 19.00% +22.58% Conversion is never a single number.
  • 19. Confidence level • Measure the reliability of an estimate – The confidence levels help us understand if the results are different merely by chance or by reason • 95% confidence level is considered good
  • 20. Analyze the results Variation Views Conversions Conversion rate Change Google First 1064 320 30.08% ± 2.32% Facebook First 1043 250 23.97% ± 2.18% -20.30% • Confidence level of 99%
  • 22. Choose the best variation • Launch – Choose the winning variation – Control the percentage of customers that receive a new feature
  • 23. Road map • What is an A/B test ? • Segmentation • Multiple Experiments http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png
  • 25. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button’s click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
  • 27. Rate us DialogFragment public class RateUsDialog extends DialogFragment { public static void show(FragmentManager fm, int color) { RateUsDialog rateUs = new RateUsDialog(); Bundle extras = new Bundle(); extras.putInt(“color”, color); rateUs.setArguments(extras); rateUs.show(fm, “my tag”); ABTester.recordEvent("Rate us dialog shown", false); }
  • 28. Rate us DialogFragment @Override public Dialog onCreateDialog(Bundle savedInstanceState) { int color = getArguments().getInt("color"); return createColoredDialog(color); }
  • 29. Rate us DialogFragment private Dialog createColoredDialog(int color) { ... .setPositiveButton("Rate", new OnClickListener() { @Override public void onClick(DialogInterface dialog, int which) { ABTester.recordEvent("Rate button click", false); } }); return myDialog;
  • 30. Rate us test 1) Asynchronously prefetching SplashActivity Default timeout is 60 seconds, and can be overriden by using preFetch(long timeout, ABTest... Test) ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color") );
  • 31. Rate us test 2) Show the dialog String fetchedColor = ABTester.getString( "Rate us test", "Rate button color", "#F5F5F5"); int color = Color.parseColor(fetchedColor); RateUsDialog.show(getFragmentManager(), color); 3) Submitting the results onPause() ABTester.submitEvents();
  • 32. Analyze the results Variation Views Conversions Conversion rate Change Confidence Control (white) 865 234 27.05% ± 1.51% Variation A (green) 904 250 27.65% ± 1.49% -0.2% Variation B (red) 830 230 27.71% ± 1.55% +2.4% 51% What can I do with these results?
  • 33. Segmentation Variation Views Conversions Conversion rate Change Confidence Control (white) 432 92 21.30% ± 1.97% Variation A (green) 464 165 35.56% ± 2.22% +66.9% 98.7% Variation B (red) 420 120 28.57% ± 2.20% +34.1% Variation Views Conversions Conversion rate Change Confidence Control (white) 433 142 32.79% ± 2.26% +22.2% 97.1% Variation A (green) 440 85 19.32% ± 1.88% -27.9% Variation B (red) 410 110 26.83% ± 2.19% -18.8% Under 40 Over 40
  • 34. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
  • 35. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – Age specific tests
  • 36. Setup the A/B test • Create a segment
  • 38. Assign the segment • In your code before the fetch – ABTester.addDimension(”age", myAge); • Dimension will be remembered forever • ABTester library will automatically add a “percentile” dimension – ABTester.addDimension(“percentile”, new Random().nextInt(100));
  • 39. Road map • What is an A/B test ? • Segmentation • Multiple Experiments http://unbounce.com/a-b-testing/shocking-results/
  • 42. Multiple Experiments • Serial tests – Run the tests one after the other, without the need to redistribute your app – More accurate but takes more time ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color", "Rate actionbar icon") );
  • 43. Multiple Experiments • Parallel tests – Run the tests together, increasing the ‘noise’ for dependent tests – Faster ABTester.preFetch( new ABTest("Rate us button", false, "Rate button color”) new ABTest("Rate us actionbar", false, "Rate button icon”) );
  • 44. Not a replacement for common sense
  • 45. Thanks, any questions ? Don’t overthink it, use Drippler’s A/B test library https://github.com/Drippler/ABTester Nir Hartmann, nhartmann@drippler.com Drippler Droidcon Tel-Aviv 2014