This presentation is from Affiliate Summit East 2015 (August 2-4, 2015 in New York, NY). Session description: Testing has hit the mainstream, but when something is popular - bad habits form. In this session you’ll learn when you should test, what to test, and how to construct a winning optimization campaign.

1. TESTING IS NOT CRO
BUT IT’S STILL DARN IMPORTANT

2. FIRST LET’S COVER
SOME
A/B TESTING BASICS

3. Types Of Tests
Synchronous Asynchronous
Sequential Testing A/B/…/N Testing Multivariate Testing
Test Types

4. Synchronous Testing
July 1st - July 15th July 16th - July 30th

5. A/b/…/n Split Testing
$$$
$$$$$$
$$$
$$$

6. A/b/…/n Testing
You’ll Get Page
Conversion
Data (Not Element By
Element)
Better For Low Traffic
Sites

7. A/b/…/n Testing
Some Rules:
Isolate Individual
Elements
If You Don’t Isolate
Elements, DO NOT
attribute the lift or loss to
any single element

8. Multivariate Testing
You’ll Get Element By
Element Data
Don’t Run If Less Than
100k Pageviews Per
Month

9. NOW LET’S GIVE
A/B TESTING SOME LOVE

10. I BUILT MY CAREER
AROUND SPLIT TESTING

12. IT’S THE ONLY WAY TO GET
DEFINITIVE ANSWERS IN A
CLEAN ENVIRONMENT

13. TESTING HELPS GUIDE
DECISIONS AND IMPROVE THE
USER EXPERIENCE WHILE
BOOSTING PROFITS

14. TESTING GOT THE
CONVERSATION
STARTED!

17. A/B TESTS ARE
SEXY

18. 89.97%
LIFT
IN
SALESSource: vwo.com

20. 12.6% LIFT
IN
SUBMISSION
S
Source: Unbounce.com

22. 18.70%
DROP
IN
SUBMISSION
SSource: ContentVerve.com

23. 19.47% LIFT
IN
SUBMISSION
S
Source: ContentVerve.com

24. AWESOME
STUFF RIGHT?

25. THERE’S ONE
BIG PROBLEM

26. EVERYONE WANTS TO TEST
EVEN WHEN THEY CAN’T

27. SIMPLY PUT:
NOT EVERYONE CAN OR
SHOULD BE SPLIT TESTING

29. NOT BEING ABLE TO
TEST DOESN’T MEAN…

30. …NOT BEING ABLE TO
OPTIMIZE

31. THERE ARE PLENTY OF WAYS
TO BOOST CONVERSIONS
OUTSIDE OF SPLIT TESTING

32. TODAY IN A NUTSHELL
1. When You Can Test
2. Should You Actually Test
3. What To Do When You
Can’t/Shouldn’t Test

33. Part
Can I Actually Run A Test

34. Testing Eligibility
• ~100 Conversion Per
Variation
• Run for at least a full week
• Not run longer than 8 weeks
• Test can reach 95%
confidence in this time frame

35. Testing Eligibility
• ~100 Conversion Per
Variation
• Run for at least a full week
• Not run longer than 8 weeks
• Test can reach 95%
confidence in this time frame

36. Conversion Per Day Chart
Minimum Number Of Conversions Per Day
Test Period
(Days)
2 Variants 3 Variants 4 Variants 5 Variants
7 29 43 58 72
14 15 22 29 36
21 10 15 20 24
28 8 11 15 18
35 6 9 12 15
42 5 8 10 12
49 5 7 9 11
56 4 6 8 9

38. Problem:
Not Enough Conversions
Solution:
Run Longer
Secondarily:
If Outside Of 8 Week Testing
Window - Don’t Test It

39. VARIABLE NUMBER ISN’T THE
ONLY THING THAT DICTATES
TIME!

40. THE PERCENT LIFT IS
A HUGE TIMING
FACTOR

43. In order for this test
to get statistical
results, it would
require a 10% lift

44. AND SO IS THE
CONFIDENCE RATEBut we’ll talk more about that
later

45. Does Your Page Get Enough Daily
Conversions?
YES No
Can’t Test - Sorry!
Is There A Big
Enough Lift
Potential?*
YES No
This Is A Testable
Page
Can’t Test - Sorry!
*With An Appropriate Confidence Rate/Index

46. Part
Should I Actually Run That
Test

48. THE CONVERSION
HIERARCHY

50. Functional
Does the page work and do
what it’s supposed to?

53. Accessible
Can people of all skill levels on
any device use the site?
What are the barriers on the
site?

56. Usable
Can people understand your
site, i.e., is it user friendly?

57. Intuitive
Does the site have a logical
flow?
Does the offer match the
source?

58. Persuasive
Is your site convincing people
to buy or take an action?

62. “For every test you run, there are an
infinite number of other tests that
you could have run” - Justindamus
120 AD

64. Part
How To Optimize If I
Can’t/Shouldn’t Test

65. You Have A Few Options
• Evaluate Your Offer
• On-Site Retargeting
• Changes Based On Qualitative
Research

66. EVALUATE
YOUR OFFER

67. It’s not a traffic problem, it’s an offer
problem

68. It’s not a testing problem, it’s an
offer problem

69. You might have to
head back to the
drawing board
completely.
Does your
product/service
provide real value?

70. You also might be
going after the
wrong traffic
source.
Identify if your ideal
customer is
‘hanging out’ on the
traffic medium

71. TRY ON SITE
RETARGETIN
G

72. What is On Site Retargeting?
Evaluating current interactions and
providing a relevant offer based on
these interactions.

73. Time On Site
Number Of Pageviews
Type Of Content Consumed
Exit Intent
Referring Source

75. Just Under Half Of
All Offer Pageviews
Sent By Optimonk

76. Typical Campaign
Conversion:
69%
With Exit pop:
80.7%

77. Using this method we got 2,689
more leads in just 14 days!

78. “Respondents targeting more than 20% of their
visitors with personalized content achieved a 5%
conversion rate, which was 2x the average
conversion rate of 2.5% in their study” [Adobe
2014 Survey]

79. PAGE CHANGES
BASED ON
OPTIMIZATION
RESEARCH

82. Here are a few things to try
• Click/Scroll Tracking
• Usability Testing
• User Surveys
• Session Recordings

83. CLICK/SCROLL
TRACKING

86. People who actually scroll to the bottom click
Not enough people are scrolling that far down
~15% of all clicks on this page are on the nav bar

87. Try a shorter variation to get people to scroll
to the CTA.
Add multiple CTA buttons at areas
where scrolling is highest.
Hide the nav bar on this page

88. USABILITY
TESTING

90. USER
SURVEYS

91. • Was it was easy to find what you were looking for?
• Were the words/vocab used to define categories/sub-categories clear to you?
• Do you have any suggestions to improve our website navigation?
• Does our website look credible to you?
• Is our pricing clear?
• Is there anything else you’d like to know before signing up with us?
• Will you shop with us again? Why/why not?
• Do you think the form has any confusing/unnecessary input fields?
QUESTION IDEAS

93. SESSION
RECORDINGS

94. Shows individual recordings of a user interacting
with your site.
Unlike heatmaps, this isn’t aggregated data

95. Really blow their
mind with a real life
user recording
with audio

96. TRICKS TO TEST
WHEN YOU DON’T
HAVE THE TRAFFIC

97. USE MICRO
CONVERSIONS AS
INDICATORS

98. Test Legnth Minimum Conversions Per Variation Per Day
7 Days 29
14 Days 15
21 Days 10
28 Days 8
35 Days 6
42 Days 5
49 Days 5
56 Days 4

102. LOWER THE
CONFIDENCE
THRESHOLD

103. Lift: 20%
Confidence Rate:
95%
_________________
Required Sample
Size:
7,682

104. Lift: 20%
Confidence Rate:
90%
_________________
Required Sample
Size:
6,052
21% Decrease

105. REMEMBER - SPLIT
TESTING IS NOT
OPTIMIZATION…

106. IT’S AN OPTIMIZATION
TOOL

107. Split Testing
Personalization
User Surveys
Funnel Analysis
Web Analytics
Personas
Neurotesting Advanced Segments
Eye Tracking
Scroll Tracking
CRO

108. SPLIT TESTING CAN
ACTUALLY HURT A
COMPANY

109. SO PRACTICE SAFE TESTING
WITH A PAGE THAT
MATTERS

110. • Email: justin.rondeau@digitalmarketer.com
• Twitter: @jtrondeau
• Website: digitalmarketer.com
Justin Rondeau
Optimization Manager,
Digital Marketer
Let’s Talk More!