6. Test Scenario 1
Experiment ID: (protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP2092
Background: Large financial institution offering mortgages to consumers.
Goal: To determine which claim of value will generate the most response.
Primary research question: Which subject line will produce the greatest
open rate and click rate?
Approach: A/B/C single-factorial split
7. Test Scenario 1
Test Protocol ID: TP2092
Version A
Looking for a lower mortgage rate?
You’re not alone.
SL Rating
63
Version B
Discover the advantages of [Brand
Name] mortgages
78
Version C
Own your home sooner with free biweekly payments
81
8. Accurate Prediction
Version B & C generated a statistically significant improvement over A
Subject Line
Tool
Rating
Open
Rate
Rel.
Diff.
A
Looking for a lower mortgage rate? You’re not alone.
63/100
18.08%
-
B
Discover the advantages of [Brand Name] mortgages
78/100
19.80%
9.5%
C
Own your home sooner with free bi-weekly payments
81/100
20.99%
16.0%
9.
Accurate Prediction
Mostly Accurate Prediction
Version C still generated a statistically significant improvement over A
B&C
statistically significant improvement over A
Subject Line
Tool
Rating
Open
Rate
Click
Rel.
Rate
Diff.
Rel.
Diff.
A
Looking for a lower mortgage rate? You’re not alone.
63/100
18.08%
0.61%
-
-
B
Discover the advantages of [Brand Name] mortgages
78/100
19.80%
0.57%
9.5%
-7.4%
C
Own your home sooner with free bi-weekly payments
81/100
20.99%
16.0%
0.68%
11.3%
10. Test Scenario 2
Experiment ID: (protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP2078
Background: A well-known news publication
Goal: To increase subscriptions from former subscribers who have been
absent between one and two years
Primary research question: Which subject line will produce the greatest
open rate?
Approach: A/B/C single-factorial split
11. Test Scenario 2
Test Protocol ID: TP2078
Version A
Choose from 2 ways to save big on
the NEW [PublicationName].com
SL Rating
56
Version B
Save 50% on your choice of two new
subscription options
59
Version C
Home Delivery comes with FREE
access to [PublicationName].com
50
12. Inaccurate Prediction
While Version A generated a statistically significant response, Version B did not
Subject Line
Tool
Rating
Open
Rate
Rel. Diff.
A
Choose from 2 ways to save big on the
NEW [PublicationName].com
56/100
9.43%
50.2%
B
Save 50% on your choice of two new
subscription options
59/100
7.01%
11.7%
C
Home Delivery comes with FREE access
to [PublicationName].com
50/100
6.28%
--
13. Inaccurate Prediction
While Version A generated a statistically significant response, Version B did not
Subject Line
Tool
Rating
Open
Rate
Click
Rel. Diff.
Rate
Rel Diff.
A
Choose from 2 ways to save big on the
NEW [PublicationName].com
56/100
9.43%
1.48%
50.2%
86%
B
Save 50% on your choice of two new
subscription options
59/100
7.01%
0.76%
11.7%
-5.3%
C
Home Delivery comes with FREE access
to [PublicationName].com
50/100
6.28%
-0.80%
--
14. Test Scenario 3
Test Protocol ID: TP2076
Version A
Target MDs with Surveys and Panels
SL Rating
94
Version B
Reach 120,000 Physicians through
Social Media
76
15. Inaccurate Prediction
Version B generated a statistically significant higher response than A
Subject Line
A Target MDs with Surveys and Panels
B
Reach 120,000 Physicians
through Social Media
Tool
Rating
Open
Rate
Rel Diff.
94/100
21.30%
--
76/100 22.92%
7.6%
16. Inaccurate Prediction
Version B generated a statistically significant higher response than A
Subject Line
A Target MDs with Surveys and Panels
B
Reach 120,000 Physicians
through Social Media
Tool
Rating
Open
Rate
Rel Diff.
94/100
21.30%
--
76/100 22.92%
7.6%
17. Experiment: Results
After rating hundreds of subject lines from our library of statistically
significant experiments…
The tool predicts a winner:
31.57%
of the time
18. The tool predicts a winner:
A coin predicts a winner:
31.57%
50%
of the time
of the time
25¢ Subject Line Rating Tool
Check Yours Now!
19. So what went wrong with the tool?
Decisions based on violations:
•
•
•
•
•
•
•
•
•
•
•
Does not contain a sense of urgency
Mobile subject line is over 35 characters
Length is over 50 characters
Contains any word that has more than 8
characters
Includes one exclamation mark
Use of a word in all caps
Includes the word “You” or “Your”
Contains comma
Starts with Save
Use of $
Contains % off
20. So what went wrong with the tool?
Decisions based on violations:
•
•
•
•
Does not contain a sense of urgency
Mobile subject line is over 35 characters
Length is over 50 characters
Contains any word that has more than 8
characters
Includes one exclamation mark
Use of a word in all caps
Includes the word “You” or “Your”
Contains comma
Starts with Save
Use of $
Contains % off
Computers are great with
•
computer logic;
•
•
But not •so great with
•
•
human logic
•
21. The computer approach
• Computer logic draws
conclusions from
algorithms, rules,
switches and settings
• Human logic draws
conclusions from
context, emotions,
desires and
belonging.
22. The computer approach
In marketing…
We tend to rely on rule sets
•
•
•
•
•
•
•
Emphasize the benefits
Avoid long copy, use bullets
Always include a call to action
Keep it above the fold
Emphasize hero shots
Minimize compliance text
Use tabs for additional info
24. The Research
MarketingExperiments was the first Internet-based
research lab to conduct experiments in optimizing
the conversion rate of sales and marketing
processes.
1992
Preliminary Research Begins
1997
Research Program Established
2001
First Research Report Published
2002
Testing of Research Partnership Model Begins
2003
Offer Response Optimization Theory Validated
2006
Patent Filings for Research Findings (10 Heuristics)
25. The Research
MarketingExperiments was the first Internet-based
research lab to conduct experiments in optimizing
the conversion rate of sales and marketing
processes.
People judge emails, websites in
the same way they judge people
1992
Preliminary Research Begins
1997
Research Program Established
2003
Offer Response Optimization Theory Validated
2006
Patent Filings for Research Findings (10 Heuristics)
It’s not the change on the page;
2001 First Research Report Published
2002 Testing of Research Partnership Model Begins
It’s the change in the mind
33. Experiment
eme = rv(of+i) – (f+a)
Experiment ID: Pier 1 Imports
Location: MarketingSherpa Research Library
Test Protocol Number: A-TP1002
Background: B2C home products company with a significant online and
retail presence
Goal: To increase the click-through rate from the email to the landing
page
Primary research question: Which email design will generate the most
clicks?
Approach: A/B/C/D split test (variable cluster)
39. 18-52% Decrease in Click-through
Each design achieved a statistically significant click-rate decrease
Email Designs
Click Rate
Rel. Diff.
Sample (client)
36.70%
-
Team A Design
17.68%
-51.83%
Team B Design
29.91%
-18.50%
Team C Design
24.07%
-34.41%
40. eme = rv(of+i) – (f+a)
Experiment
Click-through rate comparison
Team A
Team C
Team B
Client
17.68%
24.07%
29.91%
36.70%
41. eme = rv(of+i) – (f+a)
Experiment
See a progression?
Team A
Team C
Team B
Client
17.68%
24.07%
29.91%
36.70%
44. Offer
eme = rv(of+i) – (f+a)
What really is the offer in email?
• You may think that the offer is the product that you want
to sell.
• In reality it is the additional information they receive in
exchange for a click.
• What is the most effective way to get a click? Is it to give
them an entire product pitch? Or just a part?
46. Experiment
eme = rv(of+i) – (f+a)
Experiment ID: (Protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP2801
Background: A physician-only social network that allows medical product
companies to conduct first-hand research on potential purchasers.
Goal: To increase the number of leads from a rented trade publication list.
Primary research question: Which email design will generate the most
opens, clicks?
Approach: Multi-factorial sequential test on a trade publication list
47. Experiment
eme = rv(of+i) – (f+a)
Control
PROTECTED
Get started with a free
30-minute demo
Get started with a free 30-minute demo
This email assumes that
readers understand the
platform’s audience and
thus focuses on platform
uses
It also assumes intent to
purchase with multiple
“Get Started” calls to
action, one as early as
the second sentence
48. Treatment
This version
focuses less on
platform and
more on its users
The primary callto-action was
written to
synchronize with
their state of
interest
PROTECTED
50. 104% Increase in clicks
Version B achieved a statistically significant increase in click-rate
Email
Click Rate
Rel. Diff.
Control
1.55%
--
Treatment
3.16%
104%
51. Offer
F
eme = rv(of+i) – (f+a)
Key Principles
• In the body of the email, it is important to distinguish
the difference between the product offer and the clickthrough offer.
52. Offer
F
eme = rv(of+i) – (f+a)
Key Principles
• In the body of the email, it is important to distinguish
the difference between the product offer and the clickthrough offer.
• Your communication of value starts with your subject
line and should progress, like a conversation,
throughout the entire conversion process
53. Offer
eme = rv(of+i) – (f+a)
Test Protocol ID: TP2076
Not this…
Target MDs with Surveys
and Panels
But this…
Reach 120,000 Physicians
through Social Media
27.6%
Increase in clicks
57. Incentive
F
eme = rv(of+i) – (f+a)
Key Principles
• The purpose of an incentive is to increase the probability
of immediate action by tipping the balance of emotional
conflict inside the person
59. Incentive
eme = rv(of+i) – (f+a)
Also consider including…
• Instilled Deadlines
• Standard promotions
24 hours only: Free shipping on…
• Implied exclusivity
48-hour member exclusive: new fall styles…
• Contests
This is the last day to enter for your chance to…
• Limited Availability
Only 25 seats left to see register for...
60. Incentive
F
eme = rv(of+i) – (f+a)
Key Principles
• The purpose of an incentive is to increase the probability
of immediate action by tipping the balance of emotional
conflict inside the person
• In email messaging, it can also include the use of artificial
urgency, in which the immediacy associated with the
offer is controlled by the marketer.
63. Friction
F
eme = rv(of+i) – (f+a)
Key Principles
• Two common elements that create friction in the
process are:
• Length
• Difficulty
• The objective is to minimize, not eliminate friction.
If you eliminate all friction, you eliminate “the sale.”
64. Friction
eme = rv(of+i) – (f+a)
Test Protocol ID: TP2089
1
Length
But This
Not this
21.2%
Increase in clicks
65. Friction
eme = rv(of+i) – (f+a)
Test Protocol ID: TP2081
1
Length
But This
Not this
41.4%
Increase in clicks
52.5%
Increase in app starts
66. Friction
eme = rv(of+i) – (f+a)
Test Protocol ID: TP2026
2
Difficulty
But this (1 Option)
Not this (6 Options)
20%
Increase in clicks
46%
Increase in purchases
67. Friction
eme = rv(of+i) – (f+a)
Test Protocol ID: TP2026
2
Difficulty
Not this
But This
10%
Increase in clicks
68. Friction
eme = rv(of+i) – (f+a)
Test Protocol ID: A-TP1002
What else contributed to differences in performance?
Not This
But This
69. Friction
eme = rv(of+i) – (f+a)
Test Protocol ID: A-TP1002
What else contributed to differences in performance?
Not This
But This
73. Anxiety
F
eme = rv(of+i) – (f+a)
Key Principles
• Anxiety is just as lethal to conversion as friction.
• Try and mitigate anxiety rationally and you are likely to
fail…
• Though created from legitimate concern, its impact in the
mind is often disproportionate to the reality of risk.
• While website visitors are skeptical and cautious, the
natural level of concern is even greater for email offer
recipients.
74. Experiment
eme = rv(of+i) – (f+a)
Experiment ID: (Protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP2083
Background: A large event management online software provider.
Goal: To increase total lead inquiries (phone calls + form sign-ups) from
visitors who abandoned the free trial sign-up process.
Primary research question: Which email tone will result in a higher rate
of lead inquiries?
Approach: A/B single factorial split
75. Experiment
Control
Subject Line: Your Free RegOnline Access
Dear [Name],
You’re just one step away from getting FREE access to RegOnline, our award winning Event
Registration and Management Software. Quickly make an event website, try our event
marketing tools, build a registration form template or even generate custom name badges.
Click here to finish your profile and get started. Your personal profile is kept secure and we
promise NEVER to sell or misuse your information.
Have questions? Call us direct at 1-800-XXX-XXXX -- we can even set up your free access over
the phone.
Sincerely,
Jon Powell
Customer Service Representative
RegOnline
Direct: 1-800-XXX-XXXX ext. XXXX
jon.powell@RegOnline.com
• The control email uses a
typical sales tone of a
direct-response email.
76. Treatment
Subject Line: Your Free RegOnline Access
Hi [First],
I noticed that you started the process of getting free access to RegOnline but weren’t able to
finish. Are you concerned about giving out your phone number? Are you worried about high
pressure sales tactics or mandatory contracts?
We believe our product sells itself, so we’re just here to provide you with whatever
assistance you need in getting your event up and running - in whatever way works best for
you. We promise NEVER to sell or misuse your information.
Call me direct at 1-800-XXX-XXXX and I can help get you rolling. If you’d rather just try again
online, use this link instead.
Thank you in advance for your trust!
Sincerely,
[Name of Representative]
Customer Service Representative
RegOnline
Direct: 1-800-XXX-XXXX ext. XXXX
[name]@RegOnline.com
• The treatment directly addresses
anxiety, taking on a categorical
shift in tone – from a directresponse tone to a customer
service tone.
78. 349% Increase in lead inquiry rate
The service tone achieved a statistically significant increase in inquiries
Email
Inquiry Rate
Rel. Diff.
Control
1.58%
--
Treatment
7.08%
349.41%
79. Experiment
eme = rv(of+i) – (f+a)
Experiment ID: (Protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP2115
Background: National B2C, B2B financial company with an online and
retail presence
Goal: To increase account activity, engagement
Primary research question: Which subject line will generate the most
opens and clicks?
Approach: A/B/C/D split test (variable cluster)
80. Experiment
#
ID1:
ID2:
ID3:
ID4:
ID5:
ID6:
ID7:
ID8:
ID9:
ID10:
ID11:
ID12:
ID13:
eme = rv(of+i) – (f+a)
Subject Line Tested
[Name], Your Account Information is Ready To View
Save Time by Paying Your Bills Online
Did you forget your [Bank Name] Online username or password?
Manage your money more effectively with [Bank Name] Online
Your Account Information is Ready to View
Check your account balance online, anytime. Sign on now.
Sign on to your [Bank Name] checking account
It's easy to log-in to your [Bank Name] account online
Sign on to your checking account
Check your account balance online, anytime
Easily manage your accounts in one place with [Bank Name] Online
Don't forget to access your [Bank Name] account online
Save time managing your money with [Bank Name] Online
81. Experiment
#
ID1:
ID2:
ID3:
ID4:
ID5:
ID6:
ID7:
ID8:
ID9:
ID10:
ID11:
ID12:
ID13:
eme = rv(of+i) – (f+a)
Subject Line Tested
Open
[Name], Your Account Information is Ready To View
30.77%
Save Time by Paying Your Bills Online
30.70%
Did you forget your [Bank Name] Online username or password? 28.49%
Manage your money more effectively with [Bank Name] Online 28.21%
Your Account Information is Ready to View
26.97%
Check your account balance online, anytime. Sign on now.
25.23%
Sign on to your [Bank Name] checking account
25.21%
It's easy to log-in to your [Bank Name] account online
24.43%
Sign on to your checking account
21.26%
Check your account balance online, anytime
20.44%
Easily manage your accounts in one place with [Bank Name] Online19.32%
Don't forget to access your [Bank Name] account online
19.27%
Save time managing your money with [Bank Name] Online
16.53%
82. Experiment
#
ID1:
ID2:
ID3:
ID4:
ID5:
ID6:
ID7:
ID8:
ID9:
ID10:
ID11:
ID12:
ID13:
eme = rv(of+i) – (f+a)
Subject Line Tested
Open
[Name], Your Account Information is Ready To View
30.77%
Save Time by Paying Your Bills Online
30.70%
Did you forget your [Bank Name] Online username or password? 28.49%
Manage your money more effectively with [Bank Name] Online 28.21%
Your Account Information is Ready to View
26.97%
Check your account balance online, anytime. Sign on now.
25.23%
Sign on to your [Bank Name] checking account
25.21%
It's easy to log-in to your [Bank Name] account online
24.43%
Sign on to your checking account
21.26%
Check your account balance online, anytime
20.44%
Easily manage your accounts in one place with [Bank Name] Online19.32%
Don't forget to access your [Bank Name] account online
19.27%
Save time managing your money with [Bank Name] Online
16.53%
83. Experiment
#
ID3:
ID1:
ID9:
ID5:
ID7:
ID12:
ID10:
ID6:
ID8:
ID11:
ID4:
ID13:
ID2:
eme = rv(of+i) – (f+a)
Subject Line Tested
Click
Open
Did you forget your [Bank Name] Online username or password? 30.77%
4.59%
[Name], Your Account Information is Ready To View
4.44%
30.70%
Sign on to your checking account
4.40%
28.49%
Your Account Information is Ready to View
4.34%
28.21%
Sign on to your [Bank Name] checking account
3.84%
26.97%
Don't forget to access your [Bank Name] account online
3.64%
25.23%
Check your account balance online, anytime
3.59%
25.21%
Check your account balance online, anytime. Sign on now.
3.47%
24.43%
2.90%
It's easy to log-in to your [Bank Name] account online
21.26%
Easily manage your accounts in one place with [Bank Name] Online20.44%
1.94%
Manage your money more effectively with [Bank Name] Online 19.32%
1.70%
Save time managing your money with [Bank Name] Online
1.54%
19.27%
1.05%
Save Time by Paying Your Bills Online
16.53%
85. Live Optimization
eme = rv(of+i) – (f+a)
Partner ID: The New York Times
Location: MarketingExperiments Research Library
Test Protocol Number: TP2802
Background: Internationally recognized news service known for their
journalism.
Our Goal: To significantly increase subscriptions.
Primary research question: Which email will generate the greatest
subscription rate?
Approach: A/B split test (variable cluster)
86. Live Optimization
eme = rv(of+i) – (f+a)
The Subscription Product Offering
A subscription to the NY Times
• Home delivery of the Print edition
• Unlimited, unrestricted access to articles
online, in their smartphone and tablet apps
• Free digital access for one family member or
friend
• Subscriber only special offers, discounts,
invitations
Incentive
• 50% off for the first 12 weeks
107. Review
F
eme = rv(of+i) – (f+a)
Key Principles
•
There’s a story behind everything that’s bought. This occurs on two
levels: internal and external
•
It is important to distinguish the difference between the product offer
and the click-through offer.
•
Consider the use of artificial urgency as an incentive to take action
•
Aggravation in email messaging can occur on two levels: length and
difficulty
•
The natural level of concern in email is strong; do not try to mitigate it
rationally.
108. February 17-20
Aria Las Vegas
FLINT MCGLAUGHLIN
DAN ARIELY
NOAH GOLDSTEIN
The longest running
research-based
email marketing
event
Plus, brand side marketers and tactical training sessions
MECLABS.com/Email