The presentation discusses the significance of testing and how to execute a successful testing program.

1. >
Tes&ng
for
Success
<
Elements
of
a
Successful
Tes0ng
Program

2. >
Agenda
§ Why
Test?
§ Problem
Diagnosis
§ Deciding
what
to
Test
§ Test
Execu0on
and
Measurement
§ Test
Repor0ng
December
2011
©
Datalicious
Pty
Ltd
2

3. 101011010010010010101111010010010101010100001011111001010101
010100101011001100010100101001101101001101001010100111001010
010010101001001010010100100101001111101010100101001001001010
>
Why
Test?
December
2011
©
Datalicious
Pty
Ltd
3

4. 1. Why
does
your
EVERYONE’S
business/organisa0on
exist?
GOT
AN
2. How
can
your
business/
OPINION
organisa0on
improve?
December
2011
©
Datalicious
Pty
Ltd
4

5. >
Why
Test?
1. Systema0c
Innova0on
2. Avoid
costly
mistakes
3. Know
why
things
go
right,
know
why
things
go
wrong
4. BeRer
employee
engagement
§ Requires
planning
and
governance!
December
2011
©
Datalicious
Pty
Ltd
5

6. 101011010010010010101111010010010101010100001011111001010101
010100101011001100010100101001101101001101001010100111001010
010010101001001010010100100101001111101010100101001001001010
>
Problem
Diagnosis
December
2011
©
Datalicious
Pty
Ltd
6

7. >
What
is
the
business
problem?
Acquisi0on
Up-­‐Sell
Reten0on
Advocacy
Analy&cs
and
metrics
frameworks
December
2011
©
Datalicious
Pty
Ltd
7

8. >
Case
Study
December
2011
©
Datalicious
Pty
Ltd
8

9. >
Further
Diagnosis
PROBLEM:
Sales
through
online
Not
enough
site
traffic
High
home
page
bounce
rate
Low
conversion
on
product
page
Checkout
fallout
December
2011
©
Datalicious
Pty
Ltd
9

10. >
Further
Diagnosis
II
Source:
www.feng-­‐gui.com
December
2011
©
Datalicious
Pty
Ltd
10

11. >
Some&mes
the
small
things
count
December
2011
©
Datalicious
Pty
Ltd
11

12. >
Further
diagnosis
III
Wrong
message?
Wrong
channel?
Wrong
person?
Wrong
0me?
December
2011
©
Datalicious
Pty
Ltd
12

13. >
Tes&ng
as
risk
mi&ga&on
Roll-­‐out
Channel
Press
TV
Radio
Outdoor
Offer,
Crea&ve,
Call-­‐to-­‐ Offer,
Call-­‐ Offer,
Call-­‐
eDM/DM
Call-­‐to-­‐ Ac&on
to-­‐Ac&on
to-­‐Ac&on
Ac&on
Test
Paid
Channel
Search
Offer
Offer
Offer
Offer
Crea&ve,
Display
Offer,
Call-­‐
Offer,
Call-­‐
-­‐
Crea&ve
Media
to
Ac&on
to
Ac&on
December
2011
©
Datalicious
Pty
Ltd
13

14. >
Tes&ng
as
standard
prac&ce
Test
Market
Control
Market
(no
ATL)
%
Uplib
in
Sales
Time
December
2011
©
Datalicious
Pty
Ltd
14

15. 101011010010010010101111010010010101010100001011111001010101
010100101011001100010100101001101101001101001010100111001010
010010101001001010010100100101001111101010100101001001001010
>
Deciding
what
to
Test
December
2011
©
Datalicious
Pty
Ltd
15

16. Don’t
reinvent
the
wheel
December
2011
©
Datalicious
Pty
Ltd
16

17. >
What
are
the
solu&on(s)?
December
2011
©
Datalicious
Pty
Ltd
17

18. >
Consumer
Empathy
What
are
your
visitors
trying
to
achieve
by
visi2ng
your
site?
December
2011
©
Datalicious
Pty
Ltd
18

19. >
Consumer
Empathy
1. Make
it
visible
– People
can’t
convert
if
they
can’t
find
your
‘Buy
Now’
buRon
2. Make
it
relevant
– Need
to
resolve
consumer
reserva0ons/
ques0ons
3. Make
it
easy
– Easy
naviga0on,
easy
form
comple0on,
easy
to
read,
quick
page
load
December
2011
©
Datalicious
Pty
Ltd
19

20. >
Start
with
the
basics…
1.
The
headline
– Have
a
headline!
– Headline
should
be
concrete
– Headline
should
be
first
thing
visitors
look
at
2.
Call
to
ac&on
– Don’t
have
too
many
calls
to
ac0on
– Have
an
ac0onable
call
to
ac0on
– Have
a
big,
prominent,
visible
call
to
ac0on
3.
Social
proof
– Logos,
number
of
users,
tes0monials,
case
studies,
media
coverage,
etc
December
2011
©
Datalicious
Pty
Ltd
20

21. >
Start
with
the
basics…
December
2011
©
Datalicious
Pty
Ltd
21

22. >
Case
Study
December
2011
©
Datalicious
Pty
Ltd
22

23. >
Further
Examples
TEST
A
EXISTING
December
2011
©
Datalicious
Pty
Ltd
23

24. >
Further
Examples
EXISTING
TEST
December
2011
©
Datalicious
Pty
Ltd
24

25. >
Deciding
What
to
Test
Test
Selec0on
Checklist
§ Is
the
measurement
infrastructure
in
place
already?
[
✔
]
§ Can
I
readily
execute
the
solu0on?
[
✔
]
§ Do
I
have
enough
sample
to
draw
valid
conclusions?
[
✔
]
§ Will
this
prove
the
value
of
tes0ng
in
the
business?
[
✔
]
December
2011
©
Datalicious
Pty
Ltd
25

26. >
Do
you
have
the
repor&ng?
For
each
of
Segment
X,
Y
and
Z...
Test
Channel
ATL
DM
eDM
Online
Online
✔
✔
Mailroom
✔
Response
Call
Centre
Channel
Bricks
&
Mortar
Channels
in
✔
Aggregate
December
2011
©
Datalicious
Pty
Ltd
26

27. >
Offline
conversions
from
online
Tying
offline
conversions
back
to
online
campaign
and
research
behavior
using
standard
cookie
technology
by
triggering
virtual
online
order
confirma0on
pages
for
offline
sales
using
email
receipts.
Website.com
Phone
Virtual
Order
Research
Orders
@
Confirma&on
Online
Ad
Website.com
Retail
Virtual
Order
Campaign
Research
Orders
@
Confirma&on
Website.com
Online
Online
Order
Virtual
Order
Research
Orders
Confirma&on
@
Confirma&on
Cookie
Cookie
Cookie
December
2011
©
Datalicious
Pty
Ltd
27

28. >
Search
call
to
ac&on
for
offline
December
2011
©
Datalicious
Pty
Ltd
28

29. >
OTP
Response
– Different
numbers
for
different
media
channels
– Different
numbers
for
different
product
categories
– Different
numbers
for
different
conversion
steps
– Call
origin
becoming
useful
to
shape
call
script
– Feasible
to
pause
numbers
to
improve
integrity
December
2011
©
Datalicious
Pty
Ltd
29

30. >
Whose
help
do
you
need?
Technology/IT
UX Agency
Analytics!
Your boss, Your boss’ boss
Creative Agency
Customer Contact Management
December
2011
©
Datalicious
Pty
Ltd
30

31. >
Proving
the
Value
GO
BIG
December
2011
©
Datalicious
Pty
Ltd
31

32. >
How
much
sample
do
I
need?
BAU/Baseline
Conversion
Rate
#
on
Segments,
#
of
Treatments
n
Expected
Δ
in
Conversion
Time
in
Market
[Digital
Only]
December
2011
©
Datalicious
Pty
Ltd
32

33. >
Sta&s&cal
Significance
Q.
How
much
am
I
willing
to
accept
that
the
difference
in
the
results
between
my
test
group
and
control
group
may
have
been
due
to
chance?
A. Not
much.
I
want
to
be
confident
that
if
I
repeated
the
test
100
&mes,
then
I
would
observe
this
difference
95
&mes.
This
is
‘95%
confidence’
December
2011
©
Datalicious
Pty
Ltd
33

34. >
Type
I
and
Type
II
Error
Type
I:
Accept
result
to
be
true
when
it’s
actually
false
(false
posi&ves)
Type
II:
Accept
result
to
be
false
when
it’s
actually
true
(false
nega&ves)
December
2011
©
Datalicious
Pty
Ltd
34

35. >
Es&ma&ng
Sample
Size
(%s)
2 # p1 (1− p1 ) + p2 (1− p2 ) &
n = (1.645+1.282) * % 2 (
$ Δ '
Where:
n
=
es0mated
sample
size
for
each
group
p1
=
expected
conversion
rate
for
your
test
treatment
p2
=
expected
conversion
rate
for
your
control
treatment
Δ
=
expected
minimum
percentage
point
difference
between
test
and
control
results
The
value
of
1.645
reflects
that
we
accept
Type
I
error
probability
of
.05
The
value
of
1.282
reflects
that
we
accept
Type
II
error
probability
of
.10
December
2011
©
Datalicious
Pty
Ltd
35

36. >
Es&ma&ng
Sample
Size
(%s)
Typical
Champion
(control)
vs.
Challenger
(test)
A|B
test,
typical
champion
response
rate
of
2.5%.
• Only
going
to
replace
Champion
with
Challenger
if
Challenger
response
rate
is
3.0%
(0.5%
is
a
meaningful
difference)
2 ! 0.025* 0.975 + 0.030 * 0.970 $
n = (1.645+1.282) * # 2 &
" 0.005 %
Sample
size
=
18,326
For
each
of
the
Champion
and
Challenger
groups
If
1.0%
our
meaningful
difference
then
sample
size
is
only
5,378
December
2011
©
Datalicious
Pty
Ltd
36

37. >
Es&ma&ng
Sample
Size
($s)
(1.645 +1.282)2 * (s12 + s2 )
2
n=
Δ2
Where:
n
=
number
of
observa0ons
for
each
group
s1
=
expected
standard
devia0on
of
value
for
your
test
treatment
s2
=
expected
standard
devia0on
of
value
for
your
control
treatment
Δ
=
expected
minimum
difference
in
value
between
test
and
control
results
The
value
of
1.645
reflects
an
accepted
Type
I
error
probability
of
.05
The
value
of
1.282
reflects
an
accepted
Type
II
error
probability
of
.10
December
2011
©
Datalicious
Pty
Ltd
37

38. >
Standard
Devia&on
Standard
devia0on
is
measure
of
the
variability
of
your
results,
whether
some
your
results
are
quite
different
to
your
mean
(average)
result
or
whether
they
are
quite
similar.
n
∑(x − x ) i
i=1
s=
n −1
Where:
n
=
number
of
observa0ons
xi
=
the
result
for
the
ith
observa0on
x
=
mean
(average)
for
your
data
December
2011
©
Datalicious
Pty
Ltd
38

39. >
Es&ma&ng
Sample
Size
($s)
Typical
Champion
(control)
vs.
Challenger
(test)
A|B
test,
typical
champion
mean
response
value
of
$20,
typical
response
rate
of
5%
• Only
going
to
replace
Champion
with
Challenger
if
Challenger
mean
response
value
is
is
$30
($10
is
a
meaningful
difference)
• Standard
devia0on
of
Champion
results
is
$5
(based
on
past
results).
We’ll
assume
the
same
for
the
Challenger.
2 2 2
(1.645 +1.282) * (5 + 5 )
n= 2
10
Number
of
observa0ons
=
4.3
(~5)
for
each
of
the
Champion
and
Challenger
groups.
Then
divide
through
with
the
expected
response
rate
to
get
minimum
sample
size
of
86
for
each
of
Challenger
and
Control
groups
(4.3/0.05)
December
2011
©
Datalicious
Pty
Ltd
39

40. >
Further
Complexity
I
If
we
wanted
to
test
the
performance
of
Challenger
vs.
Champion
for
different
segments
of
consumers:
Response
Rate
Champion
Challenger
A
%
%
Segment
B
%
%
C
%
%
Using
same
assump0ons
as
in
earlier
example
need
18,326
per
cell,
18,326*6=109,956
in
total
.
December
2011
©
Datalicious
Pty
Ltd
40

41. >
Further
Complexity
II
If
we
wanted
to
test
the
performance
of
Challenger
vs.
Champion
for
difference
segments
of
consumers
AND
had
3
different
types
of
Champion
crea0ve:
Response
Rate
Challenger
Challenger
Challenger
Champion
#1
#2
#3
A
%
%
%
%
Segment
B
%
%
%
%
C
%
%
%
%
Using
same
assump0ons
as
in
earlier
example
need
18,326
per
cell,
18,326*12=219,912
in
total.
December
2011
©
Datalicious
Pty
Ltd
41

42. >
Further
Complexity
III
If
we
wanted
to
test
the
performance
of
Challenger
crea0ve
that
was
specifically
customised
for
difference
segments
of
consumers,
then
we’re
actually
only
running
6
tests
Response
Rate
Challenger
Challenger
Challenger
Champion
#1
#2
#3
A
%
%
Segment
B
%
%
C
%
%
Using
same
assump0ons
as
in
earlier
example
need
18,326
per
cell,
18,326*6=109.956
in
total.
December
2011
©
Datalicious
Pty
Ltd
42

43. >
Mul&variate
Tes&ng
Mul0variate
Tes0ng
(commonly
called
MVT)
is
a
term
used
for
tes0ng
different
varia0ons
of
typical
elements
of
a
landing
page,
direct
mail
leRer,
etc.
The
aim
is
to
determine
which
combina0on
delivers
the
best
result.
Element
#1:
Prominent
headline
§ Element
#1
Element
#2:
– 2
varia0ons
(1
exis0ng,
1
new)
Suppor0ng
Call
to
content
§ Element
#2
ac0on
– 2
varia0ons
(1
exis0ng,
1
new)
Element
#3:
Social
proof
/
§ Element
#3:
trust
– 2
varia0ons
(1
exis0ng,
1
new)
Terms
and
condi0ons
December
2011
©
Datalicious
Pty
Ltd
43

44. >
MVT
–
Full
Factorial
A
full
factorial
design
requires
every
unique
combina0on
of
page
elements
and
can
therefore
be
very
sample
hungry.
Element
To
calculate
the
Headline
Call
to
Ac&on
Social
Proof
number
of
1
H1
CTA1
SP1
treatments
just
need
2
H1
CTA1
SP2
to
mul0ply
the
3
H1
CTA2
SP1
number
of
varia0ons
4
H1
CTA2
SP2
Treatment
for
each
factor
5
H2
CTA1
SP1
together:
6
H2
CTA1
SP2
7
H2
CTA2
SP1
2
x
2
x
2
=
8
8
H2
CTA2
SP2
December
2011
©
Datalicious
Pty
Ltd
44

45. >
MVT
–
Frac&onal
Factorial
The
alterna0ve
is
called
a
frac0onal
factorial
design
which
is
some
smaller
set
of
elements
combina0ons.
The
design
should
be
‘balanced’
-­‐
every
varia0on
is
tested
the
same
number
of
0mes
and
each
combina0on
of
varia0ons
occurs
the
same
number
of
0mes.
Element
Headline
Call
to
Ac&on
Social
Proof
1
2
H1
CTA1
SP2
Reduced
sample
3
H1
CTA2
SP1
requirements
Treatment
4
4x18,326=73,304
5
H2
CTA1
SP1
6
7
8
H2
CTA2
SP2
December
2011
©
Datalicious
Pty
Ltd
45

46. >
Layout
Before
Content
§ Phase
#1:
A|B
test
– Test
the
same
landing
Element
#1:
Prominent
headline
page
content
in
completely
different
layouts
§ Phase
#2:
MV
test
Suppor0ng
Element
#2:
– Then
test
different
content
Call
to
ac0on
content
element
combina0ons
within
the
winning
layout
Element
#3:
Social
proof
/
trust
§ Phase
#3:
MV
test
(if
req’d)
– Test
with
reduced
set
of
Terms
and
condi0ons
elements
December
2011
©
Datalicious
Pty
Ltd
46

47. >
Case
Study
§ Yes,
the
measurement
infrastructure
is
in
place
§ I
can
readily
execute
the
test
design
§ I
have
enough
sample
to
draw
valid
conclusions
§ Yes,
this
design
will
prove
the
value
of
tes0ng
in
my
business
December
2011
©
Datalicious
Pty
Ltd
47

48. 101011010010010010101111010010010101010100001011111001010101
010100101011001100010100101001101101001101001010100111001010
010010101001001010010100100101001111101010100101001001001010
>
Execu&on
&
Measurement
December
2011
©
Datalicious
Pty
Ltd
48

49. Before
you
leap…
December
2011
©
Datalicious
Pty
Ltd
49

50. >
Sample
Selec&on
§ Each
sample
needs
to
be
alike
in
terms
of
their
predisposi0on
to
conversion
Conversion:
low
rate
credit
card
applica0on
form
comple0on
TEST
CONTROL
18-­‐34
35-­‐64
Mostly
Male
Mostly
Female
Mostly
Low
Income
Mostly
High
Income
December
2011
©
Datalicious
Pty
Ltd
50

51. >
Timing
is
Important
‘Burst’
Non
BAU
ATL
Ideal
Test
Window
Campaign
Sales
Time
December
2011
©
Datalicious
Pty
Ltd
51

52. >
A|A
Tes&ng
§ Set
a
test
that
splits
your
visitors
50/50
between
the
same
treatment
– Check
that
sample
sizes
are
actually
50/50
– Is
there
should
be
no
difference
in
your
conversion
rates
– Are
volumes
of
conversions
matching
other
repor0ng?
December
2011
©
Datalicious
Pty
Ltd
52

53. >
Measuring
your
performance
§ Propor0ons
(conversion
rates)
§ Means
(average
$s)
§ Variability
of
Means
(standard
devia0on)
Would
my
winning
treatment
s2ll
be
the
winner
across
all
my
customers/visitors/consumers?
§ Use
confidence
intervals
December
2011
©
Datalicious
Pty
Ltd
53

54. >
Confidence
Intervals
Conversion
Rate
Revenue
per
Response
A
B
C
A
B
C
Treatments
Treatments
December
2011
©
Datalicious
Pty
Ltd
54

55. >
Confidence
Intervals
December
2011
©
Datalicious
Pty
Ltd
55

56. >
Confidence
Interval
(%s)
ˆ ˆ
p(1− p)
ˆ
p ±1.96 *
n
Where:
^
p
=
response
rate
n
=
sample
size
for
treatment
The
value
of
1.96
reflects
a
95%
confidence
level
December
2011
©
Datalicious
Pty
Ltd
56

57. >
Confidence
Interval
Es&ma&on
Typical
Champion
(control)
vs.
Challenger
(test)
A|B
Test
Treatment
Champion
Challenger
Mailed
60850
52812
Responses
1055
455
Response
Rate
1.7
0.9
.017(1−.017) .009(1−.009)
1.7% ±1.96 * 0.9% ±1.96 *
60850 52812
1.7% ± 0.10% 0.9% ± 0.08%
1.69%
≤
Champion
≤
1.71%
0.82%
≤
Challenger
≤
0.98%
December
2011
©
Datalicious
Pty
Ltd
57

58. >
Confidence
Interval
Es&ma&on
p1 (1− p1 ) p2 (1− p2 )
p1 − p2 ±1.96 * +
n1 n2
Where:
p1
=
response
rate
for
challenger
p2
=
response
rate
for
champion
n1
=
sample
size
for
challenger
n2
=
sample
size
for
challenger
The
value
of
1.96
reflects
a
95%
confidence
level
December
2011
©
Datalicious
Pty
Ltd
58

59. >
Confidence
Interval
Es&ma&on
Typical
Champion
(control)
vs.
Challenger
(test)
A|B
Test
Treatment
Champion
Challenger
Mailed
60850
52812
Responses
1055
455
Response
Rate
1.7
0.9
.009(1−.009) .017(1−.017)
0.9 −1.7 ±1.96 * +
52812 60850
−0.8 ± 0.13
-­‐0.93%
≤
Difference
Between
Challenger
and
Champion
≤
-­‐0.67%
December
2011
©
Datalicious
Pty
Ltd
59

60. >
Control
Group
Sample
Size
p1 (1− p1 ) p2 (1− p2 )
p1 − p2 ±1.96 * +
n1 n2
pc (1− pc )
Rearranged:
nc = 2
" m % pt (1− pt )
$ ' −
# 1.96 & nt
Where:
nc
=
sample
size
for
control
group
nt
=
sample
size
for
test
group
pc
=
forecast
response
rate
for
control
group
nt
=
forecast
response
rate
for
test
group
m
=
desired
level
of
precision
(%
that
is
a
meaningful
difference)
The
value
of
1.96
reflects
a
95%
confidence
level
December
2011
©
Datalicious
Pty
Ltd
60

61. >
Control
Group
Sample
Size
We
have
50,000
customers
that
we
could
include
in
our
test
design,
what
would
our
control
sample
need
to
be
if
we
tested
40,000
customers,
our
‘natural’
cross-­‐sell
rate
was
1.0%
and
an
incremental
response
rate
of
1.0%
points
would
be
deemed
to
be
meaningful?
.01(1−.01)
nc = 2
" .01 % .02(.02 −.02)
$ ' −
# 1.96 & 40, 000
nc = 387
This
result
suggests
we
could
actually
test
more
of
our
available
customer
base
than
we
might
have
ini0ally
expected
(~40,600).
December
2011
©
Datalicious
Pty
Ltd
61

62. >
Confidence
intervals
($s)
s
x ±1.96 *
n
Where:
x
=
mean
revenue
among
treatment
responders
s
=
standard
devia0on
of
revenue
among
some
treatment’s
responders
n
=
number
of
responders
to
the
treatment
The
value
of
1.96
reflects
a
95%
level
of
confidence.
December
2011
©
Datalicious
Pty
Ltd
62

63. >
Standard
Devia&on
(reminder)
Standard
devia0on
is
measure
of
the
variability
of
your
results,
whether
some
your
results
are
quite
different
to
your
mean
(average)
result
or
whether
they
are
quite
similar.
n
∑(x − x ) i
i=1
s=
n −1
Where:
n
=
number
of
observa0ons
xi
=
the
result
for
the
ith
observa0on
x
=
mean
(average)
for
your
data
December
2011
©
Datalicious
Pty
Ltd
63

64. >
Confidence
intervals
($s)
2 2
s s 1 2
x1 − x2 ±1.96 * +
n1 n2
Where:
x1
=
mean
value
among
among
responders
to
a
treatment
x2
=
mean
value
among
among
responders
to
a
different
treatment
s1
=
std.
dev.
of
value
among
one
treatment’s
responders
s2
=
std.
dev.
of
value
among
the
other
treatment’s
responders
n1
=
number
of
responders
to
the
treatment
n2
=
number
of
responders
to
the
other
treatment
The
value
of
1.96
reflects
a
95%
level
of
confidence.
n1
and
n2
is
sufficiently
large
to
es0mate
the
std.
dev.
in
the
popula0on
with
the
std.
dev.
of
the
sample.
December
2011
©
Datalicious
Pty
Ltd
64

65. >
Confidence
intervals
($s)
Typical
Champion
(control)
vs.
Challenger
(test)
A|B
Test
Treatment
Champion
Challenger
Mailed
60850
52812
Responses
1055
455
Response
Rate
1.7
0.9
Total
Value
$36,925
$38,675
Mean
Value
$35
$85
Std
Dev
$30
$50
50 2 30 2
85 − 35 ±1.96 * + 50 ± 4.9
455 1055
At
a
minimum,
we
should
expect
an
incremental
$45.1
if
we
rolled
out
the
Challenger
crea0ve
as
BAU
(although
our
total
amount
of
incremental
revenue
would
be
less).
December
2011
©
Datalicious
Pty
Ltd
65

66. >
Case
Study
December
2011
©
Datalicious
Pty
Ltd
66

67. >
Main
Effects
December
2011
©
Datalicious
Pty
Ltd
67

68. >
Main
Effects
Typical
Landing
Page
Test
Element
Results
Call
to
Visitors
Conversion
Headline
Social
Proof
Conversions
Ac&on
Tested
Rate
1
H1
CTA1
SP1
1237
456
37%
2
H1
CTA1
SP2
1456
345
24%
3
H1
CTA2
SP1
1245
234
19%
4
H1
CTA2
SP2
2123
432
20%
Treatment
5
H2
CTA1
SP1
1342
234
17%
6
H2
CTA1
SP2
1102
123
11%
7
H2
CTA2
SP1
1365
700
51%
8
H2
CTA2
SP2
1243
643
52%
Treatment
#7
and
#8
were
the
clear
winners
and
It
looks
as
if
the
Headline
and
Call-­‐to-­‐Ac0on
were
much
bigger
drivers
of
posi0ve
performance
than
the
Social
Proof.
Lets
check
this!
December
2011
©
Datalicious
Pty
Ltd
68

69. >
Main
Effects
Typical
Landing
Page
Test
Element
Results
Call
to
Social
Visitors
Conversion
Headline
Ac&on
Proof
Tested
Rate
1
H1
CTA1
SP1
1237
37%
2
H1
CTA1
SP2
1456
24%
Avg
H1=24%
3
H1
CTA2
SP1
1245
19%
4
H1
CTA2
SP2
2123
20%
Treatment
5
H2
CTA1
SP1
1342
17%
6
H2
CTA1
SP2
1102
11%
7
H2
CTA2
SP1
1365
51%
Avg
H2=33%
8
H2
CTA2
SP2
1243
52%
The
Main
Effect
of
the
Headline
is
simply
the
(weighted)
average
conversion
rate
for
Headline
2
less
the
(weighted)
average
conversion
rate
for
Headline
1
(33%-­‐24%=9%)
December
2011
©
Datalicious
Pty
Ltd
69

70. >
Main
Effects
Typical
Landing
Page
Test
Main
Effect
Headline
9.4%
Element
Call
to
Ac&on
11.1%
Social
Proof
5.3%
In
actual
fact,
it
was
varia0ons
in
Call
to
Ac0on
that
had
the
most
posi0ve
impact
on
our
results,
improving
conversions
by
11.1%
points.
December
2011
©
Datalicious
Pty
Ltd
70

71. >
Interac&on
Effects
Typical
Landing
Page
Test
Element
Results
Call
to
Social
Visitors
Conversion
Headline
Ac&on
Proof
Tested
Rate
1
H1
CTA1
SP1
1237
37%
2
H1
CTA1
SP2
1456
24%
7
H2
CTA2
SP1
1365
51%
8
H2
CTA2
SP2
1243
52%
Treatment
3
H1
CTA2
SP1
1245
19%
4
H1
CTA2
SP2
2123
20%
5
H2
CTA1
SP1
1342
17%
6
H2
CTA1
SP2
1102
11%
An
interac0on
effect
is
present
where
the
performance
of
one
element
is
dependent
on
which
varia0on
of
the
another
variable
is
present.
In
this
example,
we
are
looking
at
whether
the
results
for
each
of
the
Headlines
is
dependent
on
which
Call-­‐to-­‐Ac0on.
December
2011
©
Datalicious
Pty
Ltd
71

72. >
Interac&on
Effects
Typical
Landing
Page
Test
Element
Results
Call
to
Social
Visitors
Conversion
Headline
Ac&on
Proof
Tested
Rate
1
H1
CTA1
SP1
1237
37%
2
H1
CTA1
SP2
1456
24%
Wtd
Avg
H1CTA1=30%
3
H1
CTA2
SP1
1245
19%
Wtd
Avg
H1CTA2=20%
4
H1
CTA2
SP2
2123
20%
Treatment
5
H2
CTA1
SP1
1342
17%
Wtd
Avg
H2CTA1=14%
6
H2
CTA1
SP2
1102
11%
7
H2
CTA2
SP1
1365
51%
Wtd
Avg
H2CTA2=51%
8
H2
CTA2
SP2
1243
52%
The
first
step
is
to
create
weighted
average
response
rates
between
for
each
of
the
two
factors
(ignoring
Social
Proof).
December
2011
©
Datalicious
Pty
Ltd
72

73. >
Interac&on
Effects
Typical
Landing
Page
Test
Call
to
Ac&on
CTA1
CTA2
Diff
60%
H1
30%
20%
-­‐10%
40%
CTA1
20%
Headline
H2
14%
51%
37%
CTA2
0%
Diff
-­‐16%
31%
H1
H2
The
next
step
is
to
calculate
the
difference
in
performance
of
one
factor
across
different
variants
of
the
other
factor.
If
the
difference
of
this
difference
is
non-­‐
zero
(or
not
very
close
to
zero),
then
you
have
an
interac0on
effect.
For
example,
there
is
an
interac0on
effect
between
the
Headline
and
Call
to
Ac0on
as
the
difference
in
the
difference
in
performance
is
non-­‐zero
(31%-­‐
(-­‐16%)=47%).
This
is
very
large
interac0on
when
compared
to
the
Main
Effects!
December
2011
©
Datalicious
Pty
Ltd
73

74. >
Interac&on
Effects
Typical
Landing
Page
Test
Sociol
Proof
SP1
SP2
Diff
40%
H1
28%
22%
-­‐6%
SP1
20%
Headline
H2
34%
33%
-­‐1%
SP2
0%
Diff
-­‐6%
11%
H1
H2
Sociol
Proof
40%
SP1
SP2
Diff
CTA1
27%
18%
-­‐9%
20%
SP1
Call
to
SP2
CTA2
36%
32%
-­‐4%
Ac&on
0%
Diff
9%
14%
CTA1
CTA2
December
2011
©
Datalicious
Pty
Ltd
74

75. 101011010010010010101111010010010101010100001011111001010101
010100101011001100010100101001101101001101001010100111001010
010010101001001010010100100101001111101010100101001001001010
>
Repor&ng
December
2011
©
Datalicious
Pty
Ltd
75

76. Document
Everything!
December
2011
©
Datalicious
Pty
Ltd
76

77. >
1.
Describe
the
test
§ Describe
the
outcome(s)
you’re
trying
to
influence
§ Describe
your
target
audience
§ Describe
the
different
treatments
including
copies
of
crea0ve
December
2011
©
Datalicious
Pty
Ltd
77

78. >
2.
Jus&fy
the
test
design
§ Detail
why
you’ve
chosen
the
par0cular
outcome
you’re
trying
to
influence
§ Detail
why
you’ve
chosen
the
consumers
you
are
trying
to
influence
§ Detail
why
your
interven0on
should
work
– Past
test
results/Useability
test/Case
studies
– Marketers
intui0on/logic
December
2011
©
Datalicious
Pty
Ltd
78

79. >
3.
Results
&
Conclusions
§ Detail
all
the
performance
results
§ Discuss
your
hypotheses
§ Future
tests
§ ‘Meta’
repor0ng
of
your
test
program
December
2011
©
Datalicious
Pty
Ltd
79

80. >
The
Scien&fic
Method
Knowledge
Establish
Develop
Facts
Test(s)
Data
December
2011
©
Datalicious
Pty
Ltd
80

81. >
Case
Study
December
2011
©
Datalicious
Pty
Ltd
81

82. >
List
of
(Some)
Resources
§ hRp://visualwebsiteop0mizer.com/case-­‐
studies.php
§ hRp://www.whichtestwon.com/
§ hRp://www.feng-­‐gui.com
§ hRp://www.smashingmagazine.com/
2010/06/24/the-­‐ul0mate-­‐guide-­‐to-­‐a-­‐b-­‐
tes0ng
December
2011
©
Datalicious
Pty
Ltd
82

83. Contact
us
msavio@datalicious.com
Learn
more
blog.datalicious.com
Follow
us
twizer.com/datalicious
December
2011
©
Datalicious
Pty
Ltd
83

84. Data
>
Insights
>
Ac&on