To most web designers and forward-looking experts in the field, responsive mobile design is not an optional feature. Mobile users have to be able to view your website on their devices without having to scroll everywhere to read it. But is responsive design really worth the investment? Has anyone actually pitted a responsive design against a regular desktop design? These were the questions in the minds of the marketers at a large media company faced with the mobile dilemma. To answer these questions, the marketers designed a test.

1. Responsive Design Tested
What a recent experiment reveals about the potential ROI of
mobile design

2. We’re sharing on Twitter!
#WebClinic

3. Today’s speakers
Austin McCraw
Jon Powell
Senior Editorial Analyst
MECLABS
Senior Manager, Research
and Strategy
MECLABS

4. What is responsive design?
Unresponsive
Responsive

5. What is responsive design?
The MECLABS Conversion Index
C = 4m + 3v + 2(i - f) - 2a©
Friction: Psychological resistance to a
given element in the conversion
process, typically in the form of:
• Unnecessary length
• Unnecessary difficulty

6. ?
What impact does responsive design
have on friction experienced by the
customer?

7. Experiment: Background
Experiment ID: TP1933
Record Location: MECLABS Research Library
Research Partner: [Protected]
Background: A large news media organization trying to determine whether it should
invest in responsive mobile design.
Goal: To significantly increase the number of free trial sign-ups.
Research Question: Which design will generate the highest rate of free trial sign-ups:
responsive or unresponsive?
Test Design: A/B multifactor split test

8. Experiment: Control (unresponsive)
Desktop
*Android devices also included in this test
Tablet

9. Experiment: Control (unresponsive)
Desktop
*Android devices also included in this test
Mobile

10. Experiment: Treatment (responsive)
Desktop
*Android devices also included in this test
Tablet

11. Experiment: Treatment (responsive)
Desktop
*Android devices also included in this test
Mobile

12. Experiment: Side-by-side
Unresponsive
*Android devices also included in this test
Responsive

13. Experiment: Results
56% Aggregate increase in conversion
The responsive design treatment showed an aggregate statistically significant difference in
free trial subscriptions when compared to the control.
Design
Conversion Rate
Control – Unresponsive
3.71%
Treatment – Responsive
5.80%
% Relative Change:
*Android devices are also part of these results
56.41%

14. ?
Why did the responsive design
outperform the unresponsive by so
much?

15. Experiment: Results
56% Aggregate increase in conversion
The responsive design treatment showed an aggregate statistically significant difference in
free trial subscriptions when compared to the control.
Design
Conversion Rate
Control – Unresponsive
3.71%
Treatment – Responsive
5.80%
% Relative Change:
*Android devices are also part of these results
56.41%

16. Experiment: Results
Design
Conversion Rate
56% Aggregate Increase in Conversion
Control – Unresponsive
3.71%
The responsive design treatment showed an aggregate statistically significant difference in
+ Desktop trial subscriptions when compared to the control
%
free
+ Tablet
%
+ Mobile
Treatment – Responsive
%
5.80%
+ Desktop
%
+ Tablet
%
+ Mobile
%
?
Which device makes the greatest contribution to the total difference?

17. !
Today, we are going to review test data
for each device category to determine its
total impact on perceived friction.

18. Desktop Performance
Responsive design tested

19. Experiment: Desktop control
Unresponsive
• For most screen sizes, all primary content
is accessible without action
• The primary content is framed within a
faux pop-up style box over a faded
background

20. Experiment: Desktop treatment
Responsive
• Like the control, all primary content is
accessible without action
• The faux pop-up frame, however, is
completely eliminated

21. Experiment: Desktop side-by-side
Unresponsive
Responsive

22. Experiment: Desktop results
105% Increase in conversion
The responsive design desktop treatment showed a statistically significant difference in free
trial subscriptions when compared to the control.
Design
Conversion Rate
Desktop Control – Unresponsive
2.65%
Desktop Treatment – Responsive
5.43%
% Relative Change:
104.57%

23. ?
What was actually responsible for the
difference in performance?

24. Tablet Performance
Responsive design tested

25. Experiment: Tablet control
Unresponsive
iOS:
• The majority of the content is
accessible without action
(supporting image requires
scrolling)
• A portion of the bottom of the
page shows the pop-up
Android:
• All of the content is accessible
without action; may require zoom
for certain devices
• Supporting imagery and aesthetic
quality is lost in the page load

26. Experiment: Tablet treatment
Responsive
iOS:
• The supporting image is
immediately visible and the popup background is completely
eliminated
Android:
• Significant improvement in
aesthetic and page load, though
the supporting image is still lost
• Only vertical scrolling required for
ancillary content

27. Experiment: Tablet side-by-side
Unresponsive
Responsive

28. Experiment: Tablet results
X
No significant difference in conversion
The responsive design tablet treatment showed no statistically significant difference in free
trial subscriptions when compared to the control.
Design
Conversion Rate
Tablet Control – Unresponsive
7.24%
Tablet Treatment – Responsive
8.81%
% Relative Change:
21.6%

29. ?
Why did the same treatments perform
differently on different devices?

30. Mobile Performance
Responsive design tested

31. Experiment: Mobile control
Unresponsive
iOS:
• Requires scrolling to read complete
sentences
• Also requires scrolling to see all
required fields and the primary call-toaction
Android:
• Requires one to two pinches to zoom
movements and read the content
• When zoomed in, requires horizontal
scrolling to read complete sentences

32. Experiment: Mobile treatment
Responsive
• Requires vertical scrolling to get to all fields
as well as the call-to-action
• Complete sentences are available without
action
• Generally a consistent experience across
major device platforms

33. Experiment: Mobile side-by-side
Unresponsive
Responsive

34. Experiment: Mobile results
X
No significant difference in conversion
The responsive design mobile treatment showed no statistically significant difference in free
trial subscriptions when compared to the control.
Design
Conversion Rate
Mobile Control – Unresponsive
5.14%
Mobile Treatment – Responsive
4.34%
% Relative Change:
-15.6%

35. Experiment: All results
Design
Conversion Rate
% Rel. Change
3.71%
-
+ Desktop
2.65%
-
+ Tablet
7.24%
-
+ Mobile
5.14%
-
5.80%
+56.4%
+ Desktop
5.43%
+104.6%
+ Tablet
8.81%
+21.6%
Control – Unresponsive
Treatment – Responsive
+ Mobile
?
% Relative Change:
4.34%
56.41%
-15.6%
What is the customer wisdom? Why did the treatment win in
aggregate?

36. What you need to understand
The MECLABS Conversion Index
C = 4m + 3v + 2(i - f) - 2a
©
Friction
The mobile and tablet treatments
made significant changes in layout;
they did not, however, make
significant changes in friction
experienced by the customer

37. What you need to understand
The MECLABS Conversion Index
C = 4m + 3v + 2(i - f) - 2a
©
Friction
Anxiety
The desktop treatment did something
slightly different
The move to responsive design
coincidentally reduced both friction
and anxiety by eliminating the pop-up
appearance of the control

38. Responsive design tested
F
Key Principles
1. Responsive design, like any new technology or technique, does not necessarily
increase conversion. This is because we are not optimizing design, we are
optimizing thought sequences.
2. Responsive design is only as effective as the degree of change it affects in the
mind of the customer.
3. This requires you to consider not only how responsive design will impact the
look of the page, but the customers perception on two levels:
•
•
Friction: How much will responsive design decrease perceived effort?
Anxiety: How much will responsive design decrease perceived concern?

39. EXCLUSIVE SAVINGS
Save an EXTRA
$200 on Email
Summit Tickets
Use promo code:
515-ST-1010
*Offer expires January 24

40. Experiment: Background
Experiment ID: TP1933
Record Location: MECLABS Research Library
Research Partner: [Protected]
Background: A large technology company selling software to small businesses.
Goal: To significantly increase the number of software purchases.
Research Question: Which design will generate the highest rate of software
purchases?
Test Design: A/B multifactor split test

41. Experiment: Side-by-side
Control
COLUMN #1
Treatment
COLUMN #2
COLUMN #1

42. Which treatment won?

43. To see the results …
Live: Jan. 22 at 4:00 p.m. EST
Watch live or get the full 35-minute video
replay
• What is the optimal number of columns for a
page?
• How should columns be weighted on a page?
• Which is better: a right sidebar or left
sidebar?
To watch live or view the full video replay when it is available, register at the link below:
MarketingExperiments.com/Columns

44. See how you can conduct research with us
MECLABS conducts rigorous experiments in the new science of optimization. We apply our
discoveries to help leaders optimize the financial performance of their sales and
marketing programs.
Learn more about how you may
be a fit for a MECLABS Research
Partnership:
• Select Research Partnership
Opportunities on the postwebinar survey
• Contact us directly
info@MECLABS.com
1-877-635-0565
x