Recruiting the right participants is often the most difficult and challenging aspect of conducting research studies. A well-managed recruitment process can be pivotal to the success of achieving your research objectives. Join JonDelina (“JD”) Buckley, Manager of User Experience Research at Kelley Blue Book, and Aaron Rager, Client Development Manager at Research Now, to learn about some of the best practices on recruiting the right participants for remote unmoderated user testing.

1. Webinar:
Best Practices on Recruiting
Participants for Remote User Testing
JonDelina “JD” Buckley, Kelley Blue Book
Aaron Rager, Research Now
Alfonso de la Nuez, UserZoom
#uzwebinar

2. Speakers:
JD Buckley
Manager of
User Experience
Kelley Blue Book
Speaker
Aaron Rager
Client Development
Sales Manager
Research Now
Speaker
Alfonso de la Nuez
Co-Founder and
Co-CEO
UserZoom
Moderator

3. Quick Housekeeping
• Chat box is available if you have any questions
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the
slides/recording
• Twitter hashtag: #uzwebinar
www.userzoom.com

4. Agenda for Today
1. Introductions
2. Recruiting for Remote User Research
JD Buckley, Kelley Blue Book
3. Capabilities Overview
Aaron Rager, Research Now
4. Q&A
www.userzoom.com

5. About UserZoom
 All-in-one Enterprise software solution that helps Businesses costeffectively test, measure and improve UX over websites & mobile apps.
 Offer online or remote user research & testing
solutions, saving UXers time, money, effort,
and a lot of actionable insights
 UX Consultants since ‟01, SaaS since „09
Product Suite:
 Unmoderated Remote Usability Testing
 In Sunnyvale (CA) Manchester (UK), Munich
(DE) and Barcelona (Spain)
 Remote Mobile Usability Testing
 Online Surveys (web & mobile)
 Online Card Sorting
 90% renewal rate, 50% revenue growth rate
in the last 3 years
 Tree Testing
 Screenshot Click Testing
 Screenshot Timeout Testing (5-sec test)
 Web VOC
 Mobile VOC
www.userzoom.com

6. Recruiting For Remote User Research
JD Buckley
Manager, User Experience Research
Kelley Blue Book

7. Agenda
• Who I Am & What I Do
• A Valuable Tool In The Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

8. Agenda
• Who I Am & What I Do
• A Valuable Tool In The Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

9. Who I Am, What I Do
JD Buckley – Manager of User Experience Research – Kelley Blue Book
Education
•BA – Communications/Marketing
•MA – Human Factors Psychology
Companies
•Ad Agencies - J. Walter Thompson, Lord Dentsu & Partners, Dailey &
Assoc’s,TBWAChiat/Day
•Technology Companies – DirecTV, Yahoo!
•Technology Incubators & Startups – Idealab, Dank-Muller & Co,
InsiderPages.com, Picassa.com, Evolution Robotics, Omnilux
•Health and Medical – Kaiser, Herbalife
•Automotive – Kelley Blue Book

10. Agenda
• Who I Am & What I Do
• A Valuable Tool In The Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

11. User research has traditionally been conducted like this

12. User Researchers have traditionally recruited like this
uxforthemasses.com

13. In the last several years, remote user research has become a
popular addition to the researchers toolkit:
Benefits include:
• Opportunity to collect data from a
larger sample size compared to in-lab
research
• Potential to reach special audiences not
immediately available
• Ability to create a diverse test panel
with broader geographical distribution
• Ability to simulate more realistic
behavior in real-world context
• Allows collection of reliable quantitative
as well as qualitative data

14. Agenda
• Who I Am & What I Do
• Valuable Tool In The Researchers Toolkit
• Types of Remote User Research
• Recruiting For Remote User Research
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

15. Moderated Remote Research:
Participants perform certain tasks remotely and the researcher observes
and asks questions from another location using a screen-sharing application
http://www.flickr.com/photos/boltpeters/3918152527/

16. Un-moderated/Automated Remote Research:
Participants perform certain tasks and answer survey-like questions,
researcher and participant are separated over time and space
Userzoom.com

17. Remote Self-Moderated
Participant perform tasks while their interactions and audio are recorded as
they talk “aloud” without a moderator present
Usertesting.com

18. Diary Studies
Participants chronicle their daily activities or interactions with a product for
a specific period of time, and share that information remotely with a
researcher
http://www.hcii.cmu.edu/M-HCI/2011/BOA-PlanningTools/

19. Agenda
• Who I Am & What I Do
• Valuable Tool In The Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

20. Similar to traditional in-lab research, user researchers have
two options for remote recruiting:
1. Have someone else do it – (e.g. recruitment companies)
2. Do it yourself – (e.g., recruit in house)

21. External recruiters can provide multiple options for
remote recruiting
Intercept
intercept
participants from
desktop or
mobile websites
Keynote.com
Company
Email List
Research
Panel
Custom
Recruit
Invite
participants from
opt-in email list
leverage panel of
individuals who
represent a
specific audience
create a custom
panel of
individuals with
targeted profiles

22. External Recruiting – Pros and Cons
Pros
Cons
• Frees the researcher to focus
on the study
• Can be more expensive than
self recruiting
• Can leverage multiple sources
to access your target
audience
• Requires coordination with
the external recruitment
agency
• Handle incentives and quota
fulfillment
• Study design complexity and
recruiter involvement may
increase costs
• May help with screener
development
• Quality of panelists data differ
across panels & companies

23. Self-recruiting can offer some similar and additional options
Intercept
intercept
participants from
desktop or
mobile websites
Keynote.com
Company
Email List
Invite
participants from
opt-in or rented
email list
Social Media
Online Posting
leverage social
media options to
source target
audience
Post an ad on a
blog, classifieds
site, newsletter
or via SEM

24. Self-recruiting – Pros and Cons
Pros
• Can be less expensive than
using an external resource
• Allows the researchers to
control the process and/or
quality of participants
Cons
• Can be time consuming
• Requires strong project
management/organization
skills from the researcher
• Limited options/back-up if
anything goes wrong

25. Recruitment Method and Remote Research Type Cheat Sheet
Remote
Moderated
Remote
Unmoderated
Remote
Self
Moderated
Diary Studies
Intercept
Email
Lists
Intercept
Research
Research
Type
Types
Email
Lists
Custom
Social
Research
Recruit
Media
Panel
Recruitment Methods
Custom
Recruit
Social
Media
Research
Panel
Online
Postings
Online
Postings

26. Agenda
• Who I Am & What I Do
• A Valuable Tool In The UX Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

27. Every product or service has a target audience

28. You need to understand the experience of the people who are actually going
to want to use, understand and buy your product. Anyone else’s experience
will be of marginal use, or even deceptive.
Observing the User Experience, 2nd Edition – Goodman, Kuniavsky and Moed

29. 90percentofeverything.com

30. Whether you‟ve decided to recruit for your research by using segments,
profiles or persona‟s, you should have a clear idea of:

31. Demographics data: Includes physical characteristics (age, gender,
ethnicity) employment, education and household income information
Freewaregenius.com/quantcast.com

32. Channel Usage: (Web, Mobile or Tablet)
http://www.lukew.com/ff/entry.asp?1436

33. Technology Usage
Google-Multiscreenworld

34. The presence or absence of key behaviors, needs and triggers
that help you narrow in on your appropriate research audience
Melissa - 33, married, 3 children
“After the accident, we decided to find something very safe with room for our family.”
•
•
•
•
Car for family
Other people’s opinions
Influencer / Influenced
Urgent need

35. Agenda
• Who I Am & What I Do
• Valuable Tool In The Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Research Participants
• Developing a Screener – Best Practices
• Tips and Tricks for Remote Recruiting

36. Essentially, a screener is a script that allows you to filter the general
population to an audience that reflects your specific criteria

37. The screener is probably the most important part of the recruitment
process. It’s a script that filters out people who will give you good
responses from the ones who merely match your basic criteria.
Observing The User Experience 2nd Edition – Goodman, Kuniavsky, Moed

38. Screener Best Practices:
Limit to 20 questions and a completion time of 10 minutes max

39. QX. Within the next 2 weeks, how likely are you to purchase coco pops?
Extremely Likely
A
COMPLETE: PLACE IN GROUP
Somewhat Likely
B
DO NOT RECRUIT
Not very likely
C
DO NOT RECRUIT
Not sure
D
DO NOT RECRUIT
Screener Best Practices:
Start with the questions that will screen people out

40. Screener Best Practices:
Every question should have a purpose

41. Screener Best Practices:
Be specific, clear and never use jargon

42. 1/15/12
4:42am
6lbs 14oz
Screener Best Practices:
Ask for exact dates, times, and quantities

43. Screener Best Practices:
Make sure your questions are not leading or biased

44. Example: Email Invite To Screener
Your thoughts are important us!
We would like to invite you to participate in a website evaluation that will
take approximately 20 - 30 minutes.
This website evaluation is different from other surveys you may have
taken in the past. You will be asked to view designated website(s),
perform simple tasks, and report your experience with the site(s).
Your opinions and experiences are very important, and will be used for
future website development efforts.
Screener Best Practices:
Clearly state the format of the research

45. Screener Best Practices:
Build in flexibility

46. Agenda
• Who I Am & What I Do
• Valuable Tool In The UX Researchers Toolkit
• Types of Remote User Research
• Recruiting Options
• Determining Your Target Audience
• Developing a Screener
• Tips and Tricks for Remote Recruiting

47. Tips and Tricks for Remote Research Recruiting
• Clarify the research objectives, target audience,
metrics for success, budget and timeline
• Use the Right Tool
• Recruit based on behavior first; demographic
criteria second
• Be careful not to back yourself into a corner by
“over-determining” your target audience
• Give people a reason to participate in your
research beyond the incentive
• Over recruit if possible
• Make mistakes and learn from them

48. References
Goodman, E., Kuniavsky, M., Moed, A., Observing the User Experience, A
Practitioner’s Guide to User Research, Elsevier, Morgan Kaufmann, 2012,
ISBN: 978-0-12-384869-7
Albert, B., Tedesco, D., Tullis, T., Beyond the Usability Lab, Conducting
Large-Scale Online User Experience Studies, Elsevier, Morgan Kaufmann,
2010

49. A Capabilities Overview
Research Now
social. mobile. online.
Aaron Rager
Client Development Manager
June 6, 2013

50. Who We Are
Research Now is the leading provider of global market research services.
 Experienced. Market leader in powering the insights industry, completing 2M+ interviews across 3000+
projects each month
 Innovative. Advanced digital research services including Social, Mobile, and Online
 Leading. Proprietary technologies and the largest online panels in the industry: 6M+ members across 38
countries
 Reliable. Over 2000 clients globally
 Consultative. Recognized as #1 in customer service annually since 2007*
 Worldwide. 1200+ employees in over 20 cities across four continents, with offices in Dallas, New York, Los
Angeles, Toronto, San Francisco, Chicago, London, Paris, Frankfurt, Hamburg, Athens, Madrid, Dubai,
Sydney, Melbourne, Auckland, Shanghai, Singapore, New Delhi
 Trusted. Work closely with CASRO, ESOMAR, and the MRA
50
*marketresearchcareers.com

51. Partial Client List
51

52. Our Panel Recruitment Methodology
“By-Invitation-Only” Online Panel Recruitment
–
52
A method of inviting only pre-validated individuals to enroll into a market
research panel through a controlled process.

53. Research Now: Global Reach
53

54. On-Going Panel Quality Management
Optimizing the panel member experience
•
Robust panel profiling
– 300+ profiling attributes
• Socio-demographic, life stage, interests and hobbies, shopping habits, purchase intent, etc.
•
•
Dynamic Profile Enrichment®
Business Panel - 40+ firmographic attributes
» Industry type, annual revenue, title, functional role, purchasing
role, etc.
•
Proven incentive methodology
– Pay-all
– Fair-value exchange
– Respondent reward choice
– Cash & Online currency
•
Managed Participation Levels
– Survey frequency limiters
– Member managed invitation maximums
•
Intelligent Respondent Routing – Scout & Sample Deployment (SDS) Systems
– Completed surveys routed to update profile
– Survey terminates routed to different survey
Approximately 80% Member Retention rate since inception
•
54

55. Example of Panel Reach
USA (adults 18+)
Grand Total
Sub Total Adults 18 to 34
18 to 24
25 to 34
Sub Total Adults 35 to 54
35 to 44
45 to 54
Sub Total Adults 55 to 64
55 to 64
Sub Total 65+
65+
55
© 2012 Research Now
TOTAL
2,074,994
670,745
182,848
487,897
870,673
439,254
431,419
360,701
360,701
172,875
172,875
Dated Panel Counts
North East Mid-West
South
404,613
526,530
700,359
136,131
164,756
225,382
40,551
44,747
59,672
95,580
120,009
165,710
165,496
224,270
299,671
80,060
111,069
155,240
85,436
113,201
144,431
71,356
96,236
115,935
71,356
96,236
115,935
31,630
41,268
59,371
31,630
41,268
59,371
West
443,492
144,476
37,878
106,598
181,236
92,885
88,351
77,174
77,174
40,606
40,606
Estimated Completes @ 50% Incidence 15 minute Study
TOTAL
North East Mid-West
South
West
103,750
20,231
26,327
35,018
22,175
33,537
6,807
8,238
11,269
7,224
9,142
2,028
2,237
2,984
1,894
24,395
4,779
6,000
8,286
5,330
43,534
8,275
11,214
14,984
9,062
21,963
4,003
5,553
7,762
4,644
21,571
4,272
5,660
7,222
4,418
18,035
3,568
4,812
5,797
3,859
18,035
3,568
4,812
5,797
3,859
8,644
1,582
2,063
2,969
2,030
8,644
1,582
2,063
2,969
2,030

56. Expertise in Mobile Data Collection
The Leader in Mobile Engagement For Research
Currently reaching and engaging mobile users in 12 key ways:
 Push notification
 Geo fencing
 Geo targeting
 Behavioral targeting based on usage of:
 apps, content, music, camera, browser,
phone and text
 Venue trigger (foursquare integration)
 QR code trigger
 Diary studies - calendar trigger
 Self-opted via short code SMS
 Interviewer mode for F2F / CAPI
 Email invites
 SMS invites
 In-app intercepts & mobile web intercept
56
© 2012 Research Now

57. http://www.linkedin.com/company/userzoom
https://www.facebook.com/userzoom
@userzoom
Questions & Answers
JD Buckley
Aaron Rager
Alfonso de la Nuez
Manager of
User Experience
Kelley Blue Book
Client Development
Sales Manager
Research Now
Arager@researchnow.com
214-365-3451
Co-Founder and
Co-CEO
UserZoom