Learn about Tourangeau's Four Stage Questionnaire Response model and about cognitive interviewing of surveys.

1. @uxordie
Text fields and dropdowns and radios oh my!
USABILITY TESTING ONLINE SURVEYS (AND FORMS)
Eva Kaniasty
UXPA International Conference
@uxordie

2. @uxordie@uxordie
1. Why surveys?
Isn’t user research enough?
2. Why pre-test surveys?
Let me count the ways.
3. Four stage questionnaire model
The Cognitive Psych (Tourangeau)
4. The cognitive interview
Usability testing… just for surveys. (Willis)
5. Survey Pre-Testing Process
3 scenarios.
AGENDA? AGENDA.
2

3. @uxordie
WHY SURVEYS?

4. @uxordie@uxordie
• Validate qualitative research results
• Get feedback from a larger sample
• Build a participant database
• Bridge gap between quant and qual
• Build a strong case for change
• Kill features
WHY SURVEYS?
4
“Doesn’t everybody love a survey?”
• Execs
• Business People
• Marketing People
• Sales People

5. @uxordie@uxordie
WHY SURVEYS?
5
• Surveys are easier to do
• Surveys are faster to do
• Surveys are less biased
• Surveys are more objective
• Survey data is more valid
• Survey data is statistically
significant
Why are surveys better than user research?
THE TRUTH ABOUT SURVEYS?
• Survey methodology is
well-researched
• Best practices are well-
established
THE TRUTH ABOUT SURVEYS

6. @uxordie@uxordie
MIXED METHODS RESEARCH
6
• Full picture of user behavior
• Mitigate methodological weaknesses
• Understand the what and the why
• Make better decisions
aka Triangulation

7. @uxordie@uxordie
• Complex onboarding
• Complex customization
• Data-entry driven applications
• Multi-step data collection workflows
• Skills testing
WHAT ABOUT FORMS?
7
What kinds of forms benefit from testing?
• B2B eCommerce
• Healthcare
• Education
• Government

8. @uxordie
WHY TEST SURVEYS?

9. @uxordie@uxordie
WHY TEST SURVEYS?
9
ERROR TYPES
• Sampling
•Measurement
• Coverage
• Non-Response
“There are no stupid questions… or are there?”

10. @uxordie@uxordie
WHY TEST SURVEYS?
10
“Stop doing survey research.”
- Jared Spool
https://articles.uie.com/on_surveys
DO SURVEY SUCK?
• Bias is incredibly easy to introduce.
• Bad questions can yield misleading
answers.
• You can’t adjust a survey mid-stream.
• Surveys are easy to manipulate.
• Surveys are not a substitute for qualitative
research.
• Surveys are not a shortcut.
• Surveys lack context.
• Surveys are subjective.
“How many researchers does it take to write a bad survey question?”
“Surveys are the most dangerous research
tool—misunderstood and misused. They
frequently straddle the qualita>ve and
quan>ta>ve, and at their worst represent
the worst of both..”
- Erika Hall”

11. @uxordie
FOUR STAGE
QUESTIONNAIRE
MODEL

12. @uxordie@uxordie
1. Comprehension
Do you understand the question?
2. Retrieval
Do you know or remember the answer?
3. Judgement
Can you formulate a truthful, accurate response?
4. Response
Can you map your answer to the response options
provided?
FOUR STAGE QUESTIONNAIRE MODEL
12
Roger Tourangeau

13. @uxordie@uxordie
“How many times have you participated in an online discussion about photography in the
last year?”
0 | Once or twice | 3-5 times | 6-10 times | more than 10 times
FOUR STAGES
13
COGNITIVE PROCESS
“Well, I’ve commented on things on Facebook, but I don’t think that’s what they mean. I
also took an online class, which definitely included discussion, so that
counts.” (Comprehension)
“It was still cold when I took the class, so that’s within the last year.” (Retrieval)
“There were 12 classes, but I know I missed a couple of them.” (Judgement)
“Overall, I was pretty active, so let’s just say that it’s more than 10.” (Response)

14. @uxordie
EXTREME(!) EXAMPLE

15. @uxordie@uxordie
1. COMPREHENSION
15
“How many times have you seen a provider in the last year?”
What’s a
provider?
Does it mean
a doctor?
If so, does this
include
chiropractors
…therapists…
pharmacists?
Is this
calendar
year or last
12 months?

16. @uxordie@uxordie
2. RETRIEVAL
16
“How many times have you seen a primary care doctor in the last 12 months?”
I had a checkup
a while ago, but
it may be more
than a year at
this point.

17. @uxordie@uxordie
3.JUDGEMENT
17
“How many times have you seen a primary care doctor in the last 6 months?”
I guess it’s 0. I
probably should
be taking better
care of my health
though.
I did talk to her on
the phone about
a refill, so I’ll just
say 1.

18. @uxordie@uxordie
4. RESPONSE
18
“How many times have you spoken to your primary care doctor in the last 6 months?”
Through email
On the phone
At the office
Wait a second… my
doctor has this new
telemedicine thingy, so
I did have a live chat
as well. Should I list that
as part of email?
1
0
0

19. @uxordie@uxordie
Are we done with this question yet?
19
“How many times have you spoken to your primary care doctor in the last 6 months?”
Through email
On the phone
At the office
Wait researcher, you also
forgot to randomize the
answer options, but you
won’t notice this until you
watch a few people take
the survey.
1
0
0
Other 0
Enter all that apply:
time(s)
time(s)
time(s)
time(s)

20. @uxordie@uxordie
COGNITIVE BURDEN
20
Are 30 clear questions better than 20 difficult ones?
Perhaps
Yes
Maybe
Don’t know

21. @uxordie
TESTING SURVEYS

22. @uxordie@uxordie
“The administration of draft survey questions
while collecting additional verbal information about the survey
responses,
which is used to evaluate the quality of the response
or to help determine whether the question is generating the
information that its author intends.”
COGNITIVE INTERVIEWING
22
Beatty & Willis (2007)
What is cognitive interviewing?

23. @uxordie@uxordie
Think-Aloud
+
Cognitive probes
=
Cognitive Interview
USABILITY TESTING SURVEYS
23
How is cognitive interviewing different from usability testing?

24. @uxordie@uxordie
1. Comprehension/Interpretation
“What does the term ‘leaderboard’ mean to you?”
2. Paraphrasing
“Tell me in your own words what this is asking you.”
3. Recall
“How do you know that you used it 10 times in the past 3 months?”
4. Confidence Judgement
“How sure are you that you used it at least 10 times?”
5. Specific
“What information do you review when you use the leaderboard?”
6. General
“I noticed you changed your answer. Tell me what you were thinking.”
COGNITIVE PROBES
24
Gordon B. Willis
“How frequently do you use the leaderboard feature?”

25. @uxordie
COGNITIVE PROBES

26. @uxordie@uxordie
Q. “When was the last time you visited your mother to help her with her computer?”
COGNITIVE INTERVIEW
26
A. “I live with my mother actually, so I wouldn’t be visiting her.”
P. “OK, so do you remember the last time you helped her with her computer at
home?” (General)
A. “Yes… it was about a month ago.”
P. “How sure are you it was a month ago? (Confidence Judgement)
A. “I’m pretty sure, because it was the weekend of the big snow storm at the end of March.”
P. “Does anyone else help your mom with her computer?” (Specific)
A. “Not really. I don’t have any siblings, and my dad is just as bad at technology as she is.”

27. @uxordie@uxordie
Q. “How concerned are you about lead contamination in your community garden?”
COGNITIVE INTERVIEW
27
A. “… I don’t really know. This is the first time I’ve thought about it honestly. Now that you
brought it up though, I feel like I should be very concerned.”
P. “Are you familiar with the term ‘lead contamination’?” (Comprehension)
A. “I am… I know that lead paint was used in old buildings, so sometimes it can get in the
soil, but I’m not sure why this would affect a community garden if it’s not near a building.”
P. “You said that ‘you should be very concerned’… How sure are you that you’re very
concerned?” (Confidence Judgement)
A. I guess I just assumed that this wouldn’t be a problem in my community garden. So
maybe I’m not that concerned. Honestly, I don’t know.

28. @uxordie@uxordie
Q. “Did you go to the beach last weekend?”
COGNITIVE INTERVIEW
28
A. “I did actually. I went snorkeling.”
P. “Tell me about it.” (General)
A. “I was at a conference in Puerto Rico, and I got there a few days before.”
P. “Was this the weekend that just passed, or the weekend
before?” (Interpretation)
A. “It was Saturday, so two days ago.”
P. “Do you remember the last time you went to beach back home?” (Recall)
A. …

29. @uxordie@uxordie
Q. “How often do you use our website to book a dog sitter?”
COGNITIVE INTERVIEW
29
A. “OK… In the past, I’ve traveled a few times per year, but if it’s just a weekend my friend
would watch him, so maybe… 6 times per year… so that’s. ‘once every 2-3 months’…
P. “I noticed you hesitated before choosing your answer….” (General)
A. “Yeah… I was trying to figure out what it would be on average.”
P. “So… how sure are you that it’s ‘once every 2-3 months’?” (Confidence Judgement)
A. “Well, to be honest, my dog died a few months ago… so I haven’t used it since then,
obviously.”
Once a month or more / Once every 2-3 months / Every six months / Once a year / Less than
once a year

30. @uxordie
SURVEY PRE-TESTING

31. @uxordie@uxordie
• Exploratory interview
incorporating cognitive
probes.
MODERATED INTERVIEW
31
HOW USE IT TO EXPLORE
Comprehension | Retrieval |
Judgement
• Asking the right questions
• Understanding respondent
background knowledge
• Exposing assumptions
• Setting the right response
intervals
PRO TIP
• Doubles as a round of
qualitative research.

32. @uxordie@uxordie
• Observation and cognitive
interview of draft survey.
MODERATED DRY RUN
32
HOW USE IT TO EXPLORE
Comprehension | Retrieval | Judgement
| Response
• Terminology and interpretation
issues
• Fixing confusing response
categories
• Layout issues
PRO TIP
• Probe on response categories
as well as the questions.

33. @uxordie@uxordie
• Self-administration of survey
to hand-picked, sample
before official launch.
PILOT
33
HOW USE IT TO EXPLORE
Comprehension | Response
• Some terminology issues
• Missing response categories
• Layout issues
PRO TIP
• Include open-ended
responses for reporting
problems in each survey
section.

34. @uxordie@uxordie
ANY QUESTIONS?
34