Kristen Davis and Dick Horst from UserWorks presentation slides on the "Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not" from UXPA-DC Conference Redux 2013

1. Notetaker's Perspective During Usability
Testing:
Recognizing What's Important, What's Not
UXPA-DC Conference Redux 2013
October 25, 2013
Kristen Davis ● Dick Horst
kdavis@userworks.com
●
dhorst@userworks.com
1738 Elton Rd., Suite 138, Silver Spring, MD 20903
(301) 431-0500  www.userworks.com

2. Presentation Objectives

To explore ways to become a more efficient/effective notetaker
by:
•
•
•
•
•
Developing notetaking and test planning skills
Recognizing what’s important to note and what’s not
Understanding the pros and cons of various styles
Determining what metrics to capture and how to define them
Discussing a list of best practices and tips
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
2

3. Why is Effective Notetaking Important?

To improve the data analysis process by:
•
•
•
•
Capturing the appropriate data the first time
Making analyses more efficient
Increasing accuracy and completeness of data
Identifying trends across participants
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
3

4. Typical Context

One-on-one, moderated sessions (participant in-person or
remote)
 Moderator interacting with participant, notetaker elsewhere
 Participant attempting typical task scenarios at the direction of
the moderator
 Participant thinking aloud
Some usability test principles and best practices may apply to
taking notes in focus groups, user interviews, or ethnographic
settings, but not all
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
4

5. UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013

6. Misconceptions





No preparation needed
Just take notes
Anyone can do it
Fast typing is only key skill necessary
One-size-fits-all approach
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
6

7. Set Goals




Be familiar with the product under test
Understand overall study goals
Understand specific task goals
Set goals prior to data collection
•
•
Record appropriate data
Target type(s) of data
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
7

8. Notetaking Styles/Approaches



Determine appropriate style
•
•
•
Verbatim (court stenographer) – rarely ideal
High level interpretation only, not behavior or comments
Coding behaviors in addition to free form notes
Determine number of notetakers
•
•
•
•
Moderator is only notetaker – not ideal
Multiple notetakers looking for different things
Multiple notetakers covering the same things (redundancy)
Taking notes from playback of a video/audio recording (not live)
Determine data collection tools
•
•
Checklist (word processing or spreadsheet based, or paper-based)
Automated tool (e.g., Morae, Ovo, Silverback)
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
8

9. Data Collection



Define task completion rating scale
•
•
Participant performance to be categorized
Systematic and consistent scoring
Notetaker can quickly record
•
•
•
•
Task scoring (success vs failure)
Errors, wrong paths
Pre-identified issues of interest
Occasional verbatims
Data logging software records
•
•
•
Clickpaths
Time on task
Shorthand codes for behaviors or incidents
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
9

10. UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013

11. Study Context



Product: NIHSeniorHealth.gov website
Participants: Seniors who look for health information online
Study goals:
•
•
•

What navigation path do participants use when looking for specific
pieces of information?
Do participants notice and use the pagination?
Does the visual treatment used in left hand navigation clearly
indicate location within specific health topic?
Task goals:
 Are participants able to locate information on the 2nd page?
 Are the left hand navigation labels clear?

Task: “Why do people with diabetes need to check their feet?”
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
11

12. Video Example
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
12

13. UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013

14. Study Context




Product: HealthIT.gov website
Participants: Private physicians
Study goals:
•
•
•
Are navigation labels clear?
Does information architecture make sense?
What suggestions do participants have to improve content?
Task goal:
•
What navigation path do participants use when looking for specific
pieces of information?

Task: “Medical professionals and hospitals are encouraged to
participate in the Government EHR reimbursement programs
early to receive the maximum payment. Find out when the
Government will start or has started these reimbursement
payments.”
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
14

15. Video Example
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
15

16. UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013

17. Study Context



Product: Blood glucose meter
Participants: People with diabetes who use glucose meter
Rating scale for scoring task completion:
•
•
•
•

Completed with ease: Participant easily completed the task
Completed with minor difficulty: Participant struggled somewhat
to complete the task
(e.g., attempted more than two paths)
Completed with major difficulty: Participant significantly struggled
to complete the task
(e.g., attempted numerous paths and/or used customer service)
Failed to complete: Participant was unable to complete the task
Task: Change meter default settings
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
17

18. Video Example
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
18

19. UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013

20. Pitfalls






Participant says one thing and does another
Participant thinks they have been successful when they really
haven’t
Participant doesn’t want to look bad, so bluffs
Participant initially struggles, then succeeds, and speaks
highly of product
Participant blames self and rates product highly despite
disastrous task performance
Participant is overly chatty and goes off on tangents
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
20

21. Considerations







Being selective in information noted, without introducing biases
(seeing what you want/expect to see)
Noting participant behavior versus comments
Whether to try to capture participant’s clickpath, menu choices,
data entry
Whether to capture timing
Whether to use a shorthand “code” for observations
Whether to capture participant’s non-verbal behavior
Whether to note good clips for a highlights video
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
21

22. UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013

23. "Take-Aways" for Effective Notetaking (1 of 2)





Ensure the project team values role of notetaker
Choose someone with appropriate education and experience
Include notetaker in test planning discussions and project
debriefs
Establish project-specific notetaking procedures and coding
schemes
Be sure notetaker is thoroughly familiar with:
•
•
•
•
Product
Project goals
Task scenarios
Task goals
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
23

24. "Take-Aways" for Effective Notetaking (2 of 2)







Focus on what participant does, not just what they say
Just don’t note observations; interpret what you are seeing
Review your notes and fill in any gaps as soon as possible
after sessions
Consult with the team and client throughout project
Check available storage space on recording devices before
each session
Spot check recordings throughout study in-between sessions
Discuss findings with notetaker/moderator after each session,
if time allows
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
24

25. Questions/Comments
Kristen Davis
Dick Horst
UserWorks, Inc.
UserWorks, Inc.
Senior User Experience
Specialist/ Note-Taker
Extraordinaire
President/
Principal User Experience
Specialist
kdavis@userworks.com
dhorst@userworks.com
www.userworks.com
(301) 431-0500
UXPA-DC Conference Redux 2013
UserWorks, Inc.
October 25, 2013
25