SlideShare a Scribd company logo
1 of 43
Download to read offline
Maximizing Your Moderating Game!
Susan Mercer!
Senior Experience Researcher!
smercer@madpow.com!
@susanamercer!
What is Usability Testing?!

•  It is NOT:!
     •  Market Research!
     •  The same as a Focus Group!
     •  About user preferences!
     •  “Fluffy”!

•  It is:!
     •  A systematic way to observe user behavior!
     •  A method to discover usability issues!
     •  A ongoing field of scientific inquiry!
!!
2
Topics!

•  What is think-aloud?!
•  Your roles as a moderator!
•  5 rules of great moderating!
•  Keeping sharp!
!




3!
What is think-aloud?!
Think-aloud Protocol!

•    Most commonly-used usability testing method!
•    1 participant / 1 moderator!
•    Participant does tasks and thinks aloud!
•    Origins in Cognitive Psychology!

Original protocol:!
•  To understand short-term memory!
•  Moderator invisible observer!
•  Only comments: “Keep talking”, etc.!
•  Don’t interrupt short-term memory!
5!   Source: Ericsson & Simon, 1980!
Think-aloud Protocol - Problems!

•  Many practitioners feel the strict no-
     comments moderation is not ideal:!
     •  Makes participants feel uncomfortable!
     •  Feels “unnatural” not to respond!
     •  Participants may get stuck and not be able to
       continue!




6!
Think-aloud Protocols!
     Original Think-aloud!                     Usability Testing!
                                                         Study this…!
Study this…!

                …to understand this !



                  Short term
                                        …to understand this !
                  memory!



                                             System
                                             usability




7!
Modified Think-aloud Protocol!

•  Based on Conversational Theory!
•  Set up a conversation!
     •  Participant is main speaker!
     •  Use short, neutral response phrases (continuers) !
        •  “Mhmm”, “Uh-huh”, “And now…?” !
        •  Utter phrases like questions !

     •  Ask for clarifications by repeating a phrase!
             •  Participant: “That was odd…”!
             •  Moderator: “Odd?”!
     •  Provide hints if necessary !
     •  Neutrality is key!

8!   Source: Boren & Ramey, 2000!
The Reality of Think-aloud!

•  Many usability professionals get sloppy:!
     •  Ask leading questions!
        •  Participant: “I would share this article with friends.”!
        •  Moderator: “By printing it?”!

     •  Ask closed-ended questions!
        •    Moderator: “Would you print this?”!
     •  Focus on known problems for “ammunition”!
     •  Use biasing response phrases “Good job”, “Yes,
      that’s a problem.”!
!

9!   Source: Norgaard & Hornbaek, 2006!
The Reality of Think-aloud!

•  Current practice goes beyond neutrality and
      includes probing questions!
      •  Some evidence that probing may influence what
           usability problems are discovered!
      •    Evidence is only suggestive; more research is
           needed!




10!    Source: Krahmer & Ummelen, 2004!
What does this mean?!

•  Current moderating practices are not
      consistent!
•     Moderating is often taught through
      mentorship; so bad habits can easily be
      perpetuated!
•     Moderators should be willing to examine
      their practices and adapt if needed!

•  Therefore, we need to pay attention to our
      moderation and remain neutral.   !!
11!
Your roles as a moderator!
Your Roles (Yes, plural)!

•  Gracious Host!
•  Guide!
•  Neutral Observer!
•  Curious Learner!
•  Technical Support!
!




13!
Structure of a Usability Session!

                     Intro!   Pre-Test     Tasks!   Post-Test Wrap-Up!
                              Questions!            Questions!


Gracious Host!
Guide!
Curious Learner!
Neutral Observer!
Technical Support!




14!
Structure of a Usability Session!

                     Intro!   Pre-Test     Tasks!   Post-Test Wrap-Up!
                              Questions!            Questions!


Gracious Host!
Guide!
Curious Learner!
Neutral Observer!
Technical Support!    ?!          ?!         ?!        ?!        ?!



 Challenge: Navigate the Transitions Smoothly!


15!
5 Rules of Great Moderating!
5 Rules of Great Moderating!

1.    Know your goals!
2.    Stay neutral !
3.    Be responsible!
4.    Be confident!
5.    Be genuine!
!




17!
5 Rules of Great Moderating!

1.  Know your goals!
2.  Stay neutral !
3.  Be responsible!
4.  Be confident!
5.  Be genuine!
!




18!
1. Know Your Goals!

•  What are you trying to achieve?!
      •  Formative !
      •  Summative !
•  What topics are highest priority?!
!
These will influence!
      •  Where you focus!
      •  What you skim when time runs short!
      •  How you fine-tune your moderating style!

19!
5 Rules of Great Moderating!

1.  Know your goals!
2.  Stay neutral !
3.  Be responsible!
4.  Be confident!
5.  Be genuine!
!




20!
2. Stay Neutral!

•  Be Quiet!!
•  Don’t insinuate they gave a wrong answer!
    ✘  “Why did you do that?”!
    ✘  “What made you think to click that link?”!
•  Don’t put ideas into their heads!
    ✘  “Does entering your SSN make you feel uncomfortable?”!
    ü “What do you think about entering your SSN?”!
•  Use their words!
    •  If they call the dashboard a “chart area”, then call it a
        “chart area”!
•  Keep your tone naturally curious, not derisive!
      •  “What did you expect here?”!

21!
2. Stay Neutral!

•  Avoid closed-ended (Yes/No) questions !

•  Don’t start with a verb!
      ✘ Did you see that?!
      ✘ Was that difficult?!
      ✘ Is this frustrating?!




22!
2. Stay Neutral!

•  Use Open-ended questions!

•  Start with What, When, Where, Why, How!
      ü What would you change to make this easier?!
      ü When in this process would you be ready to buy?!
      ü Where did you expect to find that information?!
      ü Why did you say this was easy?!
      ü How does that error message make you feel?!


23!
2. Stay Neutral!

•  Use neutral, “continuing” response phrases!

•  Acknowledge that they gave feedback!
•  Don’t imply whether it is good or bad!
      ✘ “Oh”, “Hmm”, “Interesting” - Assessing!
      ✘ “OK”, “Yeah”, “That’s good” - Agreeing!
      ü “Uh huh”, “Mhmm”, “Tell me more” - Continuing!




24!   Source: Boren & Ramey, 2000
5 Rules of Great Moderating!

1.  Know your goals!
2.  Stay neutral !
3.  Be responsible!
4.  Be confident!
5.  Be genuine!
!




25!
3. Be Responsible!

You have responsibilities to:!
!
•  Participant!
•  Company!
•  Future Users !
•  Study Integrity!



26!
3. Be Responsible!

•  Informed Consent:!
      1.  Voluntary!
      2.  Comprehension!
      3.  Disclosure!
        •  The purpose of the study!
        •  Any reasonably foreseeable risks!
        •  Potential benefits !
        •  Notification of recording and observers!
        •  Confidentiality protections !
        •  Ability to withdraw at any time without penalty!




27!
3. Be Responsible!

Situations happen during testing.!
You may need to juggle your responsibilities.!
!
What are your priorities?!
!
•  Study Integrity!
•  Company!
•  Future Users!
•  Participant!
28!
What to do?!

If…!
•  Participant doesn’t want to be recorded!
•  You don’t have time to ask every question!
•  The participant rants about the website and
   calls it “crap”!




29!
5 Rules of Great Moderating!

1.  Know your goals!
2.  Stay neutral !
3.  Be responsible!
4.  Be confident!
5.  Be genuine!
!




30!
4. Be Confident!

•  Project confidence!
•  If you make a mistake, keep going!
•  Don’t over-apologize!
•  Pause if you need to collect your thoughts!
•  “Fake it until you make it”!




31!
5 Rules of Great Moderating!

1.  Know your goals!
2.  Stay neutral !
3.  Be responsible!
4.  Be confident!
5.  Be genuine!
!




32!
5. Be Genuine!

•  Don’t play the notes on
   the page…!
…play the music!
!
•  Don’t just read the
   questions from the page…!
…have a conversation!



33!
5. Be Genuine!

•  LISTEN!
      •  Listen and respond!

•  Stay in the moment!
      •  Don’t worry about past !
      •  Or the future!
      •  Create a conversation!
!


34!
5 Rules of Great Moderating!

1.    Know your goals!
2.    Stay neutral !
3.    Be responsible!
4.    Be confident!
5.    Be genuine!
!




35!
Keeping sharp!
Keep Improving!

1.  Identify your improvement areas!
      •  Watch your videos!
      •  Have colleagues give you feedback!




37!
Keep Improving!

2.  Learn from others!
      •  Watch others moderate!
      •  Be a participant!
      •  Listen to talk radio interviews!




38!
Keep Improving!

3.  Practice skills in everyday life!
      •  People watch (observe) in public!
      •  Be quiet and listen in everyday conversations!
!




39!
The most important thing is…!



PRACTICE!
•  No one is perfect the first time!
•  Do your best and continually improve!
•  You’ll be awesome in no time!!

40!
Questions?

41!
Thank You!
      (Slides will be posted to Slideshare..follow me on twitter for link)


 Susan Mercer!
 Senior Experience Researcher!
 smercer@madpow.com!
 @susanamercer!

42!
References / Photo Credits!
Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE
     Transactions on Professional Communication, 43 (3), 261-278.!
Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.!
Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.!
Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3),
     215-251.!
Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two
     verbal protocols for usability testing. IEEE Transactions on Professional Communication,
     47 (2), 105-117.!
Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An
     exploratory study of think-aloud testing. DIS 2006, 209-218.!
!
!
!
Slide 4: Samantha Louras Photography !
Slide 5: flickr: torgeaux!
Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian!
Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls!
Slide 32: flickr: bberburb!
Slide 34: flickr: Tulane Public Relations!
Slide 35: flickr: apdk, flickr: marktristan!
Slide 38: flickr: Raphael Quinet!
Slide 39: flickr: l-i-n-k!
Slide 40: flickr: loppear!
!
!



43!

More Related Content

What's hot

Krueger Focus Group Research
Krueger Focus Group ResearchKrueger Focus Group Research
Krueger Focus Group ResearchLaurenShiels1
 
HHUSA Interviewing Workshop Share
HHUSA Interviewing Workshop ShareHHUSA Interviewing Workshop Share
HHUSA Interviewing Workshop ShareRaymond Jurado
 
Communication workshop cmb_may2016_contentonly
Communication workshop cmb_may2016_contentonlyCommunication workshop cmb_may2016_contentonly
Communication workshop cmb_may2016_contentonlyHannah Knapp
 
User Interview Techniques
User Interview TechniquesUser Interview Techniques
User Interview TechniquesLiz Danzico
 
Effective Presentation & Communication Skills For Business Leaders
Effective Presentation & Communication Skills For Business LeadersEffective Presentation & Communication Skills For Business Leaders
Effective Presentation & Communication Skills For Business LeadersJennifer McClure
 
3. needs identification (tools)
3. needs identification (tools)3. needs identification (tools)
3. needs identification (tools)ELPowerpoints2013
 
Investigation Interview: Elicitation Techniques
Investigation Interview: Elicitation TechniquesInvestigation Interview: Elicitation Techniques
Investigation Interview: Elicitation TechniquesCase IQ
 
RSS 2012 Interview Techniques
RSS 2012 Interview TechniquesRSS 2012 Interview Techniques
RSS 2012 Interview TechniquesWesam Abuznadah
 
Media Training for Municipal Government Officials
Media Training for Municipal Government OfficialsMedia Training for Municipal Government Officials
Media Training for Municipal Government OfficialsJohn Chambers
 
Conversations and person centred approaches
Conversations and person centred approaches Conversations and person centred approaches
Conversations and person centred approaches Jon Ralphs
 
The feedback workshop
The feedback workshopThe feedback workshop
The feedback workshopLiz Keogh
 
Communication friday presentation 2
Communication friday presentation 2Communication friday presentation 2
Communication friday presentation 2rohit bohra
 
How to succeed in Difficult Conversations
How to succeed in Difficult ConversationsHow to succeed in Difficult Conversations
How to succeed in Difficult ConversationsThe Apprentiice
 

What's hot (20)

Now habit
Now habitNow habit
Now habit
 
Let's Network!!
Let's Network!!Let's Network!!
Let's Network!!
 
Krueger Focus Group Research
Krueger Focus Group ResearchKrueger Focus Group Research
Krueger Focus Group Research
 
Ouhk comm6005 lecture 6 effective interview skills
Ouhk comm6005 lecture 6   effective interview skillsOuhk comm6005 lecture 6   effective interview skills
Ouhk comm6005 lecture 6 effective interview skills
 
HHUSA Interviewing Workshop Share
HHUSA Interviewing Workshop ShareHHUSA Interviewing Workshop Share
HHUSA Interviewing Workshop Share
 
Communication workshop cmb_may2016_contentonly
Communication workshop cmb_may2016_contentonlyCommunication workshop cmb_may2016_contentonly
Communication workshop cmb_may2016_contentonly
 
User Interview Techniques
User Interview TechniquesUser Interview Techniques
User Interview Techniques
 
Effective Presentation & Communication Skills For Business Leaders
Effective Presentation & Communication Skills For Business LeadersEffective Presentation & Communication Skills For Business Leaders
Effective Presentation & Communication Skills For Business Leaders
 
3. needs identification (tools)
3. needs identification (tools)3. needs identification (tools)
3. needs identification (tools)
 
Interviewing
InterviewingInterviewing
Interviewing
 
Investigation Interview: Elicitation Techniques
Investigation Interview: Elicitation TechniquesInvestigation Interview: Elicitation Techniques
Investigation Interview: Elicitation Techniques
 
RSS 2012 Interview Techniques
RSS 2012 Interview TechniquesRSS 2012 Interview Techniques
RSS 2012 Interview Techniques
 
Media Training for Municipal Government Officials
Media Training for Municipal Government OfficialsMedia Training for Municipal Government Officials
Media Training for Municipal Government Officials
 
Conversations and person centred approaches
Conversations and person centred approaches Conversations and person centred approaches
Conversations and person centred approaches
 
The feedback workshop
The feedback workshopThe feedback workshop
The feedback workshop
 
Communication friday presentation 2
Communication friday presentation 2Communication friday presentation 2
Communication friday presentation 2
 
How to succeed in Difficult Conversations
How to succeed in Difficult ConversationsHow to succeed in Difficult Conversations
How to succeed in Difficult Conversations
 
Public speaking made easy
Public speaking made easyPublic speaking made easy
Public speaking made easy
 
Job Interview
Job InterviewJob Interview
Job Interview
 
Conflict Resolution 07302015
Conflict Resolution 07302015Conflict Resolution 07302015
Conflict Resolution 07302015
 

Viewers also liked

My experience with english
My experience with englishMy experience with english
My experience with englishmariocp3
 
Universidad de san bueneventura –cartagena2
Universidad de san bueneventura –cartagena2Universidad de san bueneventura –cartagena2
Universidad de san bueneventura –cartagena2stefany
 
93867681 ninos-mayo
93867681 ninos-mayo93867681 ninos-mayo
93867681 ninos-mayotony_mello
 
Darlena pagan dot com power point short version
Darlena pagan dot com power point short versionDarlena pagan dot com power point short version
Darlena pagan dot com power point short versionUSA Discussion Group
 
Компанія UDS
Компанія UDSКомпанія UDS
Компанія UDS12creator
 
Silabus Pengabis Mr. Hidayat
Silabus Pengabis Mr. HidayatSilabus Pengabis Mr. Hidayat
Silabus Pengabis Mr. HidayatFeni_Triminarni
 
Yoselin barrera los lideres nacen o se hacen
Yoselin barrera los lideres nacen o se hacenYoselin barrera los lideres nacen o se hacen
Yoselin barrera los lideres nacen o se hacenYoselin Barrera
 
Proven Oilfield Cost Reduction & Environmental Results
Proven Oilfield Cost Reduction & Environmental ResultsProven Oilfield Cost Reduction & Environmental Results
Proven Oilfield Cost Reduction & Environmental ResultsBen Lucas
 
Agriculture and sustainable cultivation
Agriculture and sustainable cultivationAgriculture and sustainable cultivation
Agriculture and sustainable cultivationSnowPea Guh
 
New test
New testNew test
New testIQM123
 
Website usability ideas for business growth
Website usability ideas for business growthWebsite usability ideas for business growth
Website usability ideas for business growthJames Smith
 
사진 앨범
사진 앨범사진 앨범
사진 앨범net4you
 

Viewers also liked (20)

My experience with english
My experience with englishMy experience with english
My experience with english
 
Universidad de san bueneventura –cartagena2
Universidad de san bueneventura –cartagena2Universidad de san bueneventura –cartagena2
Universidad de san bueneventura –cartagena2
 
Tizen native application
Tizen native applicationTizen native application
Tizen native application
 
93867681 ninos-mayo
93867681 ninos-mayo93867681 ninos-mayo
93867681 ninos-mayo
 
Federico
FedericoFederico
Federico
 
Darlena pagan dot com power point short version
Darlena pagan dot com power point short versionDarlena pagan dot com power point short version
Darlena pagan dot com power point short version
 
Auryn
AurynAuryn
Auryn
 
Doma natural: Esteban Labari i Lucy Rees
Doma natural: Esteban Labari i Lucy ReesDoma natural: Esteban Labari i Lucy Rees
Doma natural: Esteban Labari i Lucy Rees
 
Компанія UDS
Компанія UDSКомпанія UDS
Компанія UDS
 
Sensorize FreeRehab
Sensorize FreeRehabSensorize FreeRehab
Sensorize FreeRehab
 
Silabus Pengabis Mr. Hidayat
Silabus Pengabis Mr. HidayatSilabus Pengabis Mr. Hidayat
Silabus Pengabis Mr. Hidayat
 
Yoselin barrera los lideres nacen o se hacen
Yoselin barrera los lideres nacen o se hacenYoselin barrera los lideres nacen o se hacen
Yoselin barrera los lideres nacen o se hacen
 
Stephen presentation
Stephen presentationStephen presentation
Stephen presentation
 
Proven Oilfield Cost Reduction & Environmental Results
Proven Oilfield Cost Reduction & Environmental ResultsProven Oilfield Cost Reduction & Environmental Results
Proven Oilfield Cost Reduction & Environmental Results
 
One pressure too many?
One pressure too many?One pressure too many?
One pressure too many?
 
Agriculture and sustainable cultivation
Agriculture and sustainable cultivationAgriculture and sustainable cultivation
Agriculture and sustainable cultivation
 
New test
New testNew test
New test
 
Website usability ideas for business growth
Website usability ideas for business growthWebsite usability ideas for business growth
Website usability ideas for business growth
 
사진 앨범
사진 앨범사진 앨범
사진 앨범
 
Secrets to Building a Stellar Research Program
Secrets to Building a Stellar Research ProgramSecrets to Building a Stellar Research Program
Secrets to Building a Stellar Research Program
 

Similar to Maximizing your moderating_game

Be Amazing: 5 Rules for Great Presentations
Be Amazing: 5 Rules for Great PresentationsBe Amazing: 5 Rules for Great Presentations
Be Amazing: 5 Rules for Great Presentationsjcasimir
 
Doing Customer Interviews Right
Doing Customer Interviews RightDoing Customer Interviews Right
Doing Customer Interviews RightJosie Scott
 
Soc semmcte
Soc semmcteSoc semmcte
Soc semmcteaj6785
 
Seek #DIGNC Digital Non-Conference Conversation starter
Seek #DIGNC Digital Non-Conference Conversation starterSeek #DIGNC Digital Non-Conference Conversation starter
Seek #DIGNC Digital Non-Conference Conversation starterSEEK Company
 
Seek #CincySM Conversation Starter
Seek #CincySM Conversation StarterSeek #CincySM Conversation Starter
Seek #CincySM Conversation StarterSEEK Company
 
Oak Hill's Sr Projects Res Step By Step PPT#3
Oak Hill's Sr Projects Res Step By Step PPT#3Oak Hill's Sr Projects Res Step By Step PPT#3
Oak Hill's Sr Projects Res Step By Step PPT#3Jeremy Young
 
Instrumentation in social science studies
Instrumentation in social science studies Instrumentation in social science studies
Instrumentation in social science studies Ashkan Toosi
 
Busniess communication
Busniess communicationBusniess communication
Busniess communicationoozair
 
Jim DeLorenzo: Public Speaking-Media Training Presentation July 2012
Jim DeLorenzo:  Public Speaking-Media Training Presentation July 2012Jim DeLorenzo:  Public Speaking-Media Training Presentation July 2012
Jim DeLorenzo: Public Speaking-Media Training Presentation July 2012Jim DeLorenzo
 
How to Effectively Lead Focus Groups: Presented at ProductTank Toronto
How to Effectively Lead Focus Groups: Presented at ProductTank TorontoHow to Effectively Lead Focus Groups: Presented at ProductTank Toronto
How to Effectively Lead Focus Groups: Presented at ProductTank TorontoTremis Skeete
 
Presentation skills for managers
Presentation skills for managersPresentation skills for managers
Presentation skills for managersClean Agent Sdn Bhd
 
Active listeningandtalkingtousers
Active listeningandtalkingtousersActive listeningandtalkingtousers
Active listeningandtalkingtousersJoan Wortman
 
Audio Reporting & Interviewing
Audio Reporting & InterviewingAudio Reporting & Interviewing
Audio Reporting & InterviewingCorinne Weisgerber
 

Similar to Maximizing your moderating_game (20)

Think epic be epic
Think epic be epicThink epic be epic
Think epic be epic
 
Be Amazing: 5 Rules for Great Presentations
Be Amazing: 5 Rules for Great PresentationsBe Amazing: 5 Rules for Great Presentations
Be Amazing: 5 Rules for Great Presentations
 
Public speaking
Public speakingPublic speaking
Public speaking
 
Doing Customer Interviews Right
Doing Customer Interviews RightDoing Customer Interviews Right
Doing Customer Interviews Right
 
Soc semmcte
Soc semmcteSoc semmcte
Soc semmcte
 
Seek #DIGNC Digital Non-Conference Conversation starter
Seek #DIGNC Digital Non-Conference Conversation starterSeek #DIGNC Digital Non-Conference Conversation starter
Seek #DIGNC Digital Non-Conference Conversation starter
 
Seek #CincySM Conversation Starter
Seek #CincySM Conversation StarterSeek #CincySM Conversation Starter
Seek #CincySM Conversation Starter
 
Oak Hill's Sr Projects Res Step By Step PPT#3
Oak Hill's Sr Projects Res Step By Step PPT#3Oak Hill's Sr Projects Res Step By Step PPT#3
Oak Hill's Sr Projects Res Step By Step PPT#3
 
Instrumentation in social science studies
Instrumentation in social science studies Instrumentation in social science studies
Instrumentation in social science studies
 
Bus370
Bus370Bus370
Bus370
 
Bus370
Bus370Bus370
Bus370
 
Busniess communication
Busniess communicationBusniess communication
Busniess communication
 
Jim DeLorenzo: Public Speaking-Media Training Presentation July 2012
Jim DeLorenzo:  Public Speaking-Media Training Presentation July 2012Jim DeLorenzo:  Public Speaking-Media Training Presentation July 2012
Jim DeLorenzo: Public Speaking-Media Training Presentation July 2012
 
Cesif week 2
Cesif week 2Cesif week 2
Cesif week 2
 
How to Effectively Lead Focus Groups: Presented at ProductTank Toronto
How to Effectively Lead Focus Groups: Presented at ProductTank TorontoHow to Effectively Lead Focus Groups: Presented at ProductTank Toronto
How to Effectively Lead Focus Groups: Presented at ProductTank Toronto
 
21tips of good speaker
21tips of good speaker21tips of good speaker
21tips of good speaker
 
Presentation skills for managers
Presentation skills for managersPresentation skills for managers
Presentation skills for managers
 
Active listeningandtalkingtousers
Active listeningandtalkingtousersActive listeningandtalkingtousers
Active listeningandtalkingtousers
 
569326.pptx
569326.pptx569326.pptx
569326.pptx
 
Audio Reporting & Interviewing
Audio Reporting & InterviewingAudio Reporting & Interviewing
Audio Reporting & Interviewing
 

Recently uploaded

Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024
Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024
Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024Ted Drake
 
AMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara Rakovska
AMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara RakovskaAMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara Rakovska
AMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara RakovskaBarusRa
 
Unlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdf
Unlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdfUnlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdf
Unlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdfIBM
 
WCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptx
WCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptxWCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptx
WCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptxHasan S
 
The future of UX design support tools - talk Paris March 2024
The future of UX design support tools - talk Paris March 2024The future of UX design support tools - talk Paris March 2024
The future of UX design support tools - talk Paris March 2024Alan Dix
 
Math Group 3 Presentation OLOLOLOLILOOLLOLOL
Math Group 3 Presentation OLOLOLOLILOOLLOLOLMath Group 3 Presentation OLOLOLOLILOOLLOLOL
Math Group 3 Presentation OLOLOLOLILOOLLOLOLkenzukiri
 
High-Quality Faux Embroidery Services | Cre8iveSkill
High-Quality Faux Embroidery Services | Cre8iveSkillHigh-Quality Faux Embroidery Services | Cre8iveSkill
High-Quality Faux Embroidery Services | Cre8iveSkillCre8iveskill
 
Create Funeral Invites Online @ feedvu.com
Create Funeral Invites Online @ feedvu.comCreate Funeral Invites Online @ feedvu.com
Create Funeral Invites Online @ feedvu.comjakyjhon00
 
Models of Disability - an overview by Marno Retief & Rantoa Letšosa
Models of Disability - an overview by Marno Retief & Rantoa LetšosaModels of Disability - an overview by Marno Retief & Rantoa Letšosa
Models of Disability - an overview by Marno Retief & Rantoa Letšosaannemarleenolthof1
 
UI UX Process for SaaS Product Design Success
UI UX Process for SaaS Product Design SuccessUI UX Process for SaaS Product Design Success
UI UX Process for SaaS Product Design SuccessThink 360 Studio
 
Embroidery design from embroidery magazine
Embroidery design from embroidery magazineEmbroidery design from embroidery magazine
Embroidery design from embroidery magazineRivanEleraki
 
How to use Ai for UX UI Design | ChatGPT
How to use Ai for UX UI Design | ChatGPTHow to use Ai for UX UI Design | ChatGPT
How to use Ai for UX UI Design | ChatGPTThink 360 Studio
 
Khushi sharma undergraduate portfolio...
Khushi sharma undergraduate portfolio...Khushi sharma undergraduate portfolio...
Khushi sharma undergraduate portfolio...khushisharma298853
 
UX Conference on UX Research Trends in 2024
UX Conference on UX Research Trends in 2024UX Conference on UX Research Trends in 2024
UX Conference on UX Research Trends in 2024mikailaoh
 
LRFD Bridge Design Specifications-AASHTO (2014).pdf
LRFD Bridge Design Specifications-AASHTO (2014).pdfLRFD Bridge Design Specifications-AASHTO (2014).pdf
LRFD Bridge Design Specifications-AASHTO (2014).pdfHctorFranciscoSnchez1
 
Designing for privacy: 3 essential UX habits for product teams
Designing for privacy: 3 essential UX habits for product teamsDesigning for privacy: 3 essential UX habits for product teams
Designing for privacy: 3 essential UX habits for product teamsBlock Party
 
Construction Documents Checklist before Construction
Construction Documents Checklist before ConstructionConstruction Documents Checklist before Construction
Construction Documents Checklist before ConstructionResDraft
 
Design mental models for managing large-scale dbt projects. March 21, 2024 in...
Design mental models for managing large-scale dbt projects. March 21, 2024 in...Design mental models for managing large-scale dbt projects. March 21, 2024 in...
Design mental models for managing large-scale dbt projects. March 21, 2024 in...Ed Orozco
 

Recently uploaded (18)

Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024
Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024
Introduce Trauma-Informed Design to Your Organization - CSUN ATC 2024
 
AMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara Rakovska
AMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara RakovskaAMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara Rakovska
AMBER GRAIN EMBROIDERY | Growing folklore elements | Barbara Rakovska
 
Unlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdf
Unlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdfUnlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdf
Unlocking Conversion_ The Art of Turning Visitors into Loyal Customers.pdf
 
WCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptx
WCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptxWCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptx
WCM Branding Agency | 210519 - Portfolio Review (F&B) -s.pptx
 
The future of UX design support tools - talk Paris March 2024
The future of UX design support tools - talk Paris March 2024The future of UX design support tools - talk Paris March 2024
The future of UX design support tools - talk Paris March 2024
 
Math Group 3 Presentation OLOLOLOLILOOLLOLOL
Math Group 3 Presentation OLOLOLOLILOOLLOLOLMath Group 3 Presentation OLOLOLOLILOOLLOLOL
Math Group 3 Presentation OLOLOLOLILOOLLOLOL
 
High-Quality Faux Embroidery Services | Cre8iveSkill
High-Quality Faux Embroidery Services | Cre8iveSkillHigh-Quality Faux Embroidery Services | Cre8iveSkill
High-Quality Faux Embroidery Services | Cre8iveSkill
 
Create Funeral Invites Online @ feedvu.com
Create Funeral Invites Online @ feedvu.comCreate Funeral Invites Online @ feedvu.com
Create Funeral Invites Online @ feedvu.com
 
Models of Disability - an overview by Marno Retief & Rantoa Letšosa
Models of Disability - an overview by Marno Retief & Rantoa LetšosaModels of Disability - an overview by Marno Retief & Rantoa Letšosa
Models of Disability - an overview by Marno Retief & Rantoa Letšosa
 
UI UX Process for SaaS Product Design Success
UI UX Process for SaaS Product Design SuccessUI UX Process for SaaS Product Design Success
UI UX Process for SaaS Product Design Success
 
Embroidery design from embroidery magazine
Embroidery design from embroidery magazineEmbroidery design from embroidery magazine
Embroidery design from embroidery magazine
 
How to use Ai for UX UI Design | ChatGPT
How to use Ai for UX UI Design | ChatGPTHow to use Ai for UX UI Design | ChatGPT
How to use Ai for UX UI Design | ChatGPT
 
Khushi sharma undergraduate portfolio...
Khushi sharma undergraduate portfolio...Khushi sharma undergraduate portfolio...
Khushi sharma undergraduate portfolio...
 
UX Conference on UX Research Trends in 2024
UX Conference on UX Research Trends in 2024UX Conference on UX Research Trends in 2024
UX Conference on UX Research Trends in 2024
 
LRFD Bridge Design Specifications-AASHTO (2014).pdf
LRFD Bridge Design Specifications-AASHTO (2014).pdfLRFD Bridge Design Specifications-AASHTO (2014).pdf
LRFD Bridge Design Specifications-AASHTO (2014).pdf
 
Designing for privacy: 3 essential UX habits for product teams
Designing for privacy: 3 essential UX habits for product teamsDesigning for privacy: 3 essential UX habits for product teams
Designing for privacy: 3 essential UX habits for product teams
 
Construction Documents Checklist before Construction
Construction Documents Checklist before ConstructionConstruction Documents Checklist before Construction
Construction Documents Checklist before Construction
 
Design mental models for managing large-scale dbt projects. March 21, 2024 in...
Design mental models for managing large-scale dbt projects. March 21, 2024 in...Design mental models for managing large-scale dbt projects. March 21, 2024 in...
Design mental models for managing large-scale dbt projects. March 21, 2024 in...
 

Maximizing your moderating_game

  • 1. Maximizing Your Moderating Game! Susan Mercer! Senior Experience Researcher! smercer@madpow.com! @susanamercer!
  • 2. What is Usability Testing?! •  It is NOT:! •  Market Research! •  The same as a Focus Group! •  About user preferences! •  “Fluffy”! •  It is:! •  A systematic way to observe user behavior! •  A method to discover usability issues! •  A ongoing field of scientific inquiry! !! 2
  • 3. Topics! •  What is think-aloud?! •  Your roles as a moderator! •  5 rules of great moderating! •  Keeping sharp! ! 3!
  • 5. Think-aloud Protocol! •  Most commonly-used usability testing method! •  1 participant / 1 moderator! •  Participant does tasks and thinks aloud! •  Origins in Cognitive Psychology! Original protocol:! •  To understand short-term memory! •  Moderator invisible observer! •  Only comments: “Keep talking”, etc.! •  Don’t interrupt short-term memory! 5! Source: Ericsson & Simon, 1980!
  • 6. Think-aloud Protocol - Problems! •  Many practitioners feel the strict no- comments moderation is not ideal:! •  Makes participants feel uncomfortable! •  Feels “unnatural” not to respond! •  Participants may get stuck and not be able to continue! 6!
  • 7. Think-aloud Protocols! Original Think-aloud! Usability Testing! Study this…! Study this…! …to understand this ! Short term …to understand this ! memory! System usability 7!
  • 8. Modified Think-aloud Protocol! •  Based on Conversational Theory! •  Set up a conversation! •  Participant is main speaker! •  Use short, neutral response phrases (continuers) ! •  “Mhmm”, “Uh-huh”, “And now…?” ! •  Utter phrases like questions ! •  Ask for clarifications by repeating a phrase! •  Participant: “That was odd…”! •  Moderator: “Odd?”! •  Provide hints if necessary ! •  Neutrality is key! 8! Source: Boren & Ramey, 2000!
  • 9. The Reality of Think-aloud! •  Many usability professionals get sloppy:! •  Ask leading questions! •  Participant: “I would share this article with friends.”! •  Moderator: “By printing it?”! •  Ask closed-ended questions! •  Moderator: “Would you print this?”! •  Focus on known problems for “ammunition”! •  Use biasing response phrases “Good job”, “Yes, that’s a problem.”! ! 9! Source: Norgaard & Hornbaek, 2006!
  • 10. The Reality of Think-aloud! •  Current practice goes beyond neutrality and includes probing questions! •  Some evidence that probing may influence what usability problems are discovered! •  Evidence is only suggestive; more research is needed! 10! Source: Krahmer & Ummelen, 2004!
  • 11. What does this mean?! •  Current moderating practices are not consistent! •  Moderating is often taught through mentorship; so bad habits can easily be perpetuated! •  Moderators should be willing to examine their practices and adapt if needed! •  Therefore, we need to pay attention to our moderation and remain neutral. !! 11!
  • 12. Your roles as a moderator!
  • 13. Your Roles (Yes, plural)! •  Gracious Host! •  Guide! •  Neutral Observer! •  Curious Learner! •  Technical Support! ! 13!
  • 14. Structure of a Usability Session! Intro! Pre-Test Tasks! Post-Test Wrap-Up! Questions! Questions! Gracious Host! Guide! Curious Learner! Neutral Observer! Technical Support! 14!
  • 15. Structure of a Usability Session! Intro! Pre-Test Tasks! Post-Test Wrap-Up! Questions! Questions! Gracious Host! Guide! Curious Learner! Neutral Observer! Technical Support! ?! ?! ?! ?! ?! Challenge: Navigate the Transitions Smoothly! 15!
  • 16. 5 Rules of Great Moderating!
  • 17. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 17!
  • 18. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 18!
  • 19. 1. Know Your Goals! •  What are you trying to achieve?! •  Formative ! •  Summative ! •  What topics are highest priority?! ! These will influence! •  Where you focus! •  What you skim when time runs short! •  How you fine-tune your moderating style! 19!
  • 20. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 20!
  • 21. 2. Stay Neutral! •  Be Quiet!! •  Don’t insinuate they gave a wrong answer! ✘  “Why did you do that?”! ✘  “What made you think to click that link?”! •  Don’t put ideas into their heads! ✘  “Does entering your SSN make you feel uncomfortable?”! ü “What do you think about entering your SSN?”! •  Use their words! •  If they call the dashboard a “chart area”, then call it a “chart area”! •  Keep your tone naturally curious, not derisive! •  “What did you expect here?”! 21!
  • 22. 2. Stay Neutral! •  Avoid closed-ended (Yes/No) questions ! •  Don’t start with a verb! ✘ Did you see that?! ✘ Was that difficult?! ✘ Is this frustrating?! 22!
  • 23. 2. Stay Neutral! •  Use Open-ended questions! •  Start with What, When, Where, Why, How! ü What would you change to make this easier?! ü When in this process would you be ready to buy?! ü Where did you expect to find that information?! ü Why did you say this was easy?! ü How does that error message make you feel?! 23!
  • 24. 2. Stay Neutral! •  Use neutral, “continuing” response phrases! •  Acknowledge that they gave feedback! •  Don’t imply whether it is good or bad! ✘ “Oh”, “Hmm”, “Interesting” - Assessing! ✘ “OK”, “Yeah”, “That’s good” - Agreeing! ü “Uh huh”, “Mhmm”, “Tell me more” - Continuing! 24! Source: Boren & Ramey, 2000
  • 25. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 25!
  • 26. 3. Be Responsible! You have responsibilities to:! ! •  Participant! •  Company! •  Future Users ! •  Study Integrity! 26!
  • 27. 3. Be Responsible! •  Informed Consent:! 1.  Voluntary! 2.  Comprehension! 3.  Disclosure! •  The purpose of the study! •  Any reasonably foreseeable risks! •  Potential benefits ! •  Notification of recording and observers! •  Confidentiality protections ! •  Ability to withdraw at any time without penalty! 27!
  • 28. 3. Be Responsible! Situations happen during testing.! You may need to juggle your responsibilities.! ! What are your priorities?! ! •  Study Integrity! •  Company! •  Future Users! •  Participant! 28!
  • 29. What to do?! If…! •  Participant doesn’t want to be recorded! •  You don’t have time to ask every question! •  The participant rants about the website and calls it “crap”! 29!
  • 30. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 30!
  • 31. 4. Be Confident! •  Project confidence! •  If you make a mistake, keep going! •  Don’t over-apologize! •  Pause if you need to collect your thoughts! •  “Fake it until you make it”! 31!
  • 32. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 32!
  • 33. 5. Be Genuine! •  Don’t play the notes on the page…! …play the music! ! •  Don’t just read the questions from the page…! …have a conversation! 33!
  • 34. 5. Be Genuine! •  LISTEN! •  Listen and respond! •  Stay in the moment! •  Don’t worry about past ! •  Or the future! •  Create a conversation! ! 34!
  • 35. 5 Rules of Great Moderating! 1.  Know your goals! 2.  Stay neutral ! 3.  Be responsible! 4.  Be confident! 5.  Be genuine! ! 35!
  • 37. Keep Improving! 1.  Identify your improvement areas! •  Watch your videos! •  Have colleagues give you feedback! 37!
  • 38. Keep Improving! 2.  Learn from others! •  Watch others moderate! •  Be a participant! •  Listen to talk radio interviews! 38!
  • 39. Keep Improving! 3.  Practice skills in everyday life! •  People watch (observe) in public! •  Be quiet and listen in everyday conversations! ! 39!
  • 40. The most important thing is…! PRACTICE! •  No one is perfect the first time! •  Do your best and continually improve! •  You’ll be awesome in no time!! 40!
  • 42. Thank You! (Slides will be posted to Slideshare..follow me on twitter for link) Susan Mercer! Senior Experience Researcher! smercer@madpow.com! @susanamercer! 42!
  • 43. References / Photo Credits! Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-278.! Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.! Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.! Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3), 215-251.! Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47 (2), 105-117.! Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An exploratory study of think-aloud testing. DIS 2006, 209-218.! ! ! ! Slide 4: Samantha Louras Photography ! Slide 5: flickr: torgeaux! Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian! Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls! Slide 32: flickr: bberburb! Slide 34: flickr: Tulane Public Relations! Slide 35: flickr: apdk, flickr: marktristan! Slide 38: flickr: Raphael Quinet! Slide 39: flickr: l-i-n-k! Slide 40: flickr: loppear! ! ! 43!