When your persona talks: Mitigating linguistic bias in voice interaction design

When your persona talks
@maryparks
Introduction
Linguistic bias
Design practices
Summary

Introduction
@maryparks
Greet your neighbor, the person to your left, to
your right, in front of you, or behind you. Introduce
yourselves. And if you happen to already know
each other, just go ahead and pretend you don’t
Stand up

Introduction
@maryparks
Sit down
Close your eyes and think about what your
neighbor sounded like. What did they say? How
did they sound? What did the sound of their voice
tell you about them? What about you yourself?
What did you say? How did you sound?

Introduction
@maryparks
This time, speak in an accent different from the
one your normally use. Use an accent from a
different region of your country, or if you have
more than one accent that you use in real life,
switch to the other one
Greet your
neighbor again

Introduction
@maryparks
What did
you notice?

@maryparks
Linguistic bias

Linguistic bias
@maryparks
Linguistic
variation
Vocabulary, pronunciation, prosody,
syntax, and other features of language
help create people’s identities, including
their regional and social group
affiliations. Language varies, but not
randomly—variation is driven by context
https://languagelog.ldc.upenn.edu/nll/?p=4676

Linguistic bias
@maryparks
Standard
language ideology
“...a bias toward an abstracted, idealized,
homogeneous spoken language which is imposed
and maintained by dominant bloc institutions and
which names as its model the written language,
but which is drawn primarily from the spoken
language of the upper middle class.”
Rosina Lippi-Green, English with an Accent; Language, ideology,
and discrimination in the United States, Routledge 1997, p 64

Linguistic bias
@maryparks
Losing a dialect,
changing an identity
When a dialect is stigmatized, people often lose
agency and experience a displacement of identity.
As one Interaction 20 attendee told me about their
original accent: “...at some point in high school or
college I worked (on my own) to change it as it
was perceived as low class, uneducated, or
uncouth....So now I have a strange unidentifiable
accent.”

Linguistic bias
@maryparks
https://twitter.com/mixedlinguist/status/1185218534810341377
“Hey y’all, this your girl Ebony. This is
the code switch thread—I wanna hear
how y’all sound with your regular voice
versus when you have to code switch or
your work voice—I wanna hear the
difference. And so I’ll go first. This is
how I sound when I’m not clocked in.
And this is how I sound when I am.
‘Good morning. Thank you for calling.
How can I help you? I don’t know what
else to say, but this is the voice that I
tend to use.’
And so now it’s y’all’s turn. I wanna hear
how do y’all sound different. Let me
know.”

Linguistic bias
@maryparks
Sandra Bland:
Talking While Black
In this Language Log post dated Aug
15, 2015, the linguists Nicole Holiday,
Rachel Burdin, and Joseph Tyler
examined how accent bias, in this case
negatively perceiving a person’s
intentions because of their accent, may
have contributed to misperceptions and
an escalation of violence in what was a
simple traffic stop
https://languagelog.ldc.upenn.edu/nll/?p=20746

Linguistic bias
@maryparks
The Accent Bias
in Britain project
“...what impact does someone’s accent
have on their opportunities and life
outcomes? Are people biased,
positively or negatively, towards
different accents? More specifically,
could accent bias affect someone’s
judgement in a professional setting?”
https://accentbiasbritain.org/

Linguistic bias
@maryparks
The Accent Bias
in Britain Project
“...accent bias is pervasive but, under certain
conditions, people in positions of power have the
capacity to resist this effect.”
https://accentbiasbritain.org/

Linguistic bias
@maryparks
Text in
speech technology
In automatic speech recognition, speech signals
are converted into text strings along with indicators
of the likelihood that the text string accurately
matches what the person really said. And when
there is speech output, whether human recordings
or a text-to-speech system, text strings are read
out loud, and they sound that way

Linguistic bias
@maryparks
Spontaneous
speech
Spontaneous speech, in contrast to text, is
information rich, complex, multilayered. Speech
and conversations are structured by intonation
units and the focus and flow of information through
sound. Speech and text strings are not the same
thing. Reading out loud and talking are not the
same thing

Linguistic bias
@maryparks
Education
background and bias
Team members working in speech, language, and
multimodal technology normally have completed
university or have advanced degrees. This
contrasts with the majority of people in the U.S for
example, who normally complete high school only.
And ~12% of people in the U.S. are proficient at
reading, which contrasts with the majority who
read at an intermediate or lower level

Linguistic bias
@maryparks
Team
language and culture
Companies, teams, and communities of practice
have their own cultures and shared language.
Members of these groups share language and
experiences that are normally distinct from those
of the people they are designing for. Most teams
are unaware of these differences and use
themselves, their language use and style, their
behaviors, or an arbitrary standard language
ideology, as a starting point and baseline

Linguistic bias
@maryparks
Performance
and usability
All these gaps result in data sets for modeling and
machine learning that have inherent bias. These
imbalanced data sets can result in a less than
acceptable performance of voice interfaces
generally, and for classes and groups of people in
particular

Linguistic bias
@maryparks
Performance
and usability
Moreover, the conceptual models behind the
systems may fit the behaviors and language use
and style of the cross-functional teams working on
the systems but risk not meeting the needs,
behaviors, or language use and style of the people
who will be using the system

Linguistic bias
@maryparks
Performance
and usability
And finally, the lack of diverse representation and
naturalness in the speech output of these systems
can lead to discomfort or fatigue and negatively
impact adoption and sustained use of voice
interfaces

@maryparks
Design practices

Design practices
@maryparks
Recipient
design
When someone talks with another person, they
spontaneously make decisions for the recipient
moment to moment. They have a model of who
they’re talking with and of what they think is
shared knowledge or lack thereof, including
location and temporal factors and anything else
that makes up the context of the conversation

Design practices
@maryparks
What makes
a conversation?
• Structure
• Identity
• Variation
• Relevance

Design practices
@maryparks
Conversations
unfold through context
Designers have to do what people do extremely
quickly every time they talk, but designers have to
do it at a much slower pace and at an adequate
level of detail to achieve objectives

Design practices
@maryparks
User
discovery
Go to where the end users are and observe them
in the environment where the interaction will take
place. Learn things like
• How they talk with one another or to themselves
• How their language and activities are tied to their identity
• How they move around and how context shapes and
constrains their language and behavior

Design practices
@maryparks
User
discovery
Use the information learned from user discovery to
start defining who the primary persona is that
you’re designing for
• Conversations are one-on-one
• Your primary persona is going to be talking to your UI
• You are not your primary persona
• Figure out who your primary persona is, and design for them

Design practices
@maryparks
Record,
transcribe, analyze
Speech goes quickly. When listening to
potential end users talk, or to any
speech really, don’t rely on real-time
notes; you’ll miss ~75% of the
information if you do. Transcription and
analysis mitigates against team bias
and provides a treasure trove of
information
http://www.linguistics.ucsb.edu/sites/secure.lsit.ucsb.edu.ling.d7/files/sitefiles/research/papers/Working%20Papers%20Vol%204.pdf

Design practices
@maryparks
Go outside
for UX research
Recruit participants from outside the company,
aiming for regional, social class, and linguistic
diversity. Perform studies outside the building,
preferably in the places where potential end-users
will be once the system is deployed

Design practices
@maryparks
Avoid
priming
One of the greater sources of priming in
voice UX research is the moderator,
whose social class standing might differ
significantly from that of participants
(photo from NYTimes article, “Why
High-Class People Get Away with
Incompetence”)
https://www.nytimes.com/2019/05/20/science/social-class-confidence.html

Design practices
@maryparks
Avoid
priming
Avoid talking to participants as much as possible,
but at the same time don't be so remote that they
feel awkward or under a microscope. Leave them
alone on tasks, in order to replicate the context of
use and give them time to complete tasks on their
own

Design practices
@maryparks
Avoid
priming
Provide cartoons or written instructions. Cartoons
and other visuals help set the scenario without
being overly prescriptive about behavior. All
instructions should be carefully crafted and
reviewed to make sure they lead the participants
on as little as possible

Design practices
@maryparks
Avoid
priming
Once tasks are complete, have participants fill out
written forms or questionnaires or perform think
aloud exercises via recording. Give them time to
be alone with the sounds of the interaction in their
memory without the influence of the moderator

Design practices
@maryparks
Create diverse
representations
These illustrations are from Designing
the Alexas and Siris of the World, by
Sean Mulholland (Ideo Design Blog,
Dec 7, 2017). The artists who made
them were from countries where Alexa
and Siri were not available. But on
hearing Alexa and Siri US English voice
recordings, these are what they
imagined Siri and Alexa to look like
https://www.ideo.com/blog/designing-the-siris-and-alexas-of-the-world

Design practices
@maryparks
Create diverse
representations
Talking is very intimate and personal, particularly
in the home. But as Sean Mulholland asks about
voice interfaces, “what kind of cultural pressures
are they carrying into the home?” People tend to
accommodate other speakers they are talking
with. When the voice UI is not speaking with the
person in a way that is more in line with the
person’s identity, then it potentially feels as though
the UI is demanding accommodation, is
exclusionary, and unworthy of trust

Design practices
@maryparks
Systematic
techniques
One final way to take bias out of design decisions
is to use systematic techniques. Though bias can
never be completely removed, the more it is
mitigated against, the more useful, usable,
inclusive, and successful the voice UI will be

Design practices
@maryparks
Define
constraints
One of the things that makes recipient design a
useful framework for voice interaction design is
that we can create a set of constraints that parallel
the types of decisions a person makes moment to
moment in a conversation

Design practices
@maryparks
Define
constraints
One of the constraints in the design is what user
discovery has shown should be expected in the
speech input. What are the types of things the
primary persona and others will say, and how will
they say it? And what has user discovery shown
that is needed in the character’s speech output.
These are practical constraints that greatly
mitigate against bias

Design practices
@maryparks
Define
constraints
Other constraints can be more conceptual—for
example something like this short set, in order of
importance (examples in parentheses are for
illustrative purposes only):
• A primary attribute of the character (sophisticated)
• High level design criteria (fast)
• The relative status of the primary persona to the character
(persona has lower status than the character)
• Level of expertise (character has greater expertise than
persona)
• Frequency of interaction (daily)
• Stylistics (casual)

Design practices
@maryparks
Brainstorm
Brainstorm a lot. It can involve any aspect of the
design, including experiences that may not be
technically feasible or don’t fit the current timeline.
Imagine a speech system that doesn’t require
push-to-talk or wake words for example. And when
it comes to language in the UI, whether for input or
output, brainstorm very large sets

Design practices
@maryparks
Brainstorm
Here’s an example of a made-up brainstorm for
expressing gratitude at one step in a flow. A real
brainstorm should have much much more
material:
• “Thanks for doing that”
• “Thanks so much”
• “Thank you”
• “You’re great”
• “That’s awesome”
• “Wow, thanks”
• “Oh thanks”
• “Thanks”
• “Nice”

Design practices
@maryparks
Evaluate
brainstormed ideas
The next step is to evaluate the brainstormed set
using the constraints. Arrange the constraints from
highest to lowest priority. Evaluate each solution
or response against each constraint. Try to
perform the evaluation from the perspective of the
primary persona you’re designing for. Later on
these types of evaluations can be validated
through user research

Design practices
@maryparks
Evaluate
brainstormed ideas
Sophisticated
character
Fast
experience (>1
syllable=bad)
Persona has
lower status
Character is
expert
They interact
daily
Style is casual
“Thanks for doing that” Y N N Y N N
“Thanks so much” Y N N N N N
“Thank you” N N N N N N
“You’re great” Y N Y N Y Y
“That’s awesome” Y N Y Y Y Y
“Wow, thanks” Y N Y Y N Y
“Oh thanks” Y N Y Y Y Y
“Thanks” Y Y Y Y Y Y
“Nice” Y Y Y Y N N

Design practices
@maryparks
Evaluate
brainstormed ideas
Sophisticated
character
Fast
experience (>1
syllable=bad)
Persona has
lower status
Character is
expert
They interact
daily
Style is casual
“Thanks for doing that” Y N (loses here) N Y N N
“Thanks so much” Y N (loses here) N N N N
“Thank you” N (loses here) N N N N N
“You’re great” Y N (loses here) Y N Y Y
“That’s awesome” Y N (loses here) Y Y Y Y
“Wow, thanks” Y N (loses here) Y Y N Y
“Oh thanks” Y N (loses here) Y Y Y Y
“Thanks” Y Y Y Y Y (wins here) Y
“Nice” Y Y Y Y N (loses here) N

Design practices
@maryparks
Evaluate
brainstormed ideas
• “Thank you” is out of the running on the constraint to
promote the perception of a sophisticated character—this
designer thinks it’s too plain; all others pass
• On the fast constraint, for simplicity measured here by
number of syllables, only “thanks” and “nice” have one
syllable, so they pass. All others are out of the running
• “Nice” finally loses on the daily basis constraint—it would be
odd to hear it frequently as a form of gratitude
• That leaves one option, “thanks,” as the optimal choice with
these constraints in this order of prioritization

Design practices
@maryparks
Evaluate
brainstormed ideas
Add or remove constraints, or change their priority,
and the results of the evaluation change.
Brainstorming and systematic evaluation such as
this help mitigate against personal, group, cultural,
and linguistic biases

@maryparks
Summary

Summary
@maryparks
Linguistic
bias
• Linguistic variation is natural and emerges from context
• Linguistic bias a) is based on arbitrary distinctions, b) has no
foundation in linguistic facts, and c) hurts individuals,
groups, and societies through harassment, discrimination,
and public policies
• Voice interaction design tends to be linguistically biased
because of a) a reliance on text strings in the underlying
technology, b) the differences in class and education
between the teams working on these systems and the
people they are designing for, and c) the nature of company
cultures and communities of practice
• The risks are experiences that perform less well than they
could and that fail to fit the diversity of human needs

Summary
@maryparks
Design
practices
• Identity, relevance, and recipient design are the foundation
of voice and multimodal experiences
• Techniques that help mitigate against linguistic bias in
design generally include a) user discovery, b) recording,
transcribing, and analyzing spontaneous discourse and
speech, c) going outside for research, d) going above and
beyond to avoid priming, d) creating diverse
representations, e) and using systematic techniques that
take the individual, group, cultural, and linguistic biases out
of design decisions

Mary Constance Parks
@maryparks
Thanks!

When your persona talks: Mitigating linguistic bias in voice interaction design

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to When your persona talks: Mitigating linguistic bias in voice interaction design

Similar to When your persona talks: Mitigating linguistic bias in voice interaction design (20)

Recently uploaded

Recently uploaded (20)

When your persona talks: Mitigating linguistic bias in voice interaction design