Improve Your Customers' Experience By Listening to Unstructured Feedback

Improve Your Customers’
Experience By Listening to
Unstructured Feedback

Good experiences. On brand. On budget.

Table of Contents
Executive Summary .................................................................................................................................................................................3
Finding the Golden Nuggets ...................................................................................................................................................................4
Capitalizing on Fresh Insight ...................................................................................................................................................................5
Emerging Market Appraoches Tipping Point ..........................................................................................................................................8
Making the Business Case .......................................................................................................................................................................9
Smart Practices for Success ..................................................................................................................................................................10
About the Author and Sponsor .............................................................................................................................................................11
Appendix: Text Mining Fundamentals ..................................................................................................................................................12

Acknowledgments
This paper was made possible with the generous sponsorship of Island Data Corp. and in-depth interviews
with its customers, including Meredith Sime, associate director, Customer Experience, for AT&T’s U-verse;
Jill Trecker, guest loyalty manager for Garden Fresh Restaurant Corp.; and Devon Child, senior product
manager for Yahoo!.

We also sincerely thank CustomerThink community members for their input in an online survey about
managing unstructured customer feedback, which provided most of the statistical information included in
this paper. Those seeking more details on text mining are encouraged to read our key background
sources: The Text-Mining Handbook by Ronen Feldman and James Sanger (Cambridge University Press,
2007) and Machine Learning in Automated Text Classification by Fabrizio Sebastiani (ACM Computing
Surveys, Vol. 34, No. 1, March 2002).

©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • sales@kana.com • www.kana.com PAGE 2


Executive Summary
Customer-centric businesses must, above all else, be committed to listening to the voice
of customers and acting on that feedback. But, these days, that “voice” includes
unstructured text that can’t be handled with conventional data analysis.

If your business is like most, you hear from your customers all the time. They send you
email. They unload earfuls to your customer service reps. They write in comments on your
product surveys. They leave feedback on your web site.

You probably would like to find out what customers are saying, but in practice, you don’t
cull through the massive amount of feedback for a variety of reasons: You’re
understaffed, there’s too much of it and you have other fires to put out.
“This kind of feedback is
giving us what the Forward-looking business leaders are now recognizing that this unstructured feedback
customer wants to tell contains the key to amplify customer listening programs. And they recognize that the
us, not what we want to effort to mine their unstructured customer feedback is worth the rewards: early warnings
ask them.” about calamities, insights about pain-points that could cause customer defection and a
view into future customer trends.
- Survey respondent
In this paper, I’ll look at the business case for mining your unstructured customer
feedback, and I'll discuss the basics of text mining, looking in simple terms at how it
works. I’ll examine the key findings from a 2007 CustomerThink survey on how businesses
are managing unstructured customer feedback, and I’ll discuss what executives from
some successful businesses say about what text mining has meant for their strategy.
Finally, I’ll discuss the practices that will work best when you do take the plunge and
adopt a text-mining solution for your organization.

A 2007 CustomerThink survey of members found that business leaders are certainly
aware of the importance of mining unstructured customer feedback, but they also
recognize they are, by and large, not doing enough about it. Here are some key findings
from that survey:

The majority of businesses surveyed received unstructured customer feedback in the
forms of comments from surveys, unsolicited email and online feedback forms.
An overwhelming majority (80 percent) of businesses surveyed said managing
unstructured customer feedback would help improve their business performance.
More than half of the businesses surveyed said that managing unstructured customer
feedback is important to top executives, and that importance will only increase in the
future.



Yet few (less than 25 percent) of the businesses we surveyed felt that their companies
were doing an “excellent” or “very good” job of capturing, analyzing, reporting or
acting on unstructured customer feedback. Only 12 percent said they used a software
tool of any kind.
Managers surveyed felt that the two most important potential benefits to effectively
managing unstructured feedback were improving products and services and improving
customer loyalty.

One emerging best practice stands out, from both executive interviews and survey
respondents. For success in managing unstructured customer feedback, the top priority is
to act on the feedback received. Otherwise, why should customers invest their time in
telling you what they think?

Finding the Golden Nuggets
Text mining, like its geological counterpart, is sifting through vast amounts of debris to
find the gold. As Ronen Feldman and James Sanger, the authors of The Text-Mining
Handbook (Cambridge University Press, 2007) note, what we call “unstructured data” isn’t
completely unstructured. Text follows some basic tenets of natural language, and text
mining involves analyzing text to 1) determine what the original author was trying to say
or 2) learn something completely new.

Like data mining, the idea is simple, but what’s “under the hood” in text mining
applications is complex. One common technique is called “categorization,” which simply
means classifying text into categories. An example would be deciding whether customer
emails represented “happy,” “unhappy” or “neutral” customers, based on the types of
words used in those emails. You can imagine that each category should be handled
differently, and it might be useful to track the percentages in each category over time. For
more details on text mining methods, please see the Appendix or read our reference
sources.

Categorization has been used since the late 1960s in areas such as medicine and news
services, where there were large stores of documents, a strong desire or need to make
sense of it and the financial wherewithal to handle the expense of the computing power
necessary to crunch through the data.

Text Mining Comes of Age
In recent years, text mining algorithms have been more widely adopted by businesses,
thanks to “Moore’s Law,” which drives continued computer performance improvements.
But equally importantly, as you’ll learn in this paper, the setup and usage of text mining
systems have become much easier with the adoption of more “turnkey” implementation
approaches, including hosted or “on-demand” applications.

Executives at companies that are using text mining to analyze their unstructured
customer feedback are enthusiastic about the benefits. Early adopters usually are. For



them, not only did text mining help them meet the challenge they knew they had, but also
they found numerous other uses and benefits. Consider these illustrations:

In one case, a floral company was getting a high volume of returns of one particular
floral arrangement. The company thought that it was because customers were
disappointed in the look of the flowers. But after mining the unstructured customer
comments, they discovered that wasn’t true after all.
In another example, a telecommunications company introducing a new service was
able to analyze unstructured feedback over a period of time. This enabled executives
to see the nature of complaints progress from predominantly service-related to
encompassing more aspects of the customer experience as the product evolved in the
marketplace.
And in yet another case, restaurant chain executives were able to see that the new
buildings they were designing for additional restaurants had some critical issues.
Executives were able to make design changes before replicating the faulty design to
multiple locations.

In every one of those cases, executives are thankful for what they see as a robust tool for
looking into the minds and experiences of the customer. In general, they see text analytics
for customer feedback as a necessary part of a customer-centric enterprise. It’s not about
just automating a process to gain efficiency, although technology does enable text
analysis that is impractical to do manually. The real key is understanding what customers
are saying in their own words and then acting on that insight as quickly as possible.

Capitalizing on Fresh Insight
If you’ve eaten at a restaurant or shopped at a store lately, you’ve no doubt seen an
invitation on your receipt to call a toll-free number and respond to a survey about your
recent experience. Surveying customers is a popular way to quickly gauge the thoughts of
motivated customers, whether they’re interested in venting about a problem or praising
your business for a great product or outstanding service.

Surveys are now commonplace tools for companies to solicit customer feedback. In a
2007 CustomerThink survey, nearly 80 percent of respondents said they were conducting
customer surveys at least annually, up from 70 percent in 2004. It’s a safe bet that nearly
all of those surveys include some questions for customers to write in their comments.



In another survey in late
2007, we learned that,
sure enough, customer
surveys are a popular
source of unstructured
feedback. But unsolicited
emails and web site
forms are also popular
sources of customer
feedback text for the
majority of respondents.

Other sources cited
included call center agent
logs, transcripts from
recorded phone calls, text
messages, chat sessions
and posts on discussion
forums or blogs.

Now, the fact that this feedback is available does not mean that it is analyzed and
acted upon! It turns out that not enough are taking advantage of it, according to
our survey. When we asked people how effective their companies were managing
unstructured customer feedback, more said they were best at capturing customer
comments. But the average ranking they gave to that activity was only 2.66, when
a rating of 3.0 is “good.” And 46.2 percent rated their efforts as less effective than
that. It's clear that businesses have a long way to go.

What about taking advantage of the feedback? Well, the story there is somewhat worse,
with fewer than 25 percent of those surveyed rating their effectiveness as “excellent” or
“good” and more than 50 percent giving their organizations a “fair” or “poor” rating.

These findings are not at all surprising, given the emerging market for text analytics
solutions. Judging from the comments from early-adopter executives, however, there’s a
tremendous opportunity for a wide variety of companies to listen better to customers and
act on the insights gained.

Listening to Vocal Customers
If you give customers a chance, they’ll communicate with you in many ways. “It’s worth
noting our guests are very vocal,” says Jill Trecker, guest loyalty manager for Garden Fresh
Restaurant Corp., a chain of 104 buffet-style restaurants known as Souplantation in
Southern California and Sweet Tomatoes in other parts of the United States’ Sun Belt. The



restaurants boast 55-foot salad bars and a selection of eight soups, baked goods and
frozen yogurts—on an “all you care to eat” basis for one price.

In addition to inviting customers on restaurant receipts to take surveys, Garden Fresh
posts a toll-free phone number in its restaurants, inviting customer feedback. They listen
to the voice prompts, and they rate the restaurant experience. Guests are also given the
opportunity to record a 60-second message. A service management company handles the
“You’re one of our responses to the structured questions and transcribes the unstructured recordings to be
favorite places because of processed by Island Data.
the healthy benefits of
your food. BUT, we have a The company also has an online “eat club.” Six weeks after customers have joined the
few suggestions: I would club, they receive a short survey with structured questions and a place for free-form
like to see raspberry responses.
vinaigrette as a regular
The philosophy at Garden Fresh is that “no news is bad news,” but with so many vocal
dressing and see you
customers, the company was overwhelmed by the volume of customer feedback. The
improve the blue cheese
company receives about 10,000 pieces of unstructured feedback a month from the
dressing, which is much
different channels. Going through them manually would be, in Trecker’s words,
too ‘mayonnaisey.’”
“prohibitive.”

“There were things we knew that we were probably missing that we didn’t have a
structured question about,” Trecker says. Additionally, Trecker and company President
Kenneth J. Keane worried that if they were soliciting feedback and not responding to
important information in that feedback, they really were wasting people’s time—or
worse—risking angering customers.

Sifting Through Mountains
Garden Fresh contracted with Island Data in 2005 to find meaning in all that feedback. It
took the company about three weeks to set up shop, beginning by taking about 1,200
records of the 60-second transcribed comments and using them to train the system to
identify different categories of praise, criticism or requests.

Now the company has dashboards and receives monthly “praise” and “complaint”
reports, both showing trends and drilling down to individual customer comments. The
data is used mainly by the marketing team, but Trecker regularly shares reports with
Keane and the rest of the executive team.

They now can look at trends “month to month to month,” says Trecker, and drill down to
investigate trends. One “aha!” moment early on was in responding to customers’
demands for more soup varieties, including vegetarian soups. The company expanded the
number of soup varieties offered, and complaints about variety lessened.



The company also was finally
able to confirm why it
typically received more
complaints in February and
March, traditionally the
months when Garden Fresh
restaurants have a higher
volume of activity. The
hunch was that when
volumes increased, attention
to cleanliness dropped, as
employees struggled to keep up with the load. Sure enough, the Island Data reports
showed people were complaining about cleanliness. In another example, as mentioned
earlier, Garden Fresh found that customers were complaining that the new restaurant
design seemed too crowded and noisy. Because of its window into customer feedback,
executives had time to make changes before it was too late.

Emerging Market Approaches
Tipping Point
It’s early yet in the market for unstructured customer feedback solutions, according to
Lane Michel, executive vice president and managing director of the Marketing
Performance Management unit at Quaero Corp. The most active applications are in
customer experience assessment and monitoring; new or upgraded product feedback; call
center performance measurement and efficiency; and competitive intelligence gathering.
“Marketing executives
cannot continue to wait Call center executives, for example, are looking to a new generation of more effective
for the cumbersome tools to address customer experience and operational performance issues. “Marketing
360-degree customer executives cannot continue to wait for the cumbersome 360-degree customer databases
databases and slow and slow analytics to generate intelligence, early warnings and pragmatic segmentation
analytics to generate schemes,” Michel says. “My guess is that we should see a steep climb in text mining tool
intelligence, early sales in the next 18 months, with a shift out of early adopters predominately purchasing
warnings and pragmatic today.”
segmentation schemes.”
In most cases, companies were already gathering feedback. The decision to actually
- Lane Michel, Quaero implement a system came when the volume of that feedback reached a tipping point. One
retailer was receiving from 4,000 responses in a slow week to 150,000 to 200,000
responses a week during peak holiday seasons.

When creating customer feedback tools, “we took a pretty decent stab at anticipating
what our customers might want to tell us," said Meredith Sime, associate director,
Customer Experience, with AT&T’s U-verse, a new platform for services provided over



Internet Protocol. "But we did not initially invest enough resources in mining the wealth
of information from an unstructured format.” And, she said, those instincts were borne
out once they started analyzing the data. “We learned a tremendous amount of
information that helped us drive improvement plans, and it taught us to ask better
questions.”

Customers, Sime said, “raised issues we had not previously considered because the
products were new to us.” Like others I spoke with, Sime would love to get near a real-
time response in the future. “We would have a first-response mechanism if any quality of
service issues were to arise,” she said.

In fact, according to Scott Austin, Island Data’s chief technology officer, the company
found immediate reception to its “early-warning” alerts, notifications that there was a
spike in complaints about a product or service. When Island Data first rolled out the
service, it was temporarily offered to customers on a complimentary basis. “Within a
week and a half, [customers] said, ‘We absolutely have to have it. We will pay you money
to make it a permanent part of what we have.’”

Making the Business Case
We asked survey respondents to rate the potential benefits of effectively managing
unstructured customer feedback. “Improve products and services” and “improve
customer loyalty” tied for top ranking, followed closely by “gain insight hidden in
customer comments” and “listen to customers in a more natural way.”

However, when we asked people
how they would justify an
investment, the top three factors in
their rationale were: 1) improving
the customer experience, 2)
improving customer loyalty and 3)
increasing revenue. Collectively,
these factors accounted for 53
percent of the weighting. Other
factors included: improving
product/service quality, supporting
a customer-centric strategy,
improving decision-making and
reducing operational costs.

Considering the survey results along with executive interviews, it seems clear that the ROI
from managing unstructured customer feedback is geared toward improving customer



retention by focusing on trends, “listening” to customers and improving the customer
experience.

One of those early-warning alerts came in handy at Yahoo!, according to Devon Child,
senior product manager for the portal’s internal feedback platform. It highlighted a
problem that product managers knew was festering but didn’t consider a priority. “They
had it on their radars.” The alert came back, and they realized that it was more serious
than they previously thought. That’s a great “aha!” moment, but does it justify
implementing a text-mining solution?

“It’s hard to argue with How do you convince the rest of your company that managing unstructured feedback is
wanting to listen to the important? I asked executives this question, and they were almost puzzled by it. Perhaps
customers – our these are stellar firms when it comes to customer-centricity, but these executives did not
customers. Their opinions encounter internal resistance to text mining.
matter.”
Their companies wanted to know what customers were thinking, which, again, is why they
- Devon Child, Yahoo! were collecting the feedback in the first place. The question for each was: Do we do it in-
house or do we outsource? Executives I interviewed said money wasn't the issue. Their
aim was to do what was best for their companies—and their customers.

At U-verse, which performed an IT assessment of the costs, comparing outsourcing with
in-house text mining, Sime saw no resistance to the question of whether to analyze the
text. “We had all this data coming in, but we had no idea what to do with it.” The ability to
cull through all the unstructured feedback has “been a godsend to us,” Sime said. “Before,
the best we could do was paste the information in quotes.”

What gems of information are hidden within the customer feedback your organization
collects? “It’s hard to argue with wanting to listen to the customers—our customers,”
Child said. “Their opinions matter.”

Smart Practices for Success
It may be a bit premature to declare the definitive list of “best practices” for success with
projects to manage unstructured customer feedback. For now, let’s call them “smart
practices.” The chart below shows the opinions of CustomerThink members, based on our
2007 online survey. Given this input, along with my interviews with early adopters, here
are five practices that will help you succeed.
1. First and foremost, it’s critical that you are truly committed to act on the feedback you
receive, whether it’s unstructured. Nothing is more disillusioning to a customer than
taking the time to provide constructive praise or criticism and then have it be ignored.



2. Make sure the insights
are correct. “Best
algorithm” was not
rated highly in our
survey, but it's very
important to the results
you get. Garbage in
means garbage out. The
analysis-and-
categorization algorithm
must be reasonably
effective; otherwise,
“The value of the you’re acting on bad
information. That said,
unstructured feedback is there is probably a point
phenomenal.” of diminishing returns in
striving for the “perfect”
- Meredith Sime, AT&T’s algorithm.
U-verse 3. You must get the “golden nuggets” of insights to the right people, at the right time, in
a format that is easy for business people to use. You can see these priorities clearly in
the survey, and several interviewees commented that ease-of-use was extremely
important.
4. Make sure the project has senior level sponsorship. Odds are you won’t be able to
create a business case that “proves” in a spreadsheet how your company will increase
revenue or cut costs by investing in text mining. You need an executive’s vision of a
customer-centric organization.
5. Finally, pick a solution that is cost effective and fits your business and IT environment.
Consider the total cost of ownership when comparing an installed to on-demand
solution. Keep in mind, however, that cost is probably not going to be the central issue
if your business strategy dictates that better listening to customers is a critical success
factor.

Follow those recommendations, and you, too, can reap the benefits of text mining. As
Sime says, “The value of the unstructured feedback is phenomenal.” You already have the
content. It’s time to put it to work to improve your customers’ experience—and your
business performance.

About the Author and Sponsor
About the Author – Bob Thompson, CustomerThink Corp.
Bob Thompson is CEO of CustomerThink Corp., an independent customer management
research and publishing firm. He is also founder of CustomerThink.com, the world’s
largest online community dedicated to helping business leaders improve customer-centric
business strategies.

Since 1998, Thompson has researched the leading industry trends, including partner
relationship management, customer value networks and customer experience
management. In January 2000, he launched CRMGuru.com (renamed CustomerThink.com
in 2007) which now serves 300,000 business leaders monthly through its web site and
email newsletters.



Thompson is a popular keynote speaker at conferences worldwide and has written
numerous articles and papers, including his most recent report, Customer Experience
Management: A Winning Business Strategy for a Flat World. Before starting
CustomerThink, he had 15 years of experience in the IT industry, including positions as
business unit executive and IT strategy consultant at IBM.

For more information, visit www.customerthink.com or contact Thompson at
bob@customerthink.com.

About the Sponsor – Island Data Corp.
Island Data Corp., the innovator in Customer Analytics solutions, delivers real-time market
and business intelligence for global online enterprises. Island Data’s software-as-a-service,
Insight RT™, allows customer-centric organizations to listen to the voice of their
customers by managing the unstructured customer feedback. Insight RT enables
executives to track key performance indicators, collect actionable intelligence and
optimize the customer experience at every touch-point. Companies employ Insight RT for
timely discovery of critical facts and issues that support continuous improvement of
product management, product quality, market and online presence, brand loyalty,
customer retention, customer service and customer satisfaction. Headquartered in
Carlsbad, California, Island Data is funded by top-tier venture capital firms Dolphin Equity
Partners and ABS Ventures.

For more information, visit www.islanddata.com.

Appendix: Text Mining Fundamentals
Text mining is the discovery of information by analyzing text. Sources could include any
written form of communication captured electronically or verbal communication
transcribed into text. Insights gleaned from this text could reflect the meaning of what the
author intended or provide entirely new information.

Whereas data mining extracts information from structured databases, text mining
extracts information from natural language text. There are two approaches: linguistic and
statistical.

The linguistic approach involves identifying linguistic elements of language and structures
that relate them to each other. Those elements are the keys to meaning. This is much like
parsing or diagramming a sentence to identify parts of speech. If you have an adjective,
for example, what noun does it modify?

You don’t need human intervention to train a classifier, but you do want to identify parts
of speech (which is done easily with a conveniently packaged list known as a “dictionary”);
words that are commonly used by people who are angry (which can come from a



dictionary or thesaurus); and information about sentence structure (if you have two
nouns, which is the subject and which is the object? Did John throw the ball or did the ball
throw John?).

In the statistical approach,
words and phrases are treated
as abstract objects. You use
purely their mathematical
relationship to each other. This
approach most often involves
machine-learning. Is your
customer angry? Is he pleased?
Is another customer talking
about your latest product?
Using a sample of the text and
assignments in a number of
categories, the computer scans
them for common elements.

The machine learns by example
based on training data assigned
by human beings. If a business
receives a million emails a day,
you would take a small
sample—say, 500 to 2,000
emails—and manually classify them. Then the computer would scan the sample to
identify relationships in the text that hold clues for whether a particular email may be
from a happy customer. People using obscene language tend to be unhappy, so simply
scanning for profanity in your sample can distinguish email from irate customers.

There are many ways to train a classifier without human intervention. One important
method is clustering, in which you look at what words naturally go together, automatically
identifying common themes just by looking at one or two emails, rather than 200,000, for
instance.

For more information, please read:

The Text-Mining Handbook by Ronen Feldman and James Sanger (Cambridge
University Press, 2007).
Machine Learning in Automated Text Classification, by Fabrizio Sebastiani (ACM
Computing Surveys, Vol. 34, No. 1, March 2002)



Copyright 2011 KANA Software, Inc. KANA and the KANA logo are registered trademarks of KANA.
Other company, product and service names may be service marks of their respective owners.
840 W California Avenue, Suite 100
Sunnyvale, CA 94086
T 650.614.8300 | F 408.736.7613
www.kana.com
Contact us at http://www.kana.com/contact-us/contactsem.php
Twitter @KANASoftware
Linkedin: http://www.linkedin.com/groups/KANA-Software-1129?mostPopular=&gid=1129
Facebook: https://www.facebook.com/pages/KANA-Software-Inc/146154198748782

All Rights Reserved. 0110-01


Improve Your Customers' Experience By Listening to Unstructured Feedback

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Viewers also liked

Viewers also liked (9)

Similar to Improve Your Customers' Experience By Listening to Unstructured Feedback

Similar to Improve Your Customers' Experience By Listening to Unstructured Feedback (20)

More from Peerasak C.

More from Peerasak C. (20)

Recently uploaded

Recently uploaded (20)

Improve Your Customers' Experience By Listening to Unstructured Feedback