Outside the Lab and Inside Their Minds: 5 Case Studies of Strategic UX Research
•
3 likes•863 views
Research that gets into your users’ heads and supports rich customer experience is most valuable and strategic when not confined to evaluating product "usage." AnswerLab shares examples of research approaches that surprise and delight by delivering experience insights beyond the scope of "could they use it?"
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Outside the Lab and Inside Their Minds: 5 Case Studies of Strategic UX Research
Similar to Outside the Lab and Inside Their Minds: 5 Case Studies of Strategic UX Research (20)
More from AnswerLab
More from AnswerLab (7)
Recently uploaded
Recently uploaded (20)
Outside the Lab and Inside Their Minds: 5 Case Studies of Strategic UX Research
- 1. Your Trusted User Experience Research Partner Amy Buckner Chowdhry AnswerLab CEO & Co-Founder February 7, 2013 Outside the Lab and Inside Their Minds: 5 Case Studies of Strategic UX Research Amy Buckner Chowdhry , AnswerLab CEO & Co-Founder February 7, 2013
- 2. AnswerLab: The User Experience Research Leader • Founded: 2004, San Francisco • Opened New York Office in 2012 • Trusted by global market leaders • Recommended by • Fastest growing research firm 4 years in a row • Recognized in Fortune Magazine 2
- 3. Amy Buckner Chowdhry CEO & Co-Founder, AnswerLab • Drives AnswerLab to fulfill its vision and strategic plan, cultivates the talent pool, and oversees financial operations • First CEO from the user experience research industry to be named by Ernst & Young as one of 10 “Entrepreneurial Winning Women.” • Before co-founding AnswerLab, Amy served as Director of Professional Services at Vividence (now Keynote). CONFIDENTIAL | PROJECT # [INPUT NUMBER HERE] 3
- 4. 5 Insights 3 Getting in Customers’ Minds 1 Why It Matters 2 Presentation Topics
- 5. Getting in Customers’ Minds
- 6. Would You Bake A Cake On A Whim?
- 7. Would You Pack For Vacation Without Knowing Your Destination?
- 8. Would You Build A House Without Examining The Foundation?
- 9. Option 1 Would You Build A Map App Without Good Source Data?
- 10. Beware Of Skipping Foundational Steps In Product Development
- 11. The Evolution Of UX Research 2005 Refine Prototypes Right Way? 2011 Understand Pre-product Right Product? 2000 Validate Live Product
- 12. Move Beyond Usability To Understanding Opportunity Gaps Environment Interaction Modes & Use Cases Understanding of Audience Holistic Research
- 13. Build the Right Thing Understand Where is the customer? Where are the gaps between product capabilities and customer activities? What is she trying to accomplish in her job? What external factors influence her? How does the customer feel about the brand? Is our product solving the right problems?
- 14. Why It Matters…
- 17. The Buzz about . . . Inadequate Understanding Of Privacy Needs Results In Product Failure
- 18. 5 UX Insights
- 19. Context is King Insight 1:
- 20. The Mobile Wallet Olympics* Who will end up on the medal stand? SPEED is required to win! *Image from Mobile Wallet Media (http://www.mobilewalletmedia.com/7_Ss_Week_5-Speed-12012.html)
- 21. • Create mock retail store and conduct mobile “shop-alongs” • Innovation games to design the ideal shopping assistant PayPal & AnswerLab Use Innovative Research Methods To Drive Digital Wallet
- 22. Key Research Findings For Digital Wallet • Consumers highly receptive to an app offering location-based deals • Point of sale is when mobile app desired • Consumer comfort with barcode scanners • Social pressures matter – concerns about perceptions of line cutting and tipping for prepayment
- 23. Click for video example PayPal Studies Customers In Context To Launch Wallet App In London • Pay on smartphone using barcode • Alerts for promotions • Works with or without signal
- 24. Insight 2: Top US Bank Needs To Measure Potential Adoption Of Online Payment
- 25. Top US Bank Needs To Measure Potential Adoption Of Online Payment • Would new & improved user experience promote adoption? • Lab setting won’t work to measure potential adoption • We used quantitative prototype research to compare existing site to future site
- 26. Key Finding: New UX Won’t Promote Adoption Among Offline Customers Because Barrier Was Not Usability Active Users Went Inactive Never Used Live Site Future Site Active Users Went Inactive Never Used Outcome Would Adopt Some Would Adopt Still Not Interested 600 Customers
- 27. Insight 3: Things Are Not Always What They Seem
- 28. • To understand new users’ experience in natural environment, we conducted a diary study and in-depth interviews • To understand factors affecting churn, we did online focus groups among former, new, & loyal subscribers Online Service Seeks to Improve Conversion From Free Trial, Reduce Churn
- 29. v Recommended Subscription Plan Monthly All Access Access All Features Every Month You Subscribe Pay Monthly $24.95/mo. Quarterly All Access Access All Features for 3 Months Pay In One Installment $44.85 ($14.95/mo.) Maintenance Plan Limited Features Pay Monthly $17.95/mo. Weekly Pass Access All Features Pay Per 2-Day Usage Pass $13.95/Week Pass v Key Finding: Subscription Model Didn’t Align With Usage Patterns; Usability/ Features Not Primary Issue Monthly All Access Access All Features Every Month You Subscribe Pay Monthly $24.95/mo. Quarterly All Access Access All Features for 3 Months Pay In One Installment $44.85 ($14.95/mo.) Existing Subscription Plan
- 30. It’s About More Than The Product Insight 4:
- 31. Ethnography Helps Software Client Evaluate Why Move from Desktop To SaaS Makes Users Unhappy
- 32. • Compensation system encourages bad data in software • In-office workflow impacts poor product experience • Poor workflow also built into software resulting in continued reliance of offline methods of data processing Key Finding: Customer Experience About More Than Product Usage
- 33. Insight 5: Get Beyond Usability To Delight
- 34. • Test various platforms including desktop applications, mobile apps, and observe in-store experiences Program Of Research To Understand End-To-End Customer Experience Of F100 Logistics Company • Online surveys to profile potential customers, determine pain points with existing services, understand new opportunities • Lab-based, in-person interviews to understand the “why” behind delights/pain points and get reactions to prototypes
- 35. • Mobile research reveals customer concerns about waiting in line, even if they could use mobile tools in advance • We recommended a kiosk approach so mobile customers could scan the QR code on their phone themselves and bypass the queue Research Uncovers Customer Expectations For Digital And In-Store Experiences
- 36. Leverage Research Techniques Beyond Usability For Deeper Customer Understanding
- 37. • Conduct research EARLY in the development cycle; ground-breaking customer insights have little impact if they are discovered late in the development game • Leverage exploratory methods; e.g., diary studies, ethnographies, to learn about customers’ behaviors in their environments and get to the bottom of the context of their usage • Encourage colleagues to be keen observers of customers’ experiences. Have them join you in ride-alongs or shop-alongs and encourage them to stay engaged during customer interviews How Do You Get Deeper Customer Understanding?
- 38. Questions? For follow-up questions about AnswerLab user experience research services, contact: info@answerlab.com
- 39. Your Trusted User Experience Research Partner
Editor's Notes
- AnswerLab is a San Francisco based user experience research firm. For those of you who may not be familiar with them, AnswerLab focusesexclusively on user experience research to understand what people see, do, think and feel when using web sites, mobile applications, and other digital products.
- Personal Background:As XXX mentioned, I’m the CEO& Co-Founder of AnswerLab. My job is to set our vision and strategic plan, and inspire our team to hit those goals. Prior to AnswerLab, I was head of Professional Services for Vividence, now Keynote Systems.
- So, here’s what I’d love to cover in today’s presentation.
- I’d like to start by posing a few questions. Would you ever bake a cake for the first time without preparing? Unless you’re Martha Stewart, it’s unlikely you can just bake a cake on a whim. Have you reviewed the recipe? Do you have the necessary ingredients? Do you need to go shopping? If you try and whip something together with what you’ve got in the house, chances are your cake will be inedible, or ugly at best.
- Would you pack your bags for vacation w/out knowing where you’re going?
- Would you build a house without first knowing what you want, ensuring you have a solid foundation, building plans, designs, etc. that factor in location?
- Would you build a map application without having the most important element nailed? It’s very hard to build a reliable mapping app if you don’t have good source data on the maps.Mapping is probably the most complex of all mobile apps. You need mountains of data, which needs constant revision in near real-time if your product is to be credible. Mistakes on maps can have massive impact. If your mobile map steers you into road construction or toward a washed out bridge while you are driving alone in the rain at night, your safety may be at risk and your unhappiness with the mapping application will be long-lasting.But making digital maps is not easy. Google has spent years working on its services, pouring all kinds of resources into the effort, including its Street View project to photograph and map the world. It will be hard to duplicate that depth and breadth. What we’ve seen lately with the iPhone 5 and the series of articles talking about Apple losing it’s edge really illustrate the perils of skipping a foundational step in anything you create.
- And, yet skipping the foundational step happens all to often when companies design things. Consider the evolution of user experience research.Most companies initially invested heavily in assessing their live products through benchmarking or other quantitative measures to evaluate their impact on the customer experience. However, many learned that investing in research at this stage was too late. Investing in evaluating / testing prototypes prior to the product being live would ensure those end measures were much more positive. Yet, a true user-centered design process places the most emphasis in the earliest stages of creating a product roadmap – in order to Understand the user. And, now we’re seeing as for 2011, many more Fortune 500 companies willing to make that investment. They understand that understanding the user yields the best outcomes.
- I’d like to encourage everyone to think more holistically about your research. Going beyond usability to understanding . . .
- So, I’d like to provide two examples of why this matters.
- [Example based on research by Bill Gribbons, Bentley University]When was the last time you waited as someone fumbled through trying to use the grocery store self check-out? We’ve all seen these and probably all used them. They were designed to solve a business problem – reducing the number of store clerks while increasing the number of open check-out stands. The plan was, stores start with one or two and eventually they do away with all the clerks. What happens when all the self check-outs have a line and someone in front can’t figure out how to scan their organic tomatoes? People start hurling food and things get ugly fast… The point is, they are no longer a convenience (in fact they may be a liability). Stores have spent 100s of millions of dollars installing this self check-out equipment which will soon be dismantled – we’re already seeing it being removed from stores in the Northeast – because it isn’t improving the shopper experience. When stores implemented these machines, they were solving the wrong problem. They were solving the business’ problem of employing grocery checkers – they weren’t solving a consumer problem. When the stores when to our friends at Bentley University and asked what the consumer problem was, they learned something entirely different was needed.
- What the research found was shoppers want something like this – and these are in stores around the country already. Smart shopping carts. I swipe my credit card, it knows who I am and it brings up my shopping list. It knows where I am in the store and as I’m walking around it will talks to me. “Amy, you haven’t purchased brown rice in two weeks, would you like some?”While I’m there, it tells me, “There’s a special on the kale you love.” These are opportunities for merchandising and innovation that didn’t exist before and this innovation that makes people’s lives easier.At the end of your shopping trip you simply walk out of the store. There are no lines at all. This is real value for the shopper and increased value in merchandising opportunities for the store. Had the super-markets focused on solving the right problem and understanding customers up front, they wouldn’t have wasted 100s of millions of dollars implementing the wrong systems.
- Google Buzz:How about Google Buzz? Who knows the Buzz about Google Buzz? Buzz was a social networking and messaging tool designed to integrate into Google's email service Gmail. It enabled users to distribute pictures, links, videos and documents in real time and to share them with all connected friends. The default setting on the tool publicly disclosed a list of the names of Gmail contacts that the user has most frequently emailed or chatted with.When Google first tested Buzz – they did not get into the minds of their users. . . not a single Google target user w/in the US or internationally gave feedback on the feature before it launched. As a result, there was a huge privacy backlash – at home and particularly abroad. There were two major challenges:No one could figure out how to turn off Google Buzz for publicly disclosing names of contacts. Privacy issues in many countries, particularly the US and Germany.It was one of the busiest holiday weekends in the lives of Buzz engineers as the scrambled to make changes to quell the backlash and make the off button more apparent for not sharing contact names.In the US, Goggle reached a settlement in a class action lawsuit regarding Buzz and have put together an $8.5 million independent fund to support privacy organizations.http://www.dw-world.de/dw/article/0,,5270758,00.htmlGoogle Buzz was shut down by Google in 2011 because Google failed to understand the problem they were solving for.
- Insight #1: Context is King.
- Whoever wins the mobile wallet Olympics is going to win BIG. Payments made through mobile phone wallets are expected to soar, and the technology will be transformative. Fortune wrote a recent article on the death of cash, in which they mention that themobile-payments revolution has arrived…Paying by phone will be as transformative as the advent of the credit card in the 1950s. It will change the way we shop and bank. With powerful smartphones and tablets taking center stage on both sides of the checkout counter, it will reshape the relationship between buyer and seller. Not only will the phone or the tablet become a wallet for consumers, but it will also turn into a credit card reader and a register for merchants. Shoppers will use their mobile device as a coupon book, a comparison-shopping tool, and a repository of those unwieldy loyalty cardsResearch Objectives: PayPal needed to understand the end-to-end in-store shopping experience, both domestically and in Europe, to drive new product innovation in mobile and online payments. To inform creation of a "digital wallet," they partnered with AnswerLab to:Gather insights into the pain and pleasure points associated with in-store shopping.Evaluate the concept of a digital wallet.Understand consumers’ mental models while shoppingIdentify features that would delight.
- So we did 2 things:We also went to Europe and conducted innovation games with consumers, where we asked them to share the contents of their wallets and form groups to design the ideal shopping assistant. Now, it’s important to do innovation games so that we get customers away from “designing” the product and instead better describing their need.AnswerLab transformed a traditional usability research lab into a mock retail store environment to conduct mobile "shop-alongs." Participants were asked to complete a series of real shopping scenarios inside the mock retail store to test a new mobile payment application. For each scenario, some rule or element of the game was altered to change the overall shopping experience, like allowing players to pay ahead using their mobile device or to find store coupons and deals on their phone.
- Across all of our studies in the US and Europe , we learned 4 Key things:1. We learned that consumers wanted to feel like“smart shoppers,” They were concerned about buyer’s remorse as the result of not always having time to research the best deals or offers prior to going to the store.So, they were highly receptive to an app that would drive deals / discounts based on their location. 2. Consumers think about “checking out” at the point of sale and would most want to present a mobile wallet that step.3. PayPal wanted to understand what mental models customers had for using their mobile phones in a retail store environment. Based on our shop-alongs and homework, we discovered that customers were most familiar with barcode scanners as a way to check prices, find store coupons, and manage loyalty cards. Some even told us they used barcode scanner apps already.4. And most importantly, they revealed something no one expected, something about social pressure . . . And they revealed this when we recreated the Starbuck’s lab: They would feel like a real jerk cutting line to get their coffee if they’d prepaid on their apps.They wanted it to be obvious they were leaving a tip.I thought you might see a version of the finished product that has launched in London.
- I thought you might see a version of the finished product that has launched in London. Customers can pay by Smartphone using the In-store PayPal app – no credit cards needed – simply use a bar codeAlerts for promotionsWorks with or without signalPayPal fundamentally understood they’d need to “get into the minds of the customers” by recreating context to build a mobile wallet app that could potentially dominate the market. . . They didn’t plan to build something and refine it later. They understood that foundational research would give them the best foot forward in winning the Mobile Wallet Olympics.
- Some online products are extremely expensive to redesign or build. .. . we’re talking millions of dollars and 1-2 year development cycles. With investments of this magnitude, it’s just not safe to proceed without first validating that your concept or idea will fly. You have to build the right thing or you’re wasting development $$s. Our customers with a lot at stake routinely get customer insight early. Here’s an example: One of our customers is one of the top 5 banks in the US. Product managers hypothesized that developing a new user experience for Bill Pay would cause more customers to adopt the feature. (In case not everyone here knows, banks actually make money from you paying your bills online.) But, how do you measure that potential adoption? Well, you can’t measure something like this in the lab setting. Have any of you ever been in a lab setting? I’m sure you’ve seen this . . .they just failed 2 tasks in a row with visible frustration. . . and then you ask, so what do you think of this? And they get this kind of spaced out look and say, I’d probably use it. Lab participants don’t want to hurt your feelings and they don’t always say what they mean.
- So, for this bank, we conducted a quantitative prototype study. We recruited 600 customers and split them into two groups. Half tried tasks on a prototype version of the live site and half tried on the future site. They did this in a fancy survey environment where we could track their behavior while feeding questions – while they were in their homes or offices (no moderator). We measured their potential adoption of bill pay both pre and post exposure to the prototypes. And, we learned some interesting things: 1) The new version of Bill Pay would continue to retain existing users of it. Yay for the bank! 2) Folks who had tried Bill Pay but stopped using would actually be enticed to try it again with the new user interface – but they would have to be marketed to. And most importantly 3) The new interface for bill pay had absolutely no impact on customers who had never used bill pay. Improving the User Interface would not drive their adoption. They had entirely different barriers to usage such as fees and preference for provider sites, where they could immediately see the impact of their payment on their bills. So, this new interface was not the ‘right thing’ for customer who had never used Bill Pay. These customers needed an entirely different value proposition. And so, the bank moved forward with the new design to help the abandoned users and has begun exploring alternatives to attracting customers who had never used Bill Pay.Again, you have to solve the right problems for customers and learn that up front before you begin building your product. Had this bank started with the question “can they use it”, they would have missed out on learning that usability wasn’t even an issue for the primary group they were targeting.
- We work with anonline business that helps people construct their family trees. They have more than 2 million users and a subscription model with a free-trial offering. Given this business model, it’s critical to the health of its business to minimize drop off and improve conversion rate from free trial.It's all about churn. If they could just convert people past 14-day free trial, they would have a big opportunity. They wanted to understand why people didn’t like the service enough to continue after the free trial. They asked us to build a research solution to help improve the initial product experience and reduce churn.Can you think of some reasons people might drop-off from using a site like this? When I’ve asked folks before, they suggest things like usability problems or lack of the right content. Well, we wanted to get to the bottom of this to see what might be wrong with the experience. Methodologies: So we conducted a diary study to understand new users' initial experience with the service in their natural environment, followed by in-depth interviews. We also conducted online focus groups among subscribers who abandoned, new subscribers, and loyal subscribers to get insights into factors affecting churn. We needed to learn what about the user experience was so poor.But things were not what they seemed. The findings surprised everyone. Drop-off / Churn had nothing to do with the usability, features or the product. It had everything to do with their research patterns. People who churned actually considered themselves "seasonal" users of the service. They enjoyed the service, and they got so involved with that they found it required a significant time commitment. They would use the service for a while, and then take a break for a while until they had time to devote to it again. I had periods when I was addicted to it. I would spend hours on end just digging up materials… it was almost overwhelming, just all the information I could discover… I just kind of felt drained. – Churner (31-60)I stopped just because we’re taking a trip to see my first grandbaby and my other son is graduating from officer school. Every penny is going towards this trip. I intend to come back. – Churner (61-100)Had the client focused on usability testing of new users or churned users, they wouldn’t not have uncovered the pattern that emerged over time in our diary studies. Holistic, exploratory research was key to uncovering this usage pattern.
- Our recommendation: Build a new revenue stream by offering a different membership model. Details/Quotes : The monthly subscription model in place was either on or off, and that didn't align with their usage patterns. We recommended a maintenance membership that was a lower cost, limited interaction model. Imagine if users had a consistent subscription model that better supported how they liked to research their family history? Again, if we can focus on understanding how people behave in the real world, the outside factors influencing how they use the product, we can build much much better customer solutions. Had the client focused on usability testing of new users or churned users, they wouldn’t not have uncovered a new revenue stream. Holistic, exploratory research was key to uncovering this opportunity.
- Bigger Picture Competitive Situation: One of our clients makes software to enable banks to automate their loan process. This client recently signed on a major bank around the time it was going through a new implementation of its product (a move from desktop to SaaS). There were numerous complaints from their customers, large and small, about their new implementation. They needed to understand to what extent the issues were environmental (the nature of the process they were trying to automate, bandwidth issues, etc) and to what extent their clients’ issues were due to the software. Essentially, was it “our fault”. Research Objectives: So we were brought in to find out if the complaints were legitimate issues with the Saas implementation or issues with the environment. We created an ethnography study to observe how loan officers, processors, underwriters and closers were using the new SaaS model. These are actual photos from the study, where we sat with the customers for days watching what they did and uncovering challenges.So, who thinks it was the bank’s “fault”? Who thinks it was the software’s “fault”?Well, it was both – the software was so confusingly designed that the officers still used their paper, offline methods to get the loan done. You can see all the offline tools they’re using here.
- "Experience" Insight: The compensation system for the loan officers encouraged them to enter as many loans as possible into the system. As a result, a closer had to go in and clean everything up.The end result was too many redundancies and inefficiencies in loan processing – the loan was getting processed 3 times. External factors – the compensation system, were contributing to inefficiency.Our insight – The customer experience is about more than just the product they’re using. In this case, the compensation system, the external factors, and the in-office workflow. When you spend time doing exploratory, holistic research, you can see what other factors influence the product experience.
- For the past year or so, AnswerLab has been conducting a series of studies to understand the holistic experience of customers of a Fortune 100 global logistics company.Our studies have spanned various platforms : tested the robust desktop applications, users of the company’s mobile app, and even spoke with brick and mortar customers.Our research approach has included online surveys survey to profile potential customers, determine pain points with existing alternatives and understand use cases for new services. Lab-based, in-person interviews to understand the “why” behind the delights and pain points discovered in the survey and to assess reactions to new productprototypes.
- By focusing on the “what”, the company has been able to uncover not only their customers expectations for the digital experience, but also expectations for what happens in physical stores. When researching mobile applications, we’ve uncovered that customers have major concerns about waiting too long in line, even if they could use mobile tools in advance. We . While this is very much a futuristic recommendation, not yet in development, it illustrates how taking a holistic approach to research can yield insights that delight, and enrich the overall customer experience.
- In summary, I’d like to encourage everyone to get beyond usability to leverage research techniques and methods that enable a deeper understanding of your customers. Make the effort and investment to “get into the minds’ of your customers.
- How can you do this?Conduct research early in your development cycle. Ground-breaking customer insights have little impact when they’re discovered late in the development game.Leverage exploratory research methods – like diary studies, ethnographies – go learn about your customers behaviors in their environments, get to the bottom of the context of their usage.And finally, encourage your colleagues to be keen observers of their customer’s experience. Coach them on how to join you in ride-alongs, or shop-alongs. Give them incentive to stay engaged during customer interviews vs. multi-tasking in the back room. The best gems, most impactful user insights can come observing what customers do, but cannot articulate.