This talk highlights the evolution of the research function at Grubhub, from the days of a small and scrappy startup to the public company you know today.
When getting a business off the ground, research is typically directed towards iterative product improvement. As the company expands (different markets, customer segments, and offerings), so does the scope and complexity of its research needs. Building an adaptable research function that is able to grow and mature with the business requires experimentation and iteration of its own.
During these transitions, key questions emerged. How can research be a shared service that benefits more than just the product team? How can we explain ourselves to the organization so that they know how to work with us? How do we reduce the time it takes to do critical exploratory studies without compromising quality? How does research impact and inform marketing decisions? What is the difference between a qualitative UX researcher and a marketing researcher? Are focus groups really the devil or are they actually a useful methodology?
To address the issues, the Grubhub research team developed some unique methodologies. Slice, the in-house restaurant worked by Grubhubbers, was created to encourage cross-departmental collaboration, inspire creativity, and build empathy for restaurant employees. The research function was split into two teams with one side focused exclusively on the product experience (UX) and the other on the full lifecycle of the customer across all touchpoints (CX).
UX and CX research are still new and often misunderstood in the corporate context. Through extensive trial and error, Grubhub landed on a solution that has made research a critical part of the decision making process throughout all departments.
2. Background
A bit about
me.
2005 - 2007
2007 - 2010
2011 -
PRESENT
1999 - 2003
Majored in
Communication
Design
2003 - 2005
Worked in a
small design
studio in a large
corporation
Got a Masters
in Interaction
Design
Worked on a
variety of exciting
mobile products
Got my break
as a researcher
3. Meet
SOURCE: GRUBHUB.COM
Grubhub is the nation's leading online and mobile food ordering company
dedicated to connecting hungry diners with local takeout restaurants. The
company’s online and mobile ordering platforms allow diners to order directly
from more than 45,000 takeout restaurants in over 1,100 U.S. cities and
London. Every order is supported by the company’s 24/7 customer service
teams. Grubhub has offices in Chicago, New York and London.
4. Meet
SOURCE: GRUBHUB.COM
Grubhub is the nation's leading online and mobile food ordering company
dedicated to connecting hungry diners with local takeout restaurants. The
company’s online and mobile ordering platforms allow diners to order directly
from more than 45,000 takeout restaurants in over 1,100 U.S. cities and
London. Every order is supported by the company’s 24/7 customer service
teams. Grubhub has offices in Chicago, New York and London.
5. Meet
SOURCE: GRUBHUB.COM
Grubhub is the nation's leading online and mobile food ordering company
dedicated to connecting hungry diners with local takeout restaurants. The
company’s online and mobile ordering platforms allow diners to order directly
from more than 45,000 takeout restaurants in over 1,100 U.S. cities and
London. Every order is supported by the company’s 24/7 customer service
teams. Grubhub has offices in Chicago, New York and London.
6. 2011 2012 2013 2014 2015 2016
90
Employees
1,300
Employees
IPO
GRUBMERGER
7.85
Million Active
Diners
271,100
Daily Average
Grubs
45,000
Restaurant
Partners
ACQ.
SERIES E
$50 MIL
SERIES D
$20 MIL
SERIES C
$11 MIL
2.58
Million Active
Diners
129,100
Daily Average
Grubs
17,000
Restaurant
Partners
Meet the rocketship
600
Employees
ACQ. ACQ.
7. 2.58
Diners
129,100
Daily Orders
17,000
Restaurants
The Evolution of Research at Grubhub
2011 2012 2013 2014 2015 2016
90
Employees
1,300
EmployeesIPO
GRUBMERGER
UX
Research
Market
Research
7.85
Diners
271,100
Daily Orders
45,000
Restaurants
Evaluative
Research
Exploratory
Research
UX
Research
CX
Research
Grubhub
hires first
researcher
ACQ.
8. UX
Research
Market
Research
Grubhub
hires first
researcher
Part 1: UX Research Scales
2011 2012 2013 2014 2015 2016
90
Employees
1,300
EmployeesIPO
GRUBMERGER
7.85
Diners
271,100
Daily Orders
45,000
Restaurants
Evaluative
UX
Research
Exploratory
UX
Research
UX
Research
CX
Research
ACQ.
Challenges
1. How do we convince
devs and Product
Managers that research
is worthwhile?
2. If we empower non-
researchers to do
research won’t they
take our jobs?
3. Should market and
UX researchers work
together or draw lines
in the sand?
9. How do you convince Devs and Product
Managers that research is worthwhile?
● Exposing developers and product managers to the user in-person is
always the quickest way to win hearts and minds.
● Give them a predictable and consistent research process that they can
count on.
● Everyone can love research. Make it fun and exciting. Rally the troops,
celebrate findings and reward participation.
CHALLENGE 1
13. Will empowering non-researchers with research
skills put you out of a job?
Heck no. That’s silly.
With proper training, non-researchers should be empowered do a variety of
research activities.
Out of bounds (examples)
Select the research method
Define research protocol
Select the size of the sample
Lead synthesis/analysis of the data
In bounds (examples)
Assist in the planning of a study
Help execute research (e.g.
moderate usability tests)
Participate in synthesis
✓
✓
✓
Ø
Ø
Ø
Ø
CHALLENGE 2
14. Should Market and UX Researchers work
together or draw lines in the sand?
Work together. The hard part is actually making it happen.
● Each group believes they own consumer data
● They prefer and find truth in different research methodologies
● Focus areas and philosophies are at odds
Macro, customer focused,
doing what’s best for the business
Micro, experience focused,
doing what’s best for the user
Understand human behavior, inform strategy
Market
Research
UX
Research
CHALLENGE 3
16. Perks
● Greater visibility into business processes and key metrics
● Better insight into brand positioning and how it relates to product
● Increased visibility with executives
Drawbacks
● Marketing thinks about people as “buyers”
● Desire for initiatives to tie back to ROI
● Don’t consider the entire customer journey,
such as customer service
Do UX Researchers belong in Marketing?
Customer?
CPA?
Attrition?
Positioning
?
LTV?
CHALLENGE 4
18. UX Researchers can support marketing programs
that directly contribute to the bottomline.
Switching cups
19. What’s the difference between a UX and
Qualitative Market Researcher?
User Experience Researchers Qualitative Market Researchers
Behavioral and Attitudinal (What people do and why they do
it)
Attitudinal (What people say)
Focuses on user needs, goals, motivations, behaviors and
perceptions involved in the use of a product or service.
Focuses on customer wants, desires, aspirations,
perceptions, opinions, logic and trends involved in purchase
decisions.
Making products or services that satisfy a need, provide a
fluid experience and allow users to seamlessly achieve their
goals.
Formulating messages that resonate with the customer so
that the brand is top of mind when the use occasion arises.
Backgrounds in anthropology, design, psychology, HCI, etc. Backgrounds in marketing, business admin., psychology, etc.
Go beyond what people say they want and uncover
underlying needs. Problem solving.
Looking for holes and opportunities in the market to sell more
product.
CHALLENGE 5
20. How did Grubhub combine Market and UX
Research forces?
CHALLENGE 6
+
+
+
Market
Researchers
Customer
Experience
Quantitative
Research
User Experience
Researchers
Service
Design
Qualitative
Research
Thanks for the wonderful introduction.
It’s a pleasure to be here.
As ____________ mentioned my name is Rhiannon (or Rhi for short) and I am a researcher at Grubhub.
This talk, When UX Research Wasn’t Enough, tells the story of the research function at Grubhub, from the days of a small and scrappy startup to the public company that you know today.
Before we dig into this story, let me give you some background on me, so you know where all of this is coming from.
Background is in design.
After graduating from CMU started my career in interaction design at Motorola
Realized I was an OK designer but had a real passion for research.
That’s where this story really begins...
Show of hands, how many of you guys have heard of Grubhub?
Ok, now how many of you have placed an order on Grubhub?
For those of you who don’t know about it or haven’t used it, it’s simple.
You enter your address into the system and Grubhub returns restaurants in your area that deliver to you.
From my home in Lincoln Park there are 668 restaurants available
From my office in the Loop there are 661 restaurants available
From the Park 55 there are 437 restaurants available
You select the restaurant, food you want to order, enter your cc information and place your order.
Grubhub sends the order to the restaurant.
The restaurant makes the food.
...and for about 20% of those orders are delivered by a Grubhub delivery partner.
From a research perspective it’s a very fun place to be:
We get to consider all sorts of interesting ordering situations:
a couple staying in for the night, watching a movie, cuddling up and ordering some food to share
a group of girls order some food to munch on as they get ready before going out for the night
Or perhaps the conference goer, who wants to spend the evening in their hotel room, hair up in a sloppy bun, watching silly amounts of HGTV because her husband won’t let her have cable.
Same goes for restaurants, we have over 45 thousand restaurant partners and there aren’t two restaurants that are alike. We have everything in our network from the hole in the wall to upper casual joints and we don’t just talk to the restaurant owners or managers, we talk to all of the employees from the hostess to the chef and everything in between watching them use our products.
When it comes to our drivers, we’re in the back seat, watching them make deliveries...
When I started on at Grubhub, little did I know how quickly Grubhub would grow over the next 6 years.
I came on board in Q1 2011, during a major growth spurt. The founders had just raised a bunch of money in a series C round and then went on to raise even more money in a series D and E.
By the end of 2011, we acquired DotMenu (included CampusFood) broadening our reach from 13 markets to hundred of markets across the country.
In 2013 merged with our biggest competitor at the time, which eliminated threats of the competition and doubled the size of the company almost instantly.
IPOed which allowed the company to raise a bunch of capital quickly to invest in things that will continue that growth.
In 2015, started acquiring a slew of delivery companies that expanded our driver network allowing us to have more control over the delivery experience.
These were some of the conditions that contributed to a growing research team.
This diagram shows the progression of the research group over time, as the company grew.
When I started on there were 90 some employees. I am the person in orange.
They were just starting to build out their UX department. I was the first hire into that group. After me they hired a UX designer.
A few of the projects I started on: we were just about to introduce pickup, so instead of getting something delivered you can pick it up instead. I tested a few of the new pickup flows.
Another example: one of the devs created an Android app over the weekend and I tested it.
The work was very tactical and focused on the product.
When we merged with Seamless we inherited a market research function that reported into the then CEO of Seamless.
For the most part UX and market research didn’t work together. In fact, when the annual diner survey came around each year and we were forced to work together we did so begrudgingly.
In early 2015, research was separated from design, whereby design was absorbed into dev teams and research acted like a service to those teams.
In 2015, with a new CMO onboard, the UX research team split into two teams: the exploratory side going into marketing and absorbed the responsibilities of the dwindling market research team, and the evaluative side staying in technology.
The exploratory side, was forward thinking, looking for new ways to advance our products and the way people eat.
The evaluative side was focused on evaluating our product concepts and designs.
By the Summer of 2016, evaluative researchers felt stifled by their titles and wanted more, plus the org was starting to think of itself as more than just a product company, but now as a service.
The timing was perfect. We were positioned to take on a new role in the organization: one we call CX. I’ll get into what this means for Grubhub later in the presentation.
To get to each phase in our growth, we were faced with different challenges.
Some of these challenges might sound very familiar, as I believe most research teams encounter them at some point or another.
I’ll start with part one, a very young, UX research capability that was scaling with the growing demands of the company.
How do we convince devs and product managers that research is worthwhile (twice, once when starting on, and again when merging with Seamless)
Should we empower non-researchers to do research or will they take our jobs?
When we merged with Seamless, we inherited a market research capability. How do we work together?
Design is always down for research, it’s those pesky devs and product managers that are tough.
Of the litany of things I’ve tried over time, these are the three things that made us the most successful at Grubhub.
Standing alone, I don’t think any one of these things is novel. But in combination, these three things are consistent across research groups, making it a sort of special sauce.
Quickly go through them.
Here’s a very real example of how we make research fun at Grubhub.
What you’re looking at here is Slice. Slice is Grubhub’s in-house restaurant that is worked, run and patroned by Grubhub employees.
The purpose of Slice is to give employees first-hand exposure to what it’s like to use our restaurant-facing products in a realistic way. It’s one thing to use these products from the comfort of your desk, and it’s another thing altogether to use them while fulfilling orders, greeting dine-in guests, handing out pick-up orders and answering the phone.
When viewed through this lens it’s almost a different product altogether.
Slice allows us to give Grubhub employees exposure to restaurant products, helps them build empathy for the restaurant (hey it’s not as easy as I thought) test new features and allows employees to think about ways to improve them insitu.
Slice runs every other Friday and teams of 6 work Slice. They start the shift with a short training on how we think about restaurants at Grubhub, they work Slice for an hour and then spend 30 minutes in a debrief talking about how things went and what could have gone better. People love it. Slice has been around since 2013.
Researchers have come to me wondering in fear. If we empower them to do research, won’t we lose our jobs?
Here’s what we’ve come across:
Your time is stretched, yet a dev team wants to test a new design and there’s no time to wait.
John Doe and Susie Q are doing some rogue research and you’re 99% sure they don’t know what they’re doing.
Someone from another department is interested in a career change and wants to learn more about research.
Absolutely. In fact, here are some stunning benefits:
They’ll understand first-hand all of the hard work that goes into a study.
You’ll make an ally in the org to help you tout the benefits of research.
Trained non-researchers can help you to expand the breadth of your projects and expand your reach. Go into other markets, speak with more participants.
They feel invested in the research and will want to socialize the learnings!
This topic was front and center when we merged with Seamless. We had this new market research function, yet we had no idea how to work with them.
They seemed to do things very differently than we did and we didn’t seem compatible.
Some argue that Market and UX Research are in fact incompatible and that they should maintain this natural tension.
However, I would argue that this tension can turn bad. These two teams are symbiotic. They need each other, in order to produce research that tells the whole story.
To get around this, and to encourage more collaboration, today these teams are united under one team. I look for researchers who have an interest in the other discipline and are looking to grow in that area.
Market Research: Aims to identify potential markets which includes understanding who will buy which product/services and opinions about a product/service.
UX Research: User research focuses on user, his needs and environment around, how and in what circumstances users use products and services, and observes what people actually do. Help explain phenomena.
Design Research is essentially more qualitative in nature - how do people use the product, what are their key needs etc. - which can further be converted into insights for conceptualizing better products ahead.
Market Research on the other hand is more quantitative in nature - number of people buying one product vis-a-vis another one etc. - which might tell you the market performance of products, but won't answer the real 'why' behind them.
Part 2:
When the exploratory team, which was largely comprised of UX researchers moved into marketing, they experienced a major culture shock. Could they hang in this brand new world?
As we started to expand the research team in New York we started to notice a new strain of UX Researcher, which were very different from the ones we were used to.
Lastly, how are we comprised today. What does the CX function look like and in what ways is it a service design function?
Sure, the culture shock sucks at times, but the change comes with clear benefits:
The world opens and you see so much more into the business and how it works. I remember working with a designer at Grubhub who wondered why the CEO wasn’t worried about the feature he was working on.
You have a better sense of how marketing thinks and works.
You have greater visibility among executives
When UX and marketing collaborate, beautiful things can happen.
This is a picture of my son and I playing a game called Yummy Rummy.
Yummy Rummy is Grubhub’s loyalty program that was developed at the end of 2011. I’d give you newer examples if it was allowed.
One of the members of the board suggested that Grubhub develop a loyalty program.
It wasn’t clear what type of program be that a points system or a punch card, but it had to be a loyalty program.
So we pulled together the UX team + a creative from marketing and went to work, locking ourselves in a room for a week.
We spent the first few days reading a bunch of secondary research, loyalty programs aren’t anything new why not learn from other’s mistakes?
We had a small concession budget to work with and when we made the calculations, the amount one would receive per point or per punch was a pittance, they’d have to work very hard to earn the rewards, so we asked the question: if we can’t change the reward, can we change the perception of the reward. Enter the idea of variable rewards.
The concept started with three bowls, the kind you’d see at…
We actually tested the concept on employees. The old office was in a food desert, with only one restaurant nearby. People would go to it because it was easy and convenient despite the higher prices. Kind of like Grubhub.
We tracked people’s purchases and setup a table near the elevator. Those with a receipt from the restaurant could play the game. We had three bowls and there was something under each bowl: there were jokes, fortunes and people had a 1 in 4 chance to win a gift card to the restaurant.
People loved the game and would watch others play and it caused quite the buzz in the office.
They would come up with these strategies. Oh, the one on the left is lucky. I always win with the one on the left. Some people liked the middle bowl. Others swore by the one on the right.
We learned a lot from that study. Not only did we see a lift in the amount that people were going to the restaurant, but we also noticed areas to improve the game. For example, 1 in 4 chance to win was confusing and didn’t map to the three bowls we used in the game. For that reason we added a 4th option. But when we did that, people weren’t able to apply their strategies as easily. For that reason we gave each of the options, which turned into cards, it’s own identity. Each card hasn’t a different dessert on the front. My son always picks chocolate, not because it’s a winner, but because it’s his favorite.
In fact, upon preliminary investigation we noticed that there was a lot of overlap between the two groups.
They use a lot of the same methodologies: ethnographic studies, interviews, direct observation, usability testing, etc.
In the same way that UX has become a loose term in the design community, we noted the same looseness on the research side.
Market researcher use many of the same methodologies community is starting to adopt UX research methodologies such as Usability Testing.
The differences on the surface may be difficult to discern.
Entire ecosystem of the user. This includes the environment, actors and artifacts, needs and desires, barriers and blocks, core values, cultural aspects, and how all these interact and influence usage and purchase decisions. Deep, focused insight.
Outside-in (UX), Inside-out (Market).
Important component for product/service strategy (UX).
Important component for business strategy (Market).
Talks of the present, gain some insight from the past, avoid talks of the future (UX).
Talks of the past, present and future are all fair game (Market).
Do you want to investigate a current or potential product, service or brand positioning? Do your want to identify strengths and weaknesses in products? Understand purchasing decisions? Study reactions to advertising or marketing campaigns? Assess the usability of a website or other interactive services? Understand perceptions about the company, brand and product? Explore reactions to packaging and design?
The CX Research team at Grubhub is an even mixture of quantitative and qualitative researchers.
The quant side is made of market researchers, and the qual side is made of former UX researchers, ready to become service designers.
This may not be the case for all organizations, but for those with leadership that is numbers minded, market research is familiar and comfortable, were as UX research acts as that fresh perspective that pushes the limits.