SlideShare a Scribd company logo
1 of 29
Download to read offline
The Importance of Culture
Building and Sustaining Effective Engineering
Organizations
Randy Shoup
@randyshoup
linkedin.com/in/randyshoup
Background
CTO at KIXEYE
• Real-time strategy games for web and mobile
Director of Engineering for Google App Engine
• World’s largest Platform-as-a-Service
Chief Engineer at eBay
• Multiple generations of eBay’s real-time search
infrastructure
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Hire and Retain the Best
Hire „A‟ Players
• In creative disciplines, top performers are 10x
more productive (!)
Confidence
• A players bring A players
• B players bring C players
Google Hiring
Goal: Only hire top talent
• False negatives are OK; false positives are
not
Hiring Process
• Famously challenging interviews
• Very detailed interviewer feedback
• Hiring committee decides whether to hire
• Separately assign new Googler to group
 Highly talented and engaged employees
Respect People
People are not interchangeable
• Different skills, interests, capabilities
• Create a Symphony, not a Factory
Most valuable and irreplaceable asset
• Treat people with care and respect
• If the company values its people, people will
provide value to the company
eBay “Train Seats”
eBay‟s development process (circa 2006)
• Design and estimate project
(“Train Seat” == 2 engineer-weeks)
• Assign engineers from common pool to implement
tasks
• Designer does not implement; implementers do not
design
 Dysfunctional engineering culture
• (-) Engineers treated as interchangeable “cogs”
• (-) No regard for skill, interest, experience
• (-) No pride of ownership in task implementation
• (-) No long-term ownership of codebase
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Service Teams
• Small, focused teams
• Single service or set of related services
• Minimal, well-defined “interface”
• Vendor – Customer relationships
• Clear “contract” between teams
• Functionality: agreed-upon scope of
responsibility
• Service levels and performance
Google Services
• All engineering groups organized into
“services”
• Gmail, App Engine, Bigtable, etc.
• Self-sufficient and autonomous
• Layered on one another
 Very small teams achieve great things
Autonomy and Accountability
• Give teams autonomy
• Freedom to choose technology, methodology,
working environment
• Responsibility for the results of those choices
• Hold team accountable for *results*
• Give a team a goal, not a solution
• Let team own the best way to achieve the
goal
KIXEYE Service Chassis
• Goal: Produce a “chassis” for building scalable
game services
• Minimal resources, minimal direction
• 3 people x 1 month
• Consider building on open source projects
 Team exceeded expectations
• Co-developed chassis, transport layer, service
template, build pipeline, red-black deployment, etc.
• Heavy use of Netflix open source projects
• 15 minutes from no code to running service in AWS
(!)
• Plan to open-source several parts of this work
Google and DevOps
Ops Support is a privilege, not a right
• Developers carry pager for first 6+ months
• Service “graduates” to SRE after intensive
review of monitoring, reliability, resilience, etc.
• SRE collaborates with service to move
forward
Everyone‟s incentives are aligned
• Everyone is responsible for production
• Everyone strongly motivated to have solid
instrumentation and monitoring
Collaboration
• One team across
engineering, product, operations, etc.
• Solve problems instead of pointing fingers
Google Co-Location
Multiple Organizations
• Engineering
• Product
• Operations
• Support
• Different reporting structures to different VPs
Virtual Team with Single Goal
• All work to make Google App Engine successful
• Coworkers are “Us”, not “Them”
• When asked which teams we need to sit next to, it
never occurred to us that other organizations were
not “our team”
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Quality over Quantity
Whole user / player experience
• Think holistically about the full end-to-end
experience of the user
• UX, functionality, performance, bugs, etc.
Less is more
• Solve 100% of one problem rather than 50%
of two
• Users prefer one great feature instead of two
partially-completed features
Institutionalize Quality
Development Practices
• Code reviews
• Continuous Testing
• Continuous Integration
Quality Automation
• Automated testing frameworks
• Canary releases to production
“Make it easy to do the right thing, and hard to do the
wrong thing”
Google Engineering Discipline
Solid Development Practices
• Code reviews before submission
• Automated tests for everything
• Single logical source code repository
 Internal Open Source Model
• Not “here is a bug report”
• Instead “here is the bug; here is the code fix;
here is the test that verifies the fix” 
Technical Tradeoffs
Make Tradeoffs Explicit
• Triangle: date vs. quality vs. features
• When you choose date and features, you
implicitly choose a level of quality
Manage Technical Debt
• Plan for how and when you will pay it off
• Maintain sustainable and well-understood level of
debt
“Don‟t have time to do it right” ?
• WRONG – Don’t have time to do it twice (!)
Building Blocks of Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Constant Learning
Any process, organization, or product can
always be improved
Mistakes are a learning opportunity
• What did you do -> What did you *learn*
• Take emotion and personalization out of it
Encourage iteration and velocity
• “Failure is not falling down but refusing to get
back up” – Theodore Roosevelt
Google Blame-Free Post-
Mortems
Post-mortem After Every Incident
• Document exactly what happened
• What went right
• What went wrong
Open and Honest Discussion
• What contributed to the incident?
• What could we have done better?
Engineers compete to take personal
responsibility (!)
Google Blame-Free Post-
Mortems
Action Items
• How will we change process, technology,
documentation, etc.
• How could we have automated the problems
away?
• How could we have diagnosed more quickly?
• How could we have restored service more
quickly?
Follow up (!)
Iteration and Experimentation
*Engineer* successes
• Constant iteration
• Launch is only the first step
• Assume you will not get it perfect on the first
try
• A / B Testing needs to be a core competence
Many small experiments sum to big wins
eBay Machine-Learned Ranking
Ranking function for search results
• Which item should appear 1st, 10th, 100th, 1000th
• Before: Small number of hand-tuned factors
• Goal: Thousands of factors
Experimentation Process
• Predictive models: query->view, view-
>purchase, etc.
• Hundreds of parallel A|B tests
• Full year of steady, incremental improvements
 2% increase in eBay revenue (~$120M)
Recap: Building Blocks of
Culture
Hiring and Retention
Ownership and Collaboration
Quality and Discipline
Learning and Experimentation
Thank you!
rshoup@kixeye.com
@randyshoup
linkedin.com/in/randyshoup

More Related Content

What's hot

DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps TransitionDOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps TransitionGene Kim
 
A CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling OrganizationsA CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling OrganizationsRandy Shoup
 
Evolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBayEvolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBayRandy Shoup
 
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine LearningRandy Shoup
 
One Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us BetterOne Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us BetterRandy Shoup
 
DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? John Willis
 
Minimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughMinimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughRandy Shoup
 
Turning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational CapitalTurning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational CapitalJohn Willis
 
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
All daydevops   2016 - Turning Human Capital into High Performance Organizati...All daydevops   2016 - Turning Human Capital into High Performance Organizati...
All daydevops 2016 - Turning Human Capital into High Performance Organizati...John Willis
 
Immutable Service Delivery Shenzhen 2016
Immutable Service Delivery   Shenzhen 2016Immutable Service Delivery   Shenzhen 2016
Immutable Service Delivery Shenzhen 2016John Willis
 
Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Sriram Angajala
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesRandy Shoup
 
DevOps Picc12 Management Talk
DevOps Picc12 Management TalkDevOps Picc12 Management Talk
DevOps Picc12 Management TalkMichael Rembetsy
 
Moving Fast at Scale
Moving Fast at ScaleMoving Fast at Scale
Moving Fast at ScaleRandy Shoup
 
Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness John Willis
 
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi WallsYulia Shcherbachova
 
Tales from the Platform Trade
Tales from the Platform TradeTales from the Platform Trade
Tales from the Platform TradeWilliam Grosso
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comAviran Mordo
 
Serverless Toronto helps Startups
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps StartupsDaniel Zivkovic
 

What's hot (20)

DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps TransitionDOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
DOES15 - Randy Shoup - Ten (Hard-Won) Lessons of the DevOps Transition
 
A CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling OrganizationsA CTO's Guide to Scaling Organizations
A CTO's Guide to Scaling Organizations
 
Evolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBayEvolving Architecture and Organization - Lessons from Google and eBay
Evolving Architecture and Organization - Lessons from Google and eBay
 
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine Learning
 
One Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us BetterOne Terrible Day at Google, and How It Made Us Better
One Terrible Day at Google, and How It Made Us Better
 
DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How?
 
Minimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughMinimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good Enough
 
Turning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational CapitalTurning Human Capital into High Performance Organizational Capital
Turning Human Capital into High Performance Organizational Capital
 
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
All daydevops   2016 - Turning Human Capital into High Performance Organizati...All daydevops   2016 - Turning Human Capital into High Performance Organizati...
All daydevops 2016 - Turning Human Capital into High Performance Organizati...
 
Immutable Service Delivery Shenzhen 2016
Immutable Service Delivery   Shenzhen 2016Immutable Service Delivery   Shenzhen 2016
Immutable Service Delivery Shenzhen 2016
 
Software devops engineer in test (SDET)
Software devops engineer in test (SDET)Software devops engineer in test (SDET)
Software devops engineer in test (SDET)
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and Microservices
 
DevOps Picc12 Management Talk
DevOps Picc12 Management TalkDevOps Picc12 Management Talk
DevOps Picc12 Management Talk
 
Moving Fast at Scale
Moving Fast at ScaleMoving Fast at Scale
Moving Fast at Scale
 
Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness Dockercon USA 2016 - Immutable Awesomeness
Dockercon USA 2016 - Immutable Awesomeness
 
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
"Transforming Enterprise Teams to DevOps Workflows" Mandi Walls
 
Tales from the Platform Trade
Tales from the Platform TradeTales from the Platform Trade
Tales from the Platform Trade
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
DevOps for CTOs
DevOps for CTOsDevOps for CTOs
DevOps for CTOs
 
Serverless Toronto helps Startups
Serverless Toronto helps StartupsServerless Toronto helps Startups
Serverless Toronto helps Startups
 

Viewers also liked

The Importance of Culture
The Importance of CultureThe Importance of Culture
The Importance of CultureTristan White
 
Company Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right TeamsCompany Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right Teamse27
 
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...Randy Shoup
 
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYEQCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYERandy Shoup
 
Concurrency at Scale: Evolution to Micro-Services
Concurrency at Scale:  Evolution to Micro-ServicesConcurrency at Scale:  Evolution to Micro-Services
Concurrency at Scale: Evolution to Micro-ServicesRandy Shoup
 
Why Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudWhy Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudRandy Shoup
 
Importance of Organizational Development
Importance of Organizational DevelopmentImportance of Organizational Development
Importance of Organizational DevelopmentAmanda Jamison
 
Nonverbalcommunication 1
Nonverbalcommunication 1Nonverbalcommunication 1
Nonverbalcommunication 1Sulman Ahmed
 
รายงานการตรวจรถ
รายงานการตรวจรถรายงานการตรวจรถ
รายงานการตรวจรถseedow
 
Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2Donald Malloy
 
Professional societies
Professional societiesProfessional societies
Professional societiesSulman Ahmed
 
Human Subjects Regulations
Human Subjects RegulationsHuman Subjects Regulations
Human Subjects RegulationsJanet Stemwedel
 
Beyond Disaster Recovery
Beyond Disaster RecoveryBeyond Disaster Recovery
Beyond Disaster Recoverymkinzie
 
Law Enforcement Oath of Honor
Law Enforcement Oath of HonorLaw Enforcement Oath of Honor
Law Enforcement Oath of HonorDiscoverPolicing
 
Discovering a Career in Law Enforcement
Discovering a Career in Law EnforcementDiscovering a Career in Law Enforcement
Discovering a Career in Law EnforcementDiscoverPolicing
 
Software Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEESoftware Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEESamsuddoha Sams
 
From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015Randy Shoup
 

Viewers also liked (20)

The Importance of Culture
The Importance of CultureThe Importance of Culture
The Importance of Culture
 
Company Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right TeamsCompany Culture And The Importance of Building The Right Teams
Company Culture And The Importance of Building The Right Teams
 
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
 
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYEQCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
 
Concurrency at Scale: Evolution to Micro-Services
Concurrency at Scale:  Evolution to Micro-ServicesConcurrency at Scale:  Evolution to Micro-Services
Concurrency at Scale: Evolution to Micro-Services
 
Why Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudWhy Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the Cloud
 
Importance of Organizational Development
Importance of Organizational DevelopmentImportance of Organizational Development
Importance of Organizational Development
 
Nonverbalcommunication 1
Nonverbalcommunication 1Nonverbalcommunication 1
Nonverbalcommunication 1
 
รายงานการตรวจรถ
รายงานการตรวจรถรายงานการตรวจรถ
รายงานการตรวจรถ
 
Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2Oath appsec sf 2015 dem rev. 2
Oath appsec sf 2015 dem rev. 2
 
Ind eng-734-doc
Ind eng-734-docInd eng-734-doc
Ind eng-734-doc
 
Katrina sigma xi oct 2009 bob gilbert
Katrina sigma xi oct 2009 bob gilbertKatrina sigma xi oct 2009 bob gilbert
Katrina sigma xi oct 2009 bob gilbert
 
Social Experimentation
Social ExperimentationSocial Experimentation
Social Experimentation
 
Professional societies
Professional societiesProfessional societies
Professional societies
 
Human Subjects Regulations
Human Subjects RegulationsHuman Subjects Regulations
Human Subjects Regulations
 
Beyond Disaster Recovery
Beyond Disaster RecoveryBeyond Disaster Recovery
Beyond Disaster Recovery
 
Law Enforcement Oath of Honor
Law Enforcement Oath of HonorLaw Enforcement Oath of Honor
Law Enforcement Oath of Honor
 
Discovering a Career in Law Enforcement
Discovering a Career in Law EnforcementDiscovering a Career in Law Enforcement
Discovering a Career in Law Enforcement
 
Software Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEESoftware Engineering code of ethics and professional practice of IEEE
Software Engineering code of ethics and professional practice of IEEE
 
From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015
 

Similar to The Importance of Culture: Building and Sustaining Effective Engineering Organizations [QCon Beijing 2014]

Java DevOps at Enterprise Scale
Java DevOps at Enterprise ScaleJava DevOps at Enterprise Scale
Java DevOps at Enterprise ScaleRyan McGuinness
 
How to create awesome customer experiences
How to create awesome customer experiencesHow to create awesome customer experiences
How to create awesome customer experiencesMorgan Simonsen
 
UX in Action: IBM Watson
UX in Action: IBM WatsonUX in Action: IBM Watson
UX in Action: IBM WatsonUserTesting
 
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisationLean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisationCraig Sullivan
 
Lean UX in an Agency Environment
Lean UX in an Agency EnvironmentLean UX in an Agency Environment
Lean UX in an Agency EnvironmentJef Bekes
 
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...Maurizio Mancini
 
Rethinking Your DevOps Strategy
Rethinking Your DevOps StrategyRethinking Your DevOps Strategy
Rethinking Your DevOps StrategyMandi Walls
 
Holistic Product Development
Holistic Product DevelopmentHolistic Product Development
Holistic Product DevelopmentGary Pedretti
 
Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...Agile India
 
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011TEST Huddle
 
Adapt your testing for Agile
Adapt your testing for AgileAdapt your testing for Agile
Adapt your testing for AgileEmma Armstrong
 
Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test AutomationNaveen Kumar Singh
 
Agile Testing - What, why and how.
Agile Testing - What, why and how.Agile Testing - What, why and how.
Agile Testing - What, why and how.Asim Kazmi
 
Ten Lessons of the DevOps Transition
Ten Lessons of the DevOps TransitionTen Lessons of the DevOps Transition
Ten Lessons of the DevOps TransitionRandy Shoup
 
Getting By Without "QA"
Getting By Without "QA"Getting By Without "QA"
Getting By Without "QA"Dave King
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Kari Kakkonen
 
Kanban testing
Kanban testingKanban testing
Kanban testingCprime
 

Similar to The Importance of Culture: Building and Sustaining Effective Engineering Organizations [QCon Beijing 2014] (20)

Java DevOps at Enterprise Scale
Java DevOps at Enterprise ScaleJava DevOps at Enterprise Scale
Java DevOps at Enterprise Scale
 
How to create awesome customer experiences
How to create awesome customer experiencesHow to create awesome customer experiences
How to create awesome customer experiences
 
UX in Action: IBM Watson
UX in Action: IBM WatsonUX in Action: IBM Watson
UX in Action: IBM Watson
 
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisationLean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
Lean UX and Optimisation - Userzoom : 24 jan 2012 - lean optimisation
 
Lean UX in an Agency Environment
Lean UX in an Agency EnvironmentLean UX in an Agency Environment
Lean UX in an Agency Environment
 
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
Getting Agile Right - Rebooting an Agile Organization in 100 days - Agile Tou...
 
Rethinking Your DevOps Strategy
Rethinking Your DevOps StrategyRethinking Your DevOps Strategy
Rethinking Your DevOps Strategy
 
Holistic Product Development
Holistic Product DevelopmentHolistic Product Development
Holistic Product Development
 
Designing for efficiency.pdf
Designing for efficiency.pdfDesigning for efficiency.pdf
Designing for efficiency.pdf
 
Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...Building and Scaling High Performing Technology Organizations by Jez Humble a...
Building and Scaling High Performing Technology Organizations by Jez Humble a...
 
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
 
Adapt your testing for Agile
Adapt your testing for AgileAdapt your testing for Agile
Adapt your testing for Agile
 
Agile ux fullday-uxpa2016
Agile ux fullday-uxpa2016Agile ux fullday-uxpa2016
Agile ux fullday-uxpa2016
 
Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test Automation
 
QA Best Practices in Agile World_new
QA Best Practices in Agile World_newQA Best Practices in Agile World_new
QA Best Practices in Agile World_new
 
Agile Testing - What, why and how.
Agile Testing - What, why and how.Agile Testing - What, why and how.
Agile Testing - What, why and how.
 
Ten Lessons of the DevOps Transition
Ten Lessons of the DevOps TransitionTen Lessons of the DevOps Transition
Ten Lessons of the DevOps Transition
 
Getting By Without "QA"
Getting By Without "QA"Getting By Without "QA"
Getting By Without "QA"
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
 
Kanban testing
Kanban testingKanban testing
Kanban testing
 

More from Randy Shoup

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
Anatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsAnatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsRandy Shoup
 
Scaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermScaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermRandy Shoup
 
Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Randy Shoup
 
Breaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsBreaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsRandy Shoup
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsRandy Shoup
 
Learning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three IncidentsLearning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three IncidentsRandy Shoup
 
Managing Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsManaging Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsRandy Shoup
 
Service Architectures at Scale
Service Architectures at ScaleService Architectures at Scale
Service Architectures at ScaleRandy Shoup
 
Managing Data in Microservices
Managing Data in MicroservicesManaging Data in Microservices
Managing Data in MicroservicesRandy Shoup
 
Effective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldEffective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldRandy Shoup
 

More from Randy Shoup (11)

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
Anatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsAnatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and Lessons
 
Scaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermScaling Your Architecture for the Long Term
Scaling Your Architecture for the Long Term
 
Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020
 
Breaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsBreaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building Teams
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and Events
 
Learning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three IncidentsLearning from Learnings: Anatomy of Three Incidents
Learning from Learnings: Anatomy of Three Incidents
 
Managing Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsManaging Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and Events
 
Service Architectures at Scale
Service Architectures at ScaleService Architectures at Scale
Service Architectures at Scale
 
Managing Data in Microservices
Managing Data in MicroservicesManaging Data in Microservices
Managing Data in Microservices
 
Effective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldEffective Microservices In a Data-centric World
Effective Microservices In a Data-centric World
 

Recently uploaded

TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSedrianrheine
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxnaveenithkrishnan
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Shubham Pant
 
world Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptxworld Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptxnaveenithkrishnan
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteMavein
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfmchristianalwyn
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpressssuser166378
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdfShreedeep Rayamajhi
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilitiesalihassaah1994
 
Niche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus BonusNiche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus BonusSkylark Nobin
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...APNIC
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024Jan Löffler
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsRoxana Stingu
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSlesteraporado16
 
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptxA_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptxjayshuklatrainer
 

Recently uploaded (15)

TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptx
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024
 
world Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptxworld Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptx
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a Website
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpress
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilities
 
Niche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus BonusNiche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus Bonus
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
 
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptxA_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
 

The Importance of Culture: Building and Sustaining Effective Engineering Organizations [QCon Beijing 2014]

  • 1. The Importance of Culture Building and Sustaining Effective Engineering Organizations Randy Shoup @randyshoup linkedin.com/in/randyshoup
  • 2. Background CTO at KIXEYE • Real-time strategy games for web and mobile Director of Engineering for Google App Engine • World’s largest Platform-as-a-Service Chief Engineer at eBay • Multiple generations of eBay’s real-time search infrastructure
  • 3. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 4. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 5. Hire and Retain the Best Hire „A‟ Players • In creative disciplines, top performers are 10x more productive (!) Confidence • A players bring A players • B players bring C players
  • 6. Google Hiring Goal: Only hire top talent • False negatives are OK; false positives are not Hiring Process • Famously challenging interviews • Very detailed interviewer feedback • Hiring committee decides whether to hire • Separately assign new Googler to group  Highly talented and engaged employees
  • 7. Respect People People are not interchangeable • Different skills, interests, capabilities • Create a Symphony, not a Factory Most valuable and irreplaceable asset • Treat people with care and respect • If the company values its people, people will provide value to the company
  • 8. eBay “Train Seats” eBay‟s development process (circa 2006) • Design and estimate project (“Train Seat” == 2 engineer-weeks) • Assign engineers from common pool to implement tasks • Designer does not implement; implementers do not design  Dysfunctional engineering culture • (-) Engineers treated as interchangeable “cogs” • (-) No regard for skill, interest, experience • (-) No pride of ownership in task implementation • (-) No long-term ownership of codebase
  • 9. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 10. Service Teams • Small, focused teams • Single service or set of related services • Minimal, well-defined “interface” • Vendor – Customer relationships • Clear “contract” between teams • Functionality: agreed-upon scope of responsibility • Service levels and performance
  • 11. Google Services • All engineering groups organized into “services” • Gmail, App Engine, Bigtable, etc. • Self-sufficient and autonomous • Layered on one another  Very small teams achieve great things
  • 12. Autonomy and Accountability • Give teams autonomy • Freedom to choose technology, methodology, working environment • Responsibility for the results of those choices • Hold team accountable for *results* • Give a team a goal, not a solution • Let team own the best way to achieve the goal
  • 13. KIXEYE Service Chassis • Goal: Produce a “chassis” for building scalable game services • Minimal resources, minimal direction • 3 people x 1 month • Consider building on open source projects  Team exceeded expectations • Co-developed chassis, transport layer, service template, build pipeline, red-black deployment, etc. • Heavy use of Netflix open source projects • 15 minutes from no code to running service in AWS (!) • Plan to open-source several parts of this work
  • 14. Google and DevOps Ops Support is a privilege, not a right • Developers carry pager for first 6+ months • Service “graduates” to SRE after intensive review of monitoring, reliability, resilience, etc. • SRE collaborates with service to move forward Everyone‟s incentives are aligned • Everyone is responsible for production • Everyone strongly motivated to have solid instrumentation and monitoring
  • 15. Collaboration • One team across engineering, product, operations, etc. • Solve problems instead of pointing fingers
  • 16. Google Co-Location Multiple Organizations • Engineering • Product • Operations • Support • Different reporting structures to different VPs Virtual Team with Single Goal • All work to make Google App Engine successful • Coworkers are “Us”, not “Them” • When asked which teams we need to sit next to, it never occurred to us that other organizations were not “our team”
  • 17. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 18. Quality over Quantity Whole user / player experience • Think holistically about the full end-to-end experience of the user • UX, functionality, performance, bugs, etc. Less is more • Solve 100% of one problem rather than 50% of two • Users prefer one great feature instead of two partially-completed features
  • 19. Institutionalize Quality Development Practices • Code reviews • Continuous Testing • Continuous Integration Quality Automation • Automated testing frameworks • Canary releases to production “Make it easy to do the right thing, and hard to do the wrong thing”
  • 20. Google Engineering Discipline Solid Development Practices • Code reviews before submission • Automated tests for everything • Single logical source code repository  Internal Open Source Model • Not “here is a bug report” • Instead “here is the bug; here is the code fix; here is the test that verifies the fix” 
  • 21. Technical Tradeoffs Make Tradeoffs Explicit • Triangle: date vs. quality vs. features • When you choose date and features, you implicitly choose a level of quality Manage Technical Debt • Plan for how and when you will pay it off • Maintain sustainable and well-understood level of debt “Don‟t have time to do it right” ? • WRONG – Don’t have time to do it twice (!)
  • 22. Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation
  • 23. Constant Learning Any process, organization, or product can always be improved Mistakes are a learning opportunity • What did you do -> What did you *learn* • Take emotion and personalization out of it Encourage iteration and velocity • “Failure is not falling down but refusing to get back up” – Theodore Roosevelt
  • 24. Google Blame-Free Post- Mortems Post-mortem After Every Incident • Document exactly what happened • What went right • What went wrong Open and Honest Discussion • What contributed to the incident? • What could we have done better? Engineers compete to take personal responsibility (!)
  • 25. Google Blame-Free Post- Mortems Action Items • How will we change process, technology, documentation, etc. • How could we have automated the problems away? • How could we have diagnosed more quickly? • How could we have restored service more quickly? Follow up (!)
  • 26. Iteration and Experimentation *Engineer* successes • Constant iteration • Launch is only the first step • Assume you will not get it perfect on the first try • A / B Testing needs to be a core competence Many small experiments sum to big wins
  • 27. eBay Machine-Learned Ranking Ranking function for search results • Which item should appear 1st, 10th, 100th, 1000th • Before: Small number of hand-tuned factors • Goal: Thousands of factors Experimentation Process • Predictive models: query->view, view- >purchase, etc. • Hundreds of parallel A|B tests • Full year of steady, incremental improvements  2% increase in eBay revenue (~$120M)
  • 28. Recap: Building Blocks of Culture Hiring and Retention Ownership and Collaboration Quality and Discipline Learning and Experimentation