A Retrospective on Leadpages’ Microservicization Journey

1. THE COST OF MICROSERVICES
A RETROSPECTIVE ON LEADPAGES’ MICROSERVICIZATION JOURNEY
MICHAEL NADEL / ENGINEERING MANAGER, LEADPAGES / @MNADEL

2. WHO’S LEADPAGES?
▸ Minneapolis-based marketing technology startup founded
in 2012
▸ We make software that lets small businesses quickly build
high-converting landing pages and other lead capture
tools

3. WHAT’S A LANDING PAGE?
▸ A standalone web page
▸ Designed for a single purpose
▸ Direct-marketing technique to grow engagement with and
commitment from customers

6. LEADPAGES INTEGRATIONS
▸ Add email address to a mailing list
▸ And also add the contact to your CRM
▸ And also integrate with Stripe

7. ERA OF THE
MONOLITH
2012

8. 2012 — 2014
GROWTH
▸ Double-digit month-over-month top line growth
▸ Raised $38M venture capital
▸ Engineering grew from 1 → 24

9. EARLY 2015
ENGINEERING ORGANIZATION
▸ 2 teams of 12 developers
▸ Separate QA & PM functions
▸ Still rockin’ the monolith

10. MID 2015
PAIN POINTS
▸ Outgrown the PaaS monolith

11. MID 2015
PAIN POINTS
▸ Outgrown the PaaS monolith
▸ Scaling a monolith on PaaS

12. END 2015

13. END 2015
GOALS
▸ Decouple development teams
▸ Develop a clear ownership model
▸ Better control over costs

14. YEAR OF THE
MICROSERVICE
2016

15. 2016

16. 2016
THERE WERE CHALLENGES
▸ Organizational
▸ Technical
▸ Cultural

17. 2016
ORGANIZATIONAL CHALLENGES
▸ Centralized QA & PM no longer serving us well

18. 2016
ORGANIZATIONAL CHALLENGES
▸ Centralized QA & PM no longer serving us well
▸ Lack of focus on operational concerns

19. 2016
ORGANIZATIONAL CHALLENGES
▸ Centralized QA & PM no longer serving us well
▸ Lack of focus on operational concerns
▸ Diverted a meaningful portion of our development
resources

20. 2016
TECHNICAL CHALLENGES
▸ There are always challenges
▸ The way you navigate them…

21. 2016
CULTURAL CHALLENGES

22. 2016
CULTURAL CHALLENGES
It’s a paradigm shift!

23. 2016
CULTURAL CHALLENGES
It’s a paradigm shift!
It will require people to change.

24. 2016
CULTURAL CHALLENGES
It’s a paradigm shift!
It will require people to change.
Change is hard.

25. 2016
CULTURAL CHALLENGES
It’s a paradigm shift!
It will require people to change.
Change is hard.
It really is a paradigm shift.

26. 2016
CULTURAL CHALLENGES
It’s a paradigm shift!
It will require people to change.
Change is hard.
It really is a paradigm shift.
Change is really hard.

27. 2016
PARADIGM SHIFT
▸ New programming model

28. 2016
PARADIGM SHIFT
▸ New programming model
▸ Brand new failure modes

29. 2016
WE WERE ASKING PEOPLE TO…
▸ Unlearn and abandon the familiar
▸ Move out of their comfort zone
▸ Climb a steep learning curve

30. 2016
CHANGE IS HARD
▸ People can be resistant to change
▸ Consensus-building eventually gave way to mandates
▸ People unintentionally undermining progress

31. 2016
THE HARD TRUTH
▸ Goals/aspirations were aligned to the monolithic
paradigm

32. 2016
THE HARD TRUTH
▸ Goals/aspirations were aligned to the monolithic
paradigm
▸ It’s a learning curve; not everyone willing to invest in
climbing our specific curve

33. 2016
WHERE WE LANDED
▸ 35 — 40 services in production
▸ 36 engineers across 4 teams
▸ Developers owned quality, practiced TDD
▸ PM → Product Owners, embedded in team
▸ Infra team grew from 0 → 4

34. 2016
HOW’D WE DO AGAINST OUR GOALS?

35. 2016
HOW’D WE DO AGAINST OUR GOALS?
▸ Decoupled teams

36. 2016
HOW’D WE DO AGAINST OUR GOALS?
▸ Decoupled teams
▸ Physically isolated APIs

37. 2016
HOW’D WE DO AGAINST OUR GOALS?
▸ Decoupled teams
▸ Physically isolated APIs
▸ Clear ownership model

38. 2016
HOW’D WE DO AGAINST OUR GOALS?
▸ Decoupled teams
▸ Physically isolated APIs
▸ Clear ownership model
▸ Yeah…about that…

39. 2016
HOW’D WE DO AGAINST OUR GOALS?
▸ Decoupled teams
▸ Physically isolated APIs
▸ Clear ownership model
▸ Yeah…about that…
▸ Control over costs

40. 2016
HOW’D WE DO AGAINST OUR GOALS?
▸ Decoupled teams
▸ Physically isolated APIs
▸ Clear ownership model
▸ Yeah…about that…
▸ Control over costs
▸ IaaS more flexible than PaaS

41. BUT…

42. END 2016

43. END 2016
PHYSICAL ARCHITECTURE
HAProxy
API API
Worker Worker

44. END 2016
NEW COST MODEL
▸ 5 VMs per service

45. END 2016
NEW COST MODEL
▸ 5 VMs per service
▸ 40 services

46. END 2016
NEW COST MODEL
▸ 5 VMs per service
▸ 40 services
▸ 2 environments

47. END 2016
NEW COST MODEL
▸ 5 VMs per service
▸ 40 services
▸ 2 environments
▸ = 400 VMs

48. END 2016
NEW HARD TRUTH
▸ Operational complexity

49. END 2016
NEW HARD TRUTH
▸ Operational complexity
▸ + Risky deployments

50. END 2016
NEW HARD TRUTH
▸ Operational complexity
▸ + Risky deployments
▸ = Reduced overall velocity

51. END 2016
NEW GOALS!
▸ Simplify to reduce complexity and increase velocity
▸ Reduce operational expenses

52. HOW DO WE CONVINCE PEOPLE THAT IN
PROGRAMMING SIMPLICITY AND CLARITY…
ARE NOT A DISPENSABLE LUXURY, BUT A
CRUCIAL MATTER THAT DECIDES BETWEEN
SUCCESS AND FAILURE?
Edsger Dijkstra

53. 2017
LEVERS WE FOCUSED ON
▸ Simplification
▸ Reduce deployment risk
▸ Cost reduction
▸ Increase resource utilization

54. THE YEAR OF
KUBERNETES
2017

55. 2017
KUBERNETES — SIMPLIFICATION
▸ Google Kubernetes Engine
▸ Fully-managed Kubernetes

56. 2017
KUBERNETES — SIMPLIFICATION
▸ Google Kubernetes Engine
▸ Fully-managed Kubernetes
▸ Containers
▸ Fixed dependency hell & deployment pipelines

57. 2017
KUBERNETES — SIMPLIFICATION
▸ Google Kubernetes Engine
▸ Fully-managed Kubernetes
▸ Containers
▸ Fixed dependency hell & deployment pipelines
▸ Chef/Packer/Terraform/Ansible
▸ Dockerfile + Templatized K8s manifests

58. 2017
KUBERNETES WINS
▸ Resource utilization
▸ 2 VMs staging, 6 VMs production

59. 2017
KUBERNETES WINS
▸ Resource utilization
▸ 2 VMs staging, 6 VMs production
▸ 60 — 75% compute savings

60. 2017
KUBERNETES WINS
▸ Resource utilization
▸ 2 VMs staging, 6 VMs production
▸ 60 — 75% compute savings
▸ Faster autoscaling

61. 2017
KUBERNETES WINS
▸ Resource utilization
▸ 2 VMs staging, 6 VMs production
▸ 60 — 75% compute savings
▸ Faster autoscaling
▸ True high availability

62. END 2017

63. RETRO!

64. END 2017
TAKEAWAYS
▸ Cultural
▸ Technical
▸ Financial

65. END 2017
TAKEAWAYS — CULTURAL
▸ You cannot overestimate the cultural/people impact

66. END 2017
TAKEAWAYS — CULTURAL
▸ You cannot overestimate the cultural/people impact
▸ Give people room to unlearn, learn, and relearn

67. END 2017
TAKEAWAYS — CULTURAL
▸ You cannot overestimate the cultural/people impact
▸ Give people room to unlearn, learn, and relearn
▸ Actively seek out diverse perspectives

68. END 2017
TAKEAWAYS — CULTURAL
▸ You cannot overestimate the cultural/people impact
▸ Give people room to unlearn, learn, and relearn
▸ Actively seek out diverse perspectives
▸ Ensure you have 100% commitment from leadership

69. END 2017
TAKEAWAYS — TECHNICAL
▸ Minimize the slope of the learning curve

70. END 2017
TAKEAWAYS — TECHNICAL
▸ Minimize the slope of the learning curve
▸ Know your tools/platform and their constraints/tradeoffs

71. END 2017
TAKEAWAYS — TECHNICAL
▸ Minimize the slope of the learning curve
▸ Know your tools/platform and their constraints/tradeoffs
▸ Be afraid of new technologies

72. END 2017
TAKEAWAYS — TECHNICAL
▸ Minimize the slope of the learning curve
▸ Know your tools/platform and their constraints/tradeoffs
▸ Be afraid of new technologies
▸ Don’t be afraid to experiment

73. END 2017
TAKEAWAYS — FINANCIAL
▸ Pay attention to resource utilization

74. END 2017
TAKEAWAYS — FINANCIAL
▸ Pay attention to resource utilization
▸ Consider the compute density of your stack

75. END 2017
TAKEAWAYS — FINANCIAL
▸ Pay attention to resource utilization
▸ Consider the compute density of your stack
▸ Understand the cost model of your physical architecture

76. PROBABLY I AM VERY NAIVE, BUT I
ALSO THINK I PREFER TO REMAIN SO,
AT LEAST FOR THE TIME BEING AND
PERHAPS FOR THE REST OF MY LIFE.
Edsger Dijkstra

77. THANKS FOR LISTENING!
Michael Nadel