OSDC 2019 | Evolution of a Microservice-Infrastructure by Jan Martens
•
0 likes•185 views
This talk is about our journey from Nginx & Docker Swarm to Traefik & Nomad. With the increase of load & traffic on our container-environment over the time, we experienced some issues that were unnoticed when built the environment. Due to the way we dynamically configured Nginx with Consul-Template we started to experience a lot of dropped keepalive connections and connection resets. Also the traffic wasn’t distributed evenly throughout our container-infrastructure which led to single instances receiving most of the load. This is why made the decision to change to a reverse-proxy that can get dynamically configured. We were aware of the shortcommings of Docker Swarm (standalone) and seeked for a tool that would allow us to distribute containers more evenly without totally reconstructing everything and provide us with self healing capabilities. Performing these changes under the hood, transparent for our developers, was one of our key objectives.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to OSDC 2019 | Evolution of a Microservice-Infrastructure by Jan Martens
Similar to OSDC 2019 | Evolution of a Microservice-Infrastructure by Jan Martens (20)
Recently uploaded
Recently uploaded (20)
OSDC 2019 | Evolution of a Microservice-Infrastructure by Jan Martens
- 1. Evolution of a Microservice Infrastructure Jan Martens OSDC 2019, Berlin
- 2. 2 Details REWE Group Turnover >57 bn History >90 years Employees >345.000 Industries Food Retail, Tourism, DIY Shops >15.000
- 3. 3 What do we actually run?
- 4. Our history
- 5. Our history at REWE Digital
- 6. Service/team growth 2014 40 15 100 28 200 46 2015 2016 2017 # Services # Dev Teams 1 2 2018 270 48
- 7. Recap
- 8. We’re operating a custom Docker-Environment consisting of: Everything was cool. Developers could bring Code live. All was well. Recap The current state of 2018, achievements so far
- 9. … and looks like Ingress-Nodes Consul-Server a.k.a Master-Nodes Docker-Hosts a.k.a. Worker-Nodes Other “aaS” * nS1 S1 S2 S2 S1 S1S1 S2
- 10. … and looks like Ingress-Nodes ● Nginx-config written by consul-template on change of Consul-information ● Routes external Hostnames Ingress-Nodes Consul-Server a.k.a Master-Nodes Docker-Hosts a.k.a. Worker-Nodes Other “aaS” * nS1 S1 S2 S2 S1 S1S1 S2
- 11. … and looks like Ingress-Nodes Consul-Server a.k.a Master-Nodes Docker-Hosts a.k.a. Worker-Nodes Other “aaS” * nS1 S1 S2 S2 S1 S1S1 S2 Docker-Host ● Nginx-config written by consul-template on change of Consul-information ● Routes internal Hostnames to containers ● Runs containers
- 12. … and looks like Ingress-Nodes Consul-Server a.k.a Master-Nodes Docker-Hosts a.k.a. Worker-Nodes Other “aaS” * nS1 S1 S2 S2 S1 S1S1 S2 Consul-Server & Swarm-Master ● Contain knowledge of all services ● Deployments are started from here ● Act as DNS-Servers for service-discovery
- 13. … and looks like Ingress-Nodes Consul-Server a.k.a Master-Nodes Docker-Hosts a.k.a. Worker-Nodes Other “aaS” * n Other centrally managed “platform-services” ● Kafka ● Databases ● ELK-Stack ● Prometheus & Grafana ● ... S1 S1 S2 S2 S1 S1S1 S2
- 14. Deployments
- 15. ● Blue-Green ○ only one color can be active ● Independent set of containers per color ○ started via docker-compose Deployments how we do them
- 16. ● Teams can deploy themselves ● Color Switch per Service ● Both colors addressed via the same DNS record Deployments how we do them
- 19. Deployments docker-2docker-1 docker-3 Consul Key/Value S1 S2 S2 S1 deploy basket blue basket
- 20. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value S1 S2 S2 S1 deploy basket blue basket
- 21. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value S1 S2 S2 S1 basket reload Nginx
- 22. Deployments docker-2docker-1 docker-3 basket: - active: blue Consul Key/Value S1 S2 S2 S1 basket active: blue
- 23. Deployments docker-2docker-1 docker-3 basket: - active: blue Consul Key/Value S1 S2 S2 S1 basket reload Nginx active: blue
- 24. Deployments docker-2docker-1 docker-3 basket: - active: blue Consul Key/Value S1 S2 S2 S1 basket reload Nginx active: blue deploy basket blue
- 25. ● Both colors have the same DNS record ○ Consul will return IPs for all hosts where the Service is running ● Nginx running on each Worker Node ○ routes to color depending on used Port Request routing how can services be addressed
- 34. ProblemsNginx
- 35. ● Requests which never reached their destination ● Keepalive connections dropped after short time Always happened at the time of deployments Problems with Nginx increased with the size of the environment
- 36. ● Requests which never reached their destination ● Keepalive connections dropped after short time Always happened at the time of deployments ● Consul-template would reload all Nginx instances at the same time Problems with Nginx increased with the size of the environment
- 37. ● Look for different reverse proxy ○ no reload on config change ○ dynamic configuration Problems with Nginx looking for solutions
- 38. Problems with Nginx possible replacements
- 39. ● Dynamically configurable ● Live reloading of configuration ● Lots of metrics ● Nice web ui ● Single Go binary Traefik
- 40. 1. Install alongside Nginx on Worker and Ingress Nodes ○ listen on different ports 2. Check that configured routes are correct and work 3. Change port mapping host by host 4. Remove Nginx Traefik how to migrate
- 42. Traefik how to migrate :80 docker-1 basket basket-service.rewe docker-1.rewe docker-2.rewe :10080
- 43. Traefik how to migrate :80 docker-1 basket basket-service.rewe docker-1.rewe docker-2.rewe :10080
- 45. ● Keepalive and connection problems immediately went away ● Almost real time data about service response time ● Web UI to check routes ● Rich access logs Traefik Benefits
- 46. Traefik Benefits
- 47. Traefik Benefits
- 48. ProblemsSwarm
- 49. ● Poor container spread ○ all service instances running on one host ● No self healing ● Manual node draining ○ dependent on docker-compose files Problems with standalone Swarm also increased
- 50. ● Look for different container Orchestrator ○ self healing ○ proper container spread Problems with standalone Swarm looking for solutions
- 51. Problems with standalone Swarm possible replacements
- 52. ● Seamless Consul integration ○ almost no setup needed ● Self healing ● Bin packing ● Single Go binary ● Nice Web UI Nomad
- 53. 1. Install alongside Swarm on Worker and Master Nodes ○ agnostic of other Docker Containers 2. Modify deployment Jobs ○ Start new deployments via Nomad 3. Remove Swarm Nomad how to migrate
- 54. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value S1 S2 S2 S1 deploy basket blue basket
- 55. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value S1 S2 S2 S1 deploy basket blue basket
- 56. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value S1 S2 S2 S1 deploy basket blue
- 57. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value S1 S2 S2 S1 deploy basket blue basket
- 58. Deployments docker-2docker-1 docker-3 basket: - active: green Consul Key/Value deploy basket blue basket S1 S2 S2 S1
- 59. ● Ensures proper container spread ● Self healing on Node outage ● ACLs ● Job history Nomad Benefits
- 60. ● Not limited to Docker ○ Rkt and LXC ● Not limited to Containers ○ Jar files ○ Binaries ○ VMs Nomad Benefits
- 61. Nomad Benefits
- 62. Nomad Benefits
- 63. Nomad Benefits
- 64. What we Learned
- 65. ● Having a centralised deployment-toolset ○ perform all changes for all teams / developers at the same time ● Do Canary-like changes on our infrastructure ○ fully interoperable changes ○ nginx <-> Traefik What helped us most?
- 66. ● You might not need Kubernetes What did we learn?
- 67. ● You might not need Kubernetes ● Keeping your architecture pluggable helps What did we learn?
- 68. ● You might not need Kubernetes ● Keeping your architecture pluggable helps ● Computing resources are finite ○ Setting resource limits can be difficult What did we learn?
- 69. ● You might not need Kubernetes ● Keeping your architecture pluggable helps ● Computing resources are finite ○ Setting resource limits can be difficult ● Distributed systems can be hard What did we learn?
- 70. Thanks go to:
- 71. Thank You! Jan Martens | github.com/jan-martens www.rewe-digital.com | @rewedigitaltech All background pictures are licensed under CC0. Source: pexels.com Evolution of a Microservice Infrastructure OSDC 2019, Berlin