In this presentation of Venkata Ramana Gollamudi, Principal Architect at Flipkart, as our esteemed speaker. With his expertise and experience, he'll guide us through the intricate process of “Scaling web service to handle millions of requests per second”. Key Takeaways: Understanding the specific challenges of scaling for Indian users Strategies to ensure service availability, reduce latency, and maintain reliability Leveraging tools like ELBs, K8s, message queues, and NoSQL databases effectively Don't miss this unparalleled opportunity to gain invaluable insights and elevate your expertise in scaling web services to new heights. We eagerly await your participation in this transformative webinar. Let's embark on the journey to mastering scalability together!

1. www.webstackacademy.com
Scaling Web Service
to Handle Millions of Requests per Second
Presentation – 23rd
March 2024
Venkata Ramana Gollamudi (Senior Principal Architect @ Flipkart)

2. www.webstackacademy.com
About Me
With 20+ Years of Software development Experience.
Senior Principal Architect @ Flipkart solutioning Data Platform and Data
Engineering. This platform support data and real time analytics on 100s of PB of
data.
Open Source Apache Committer.
Key expertise in designing scalable distributed processing and analytical systems.
New Interest is on NLP based data analytics on Big Data

3. www.webstackacademy.com
Agenda
1. Scale for India, expectations (impact)
2. Case1: Journey of scaling a simple lookup web service
a. Challenges
b. Using K8s
c. Using NoSQL stores
3. Case2: Journey of scaling a simple order placement service
a. Challenges
b. Queues and Event driven Architecture
4. Typical network Scalability model

4. www.webstackacademy.com
Scale for India
1. 1.4 billion visits during Big Billion Day 2023 Sales
2. 4 million packages delivered in first 4 days of Sale.

5. www.webstackacademy.com
www.webstackacademy.com
Case 1: Journey of scaling a simple
lookup web service

6. www.webstackacademy.com
Problem statement
Webservice
GetProductDetail
MySQL
1 VM
10 Requests/sec
10 Requests/sec

7. www.webstackacademy.com
Try Scaling - Challenges
Load Balancer
Webservice
GetProductDetail
MySQL
10 VM
100 Requests/sec
100 Requests/sec
Challenges
- Request routing across VMS
- Each VM load protection
- Images and common data needs
catching
- VM is down
- Load protection on MySQL
- MySQL Instance is down
vm1 vm2 vm10
…
Multi Master,
Master slave replica etc
MySQL Router /
Load Balancer

8. www.webstackacademy.com
Scale further - Challenges
Load Balancer
Webservice
GetProductDetail
MySQL
1000 VMs
10000 Requests/sec
10000 Requests/sec
Challenges
- Request routing across VMs
- Each VM load protection
- Images and common data needs
catching
- VM is down
- Load protection on MySQL
- MySQL Instance is down
- Cannot scale horizontally
vm1 vm2 vm1000
…
100s of Nodes
Multi Master,
Master slave replica etc
MySQL Router /
Load Balancer

9. www.webstackacademy.com
Scaling Web service using K8s
Replaced
with
What it brings in
● Containerization
● Scalability
● High Availability
● Resource Efficiency
● Self-Healing
● Portability
● DevOps Enablement &
Deployment Spec
Webservice
vm1 vm2 vm1000
…

10. www.webstackacademy.com
Scaling Web service using K8s (contd..)
Load Balancer
Container1
(Bench marked)
Container2
Container
1000
…
● Liveness Probes
● Readiness Probe
● Startup Probe
Readiness
Probes
Control Plane (Autoscaler)
Startup probe
Liveness probe

11. www.webstackacademy.com
Scaling Database using NoSQL Database
Replaced
with
What it brings in
● Key-Value Store
● High Scalability
● High Availability
● Efficiency
● Row level Consistency
● In-Memory Cache
MySQL
100s of Nodes
Multi Master,
Master slave replica etc
MySQL Router /
Load Balancer
RegionServer
Region
Region
RegionServer
Region
Region
…
RegionServer
Region
Region
1000s of Nodes
Web Service
(Clients)

12. www.webstackacademy.com
Scaling Database using NoSQL Database (contd)
What is brings in
● Key Sharding
● Fast lookup
● Fast Update
Key Range 00 - 15 16 - 30 31 - 45 46 - 60 61 - 75 75 - 100

13. www.webstackacademy.com
Scaling Database using NoSQL Database (contd)
Load Balancer
Web service
GetProductDetail
1000s of Containers
Million Requests/sec
Million Requests/sec
Cont1 …
CDN
Cont1 Cont1k
RegionServer
Region
Region
RegionServer
Region
Region
…
RegionServer
Region
Region
1000s of Nodes

14. www.webstackacademy.com
www.webstackacademy.com
Case 2: Journey of scaling a
simple order placement service

15. www.webstackacademy.com
Problem statement
Order Service
PlaceOrder
MySQL
Track Order
Deliver Order
1 VM
10 Requests/sec
10 Requests/sec
Read/Write

16. www.webstackacademy.com
Try scaling
Order Service
PlaceOrder Track Order
Deliver
Order 1000s Place Order Requests/sec
1000s Deliver Order Requests/sec
Million Track Order Requests/Sec
1000s of Containers
1000s Write Requests/sec
Million Read Requests/sec
Read/Write
NoSQL
Load Balancer

17. www.webstackacademy.com
Try scaling - Challenges
Order Service
PlaceOrder Track Order
Deliver
Order 1000s Place Order Requests/sec
1000s Deliver Order Requests/sec
Million Track Order Requests/Sec
1000s of Containers
1000s Write Requests/sec
Million Read Requests/sec
Read/Write
NoSQL
Load Balancer
Challenges
● Different Services requests have
different resources requirement
● Adding new Service is tough
● Latency of Place Order is high
● Track Order and Place Order have
different scale requirements
● Not all operations are synchronous

18. www.webstackacademy.com
User Queues and Event based design
Order Service
PlaceOrder
Track
Order
Deliver
Order
NoSQL
Load Balancer
Order Tracking
Service
NoSQL
Fulfillment
Service
NoSQL
Load Balancer Load Balancer
Fraud Detection
Service
Graph Database
Load Balancer
What it brings in
● Independent scaling
● Low latency Async Processing
● High throughput Scalable
Queues
● Resource Efficiency
● Independent tech stack
● Easy New service addition
● DevOps independence
Advertisement
Service
NoSQL Database
Load Balancer

19. www.webstackacademy.com
www.webstackacademy.com
Typical Network scalability Model

20. www.webstackacademy.com
Typical network scalability model

21. www.webstackacademy.com
Typical network CDN placement

22. www.webstackacademy.com
www.webstackacademy.com ‹#›
Thank you

23. www.webstackacademy.com
MCQ Questions
1. Which Databases preferred to Scale to Million Read/Write Requests per sec
a. Transactional Databases
b. NoSQL databases
c. File Based Stores
d. None of the above
Answer: b

24. www.webstackacademy.com
MCQ Questions
2. K8s brings in
a. Scalability
b. Self Healing
c. Containerization and portability
d. All of the above
Answer: d

25. www.webstackacademy.com
MCQ Questions
3. Example of NoSQL databases
a. MySQL
b. HBase
c. K8s
d. Kafka
Answer: b

26. www.webstackacademy.com
MCQ Questions
4. HBase scalability is achieved using
a. Master Slave model
b. Sharding key space
c. Multiple clusters
d. Distributed transactions
e. All of the above
Answer: b

27. www.webstackacademy.com
MCQ Questions
5. Load balancer will identify Active Containers to route traffic using
a. Readiness Probe
b. Round robin distribution
c. Errors
d. Latency of request
Answer: a

28. www.webstackacademy.com
MCQ Questions
6. Event based async architecture requires ______ for inter service communication
a. K8s
b. NoSql Databases
c. Queues
d. Load balancer
Answer: c

29. www.webstackacademy.com
MCQ Questions
7. Event based async architecture benefits
a. Services can independently scale
b. Addition of new service is easy
c. Can choose different tech stack and stores based on need
d. All of the above
Answer: d

30. www.webstackacademy.com
MCQ Questions
8. Popular implementation of Queues
a. HBase
b. Cassandra
c. Apache kafka
d. K8s
Answer: c

31. www.webstackacademy.com
MCQ Questions
9. CDN useful to store
a. Images and Videos
b. Order status information
c. Delivery status information
d. All of the above
Answer: a

32. www.webstackacademy.com
MCQ Questions
10. Typical path for mobile to web service traffic is
a. Mobile -> POP endpoint -> Internet -> Data center
b. Mobile -> Internet -> POP endpoint -> Data center
c. Mobile -> Internet -> POP endpoint -> Internet -> Data center
d. All of the above
Answer: b