We had an incident which did occur in our warehouse at KRUU. The downloading of the photos was very slow from one to the other day - well we thought that the problem started on this day. Actually we did notice this very late and the problem started two years ago but we did notice this very late due to the reduced rentals because of Covid19. This will never happen again thanks to our new metrics and alerting powered by the OpenSearch!

1. Hi, I am Michi!
Head of Code at
@michilehr

2. YOU CAN’T TEST EVERYTHING
BUT YOU SHOULD MONITOR IT!
11. October 2022 @ Nerd-BBQ

3. "As Europe's leading photo booth provider, we have
made it our mission to help our brides and grooms
with their complete journey to their dream wedding.
This is something we work tirelessly on with our
team."
What is doing?
Philipp Schreiber - Co-Founder, KRUU.com

5. Photo Booth
Cycle

7. The Incident

8. Good
~12 MB/s

9. 0.95 MB/s
Bad

11. 1. When did it start?
2. Why did it happen?
3. How to prevent?
4. How to notice early?
Investigate

12. When did it start?
We had data in our Slack Channel, but…

13. 1. Write a script to extract the data as CSV
2. Import data to MySQL
3. Write query to aggregate by day
4. Create nice graph

15. Started long time ago…

16. What happened?
Network configuration error

17. How to prevent?
No idea. Things like this happen

18. How to notice early?

22. How to notice early?

23. Alerts!

24. Query

25. Trigger

26. Notification

29. ● 404 alert by threshold
● Auth failure alert by threshold to
detect brute force
● …
What next?

30. Thank you for your time!
Questions?
Feedback?
Notes?