We recently started researching and developing a Module for Icinga to monitor Kubernetes environments. During the past months we learned a lot about the platform and how we can monitor Kubernetes with Icinga efficiently. In this talk I will present our challenges but also the progress that we made. The talk will include a sneak peak into the current state of the Module and outline our vision of monitoring Kubernetes with Icinga.

1. Eric Lippmann | Open Source Camp on Kubernetes | Apr 18, 2024
Monitor K8s with Icinga

2. Eric, CTO @ Icinga

3. Traditional Monitoring
• Hosts
• Physical servers, virtual machines
• Cloud instances to some extent
• Services
• Resource usage
• Applications

4. Traditional Monitoring
• Configuration defines what to monitor
and how
• Automation
• Configuration Management
• Icinga Director
• Icinga APIs

5. Traditional Monitoring
• Logs
• Check Plugins
• Metrics
• Alerts

6. Check Plugins
• Logic how to get and interpret data
• Common understanding of what is wrong
• Reasonable defaults for thresholds
• State, text output
• Metrics

7. Kubernetes
Monitoring in the container deployment era

9. K8s Monitoring – Complexity
• Loads of resource types
• Multiple components and layers
• Different failure points
• Understanding of the entire stack
Via hosts, services and check plugins?

10. K8s Monitoring – Challenges
Run
Fail
Respawn
Run
Fail
....

11. K8s Monitoring– What?
• Nodes (where K8s components run)
• K8s itself
• Resources
• Key metrics
• Applications

12. K8s Monitoring – Use Probes
Liveness probes periodically check container liveness and
restart containers that fail it.
Readiness probes indicate container readiness and remove
failing ones from their service endpoints.
Startup probes defer the execution of liveness and readiness
probes and restarts containers that fail it.

13. K8s Monitoring – Approaches
• Poll K8s APIs
• Agent per node via DaemonSet
• Agent per pod (sidecar/ephemeral containers)
• Events
• Metrics
• Logs

14. K8s Monitoring – Metrics
• Various metric sources, e.g cAdvisor,
/metrics, kube-state-metrics, Prometheus
node exporter
• Loads of metrics
Understand which metrics there are,
what they mean and how to interpret
them.
Cluster
Node
Workload Management
Pod
Container

15. K8s Monitoring – Health
Health
Expectations
Conditions,
State
Metrics
Events

16. Monitoring Kubernetes
with Icinga

17. Icinga K8s Monitoring
• Collects K8s resources and their
• Configuration
• State
• Logs
• Events
• Metrics
• Visualizes K8s resources and hierarchies

18. Icinga K8s Monitoring Architecture
• Icinga Web Module (PHP)
• View resources and hierarchies
• Daemon (Go)
• Collect resources, health, events,
logs and certain metrics
• Database
• Stores configuration, resources, health

19. Icinga K8s Monitoring Deploy Options
• Can run either inside or outside Kubernetes clusters
• Installation via packages
• Container
• Helm chart

20. Demo

21. Icinga K8s Monitoring
• https://github.com/icinga/icinga-kubernetes
• https://github.com/icinga/icinga-Kubernetes-web
• https://github.com/Icinga/helm-charts

22. What’s Next
• Correlate health, logs, metrics and events
• Provide alerts
• Refine UI
• Add charts

23. x.com/icinga github.com/icinga facebook.com/icinga
icinga.com
Thank You!