We recently started researching and developing a Module for Icinga to monitor Kubernetes environments. During the past months we learned a lot about the platform and how we can monitor Kubernetes with Icinga efficiently. In this talk I will present our challenges but also the progress that we made. The talk will include a sneak peak into the current state of the Module and outline our vision of monitoring Kubernetes with Icinga.
6. Check Plugins
• Logic how to get and interpret data
• Common understanding of what is wrong
• Reasonable defaults for thresholds
• State, text output
• Metrics
9. K8s Monitoring – Complexity
• Loads of resource types
• Multiple components and layers
• Different failure points
• Understanding of the entire stack
Via hosts, services and check plugins?
12. K8s Monitoring – Use Probes
Liveness probes periodically check container liveness and
restart containers that fail it.
Readiness probes indicate container readiness and remove
failing ones from their service endpoints.
Startup probes defer the execution of liveness and readiness
probes and restarts containers that fail it.
13. K8s Monitoring – Approaches
• Poll K8s APIs
• Agent per node via DaemonSet
• Agent per pod (sidecar/ephemeral containers)
• Events
• Metrics
• Logs
14. K8s Monitoring – Metrics
• Various metric sources, e.g cAdvisor,
/metrics, kube-state-metrics, Prometheus
node exporter
• Loads of metrics
Understand which metrics there are,
what they mean and how to interpret
them.
Cluster
Node
Workload Management
Pod
Container
15. K8s Monitoring – Health
Health
Expectations
Conditions,
State
Metrics
Events