4. No centralised logging
User needs OS
knowledge
Distribution
Of keys
Enemy of
autoscaling
Log
rotation
Users download
logs unnecessarily
Doesn’t scale
To many servers
Slow to
find issues
Alerting
is hard
Sshing to
servers :(
7. SteamhausKELK ON AWS
KELK on AWS
• Low maintenance - No ec2, Uses entirely AWS serverless technologies and services
• ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs
• Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short
term analytics
• Automated with Terraform
• Open source
Kinesis: buffering and delivering instance logs
Elasticsearch: Indexing and log storage
Lambda: processing and delivering S3 logs
Kibana: Search and analytics
21. SteamhausKELK ON AWS
Automation
code
Sample
Web Stack
VPC
ALB
EC2
Logging
Stack
Kinesis
Elasticsearch
Service
Lambda
S3
CloudfrontPython
Terraform
Do try this at home!
github.com/steamhaus/kelk-example
22. SteamhausKELK ON AWS
Callouts from the build
• It’s not production ready, built for readability
• Nailing iam and bucket policies can take a while!
• Testing lambda - create a test event in the UI
• Use Terraform, rinse and repeat