3. • Why do we need Logging ?
– Troubleshoot Issues
– Security
• Analyze logs to detect patterns
• Detect Malware Activity - Intrusion Detection, Denial of Service
• Unauthorized Resource Usage
– Monitoring
• Monitor Resource Usage
• Developers and Logging
– Logging Aids in Development ?
– Forget about Production !!!!!
Logging
4. • “Capture-it-all” Approach
• What to Log? Everything
• DevOps Movement
• Logs are archived for years
• Big Data
• Application Usage Statistics
Logging
5. • Searching the logs
– Command line, cat, tail, sed, grep, awk
– Regular Expressions
• Multiple Servers behind the load balancer
• Multi-Tier Architecture
– Web Application
– Service Layer
– Correlation between various components in a System
• Geographically distributed
– Timestamps
Log management
6. • Centralize all the Logs
– Too much information to go through
– Increasingly hard to correlate the contextual Data
• Add Searching and Indexing Technology
– grep
– Custom logging frameworks , custom integration of logging, searching
technologies
• Monitor the Logs
Log management
7. • Logstash to the Rescue
–Integration Framework
• Log Collection
• Centralization
• Parsing
• Storage and Search
Logstash
8. • JRuby
– Run on Java Virtual Machine (JVM)
– Simple Message Based Architecture
– Single Agent that can be configured for multiple things
– OPEN SOURCE
• Four Components
– Shipper
– Broker and Indexer
– Search and Storage
– Web Interface
Logstash
10. Architecture - Broker
• Acts as Temp Buffer between Logstash Agents
and the Central server
– Enhance Performance by providing caching buffer
for log events
– Adds Resiliency
• Incase the Indexing fails, the events are held in a queue
instead of getting lost
• AMQP,0MQ, Redis
11. • Indexing and Searching Tool
– Built on Lucene
• Search and Index data available Restfully as JSON over HTTP
• Comes bundled with Logstash – embedded
• Text indexing Search Engine
– Searches on the Index rather than on the content
• Creates Indexes of the incoming content
– Uses Apache Lucene to create Indexes
• ElasticSearch can have a schema – Fields on which Indexes are
created
ElasticSearch
12. • Indexes are stored in Lucene Instances called
“Shards”
• ElasticSearch can have multiple nodes
• Two Types of Shards
– Primary
– Replica
• Replicas of Primary Shards
– Protect the data
– Make Searches Faster
ElasticSearch
13. • Wouldn’t it be good to have a webpage to do search on
ElasticSearch instead of searching it through a Service
• Kibana provides a Simple but Powerful web Interface
– Customizable Dashboards
– Search the log events
• Support Lucene Query Syntax
– Creation of tables, graphs and sophisticated visualizations
Kibana