As a small startup team of developers, release engineering and quality assurance was inherently problematic. To optimize these processes, Sensu implemented a full automated test infrastructure for staging and end-to-end testing, which later became known as the QA Crucible. To increase observability on the overall ecosystem, Nikki started to research the intersection of monitoring and testing in a CI/CD pipeline, and sought to implement existing tooling to further optimize the workflow. By instrumenting JSON test results within a monitoring solution, she discovered a major upside to how tests are run, visualized, responded to, and remediated. Not only can tests be run continuously, but results can also be routed through an event pipeline for data manipulation, visibility and alerting. This pattern allows operators to treat test failures as incidents, and persists test results as metrics in a time-series database for analysis. This technology stack uses Rspec for automated tests, Sensu as an event pipeline, InfluxDB for metric storage, and Grafana for visual dashboards.
13. #1: Eliminate bias by adding
instruction.
#2: Define acceptance criteria.
#3: Automate, but don’t
over-automate.
#4: Implement test case
management.
#5: Hire employees with
diverse experience.
nikki.dev @nikkixdev
47. The monitoring event pipeline
The Sensu Go agent is a powerful cross-platform event producer.
Sensu Event Pipeline
Filter HandlerTransform
Sensu Client
Events
Agent
(event
producer)
Nagios Plugin
(service check)
StatsD Metrics
(instrumentation
library)
Prometheus
(/metrics endpoint)