The O11y toolkit is a set of utilities that help you debug, augment, and manage your open source observability stack.

1. The O11y toolkit
Julien Pivotto
February 2023
O11y

2. Who am I
• Julien Pivotto
• Prometheus server maintainer
• 15+ years in Open Source
• 10+ years in monitoring
O11y 1

3. About O11y
• Prometheus customers support
• Covering the Prometheus ecosystem
• Contributing in upstream Prometheus
O11y 2

4. What is a toolkit?
• Years of hands-on experience with prometheus
• Know what’s missing
• I know no one wants more tool... but
• Some things don’t fit
• Advanced/Unique features
O11y 3

5. The O11y toolkit
• Open Source
• Help you with your Prometheus stack
• Debug
• Enhance
O11y 4

6. Principles of the O11y toolkit
• Tools are available individually
• CLI and in-browser tools
• Use the HTTP API rather than local files when possible
• Share common CLI options
O11y 5

7. O11y 6

8. O11y 7

9. oy-expose
• Exposes a file to be scraped by Prometheus
• Parses them
• Adds Node-Exporter-like metadata (mtime)
O11y 8

10. O11y 9

11. Why o11y-scrape-jitter
O11y 10

12. After adapting the timestamp tolerance
--scrape.timestamp-tolerance=49ms
O11y 11

13. Outcome
40% disk usage reduction!
O11y 12

14. Configuration
--client.config
• Authenticate to Prometheus
• Add user/password
• TLS certificates
• proxy
O11y 13

15. Web tools
• In browser
• WASM
• No data sent to us
• Uses the original go code of Prometheus
O11y 14

16. O11y 15

17. Example generated file
Use with --web.config.file="web.yml"
basic_auth_users:
julien: $2a$10$sbf6u5Y4E.1KuGtgRUehE.IyH0j3k2lpIb9QpM6IdEDbI3j6VjoN6
kris: $2a$10$dgRYqya5Rn6Mf3NntphomOMN9DyQDvHApeiorMby.gh8vRkJdjHxi
O11y 16

18. O11y 17

19. O11y 18

20. Coming soon
O11y 19

21. https:/
/github.com/o11ydev/oy-toolkit
O11y

22. Get the O11y toolkit
• Downloads & in-browser
• deb
• rpm
• docker
• nix
O11y 20

23. https:/
/O11y.tools
O11y

24. O11y
https:/
/o11y.eu/prometheus-support/
O11y 21