This document provides instructions for monitoring additional metrics from clusters and applications using Grafana, Prometheus, JMX, and PushGateway. It includes steps to export JMX metrics from Kafka and NiFi, setup and use PushGateway to collect and expose custom metrics, and create Grafana dashboards to visualize the metrics.

1. How-To Go
The Extra Mile
on Monitoring
Grafana + Prometheus + JMX
+ PushGateway

2. Introduction
With this presentation, you will be able get even more metrics from your
Cluster/Applications.

3. Pre-Assumptions
1. Grafana Installed
2. Prometheus Installed

4. JMX
Create and Download JMX Exporter Directory
sudo mkdir -p /opt/jmx-exporter;
sudo cd /opt/jmx-exporter;
sudo wget
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-
0.15.0.jar;
Note: just need to change the highlighted values for your own values.

5. JMX - Use Case Kafka
Download Kafka YML
sudo cd /opt/jmx-exporter;
wget https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml;
mv kafka-2_0_0.yml kafka_broker.yml;
[Cloudera] CM - Kafka Configuration (broker_java_opts)
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -
javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.15.0.jar=7072:/opt/prometheus/kafka_broker.yml
Restart Service and Check HTTP Metrics
Note: just need to change the highlighted values for your own values.

6. JMX - Use Case Kafka
Add Job to Prometheus Config YML
sudo nano /etc/prometheus/prometheus.yml;
- job_name: 'kafka_jmx'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'
scrape_interval: 5s
static_configs:
- targets: ['10.111.22.33:7072']
Restart Prometheus Service
sudo systemctl restart prometheus;
Note: just need to change the highlighted values for your own values.

7. JMX - Use Case NiFi
Create NiFi Metrics Config YML
sudo nano /usr/hdp/nifi/conf/nifi.yml;
lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:- pattern: ".*"
Add to Nifi Bootstrap Config File
sudo nano /usr/hdp/nifi/conf/bootstrap.conf; java.arg.18=-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-
0.15.0.jar=7474:/usr/hdp/nifi/conf/nifi.yml
Note: java.arg.XX the XX value should be unique!
Restart Prometheus Service
sudo systemctl restart prometheus;
Note: just need to change the highlighted values for your own values.
Add Job to Prometheus Config YML
sudo nano /etc/prometheus/prometheus.yml;
- job_name: 'nifi_jmx'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'
scrape_interval: 5s
static_configs:
- targets: ['10.111.22.44:7474']

8. PushGateway
Download PushGateway
sudo cd /u01/downloads;
sudo wget https://github.com/prometheus/pushgateway/releases/download/v1.4.0/pushgateway-1.4.0.linux-amd64.tar.gz;
sudo tar xvzf pushgateway-1.4.0.linux-amd64.tar.gz && mv pushgateway-1.4.0.linux-amd64/pushgateway
/usr/local/bin/pushgateway;

9. PushGateway
Create User
sudo useradd -m -s /bin/false pushgateway && id pushgateway;
sudo chown pushgateway:pushgateway /usr/local/bin/pushgateway;
Create User
sudo nano /etc/prometheus/prometheus.yml;
global:
scrape_interval: 1s
scrape_configs:
- job_name: 'pushgateway'
honor_labels: true
Static_configs:
- targets: ['10.111.22.33:9091']
Adjust the firewall
sudo firewall-cmd --add-port=9091/tcp --permanent && firewall-cmd --reload;
Note: just need to change the highlighted values for your own values.

10. PushGateway
Creating a Systemd service file for Prometheus Server
sudo nano /etc/systemd/system/pushgateway.service;
[Unit]
Description=Custom Metrics on PushGateway
Wants=network-online.target
After=network-online.target
[Service]
User=pushgateway
Group=pushgateway
Type=simple
ExecStart=/usr/local/bin/pushgateway
--web.listen-address=:9091
--web.enable-admin-api
--web.telemetry-path=/metrics
--persistence.file=/tmp/metric.store
--persistence.interval=5m
[Install]
WantedBy=multi-user.target
Note: just need to change the highlighted values for your own values.

11. PushGateway
Reload the Systemctl
sudo systemctl daemon-reload;
Now start and enable Prometheus to run on boot
sudo systemctl start pushgateway && sudo systemctl enable pushgateway;
Confirm Pushgateway is UP
sudo netstat -tunlp | grep pushgateway;
Note: just need to change the highlighted values for your own values.

12. PushGateway
Create Top Custom Metrics Script
sudo nano metricsTOP;
#!/bin/bash
# CPU Usage %
z=$(ps aux | grep -v '[' )
while read -r z
do
var=$var$(awk 'NR>1 {print "cpu_usage{user=""$1"" , pid=""$2"" , process=""$11"", arg=""$12" "$13" "$14" "$15"" }", $3z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/hostname01
# Memory Usage Bytes
x=$(ps aux | grep -v '[' )
while read -r x
do
var2=$var2$(awk 'NR>1 {print "memory_usage{user=""$1"" , pid=""$2"" , process=""$11"", arg=""$12" "$13" "$14" "$15"" }", $6x}');
done <<< "$x"
curl -X POST -H "Content-Type: text/plain" --data "$var2
" http://localhost:9091/metrics/job/top/instance/hostname01
sudo chmod u+x metricsTOP && ./metricsTOP;

13. PushGateway
Create Crontab Job
sudo contrab -e;
* * * * * for i in {1..60}; do /path/metricsTOP & sleep 1; done
Create IOTop Custom Metrics Script
sudo nano metricsTOP;
* * * * * for i in {1..60}; do /path/metricsTOP & sleep 1; done

14. PushGateway
Create IOTop Custom Metrics Script
# IO Usage %
t=$(iotop -Pkqbn 1 | grep -v '[' )
while read -r t
do
var3=$var3$(awk 'NR>2 {print "iotop_usage{user=""$3"" , pid=""$1"" , stream="%" ,process=""$12"", arg=""$13" "$14" "$15" "$16"" }", $10}');
done <<< "$t"
curl -X POST -H "Content-Type: text/plain" --data "$var3
" http://localhost:9091/metrics/job/iotop/instance/hostname01
# IO Read Kbs
u=$(iotop -Pkqbn 1 | grep -v '[' )
while read -r u
do
var4=$var4$(awk 'NR>2 {print "iotop_read{user=""$3"" , pid=""$1"" , stream="read" ,process=""$12"", arg=""$13" "$14" "$15" "$16"" }", $4}');
done <<< "$u"
curl -X POST -H "Content-Type: text/plain" --data "$var4
" http://localhost:9091/metrics/job/iotop/instance/hostname01
# IO Write Kbs
v=$(iotop -Pkqbn 1 | grep -v '[' )
while read -r v
do
var5=$var5$(awk 'NR>2 {print "iotop_write{user=""$3"" , pid=""$1"" , stream="write" ,process=""$12"", arg=""$13" "$14" "$15" "$16"" }", $6}');
done <<< "$v"
curl -X POST -H "Content-Type: text/plain" --data "$var5
" http://localhost:9091/metrics/job/iotop/instance/hostname01
Note: just need to change the highlighted values for your own values.

15. Grafana

16. Grafana

17. Thanks
Big Data Engineer
Tiago Simões