Infrastructure — Kafka, RabbitMQ and ActiveMQ

Kafka, RabbitMQ and ActiveMQ metrics are collected through a Tower. The add steps are the same as for databases.

#Kafka

Quick install

1. Run the command on a Kafka broker, answer the questions.

   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/kafka.sh?t=<token>" -o /tmp/sf-kafka.sh && bash /tmp/sf-kafka.sh

2. When you see this line, you are done.

   textCopy

   OK: Kafka added to Spitfire, first metrics within ~1 min

Manual install

1. With SCRAM, create the user.

   bashCopy

   /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type users --entity-name spitfire_monitor --add-config 'SCRAM-SHA-512=[iterations=8192,password=<password>]'

2. With ACLs, grant Describe.

   bashCopy

   /opt/kafka/bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:spitfire_monitor --operation Describe --cluster
   /opt/kafka/bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:spitfire_monitor --operation Describe --topic '*'
   /opt/kafka/bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:spitfire_monitor --operation Describe --group '*'

3. Check that the broker answers.

   bashCopy

   /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server <host>:9092 >/dev/null && echo 'OK: Kafka answered'

4. Add it in Infrastructure sources → Kafka → Manual install.

#Broker metrics (JMX exporter, optional)

Bytes and message rates, request p99, under-replicated and offline partitions, the controller, heap and GC come from the broker's JMX. Each broker runs the Prometheus JMX exporter as a javaagent. Do the steps on each broker:

1. Download the javaagent:

   bashCopy

   sudo mkdir -p /opt/jmx-exporter && sudo curl -fsSL -o /opt/jmx-exporter/jmx_prometheus_javaagent.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar

2. Write the config file (also in the Broker metrics (JMX exporter) box on the page):

   bashCopy

   sudo tee /opt/jmx-exporter/kafka-jmx.yml >/dev/null <<'EOF'
   # Spitfire: Kafka broker metrics for the Prometheus JMX exporter javaagent.
   lowercaseOutputName: false
   includeObjectNames:
     - "kafka.server:type=BrokerTopicMetrics,*"
     - "kafka.server:type=ReplicaManager,*"
     - "kafka.controller:type=KafkaController,*"
     - "kafka.network:type=RequestMetrics,name=TotalTimeMs,*"
     - "java.lang:type=Memory"
     - "java.lang:type=GarbageCollector,*"
   rules:
     - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec><>Count'
       name: kafka_broker_bytes_in
       type: GAUGE
     - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec><>Count'
       name: kafka_broker_bytes_out
       type: GAUGE
     - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>Count'
       name: kafka_broker_messages_in
       type: GAUGE
     - pattern: 'kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec><>Count'
       name: kafka_broker_produce_requests
       type: GAUGE
     - pattern: 'kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec><>Count'
       name: kafka_broker_fetch_requests
       type: GAUGE
     - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>99thPercentile'
       name: kafka_broker_request_p99_ms
       labels:
         request: "$1"
       type: GAUGE
     - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
       name: kafka_broker_under_replicated_partitions
       type: GAUGE
     - pattern: 'kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value'
       name: kafka_broker_offline_partitions
       type: GAUGE
     - pattern: 'kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value'
       name: kafka_broker_active_controllers
       type: GAUGE
     - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used'
       name: kafka_broker_heap_used_bytes
       type: GAUGE
     - pattern: 'java.lang<(?:type=GarbageCollector, name=([^,>]+)|name=([^,>]+), type=GarbageCollector)><>CollectionTime'
       name: kafka_broker_gc_time_ms
       labels:
         gc: "$1$2"
       type: GAUGE
   EOF

3. With Kafka under systemd, add the agent and restart (change kafka if the service has another name):

   bashCopy

   sudo mkdir -p /etc/systemd/system/kafka.service.d && sudo tee /etc/systemd/system/kafka.service.d/spitfire-jmx.conf >/dev/null <<'EOF'
   [Service]
   Environment="KAFKA_OPTS=-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/kafka-jmx.yml"
   EOF
   sudo systemctl daemon-reload && sudo systemctl restart kafka

   With Docker compose, add to the broker service, then docker compose up -d:

   yamlCopy

       environment:
         KAFKA_OPTS: "-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/kafka-jmx.yml"
       volumes:
         - /opt/jmx-exporter:/opt/jmx-exporter:ro
       ports:
         - "7071:7071"

4. Check (the number must be above 0):

   bashCopy

   curl -s http://localhost:7071/metrics | grep -c kafka_broker_

5. In the Kafka source, write the exporter addresses comma separated in JMX exporters (kafka1:7071,kafka2:7071) → Save.

#Consumer lag table

Below the Kafka dashboard the Consumer lag table shows each consumer group's total lag, the furthest behind first. Click a topic to open its partitions. The values are the newest points of the last 15 minutes; a group over the lag limit (10,000) shows in the warning colour.

#RabbitMQ

Quick install (on a RabbitMQ node)

1. Copy the command, paste it into the terminal and run it.

   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/rabbitmq.sh?t=<token>" -o /tmp/sf-rabbitmq.sh && bash /tmp/sf-rabbitmq.sh

2. On Windows, run this in PowerShell instead.

   powershellCopy

   Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/rabbitmq.ps1?t=<token>" -OutFile "$env:TEMP\sf-rabbitmq.ps1"; & "$env:TEMP\sf-rabbitmq.ps1"

3. When you see OK: rabbitmq ..., you are done.

Manual install

1. Enable the management plugin.

   bashCopy

   rabbitmq-plugins enable rabbitmq_management

2. Create the monitoring user.

   bashCopy

   rabbitmqctl add_user spitfire_monitor '<password>' && rabbitmqctl set_user_tags spitfire_monitor monitoring

3. Grant read on every vhost.

   bashCopy

   for v in $(rabbitmqctl -q --no-table-headers list_vhosts name); do rabbitmqctl set_permissions -p "$v" spitfire_monitor '' '' '.*'; done

4. Test the management API.

   bashCopy

   curl -fsS -o /dev/null -u spitfire_monitor:'<password>' http://<host>:15672/api/overview && echo "OK: spitfire_monitor"

5. Add it under Infrastructure sources → RabbitMQ, address http://<host>:15672.

#Live view

The RabbitMQ dashboard has a Live view card. Read now shows every queue with its ready and unacked messages, consumers and publish/deliver rates; click a column header to sort by it (again to flip). A queue with messages and no consumer is marked yellow. Next to it: the nodes with their memory, disk, file descriptors and sockets.

#ActiveMQ

ActiveMQ Classic and Artemis are one ActiveMQ monitored system (Beta). The Tower reads the Jolokia REST API of the broker's web console itself; no Collector is needed. A broker or a cluster counts as one monitored system: give the addresses of the cluster, or of a primary and its backup, comma separated.

Quick install (on the broker host)

1. Copy the command, paste it into a terminal and run it.

   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/activemq.sh?t=<token>" -o /tmp/sf-activemq.sh && bash /tmp/sf-activemq.sh

2. On Windows run this in an Administrator PowerShell instead.

   powershellCopy

   Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/activemq.ps1?t=<token>" -OutFile "$env:TEMP\sf-activemq.ps1"; & "$env:TEMP\sf-activemq.ps1"

3. When you see OK: activemq ..., you're done.

Manual install: ActiveMQ Classic (/opt/apache-activemq)

1. Back up the files.

   bashCopy

   for f in users.properties groups.properties jolokia-access.xml; do cp -n /opt/apache-activemq/conf/$f /opt/apache-activemq/conf/$f.spitfire.bak; done

2. Add the monitor user.

   bashCopy

   grep -q '^spitfire_monitor=' /opt/apache-activemq/conf/users.properties || printf '\nspitfire_monitor=%s\n' '<password>' >> /opt/apache-activemq/conf/users.properties

3. Add the user to the users group.

   bashCopy

   grep -q '^users=.*spitfire_monitor' /opt/apache-activemq/conf/groups.properties || { grep -q '^users=' /opt/apache-activemq/conf/groups.properties && sed -i 's/^users=\(..*\)$/&,spitfire_monitor/; s/^users=$/users=spitfire_monitor/' /opt/apache-activemq/conf/groups.properties || printf '\nusers=spitfire_monitor\n' >> /opt/apache-activemq/conf/groups.properties; }

4. Make Jolokia read-only.

   bashCopy

   printf '%s\n' '<?xml version="1.0" encoding="UTF-8"?>' '<restrict>' '  <cors><strict-checking/></cors>' '  <commands><command>read</command><command>list</command><command>version</command><command>search</command></commands>' '</restrict>' > /opt/apache-activemq/conf/jolokia-access.xml

5. Restart the broker.

   bashCopy

   /opt/apache-activemq/bin/activemq restart

6. Test a read.

   bashCopy

   curl -fsS -u 'spitfire_monitor:<password>' -H 'Origin: http://localhost' 'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=*/BrokerVersion' | grep -q '"status":200' && echo "OK: spitfire_monitor"

7. Add it under Infrastructure sources → ActiveMQ: flavour classic, address http://<host>:8161.

Manual install: ActiveMQ Artemis (/var/lib/artemis-instance)

1. Back up the files.

   bashCopy

   for f in artemis-users.properties artemis-roles.properties artemis.profile management.xml; do cp -n /var/lib/artemis-instance/etc/$f /var/lib/artemis-instance/etc/$f.spitfire.bak; done

2. Add the monitor user with a hashed password.

   bashCopy

   grep -q '^spitfire_monitor *=' /var/lib/artemis-instance/etc/artemis-users.properties || printf '\nspitfire_monitor = ENC(%s)\n' "$(/var/lib/artemis-instance/bin/artemis mask --hash '<password>' | sed -n 's/^result: //p')" >> /var/lib/artemis-instance/etc/artemis-users.properties

3. Give it the spitfire role.

   bashCopy

   grep -q '^spitfire *=' /var/lib/artemis-instance/etc/artemis-roles.properties || printf '\nspitfire = spitfire_monitor\n' >> /var/lib/artemis-instance/etc/artemis-roles.properties

4. Add the role to the console.

   bashCopy

   grep -q '^HAWTIO_ROLES=.*spitfire' /var/lib/artemis-instance/etc/artemis.profile || sed -i "s/^HAWTIO_ROLES='\([^']*\)'/HAWTIO_ROLES='\1,spitfire'/" /var/lib/artemis-instance/etc/artemis.profile

5. Give the role read-only MBean access.

   bashCopy

   grep -q 'roles="amq,spitfire"' /var/lib/artemis-instance/etc/management.xml || sed -i 's#<match domain="org.apache.activemq.artemis">#&<access method="getFirstMessageAsJSON" roles="amq"/><access method="get*" roles="amq,spitfire"/><access method="is*" roles="amq,spitfire"/>#' /var/lib/artemis-instance/etc/management.xml

6. Restart the broker.

   bashCopy

   /var/lib/artemis-instance/bin/artemis-service restart

7. Test a read.

   bashCopy

   curl -fsS -u 'spitfire_monitor:<password>' -H 'Origin: http://localhost' 'http://localhost:8161/console/jolokia/read/org.apache.activemq.artemis:broker=*/Version' | grep -q '"status":200' && echo "OK: spitfire_monitor"

8. Add it under Infrastructure sources → ActiveMQ: flavour artemis, address http://<host>:8161.

#Queue table

The ActiveMQ dashboard has a table of the 20 deepest queues: pending messages, consumers, in-flight, the age of the oldest message (Artemis) and the DLQs. A queue with messages and no consumer is marked no consumer. A backup broker counts only as up; its metrics come from the primary.

#Threshold findings

#Success check

1. Last test = Connected.
2. On the Kafka dashboard Brokers equals your broker count; on RabbitMQ Consumers is drawn (RabbitMQ metrics are per queue: with no queue the dashboard stays empty).
3. On the ActiveMQ dashboard Broker reachable is 1 and the queue table is filled.