SpitfireLoad Testing & APMWhat's new

Infrastructure: Hazelcast, Kafka Connect, Keycloak, Vault, IBM MQ, NATS

The Tower reads these six over their own HTTP APIs every 30 seconds; no Collector config is needed. One cluster is one source: list all of its members or nodes, comma-separated, in that source. Access is read-only; Spitfire never reads map data, connector configs, users, secrets or messages. Kafka Connect and NATS are tested on real versions, the others are Beta.

Every product has two paths. Quick install: run the command, answer the questions, and when you see OK: ... you are done. The manual install does the same work step by step.

Hazelcast

Hazelcast 5.x. Spitfire reads /hazelcast/health (port 5701) and the Prometheus page (PROMETHEUS_PORT, 8080 by default) of every member.

Quick install: run the command on every member; on the last member answer 1 to add the cluster.

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/hazelcast.sh?t=<token>" -o /tmp/sf-hazelcast.sh && bash /tmp/sf-hazelcast.sh

Manual install

  1. Docker or Kubernetes: add these three lines to every member's environment, recreate the members one at a time.
bash
HZ_NETWORK_RESTAPI_ENABLED=true
HZ_NETWORK_RESTAPI_ENDPOINTGROUPS_HEALTHCHECK_ENABLED=true
PROMETHEUS_PORT=8080
  1. Package install (systemd unit hazelcast): write the same as a drop-in and restart the member.
bash
sudo mkdir -p /etc/systemd/system/hazelcast.service.d && printf '%s\n' '[Service]' 'Environment=HZ_NETWORK_RESTAPI_ENABLED=true' 'Environment=HZ_NETWORK_RESTAPI_ENDPOINTGROUPS_HEALTHCHECK_ENABLED=true' 'Environment=PROMETHEUS_PORT=8080' | sudo tee /etc/systemd/system/hazelcast.service.d/spitfire.conf && sudo systemctl daemon-reload && sudo systemctl restart hazelcast
  1. Check the health check and the metrics on the member.
bash
curl -fsS http://localhost:5701/hazelcast/health | grep -q '"nodeState"' && curl -fsS http://localhost:8080/metrics | grep -q '^com_hazelcast_Metrics' && echo 'OK: Hazelcast health and metrics ready'
  1. Spitfire → Infrastructure sources → Hazelcast; address: http://hz1:5701,http://hz2:5701,http://hz3:5701, Prometheus port 8080.
Warning

Restart members one at a time and wait until the cluster is safe again (clusterSafe: true), or partitions may lose their backups.

Kafka Connect

Spitfire sends only GET requests to the Connect REST API (port 8083): worker, connector and task states, and the first line of a failed task's trace. Passwords, tokens and sasl.jaas.config values in the trace are masked before they leave the Tower. Tested with Apache Kafka 3.7, 3.8, 3.9 and 4.1 Connect (basic auth included). JMX metrics are not read.

Quick install: run the command on any machine that reaches a worker.

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/kafkaconnect.sh?t=<token>" -o /tmp/sf-kafkaconnect.sh && bash /tmp/sf-kafkaconnect.sh

Manual install

  1. Check the REST API. Without a REST password nothing else is needed.
bash
curl -fsS http://localhost:8083/ | grep -q '"kafka_cluster_id"' && curl -fsS 'http://localhost:8083/connectors?expand=status' >/dev/null && echo 'OK: Kafka Connect REST API ready'
  1. Only with BasicAuthSecurityRestExtension: add a user to the workers' password file (the file= path of the JAAS config), restart the workers one at a time. Replace <password>. Connect has no read-only REST role; keep that password for Spitfire only.
bash
grep -q '^spitfire:' /etc/kafka/connect.password || echo 'spitfire: <password>' | sudo tee -a /etc/kafka/connect.password >/dev/null && echo 'OK: user added'
  1. Spitfire → Infrastructure sources → Kafka Connect; address: http://connect1:8083,http://connect2:8083; user and password only with step 2.

Keycloak

Keycloak 25 or later. Spitfire reads /health/ready, /health/live and /metrics on the management port (9000). event-metrics-user-enabled adds the login and failed-login counts per realm. Keycloak's metrics carry no active session count, so Spitfire shows none.

Quick install: run the command on every node; on the last node answer 1.

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/keycloak.sh?t=<token>" -o /tmp/sf-keycloak.sh && bash /tmp/sf-keycloak.sh

Manual install

  1. Back up keycloak.conf.
bash
sudo cp -n /opt/keycloak/conf/keycloak.conf /opt/keycloak/conf/keycloak.conf.spitfire.bak && echo 'OK: backup taken'
  1. Turn on health, metrics and the login counts.
bash
for o in health-enabled metrics-enabled event-metrics-user-enabled; do grep -q "^$o=" /opt/keycloak/conf/keycloak.conf || echo "$o=true" | sudo tee -a /opt/keycloak/conf/keycloak.conf; done; echo 'OK: options set'
  1. Restart the node. With Docker or Kubernetes set KC_HEALTH_ENABLED=true, KC_METRICS_ENABLED=true and KC_EVENT_METRICS_USER_ENABLED=true instead and recreate the node.
bash
sudo systemctl restart keycloak && echo 'OK: restarted'
  1. Check the management port.
bash
curl -fsS http://localhost:9000/health/ready | grep -q '"UP"' && curl -fsS http://localhost:9000/metrics | grep -q '^jvm_' && echo 'OK: Keycloak health and metrics ready'
  1. Spitfire → Infrastructure sources → Keycloak; address: http://kc1:9000,http://kc2:9000.

HashiCorp Vault

Spitfire reads /v1/sys/health of every node without a token and /v1/sys/metrics with a token whose policy reads only sys/metrics. The token is periodic (32 days), the Tower renews it every 12 hours, and it is stored encrypted and never logged.

Before you start: telemetry. Outside dev mode Vault serves Prometheus metrics only with a telemetry block in its server configuration; both installs stop if it is missing. On every node (package install, /etc/vault.d/vault.hcl):

bash
sudo grep -q '^telemetry' /etc/vault.d/vault.hcl || printf '%s\n' '' 'telemetry {' '  prometheus_retention_time = "24h"' '  disable_hostname = true' '}' | sudo tee -a /etc/vault.d/vault.hcl >/dev/null; sudo grep -q 'prometheus_retention_time' /etc/vault.d/vault.hcl && echo 'OK: telemetry block in place' || echo 'A telemetry block exists without prometheus_retention_time: add prometheus_retention_time = "24h" inside it'

Then restart the nodes one at a time (the standbys first, the active node last) and unseal each one:

bash
sudo systemctl restart vault && echo 'OK: restarted, now unseal this node'

With Docker/Kubernetes add the same block to the configuration the container reads (-config= or the ConfigMap) and recreate the nodes one at a time.

Quick install: run the command once per cluster, against the active node; give a token that may write policies (it is used in memory only).

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/vault.sh?t=<token>" -o /tmp/sf-vault.sh && bash /tmp/sf-vault.sh

Manual install

  1. Log in with a token that may write policies (replace the address).
bash
export VAULT_ADDR=https://vault1:8200 && vault login && echo 'OK: logged in'
  1. Write the read-only policy.
bash
vault policy write spitfire-metrics - <<'EOF'
path "sys/metrics" {
  capabilities = ["read"]
}
path "auth/token/renew-self" {
  capabilities = ["update"]
}
EOF
vault policy read spitfire-metrics >/dev/null && echo 'OK: policy written'
  1. Create the token and copy what it prints.
bash
vault token create -policy=spitfire-metrics -no-default-policy -orphan -period=768h -display-name=spitfire -field=token
  1. Check sys/metrics with that token (replace <token>).
bash
curl -fsS -H 'X-Vault-Token: <token>' "$VAULT_ADDR/v1/sys/metrics?format=prometheus" | grep -q '^vault_core_unsealed' && echo 'OK: Vault metrics with the Spitfire token'
  1. Spitfire → Infrastructure sources → HashiCorp Vault; address: https://vault1:8200,https://vault2:8200,https://vault3:8200, the token from step 3.

If the Tower cannot reach Vault for 32 days the token expires; create a new one with step 3 and paste it into the source.

IBM MQ

IBM MQ 9.2 or later (tested with 10.0.1.0). One source is one queue manager. Spitfire reads it over the mqweb administrative REST API (https://host:9443/ibmmq/rest/v2) with a user in the MQWebAdminRO role. That role only runs DISPLAY; no message is browsed. For a multi-instance queue manager list both hosts: the one it runs on is read, the other counts as standby.

Quick install: run the command on every host of the queue manager (both hosts of a multi-instance pair). It asks for the queue manager name and a password; leave it empty and it generates a strong one.

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/ibmmq.sh?t=<token>" -o /tmp/sf-ibmmq.sh && bash /tmp/sf-ibmmq.sh

With the queue manager in Docker (the official image has no curl or sudo), run the command on the Docker host and give the container's name to the first question (or SF_CONTAINER=<name> bash /tmp/sf-ibmmq.sh). The script makes mqweb listen on every interface (httpHost *). Create the container with -p 9443:9443 and keep /var/mqm on a volume so the user survives a recreate.

Manual install

  1. On every host add the spitfire user to the mqweb basic registry and to the MQWebAdminRO role. The block first keeps a backup (mqwebuser.xml.spitfire.bak) and asks for the password; mqweb reloads the file itself.
bash
MQWEB=/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml && { sudo test -e "$MQWEB.spitfire.bak" || sudo cp "$MQWEB" "$MQWEB.spitfire.bak"; } && read -rs -p 'Password for spitfire: ' PW && echo && HASH=$(sudo JAVA_HOME=/opt/mqm/java/jre64/jre /opt/mqm/web/bin/securityUtility encode --encoding=hash "$PW") && { sudo grep -q '<user name="spitfire"' "$MQWEB" || { sudo awk -v h="$HASH" '
/<basicRegistry/ && !u { print; print "        <user name=\"spitfire\" password=\"" h "\"/>"; u = 1; next }
/id="com.ibm.mq.rest"/ { rest = 1 }
rest && !r && /<security-role name="MQWebAdminRO">/ { r = 1; if (sub(/<\/security-role>/, "<user name=\"spitfire\" realm=\"defaultRealm\"/></security-role>")) { print } else { print; print "                <user name=\"spitfire\" realm=\"defaultRealm\"/>" }; next }
rest && !r && /<\/application-bnd>/ { print "            <security-role name=\"MQWebAdminRO\"><user name=\"spitfire\" realm=\"defaultRealm\"/></security-role>"; r = 1 }
rest && /<\/enterpriseApplication>/ { rest = 0 }
{ print }' "$MQWEB" | sudo tee "$MQWEB.spitfire.new" >/dev/null && sudo mv "$MQWEB.spitfire.new" "$MQWEB"; }; } && sudo grep -q 'MQWebAdminRO.*spitfire\|<user name="spitfire" realm' "$MQWEB" && echo 'OK: spitfire in the registry and in MQWebAdminRO'
  1. Only when the Tower runs on another host: mqweb listens on localhost by default.
bash
sudo /opt/mqm/bin/setmqweb properties -k httpHost -v '*' && echo 'OK: mqweb listens on every interface'
  1. Check the REST API with the user (replace QM1; type the password again).
bash
read -rs -p 'Password for spitfire: ' PW && echo && curl -fsSk -u "spitfire:$PW" https://localhost:9443/ibmmq/rest/v2/admin/qmgr/QM1 | grep -q '"state"' && echo 'OK: IBM MQ REST API with the Spitfire user'
  1. Optional, for the oldest message's age: turn queue monitoring on. This changes the queue manager; replace QM1.
bash
echo 'ALTER QMGR MONQ(MEDIUM)' | sudo -u mqm /opt/mqm/bin/runmqsc QM1 | grep -q 'AMQ8005I' && echo 'OK: queue monitoring on'
  1. Spitfire → Infrastructure sources → IBM MQ; address: https://mq1:9443,https://mq2:9443, queue manager QM1, user spitfire, the password from step 1. The default mqweb certificate is self-signed: tick "skip certificate verification" unless you replaced it.

With an LDAP registry skip step 1 and add the user to the LDAP group mapped to MQWebAdminRO. Do not put the user in the MQWebUser role: that role runs commands as the user, needs MQ authorities of its own, and the request is refused with reason 2035.

NATS

NATS 2.2 or later. Spitfire sends only GET requests to every server's HTTP monitoring port (8222): /varz, /healthz, /routez, /subsz, /connz and, with JetStream on, /jsz (lag, ack pending, redeliveries). Messages are never read. The monitoring port is off by default; 4222 is the client port and does not work here. The NATS entry under Connections is something else: the connection test steps use to publish and request.

Quick install: turn on the monitoring port first (manual step 1), then run the command on any machine that reaches a server.

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/nats.sh?t=<token>" -o /tmp/sf-nats.sh && bash /tmp/sf-nats.sh

Manual install

  1. On every server turn on the monitoring port and restart the servers one at a time. The config file defaults to /etc/nats/nats-server.conf; change the path if yours is elsewhere.
bash
CONF=/etc/nats/nats-server.conf; sudo grep -q '^http_port' "$CONF" || echo 'http_port: 8222' | sudo tee -a "$CONF" >/dev/null && sudo systemctl restart nats-server && echo 'OK: http_port set'

With Docker, add -m 8222 to the server's command and publish the port:

bash
docker run -d --name nats -p 4222:4222 -p 8222:8222 nats:2.12 -js -m 8222
  1. Check the port on each server (replace localhost with the server when you run it elsewhere).
bash
curl -fsS http://localhost:8222/varz | grep -q '"server_id"' && curl -fsS http://localhost:8222/healthz | grep -q '"ok"' && echo 'OK: NATS monitoring port ready'
  1. Spitfire → Infrastructure sources → NATS monitoring; address: every server of the cluster, http://nats1:8222,http://nats2:8222,http://nats3:8222 (each server reports only itself). User and password only when a reverse proxy with basic auth is in front.

The monitoring port has no password and shows connection names, IPs and subjects. Our advice: open it only to the Tower (a firewall rule, or http: 10.0.0.5:8222 on a private interface).

Uninstalling

For each product first delete the source: All integrations In Spitfire: /admin/integrations#observability → the bin on its row → Delete. The Tower stops reading; metrics already collected stay. Then undo only what that product's setup changed. We use the default names; if you chose another, use it. A run that stopped halfway can leave ~/.spitfire-setup/<product>.state (rm -f ~/.spitfire-setup/hazelcast.state). The Kafka Connect and NATS scripts do not touch the product.

Hazelcast

The setup adds the drop-in /etc/systemd/system/hazelcast.service.d/spitfire.conf on every member. Do the members one at a time and wait for clusterSafe: true after each; put your unit name in place of hazelcast.

bash
UNIT=hazelcast; sudo rm -f /etc/systemd/system/$UNIT.service.d/spitfire.conf && sudo systemctl daemon-reload && sudo systemctl restart $UNIT && ! systemctl show $UNIT -p Environment | grep -q PROMETHEUS_PORT && echo 'OK: drop-in removed'

With Docker or Kubernetes remove the three environment lines and recreate the members one at a time (keep them if something else uses the REST API).

Kafka Connect

Only if you added a user in manual step 2: remove it on every worker, restart the workers one at a time.

bash
sudo sed -i '/^spitfire:/d' /etc/kafka/connect.password && ! sudo grep -q '^spitfire:' /etc/kafka/connect.password && echo 'OK: user removed'

Keycloak

The setup adds three lines to keycloak.conf and first keeps the backup keycloak.conf.spitfire.bak. On every node, one at a time, put the backup back. If you changed the file for something else after the backup, remove only those three lines.

bash
sudo cp -p /opt/keycloak/conf/keycloak.conf.spitfire.bak /opt/keycloak/conf/keycloak.conf && sudo systemctl restart keycloak && echo 'OK: keycloak.conf restored'

With Docker or Kubernetes remove the three KC_* variables and recreate the nodes. Keep health or metrics on if Kubernetes probes use them.

HashiCorp Vault

The setup creates the spitfire-metrics policy and a periodic orphan token with display name spitfire. Delete the policy; the token can then read nothing and expires within 32 days.

bash
export VAULT_ADDR=https://vault1:8200 && vault login && vault policy delete spitfire-metrics && ! vault policy read spitfire-metrics >/dev/null 2>&1 && echo 'OK: policy deleted'

If you still have the token (manual step 3), you can revoke it right away:

bash
vault token revoke '<token>' && echo 'OK: token revoked'

The telemetry block is harmless and may stay. If you remove it, restart and unseal the nodes one at a time.

IBM MQ

The setup adds spitfire to mqwebuser.xml on every host (backup: mqwebuser.xml.spitfire.bak). Remove the user; mqweb reloads the file itself.

bash
MQWEB=/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml && sudo sed -i -e '/^[[:space:]]*<user name="spitfire"[^>]*\/>[[:space:]]*$/d' -e 's#<user name="spitfire"[^>]*/>##g' "$MQWEB" && ! sudo grep -q '<user name="spitfire"' "$MQWEB" && echo 'OK: spitfire removed from mqwebuser.xml'

In a container, on the Docker host (replace <container>):

bash
docker exec <container> sh -c 'MQWEB=/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml && sed -i -e "/^[[:space:]]*<user name=\"spitfire\"[^>]*\/>[[:space:]]*$/d" -e "s#<user name=\"spitfire\"[^>]*/>##g" "$MQWEB" && ! grep -q "<user name=\"spitfire\"" "$MQWEB"' && echo 'OK: spitfire removed from mqwebuser.xml'

Only if you opened httpHost for Spitfire and nobody uses the MQ Console from another host, undo it (in a container prefix docker exec <container>, without sudo):

bash
sudo /opt/mqm/bin/setmqweb properties -k httpHost -d && echo 'OK: httpHost removed'

If you turned queue monitoring on only for Spitfire, turn it off with echo 'ALTER QMGR MONQ(OFF)' | sudo -u mqm /opt/mqm/bin/runmqsc QM1. With LDAP, take the user out of the group.

NATS

Only if you opened the monitoring port for Spitfire: remove the http_port line and restart the servers one at a time (with Docker remove -m 8222 and the port mapping and recreate).

bash
CONF=/etc/nats/nats-server.conf; sudo sed -i '/^http_port/d' "$CONF" && sudo systemctl restart nats-server && echo 'OK: http_port removed'