Infrastructure: Hazelcast, Kafka Connect, Keycloak, Vault, IBM MQ, NATS
The Tower reads these six over their own HTTP APIs every 30 seconds; no Collector config is needed. One cluster is one source: list all of its members or nodes, comma-separated, in that source. Access is read-only; Spitfire never reads map data, connector configs, users, secrets or messages. Kafka Connect and NATS are tested on real versions, the others are Beta.
Every product has two paths. Quick install: run the command, answer the questions, and when you see
OK: ... you are done. The manual install does the same work step by step.
Hazelcast
Hazelcast 5.x. Spitfire reads /hazelcast/health (port 5701) and the Prometheus page
(PROMETHEUS_PORT, 8080 by default) of every member.
Quick install: run the command on every member; on the last member answer 1 to add the cluster.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/hazelcast.sh?t=<token>" -o /tmp/sf-hazelcast.sh && bash /tmp/sf-hazelcast.shManual install
- Docker or Kubernetes: add these three lines to every member's environment, recreate the members one at a time.
HZ_NETWORK_RESTAPI_ENABLED=true
HZ_NETWORK_RESTAPI_ENDPOINTGROUPS_HEALTHCHECK_ENABLED=true
PROMETHEUS_PORT=8080- Package install (systemd unit
hazelcast): write the same as a drop-in and restart the member.
sudo mkdir -p /etc/systemd/system/hazelcast.service.d && printf '%s\n' '[Service]' 'Environment=HZ_NETWORK_RESTAPI_ENABLED=true' 'Environment=HZ_NETWORK_RESTAPI_ENDPOINTGROUPS_HEALTHCHECK_ENABLED=true' 'Environment=PROMETHEUS_PORT=8080' | sudo tee /etc/systemd/system/hazelcast.service.d/spitfire.conf && sudo systemctl daemon-reload && sudo systemctl restart hazelcast- Check the health check and the metrics on the member.
curl -fsS http://localhost:5701/hazelcast/health | grep -q '"nodeState"' && curl -fsS http://localhost:8080/metrics | grep -q '^com_hazelcast_Metrics' && echo 'OK: Hazelcast health and metrics ready'- Spitfire → Infrastructure sources → Hazelcast; address:
http://hz1:5701,http://hz2:5701,http://hz3:5701, Prometheus port8080.
Restart members one at a time and wait until the cluster is safe again
(clusterSafe: true), or partitions may lose their backups.
Kafka Connect
Spitfire sends only GET requests to the Connect REST API (port 8083): worker, connector and task
states, and the first line of a failed task's trace. Passwords, tokens and sasl.jaas.config values
in the trace are masked before they leave the Tower. Tested with Apache Kafka 3.7, 3.8, 3.9 and 4.1
Connect (basic auth included). JMX metrics are not read.
Quick install: run the command on any machine that reaches a worker.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/kafkaconnect.sh?t=<token>" -o /tmp/sf-kafkaconnect.sh && bash /tmp/sf-kafkaconnect.shManual install
- Check the REST API. Without a REST password nothing else is needed.
curl -fsS http://localhost:8083/ | grep -q '"kafka_cluster_id"' && curl -fsS 'http://localhost:8083/connectors?expand=status' >/dev/null && echo 'OK: Kafka Connect REST API ready'- Only with
BasicAuthSecurityRestExtension: add a user to the workers' password file (thefile=path of the JAAS config), restart the workers one at a time. Replace<password>. Connect has no read-only REST role; keep that password for Spitfire only.
grep -q '^spitfire:' /etc/kafka/connect.password || echo 'spitfire: <password>' | sudo tee -a /etc/kafka/connect.password >/dev/null && echo 'OK: user added'- Spitfire → Infrastructure sources → Kafka Connect; address:
http://connect1:8083,http://connect2:8083; user and password only with step 2.
Keycloak
Keycloak 25 or later. Spitfire reads /health/ready, /health/live and /metrics on the
management port (9000). event-metrics-user-enabled adds the login and failed-login counts per
realm. Keycloak's metrics carry no active session count, so Spitfire shows none.
Quick install: run the command on every node; on the last node answer 1.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/keycloak.sh?t=<token>" -o /tmp/sf-keycloak.sh && bash /tmp/sf-keycloak.shManual install
- Back up keycloak.conf.
sudo cp -n /opt/keycloak/conf/keycloak.conf /opt/keycloak/conf/keycloak.conf.spitfire.bak && echo 'OK: backup taken'- Turn on health, metrics and the login counts.
for o in health-enabled metrics-enabled event-metrics-user-enabled; do grep -q "^$o=" /opt/keycloak/conf/keycloak.conf || echo "$o=true" | sudo tee -a /opt/keycloak/conf/keycloak.conf; done; echo 'OK: options set'- Restart the node. With Docker or Kubernetes set
KC_HEALTH_ENABLED=true,KC_METRICS_ENABLED=trueandKC_EVENT_METRICS_USER_ENABLED=trueinstead and recreate the node.
sudo systemctl restart keycloak && echo 'OK: restarted'- Check the management port.
curl -fsS http://localhost:9000/health/ready | grep -q '"UP"' && curl -fsS http://localhost:9000/metrics | grep -q '^jvm_' && echo 'OK: Keycloak health and metrics ready'- Spitfire → Infrastructure sources → Keycloak; address:
http://kc1:9000,http://kc2:9000.
HashiCorp Vault
Spitfire reads /v1/sys/health of every node without a token and /v1/sys/metrics with a token
whose policy reads only sys/metrics. The token is periodic (32 days), the Tower renews it every 12
hours, and it is stored encrypted and never logged.
Before you start: telemetry. Outside dev mode Vault serves Prometheus metrics only with a
telemetry block in its server configuration; both installs stop if it is missing. On every node
(package install, /etc/vault.d/vault.hcl):
sudo grep -q '^telemetry' /etc/vault.d/vault.hcl || printf '%s\n' '' 'telemetry {' ' prometheus_retention_time = "24h"' ' disable_hostname = true' '}' | sudo tee -a /etc/vault.d/vault.hcl >/dev/null; sudo grep -q 'prometheus_retention_time' /etc/vault.d/vault.hcl && echo 'OK: telemetry block in place' || echo 'A telemetry block exists without prometheus_retention_time: add prometheus_retention_time = "24h" inside it'Then restart the nodes one at a time (the standbys first, the active node last) and unseal each one:
sudo systemctl restart vault && echo 'OK: restarted, now unseal this node'With Docker/Kubernetes add the same block to the configuration the container reads (-config= or
the ConfigMap) and recreate the nodes one at a time.
Quick install: run the command once per cluster, against the active node; give a token that may write policies (it is used in memory only).
curl -fsSL "https://<spitfire>/api/v1/setup/ds/vault.sh?t=<token>" -o /tmp/sf-vault.sh && bash /tmp/sf-vault.shManual install
- Log in with a token that may write policies (replace the address).
export VAULT_ADDR=https://vault1:8200 && vault login && echo 'OK: logged in'- Write the read-only policy.
vault policy write spitfire-metrics - <<'EOF'
path "sys/metrics" {
capabilities = ["read"]
}
path "auth/token/renew-self" {
capabilities = ["update"]
}
EOF
vault policy read spitfire-metrics >/dev/null && echo 'OK: policy written'- Create the token and copy what it prints.
vault token create -policy=spitfire-metrics -no-default-policy -orphan -period=768h -display-name=spitfire -field=token- Check sys/metrics with that token (replace
<token>).
curl -fsS -H 'X-Vault-Token: <token>' "$VAULT_ADDR/v1/sys/metrics?format=prometheus" | grep -q '^vault_core_unsealed' && echo 'OK: Vault metrics with the Spitfire token'- Spitfire → Infrastructure sources → HashiCorp Vault; address:
https://vault1:8200,https://vault2:8200,https://vault3:8200, the token from step 3.
If the Tower cannot reach Vault for 32 days the token expires; create a new one with step 3 and paste it into the source.
IBM MQ
IBM MQ 9.2 or later (tested with 10.0.1.0). One source is one queue manager. Spitfire reads it over
the mqweb administrative REST API (https://host:9443/ibmmq/rest/v2) with a user in the
MQWebAdminRO role. That role only runs DISPLAY; no message is browsed. For a multi-instance queue
manager list both hosts: the one it runs on is read, the other counts as standby.
Quick install: run the command on every host of the queue manager (both hosts of a multi-instance pair). It asks for the queue manager name and a password; leave it empty and it generates a strong one.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/ibmmq.sh?t=<token>" -o /tmp/sf-ibmmq.sh && bash /tmp/sf-ibmmq.shWith the queue manager in Docker (the official image has no curl or sudo), run the command on the
Docker host and give the container's name to the first question (or
SF_CONTAINER=<name> bash /tmp/sf-ibmmq.sh). The script makes mqweb listen on every interface
(httpHost *). Create the container with -p 9443:9443 and keep /var/mqm on a volume so the user
survives a recreate.
Manual install
- On every host add the
spitfireuser to the mqweb basic registry and to theMQWebAdminROrole. The block first keeps a backup (mqwebuser.xml.spitfire.bak) and asks for the password; mqweb reloads the file itself.
MQWEB=/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml && { sudo test -e "$MQWEB.spitfire.bak" || sudo cp "$MQWEB" "$MQWEB.spitfire.bak"; } && read -rs -p 'Password for spitfire: ' PW && echo && HASH=$(sudo JAVA_HOME=/opt/mqm/java/jre64/jre /opt/mqm/web/bin/securityUtility encode --encoding=hash "$PW") && { sudo grep -q '<user name="spitfire"' "$MQWEB" || { sudo awk -v h="$HASH" '
/<basicRegistry/ && !u { print; print " <user name=\"spitfire\" password=\"" h "\"/>"; u = 1; next }
/id="com.ibm.mq.rest"/ { rest = 1 }
rest && !r && /<security-role name="MQWebAdminRO">/ { r = 1; if (sub(/<\/security-role>/, "<user name=\"spitfire\" realm=\"defaultRealm\"/></security-role>")) { print } else { print; print " <user name=\"spitfire\" realm=\"defaultRealm\"/>" }; next }
rest && !r && /<\/application-bnd>/ { print " <security-role name=\"MQWebAdminRO\"><user name=\"spitfire\" realm=\"defaultRealm\"/></security-role>"; r = 1 }
rest && /<\/enterpriseApplication>/ { rest = 0 }
{ print }' "$MQWEB" | sudo tee "$MQWEB.spitfire.new" >/dev/null && sudo mv "$MQWEB.spitfire.new" "$MQWEB"; }; } && sudo grep -q 'MQWebAdminRO.*spitfire\|<user name="spitfire" realm' "$MQWEB" && echo 'OK: spitfire in the registry and in MQWebAdminRO'- Only when the Tower runs on another host: mqweb listens on localhost by default.
sudo /opt/mqm/bin/setmqweb properties -k httpHost -v '*' && echo 'OK: mqweb listens on every interface'- Check the REST API with the user (replace
QM1; type the password again).
read -rs -p 'Password for spitfire: ' PW && echo && curl -fsSk -u "spitfire:$PW" https://localhost:9443/ibmmq/rest/v2/admin/qmgr/QM1 | grep -q '"state"' && echo 'OK: IBM MQ REST API with the Spitfire user'- Optional, for the oldest message's age: turn queue monitoring on. This changes the queue
manager; replace
QM1.
echo 'ALTER QMGR MONQ(MEDIUM)' | sudo -u mqm /opt/mqm/bin/runmqsc QM1 | grep -q 'AMQ8005I' && echo 'OK: queue monitoring on'- Spitfire → Infrastructure sources → IBM MQ; address:
https://mq1:9443,https://mq2:9443, queue managerQM1, userspitfire, the password from step 1. The default mqweb certificate is self-signed: tick "skip certificate verification" unless you replaced it.
With an LDAP registry skip step 1 and add the user to the LDAP group mapped to MQWebAdminRO. Do not
put the user in the MQWebUser role: that role runs commands as the user, needs MQ authorities of
its own, and the request is refused with reason 2035.
NATS
NATS 2.2 or later. Spitfire sends only GET requests to every server's HTTP monitoring port (8222):
/varz, /healthz, /routez, /subsz, /connz and, with JetStream on, /jsz (lag, ack pending,
redeliveries). Messages are never read. The monitoring port is off by default; 4222 is the client
port and does not work here. The NATS entry under Connections is something else: the connection
test steps use to publish and request.
Quick install: turn on the monitoring port first (manual step 1), then run the command on any machine that reaches a server.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/nats.sh?t=<token>" -o /tmp/sf-nats.sh && bash /tmp/sf-nats.shManual install
- On every server turn on the monitoring port and restart the servers one at a time. The config
file defaults to
/etc/nats/nats-server.conf; change the path if yours is elsewhere.
CONF=/etc/nats/nats-server.conf; sudo grep -q '^http_port' "$CONF" || echo 'http_port: 8222' | sudo tee -a "$CONF" >/dev/null && sudo systemctl restart nats-server && echo 'OK: http_port set' With Docker, add -m 8222 to the server's command and publish the port:
docker run -d --name nats -p 4222:4222 -p 8222:8222 nats:2.12 -js -m 8222- Check the port on each server (replace
localhostwith the server when you run it elsewhere).
curl -fsS http://localhost:8222/varz | grep -q '"server_id"' && curl -fsS http://localhost:8222/healthz | grep -q '"ok"' && echo 'OK: NATS monitoring port ready'- Spitfire → Infrastructure sources → NATS monitoring; address: every server of the cluster,
http://nats1:8222,http://nats2:8222,http://nats3:8222(each server reports only itself). User and password only when a reverse proxy with basic auth is in front.
The monitoring port has no password and shows connection names, IPs and subjects. Our advice: open
it only to the Tower (a firewall rule, or http: 10.0.0.5:8222 on a private interface).
Uninstalling
For each product first delete the source: All integrations In Spitfire: /admin/integrations#observability →
the bin on its row → Delete. The Tower stops reading; metrics already collected stay. Then undo
only what that product's setup changed. We use the default names; if you chose another, use it. A run
that stopped halfway can leave ~/.spitfire-setup/<product>.state
(rm -f ~/.spitfire-setup/hazelcast.state). The Kafka Connect and NATS scripts do not touch the product.
Hazelcast
The setup adds the drop-in /etc/systemd/system/hazelcast.service.d/spitfire.conf on every member.
Do the members one at a time and wait for clusterSafe: true after each; put your unit name in place
of hazelcast.
UNIT=hazelcast; sudo rm -f /etc/systemd/system/$UNIT.service.d/spitfire.conf && sudo systemctl daemon-reload && sudo systemctl restart $UNIT && ! systemctl show $UNIT -p Environment | grep -q PROMETHEUS_PORT && echo 'OK: drop-in removed'With Docker or Kubernetes remove the three environment lines and recreate the members one at a time (keep them if something else uses the REST API).
Kafka Connect
Only if you added a user in manual step 2: remove it on every worker, restart the workers one at a time.
sudo sed -i '/^spitfire:/d' /etc/kafka/connect.password && ! sudo grep -q '^spitfire:' /etc/kafka/connect.password && echo 'OK: user removed'Keycloak
The setup adds three lines to keycloak.conf and first keeps the backup keycloak.conf.spitfire.bak.
On every node, one at a time, put the backup back. If you changed the file for something else after
the backup, remove only those three lines.
sudo cp -p /opt/keycloak/conf/keycloak.conf.spitfire.bak /opt/keycloak/conf/keycloak.conf && sudo systemctl restart keycloak && echo 'OK: keycloak.conf restored'With Docker or Kubernetes remove the three KC_* variables and recreate the nodes. Keep health or
metrics on if Kubernetes probes use them.
HashiCorp Vault
The setup creates the spitfire-metrics policy and a periodic orphan token with display name
spitfire. Delete the policy; the token can then read nothing and expires within 32 days.
export VAULT_ADDR=https://vault1:8200 && vault login && vault policy delete spitfire-metrics && ! vault policy read spitfire-metrics >/dev/null 2>&1 && echo 'OK: policy deleted'If you still have the token (manual step 3), you can revoke it right away:
vault token revoke '<token>' && echo 'OK: token revoked'The telemetry block is harmless and may stay. If you remove it, restart and unseal the nodes one at
a time.
IBM MQ
The setup adds spitfire to mqwebuser.xml on every host (backup: mqwebuser.xml.spitfire.bak).
Remove the user; mqweb reloads the file itself.
MQWEB=/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml && sudo sed -i -e '/^[[:space:]]*<user name="spitfire"[^>]*\/>[[:space:]]*$/d' -e 's#<user name="spitfire"[^>]*/>##g' "$MQWEB" && ! sudo grep -q '<user name="spitfire"' "$MQWEB" && echo 'OK: spitfire removed from mqwebuser.xml'In a container, on the Docker host (replace <container>):
docker exec <container> sh -c 'MQWEB=/var/mqm/web/installations/Installation1/servers/mqweb/mqwebuser.xml && sed -i -e "/^[[:space:]]*<user name=\"spitfire\"[^>]*\/>[[:space:]]*$/d" -e "s#<user name=\"spitfire\"[^>]*/>##g" "$MQWEB" && ! grep -q "<user name=\"spitfire\"" "$MQWEB"' && echo 'OK: spitfire removed from mqwebuser.xml'Only if you opened httpHost for Spitfire and nobody uses the MQ Console from another host, undo it
(in a container prefix docker exec <container>, without sudo):
sudo /opt/mqm/bin/setmqweb properties -k httpHost -d && echo 'OK: httpHost removed'If you turned queue monitoring on only for Spitfire, turn it off with
echo 'ALTER QMGR MONQ(OFF)' | sudo -u mqm /opt/mqm/bin/runmqsc QM1. With LDAP, take the user out of
the group.
NATS
Only if you opened the monitoring port for Spitfire: remove the http_port line and restart the
servers one at a time (with Docker remove -m 8222 and the port mapping and recreate).
CONF=/etc/nats/nats-server.conf; sudo sed -i '/^http_port/d' "$CONF" && sudo systemctl restart nats-server && echo 'OK: http_port removed'