SpitfireLoad Testing & APMWhat's new

Infrastructure: Kong, Envoy, Traefik

The Tower reads the metrics endpoint of API gateways and proxies (Beta); it never touches the gateway's config. Each gateway cluster is one monitored system: list all of its nodes, comma-separated, in that one source. Panels show requests, 5xx ratio, latency, upstream health and config state.

Kong

Quick install

  1. Run the command on one Kong node and answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/kong.sh?t=<token>" -o /tmp/sf-kong.sh && bash /tmp/sf-kong.sh
  1. You are done when you see the OK: ... line.

Manual install (Kong with a database)

  1. Turn on the Prometheus plugin through the Admin API (once per cluster).
bash
curl -fsS -X POST http://127.0.0.1:8001/plugins -H 'Content-Type: application/json' --data '{"name":"prometheus","config":{"status_code_metrics":true,"latency_metrics":true,"bandwidth_metrics":true,"upstream_health_metrics":true}}' && echo 'OK: plugin on'
  1. Open the Status API on every node (kong.conf).
bash
grep -q '^status_listen' /etc/kong/kong.conf || echo 'status_listen = 0.0.0.0:8100' | sudo tee -a /etc/kong/kong.conf
  1. Reload Kong with this kong.conf (without -c, kong reload ignores the change in the file).
bash
sudo kong reload -c /etc/kong/kong.conf
  1. Check the metrics.
bash
curl -fsS http://127.0.0.1:8100/metrics | grep -q '^kong_datastore_reachable' && echo 'OK: Kong metrics ready'
  1. Spitfire → Infrastructure sources → Kong; address: kong1:8100,kong2:8100.

Manual install (DB-less Kong)

  1. Add the plugin to kong.yml.
yaml
plugins:
- name: prometheus
  config:
    status_code_metrics: true
    latency_metrics: true
    bandwidth_metrics: true
    upstream_health_metrics: true
  1. Set KONG_STATUS_LISTEN=0.0.0.0:8100 in the environment, then reload Kong.
bash
sudo kong reload
  1. Check the metrics.
bash
curl -fsS http://127.0.0.1:8100/metrics | grep -q '^kong_datastore_reachable' && echo 'OK: Kong metrics ready'
  1. Spitfire → Infrastructure sources → Kong; address: kong1:8100,kong2:8100.

The Status API only reads. We recommend never giving port 8001 (the Admin API) to any monitoring tool.

Envoy

Quick install

  1. Run the command on every Envoy host.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/envoy.sh?t=<token>" -o /tmp/sf-envoy.sh && bash /tmp/sf-envoy.sh
  1. You are done when you see the OK: ... line.

Manual install

  1. Make sure the bootstrap config has an admin block on a private address.
yaml
admin:
  address:
    socket_address: { address: <private-ip>, port_value: 9901 }
  1. Restart Envoy (hot restart or the service).
bash
sudo systemctl restart envoy
  1. Check the metrics.
bash
curl -fsS 'http://<private-ip>:9901/stats/prometheus?usedonly' | grep -q '^envoy_server_live' && echo 'OK: Envoy metrics ready'
  1. Spitfire → Infrastructure sources → Envoy; address: <envoy1-ip>:9901,<envoy2-ip>:9901.
Warning

The admin interface can stop Envoy (/quitquitquit). Open port 9901 to the Tower only.

Traefik

Quick install

  1. Run the command on every Traefik host.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/traefik.sh?t=<token>" -o /tmp/sf-traefik.sh && bash /tmp/sf-traefik.sh
  1. You are done when you see the OK: ... line.

Manual install

  1. Add these flags to the static config (CLI or the container command).
text
--entrypoints.metrics.address=:8082
--metrics.prometheus=true
--metrics.prometheus.entryPoint=metrics
--metrics.prometheus.addEntryPointsLabels=true
--metrics.prometheus.addRoutersLabels=true
--metrics.prometheus.addServicesLabels=true
  1. Restart Traefik.
bash
sudo systemctl restart traefik
  1. Check the metrics.
bash
curl -fsS http://127.0.0.1:8082/metrics | grep -q '^traefik_config_reloads_total' && echo 'OK: Traefik metrics ready'
  1. Spitfire → Infrastructure sources → Traefik; address: traefik1:8082,traefik2:8082.

The API entry point (8080) works too instead of a separate one; the address is then traefik1:8080.

Panels and findings

Source Finding Threshold
All Node down up = 0
All 5xx ratio (Kong service, Envoy listener and cluster, Traefik router/service/entry point) > 1%, at least 20 requests
Kong Database unreachable kong_datastore_reachable = 0
Kong Unhealthy upstream target / every target of an upstream unhealthy -
Envoy Not live, cluster without a healthy member, unhealthy member, open circuit breaker -
Envoy Rejected config (CDS/LDS update_rejected) 15 min
Traefik Service with no server up, server down -
Traefik Last config reload failed (v2 only) -
Traefik TLS certificate expiring < 14 days

p95 values are computed from histogram buckets over a 5-minute window.

Uninstalling

For each gateway, first delete the source in Spitfire: All integrations In Spitfire: /admin/integrations#observability → the bin on the source's row → Delete. The Tower stops reading; collected metrics stay. An interrupted run can leave a state file in your home directory: rm -f ~/.spitfire-setup/kong.state (the same for envoy, traefik).

Kong

On a Kong with a database, the script adds the global Prometheus plugin through the Admin API (it leaves an existing one alone) and changes no config file. If the script added the plugin and nothing else reads /metrics, delete it on one Kong node (once per cluster; needs jq: apt install jq / dnf install jq). It should print OK: plugin deleted.

bash
for id in $(curl -fsS 'http://127.0.0.1:8001/plugins?size=1000' | jq -r '.data[] | select(.name == "prometheus") | .id'); do curl -fsS -X DELETE "http://127.0.0.1:8001/plugins/$id"; done; curl -fsS 'http://127.0.0.1:8001/plugins?size=1000' | jq -e '[.data[] | select(.name == "prometheus")] | length == 0' >/dev/null && echo 'OK: plugin deleted'

On a DB-less Kong, or with the manual install, you added the plugin and the status_listen = 0.0.0.0:8100 line (or KONG_STATUS_LISTEN) yourself: delete the - name: prometheus block from kong.yml and the line from kong.conf, then run sudo kong reload -c /etc/kong/kong.conf. The Status API can stay if something else uses it.

Envoy

The script changes nothing in Envoy; deleting the source is enough. If you added the admin block only for Spitfire, delete it from the bootstrap, restart Envoy and close 9901 in the firewall.

Traefik

The script changes nothing in Traefik; deleting the source is enough. If you added the --metrics.prometheus flags only for Spitfire, delete them from the static config and restart Traefik.