SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Infrastructure — ZooKeeper, etcd, Consul

Coordination clusters are read through the Tower. Each cluster is one monitored system; list all of its members, comma-separated, in that one source. Panels and findings show leader, quorum, raft and disk (WAL fsync) health.

Note

These sources are Beta. All access is read-only; Spitfire neither reads nor writes data.

ZooKeeper

Quick install

  1. Run the command on every member, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/zookeeper.sh?t=<token>" -o /tmp/sf-zookeeper.sh && bash /tmp/sf-zookeeper.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Back up zoo.cfg.
bash
sudo cp -n /opt/zookeeper/conf/zoo.cfg /opt/zookeeper/conf/zoo.cfg.spitfire.bak
  1. Turn on the Prometheus metrics provider.
bash
grep -q '^metricsProvider.className=' /opt/zookeeper/conf/zoo.cfg || printf '%s\n' 'metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider' 'metricsProvider.httpHost=0.0.0.0' 'metricsProvider.httpPort=7000' | sudo tee -a /opt/zookeeper/conf/zoo.cfg
  1. Restart the member, the leader last.
bash
sudo /opt/zookeeper/bin/zkServer.sh restart
  1. Check the metrics.
bash
curl -fsS http://localhost:7000/metrics | grep -q '^znode_count' && echo 'OK: ZooKeeper metrics ready'
  1. Spitfire → Infrastructure sources → ZooKeeper; address: zk1:7000,zk2:7000,zk3:7000.
Warning

Restart members one at a time; with two down at once quorum is lost.

etcd

Quick install

  1. Run the command on every control-plane node.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/etcd.sh?t=<token>" -o /tmp/sf-etcd.sh && bash /tmp/sf-etcd.sh
  1. When you see OK: ..., you are done.

Manual install (kubeadm)

  1. Back up the manifest outside its directory.
bash
sudo cp -n /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/etcd.yaml.spitfire.bak
  1. Open the metrics listener on the node IP.
bash
sudo sed -i 's#--listen-metrics-urls=http://127.0.0.1:2381$#--listen-metrics-urls=http://127.0.0.1:2381,http://<node-ip>:2381#' /etc/kubernetes/manifests/etcd.yaml
  1. Check the metrics (after ~30 s).
bash
curl -fsS http://<node-ip>:2381/metrics | grep -q '^etcd_server_has_leader' && echo 'OK: etcd metrics ready'
  1. Spitfire → Infrastructure sources → etcd; address: <node1-ip>:2381,<node2-ip>:2381,<node3-ip>:2381.
Note

2381 serves only /metrics and /health. To use 2379, turn on TLS and enter the CA, client certificate and key (PEM); on kubeadm they are under /etc/kubernetes/pki/etcd.

Warning

With etcd auth off, a CA-signed client certificate gives full KV access. That is why 2381 is recommended.

Consul

Quick install

  1. Run the command on every agent, enter the management token.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/consul.sh?t=<token>" -o /tmp/sf-consul.sh && bash /tmp/sf-consul.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Write the telemetry setting.
bash
printf '%s\n' 'telemetry {' '  prometheus_retention_time = "120s"' '  disable_hostname = true' '}' | sudo tee /etc/consul.d/spitfire-telemetry.hcl
  1. Restart the agent.
bash
sudo systemctl restart consul
  1. Write the read-only policy file.
bash
printf '%s\n' 'agent_prefix "" { policy = "read" }' 'node_prefix "" { policy = "read" }' 'service_prefix "" { policy = "read" }' 'operator = "read"' > /tmp/spitfire-metrics.hcl
  1. Create the policy.
bash
CONSUL_HTTP_TOKEN='<management-token>' consul acl policy create -name spitfire-metrics -rules @/tmp/spitfire-metrics.hcl
  1. Create the token, copy its SecretID.
bash
CONSUL_HTTP_TOKEN='<management-token>' consul acl token create -description spitfire -policy-name spitfire-metrics
  1. Check the metrics.
bash
curl -fsS -H 'X-Consul-Token: <secret-id>' 'http://localhost:8500/v1/agent/metrics?format=prometheus' | grep -q '^consul_runtime_alloc_bytes' && echo 'OK: Consul metrics ready'
  1. Spitfire → Infrastructure sources → Consul; the address and the ACL token (SecretID).
Note

The management token is used only in steps 4 and 5 and is never stored. The token given to Spitfire only reads.

Panels and findings

Source Finding Threshold
ZooKeeper Member down / no leader (quorum) up = 0 / no leader_uptime
ZooKeeper Outstanding requests, average latency > 10, > 100 ms
ZooKeeper Unsynced learner synced / learners < 1
etcd Member down, no leader, leader changed, proposal failed 15 min
etcd WAL fsync p99, backend commit p99 > 10 ms, > 25 ms
etcd DB quota usage > 80%
Consul Agent down, autopilot unhealthy, no leader, failed member —
Consul Leader last contact p99 > 200 ms

p99 values are computed from the histogram buckets over a 5-minute window.