SpitfireLoad Testing & APMWhat's new

Infrastructure map: load balancers, hosts, data tier

The Infrastructure map In Spitfire: /infrastructure/map (Infrastructure → Infrastructure map) shows hosts, the load balancers in front of them and the data tier on one screen. It is on Starter and above.

Setup

  1. Add a load balancer data source:
    • HAProxy and Nginx: Web servers
    • NetScaler: NetScaler and Citrix
    • Kubernetes Service, EndpointSlice and Ingress: Kubernetes and OpenShift (the same token; read access to services and ingresses in the RBAC)
    • AWS ALB/NLB target groups: Cloud (IAM: elasticloadbalancing:DescribeLoadBalancers, DescribeTargetGroups, DescribeTargetHealth)
  2. Install the server Collector on the servers the members run on (Server monitoring); the IPs the agent reports match the members by themselves.
  3. Optional: APM for host to host calls and the "held at load balancer" estimate; the per-request syslog line for NetScaler's own 5xx answers (Log collection → NetScaler).

Success check

The first read arrives within 5 minutes (AWS is read every 5 minutes). On the Load balancers tab each load balancer's members are listed and the Host column shows a host name or "Unknown address"; on the Map tab dashed lines run from the load balancer to its members.

Matching order

Each member's address is compared in order, and the first that fits wins: a manual mapping, Agent IP (the server Collector), Pod IP (the pod in the EndpointSlice and its node), APM (the host name and address it saw), DNS (when the member is written as a name, e.g. HAProxy server web-1 web-1:8080). When none fits, the member shows as Unknown address. AWS instance targets and HAProxy members written as IPs often need a manual mapping.

Manual mapping

  1. On the Load balancers tab, click map on the member's row (only workspace admins see it).
  2. Pick or type the host; optionally the service too.
  3. Save. The mapping applies to this address on this load balancer; Remove the mapping in the same window removes it.

Mappings and removals are written to the audit log (lb.map.set, lb.map.delete).

Map

  • Nodes: host (the highest of CPU, memory and disk: above 85% yellow, above 95% red; the number in the corner is the open problems), load balancer, data tier (the databases and queues APM saw), unknown address.
  • Lines: load balancer → member (dashed, the member's state) and host → host (APM calls: calls per minute, error rate, p95). Netflow is not used.
  • Clicking a node centres the map on it: callers on the left, the called on the right; −/+ picks how many steps to show, and clicking the centred host again opens its host page.
  • Narrow by zone (the Kubernetes cluster or AWS region), node kind and, with APM on, environment.

The Load balancers in front card on the server page shows which pools the host is a member of.

Held at load balancer (estimate)

For NetScaler vservers, the requests APM saw on the hosts behind (divided by the sampling share) are subtracted from the requests the load balancer counted. When the difference is above 5 requests and 2%, the Held at load balancer (estimate) badge shows, with the evidence next to it: the surge queue, spillover, DOWN members, the load balancer's own 5xx answers and busy errors. This is an estimate, not a measurement.

  • When it cannot be computed the reason shows, e.g. cannot be computed: 2 members not matched; map the members and the estimate starts.
  • If your services send a part of the requests (e.g. 10%), enter the APM sampling share on this tab (audit log: lb.sampling.set).
  • Click a vserver for its per-minute charts; This vserver's top requests filters the list to that vserver's hosts. The same table is on the Load balancer tab of APM → Top lists.

Rule

The built-in Requests held at the load balancer (estimate) rule is off by default; turn it on under Event policies → Built-in In Spitfire: /events/policies?tab=builtin → Network. Thresholds: 5% warning, 20% critical, for 5 minutes. Its events count as the infrastructure layer in the root cause estimate.