SpitfireLoad Testing & APMWhat's new

Add load balancers and proxies to the trace (Nginx, Envoy, HAProxy)

Nginx and Envoy write their own span; HAProxy only carries the context and the client address to the service behind it. The topology then shows a proxy node, the waterfall the time spent in the proxy and the span detail the client chain (Call path). Even when you do nothing, Spitfire shows the load balancer between two instrumented services with its hop time and via address; this guide adds the proxy's own span to that.

What you need

  • Root or sudo on the proxy server.
  • A Collector on the same server listening for OTLP gRPC on 127.0.0.1:4317 (setup). The Collector carries the key and the Spitfire address, and no key goes into the proxy's configuration.

Check that the Collector listens:

bash
sudo ss -ltnp | grep 4317

OK: the output has a 127.0.0.1:4317 or 0.0.0.0:4317 line.

Nginx

Verified: nginx 1.27 and nginx-module-otel from the nginx.org repository, Ubuntu 24.04.

  1. Install the module (the nginx.org repository must be set up):

    bash
    sudo apt-get update && sudo apt-get install -y nginx-module-otel

    OK: ls /usr/lib/nginx/modules/ngx_otel_module.so lists the file.

  2. Load the module (adds one line at the very top of /etc/nginx/nginx.conf):

    bash
    sudo grep -q ngx_otel_module /etc/nginx/nginx.conf || sudo sed -i '1i load_module modules/ngx_otel_module.so;' /etc/nginx/nginx.conf

    OK: sudo head -1 /etc/nginx/nginx.conf prints load_module modules/ngx_otel_module.so;.

  3. Write the trace settings (the files in /etc/nginx/conf.d/ go into the http block):

    bash
    sudo tee /etc/nginx/conf.d/spitfire-otel.conf >/dev/null <<'CONF'
    otel_exporter {
        endpoint 127.0.0.1:4317;
    }
    otel_service_name nginx-edge;
    otel_trace on;
    otel_trace_context propagate;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Request-Id $request_id;
    CONF

    OK: sudo nginx -t prints syntax is ok and test is successful.

  4. Reload Nginx:

    bash
    sudo systemctl reload nginx && curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/

    OK: it prints an HTTP status code (e.g. 200). Within a minute APM → Topology shows the nginx-edge node with a proxy · nginx badge.

You can change otel_service_name to the proxy's name. When a location block uses proxy_set_header, Nginx does not take the http level ones in that block; add the two proxy_set_header lines to that block too.

Envoy

Verified: Envoy 1.32, a setup that runs with /etc/envoy/envoy.yaml.

  1. In /etc/envoy/envoy.yaml, add this under the typed_config of every envoy.filters.network.http_connection_manager block:

    yaml
              tracing:
                provider:
                  name: envoy.tracers.opentelemetry
                  typed_config:
                    "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
                    grpc_service:
                      envoy_grpc:
                        cluster_name: spitfire_otel
                      timeout: 0.25s
                    service_name: envoy-edge
  2. In the same file, add this cluster to the static_resources.clusters list:

    yaml
      - name: spitfire_otel
        type: STATIC
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            explicit_http_config:
              http2_protocol_options: {}
        load_assignment:
          cluster_name: spitfire_otel
          endpoints:
            - lb_endpoints:
                - endpoint:
                    address:
                      socket_address: { address: 127.0.0.1, port_value: 4317 }
  3. Validate the file:

    bash
    envoy --mode validate -c /etc/envoy/envoy.yaml

    OK: the output ends with configuration '/etc/envoy/envoy.yaml' OK.

  4. Restart Envoy:

    bash
    sudo systemctl restart envoy && sudo systemctl is-active envoy

    OK: active. After a request APM → Topology shows the envoy-edge node with a proxy · envoy badge.

On Istio these steps are not needed: point the mesh's trace provider at the same Collector; the istio-ingressgateway node is recognised as a proxy.

HAProxy

HAProxy's standard packages write no spans; Spitfire shows HAProxy as the hop between two services. These steps carry the context and the client address to the service behind it.

Verified: HAProxy 2.8 and 3.0, /etc/haproxy/haproxy.cfg.

  1. In /etc/haproxy/haproxy.cfg, add these lines to every frontend block:

    text
        option forwardfor
        unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
        http-request set-header X-Request-Id %[unique-id] unless { req.hdr(x-request-id) -m found }

    Leave the traceparent header alone: HAProxy passes it on as it is.

  2. Validate the file:

    bash
    sudo haproxy -c -f /etc/haproxy/haproxy.cfg

    OK: Configuration file is valid.

  3. Reload HAProxy:

    bash
    sudo systemctl reload haproxy && sudo systemctl is-active haproxy

    OK: active.

Have the headers written on the span

For the client chain and the request id to show in the span detail, the service behind must write these headers on its span. For the Java agent, add to the application's environment:

bash
OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS=x-forwarded-for,forwarded,x-real-ip,x-request-id,via

For .NET the same list goes in OTEL_DOTNET_AUTO_TRACES_ASPNETCORE_INSTRUMENTATION_CAPTURE_REQUEST_HEADERS; for Node.js and Python it is the header setting in their guides. Restart the application.

OK: in APM → Traces, open the server span of a request that went through the proxy; the Call path section shows Forwarding headers.

Undoing it

  • Nginx: sudo rm /etc/nginx/conf.d/spitfire-otel.conf && sudo sed -i '/ngx_otel_module/d' /etc/nginx/nginx.conf && sudo nginx -t && sudo systemctl reload nginx
  • Envoy: delete the tracing block and the spitfire_otel cluster you added, validate with envoy --mode validate -c /etc/envoy/envoy.yaml and restart.
  • HAProxy: delete the three lines you added, then sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy.