Add load balancers and proxies to the trace (Nginx, Envoy, HAProxy)
Nginx and Envoy write their own span; HAProxy only carries the context and the client address to the service behind it. The topology then shows a proxy node, the waterfall the time spent in the proxy and the span detail the client chain (Call path). Even when you do nothing, Spitfire shows the load balancer between two instrumented services with its hop time and via address; this guide adds the proxy's own span to that.
What you need
- Root or
sudoon the proxy server. - A Collector on the same server listening for OTLP gRPC on
127.0.0.1:4317(setup). The Collector carries the key and the Spitfire address, and no key goes into the proxy's configuration.
Check that the Collector listens:
sudo ss -ltnp | grep 4317OK: the output has a 127.0.0.1:4317 or 0.0.0.0:4317 line.
Nginx
Verified: nginx 1.27 and nginx-module-otel from the nginx.org repository, Ubuntu 24.04.
Install the module (the nginx.org repository must be set up):
sudo apt-get update && sudo apt-get install -y nginx-module-otelOK:
ls /usr/lib/nginx/modules/ngx_otel_module.solists the file.Load the module (adds one line at the very top of
/etc/nginx/nginx.conf):sudo grep -q ngx_otel_module /etc/nginx/nginx.conf || sudo sed -i '1i load_module modules/ngx_otel_module.so;' /etc/nginx/nginx.confOK:
sudo head -1 /etc/nginx/nginx.confprintsload_module modules/ngx_otel_module.so;.Write the trace settings (the files in
/etc/nginx/conf.d/go into thehttpblock):sudo tee /etc/nginx/conf.d/spitfire-otel.conf >/dev/null <<'CONF' otel_exporter { endpoint 127.0.0.1:4317; } otel_service_name nginx-edge; otel_trace on; otel_trace_context propagate; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Request-Id $request_id; CONFOK:
sudo nginx -tprintssyntax is okandtest is successful.Reload Nginx:
sudo systemctl reload nginx && curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/OK: it prints an HTTP status code (e.g.
200). Within a minute APM → Topology shows thenginx-edgenode with a proxy · nginx badge.
You can change otel_service_name to the proxy's name. When a location block uses proxy_set_header, Nginx does not take the http level ones in that block; add the two proxy_set_header lines to that block too.
Envoy
Verified: Envoy 1.32, a setup that runs with /etc/envoy/envoy.yaml.
In
/etc/envoy/envoy.yaml, add this under thetyped_configof everyenvoy.filters.network.http_connection_managerblock:tracing: provider: name: envoy.tracers.opentelemetry typed_config: "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig grpc_service: envoy_grpc: cluster_name: spitfire_otel timeout: 0.25s service_name: envoy-edgeIn the same file, add this cluster to the
static_resources.clusterslist:- name: spitfire_otel type: STATIC typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions explicit_http_config: http2_protocol_options: {} load_assignment: cluster_name: spitfire_otel endpoints: - lb_endpoints: - endpoint: address: socket_address: { address: 127.0.0.1, port_value: 4317 }Validate the file:
envoy --mode validate -c /etc/envoy/envoy.yamlOK: the output ends with
configuration '/etc/envoy/envoy.yaml' OK.Restart Envoy:
sudo systemctl restart envoy && sudo systemctl is-active envoyOK:
active. After a request APM → Topology shows theenvoy-edgenode with a proxy · envoy badge.
On Istio these steps are not needed: point the mesh's trace provider at the same Collector; the istio-ingressgateway node is recognised as a proxy.
HAProxy
HAProxy's standard packages write no spans; Spitfire shows HAProxy as the hop between two services. These steps carry the context and the client address to the service behind it.
Verified: HAProxy 2.8 and 3.0, /etc/haproxy/haproxy.cfg.
In
/etc/haproxy/haproxy.cfg, add these lines to everyfrontendblock:option forwardfor unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid http-request set-header X-Request-Id %[unique-id] unless { req.hdr(x-request-id) -m found }Leave the
traceparentheader alone: HAProxy passes it on as it is.Validate the file:
sudo haproxy -c -f /etc/haproxy/haproxy.cfgOK:
Configuration file is valid.Reload HAProxy:
sudo systemctl reload haproxy && sudo systemctl is-active haproxyOK:
active.
Have the headers written on the span
For the client chain and the request id to show in the span detail, the service behind must write these headers on its span. For the Java agent, add to the application's environment:
OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS=x-forwarded-for,forwarded,x-real-ip,x-request-id,viaFor .NET the same list goes in OTEL_DOTNET_AUTO_TRACES_ASPNETCORE_INSTRUMENTATION_CAPTURE_REQUEST_HEADERS; for Node.js and Python it is the header setting in their guides. Restart the application.
OK: in APM → Traces, open the server span of a request that went through the proxy; the Call path section shows Forwarding headers.
Undoing it
- Nginx:
sudo rm /etc/nginx/conf.d/spitfire-otel.conf && sudo sed -i '/ngx_otel_module/d' /etc/nginx/nginx.conf && sudo nginx -t && sudo systemctl reload nginx - Envoy: delete the
tracingblock and thespitfire_otelcluster you added, validate withenvoy --mode validate -c /etc/envoy/envoy.yamland restart. - HAProxy: delete the three lines you added, then
sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy.