SpitfireLoad Testing & APMWhat's new

Call path (messaging, load balancers, external calls)

The whole path of a request between services: messages through Kafka, RabbitMQ, IBM MQ, ActiveMQ, SQS, Service Bus or Pub/Sub, the load balancers and proxies in between, and the calls to third party services nobody instruments. It works for services that send OTel straight and for the traces the Spitfire Collector turns on. Menu: APM → Services → Messaging and APM → Services → External calls.

Messaging

When a consumer reads a message, most instrumentations start a new trace and put a span link to the producer's span. Spitfire follows these links. The Linked traces line on top of the trace page shows the traces before (Continued from) and after (Continues in). The End to end button next to it opens the producer → topic or queue → consumer chain on one page (at most 20 traces).

The Messaging page lists, for each topic, queue and consumer group, messages per minute, error rate, delay (p50/p95/p99) and processing time. The Call path section of the span detail shows partition, offset, consumer group and routing key.

Delay = the consumer span's start − the producer span's end. When the producer is not instrumented, the Java agent's Kafka record time is used. The producer is looked up at most 6 hours back. Old and new OTel names are read alike (messaging.destination and messaging.destination.name and so on).

The most common case: the API answers fast but the work finishes late. The cause is usually a consumer falling behind, and the delay p95 on the Messaging page is the first place that shows it.

Load balancers and proxies

  • Hop time: when a client span calls another service's server span, the difference (client duration − server duration) is the time spent in the network, the load balancer or the proxy. The waterfall shows it as an LB/proxy +… ms badge.
  • Via: when the address the caller asked (server.address) differs from the host that answered, that address is a load balancer, a Kubernetes service or a gateway. The topology writes via … on the line.
  • Proxy nodes: Nginx, Envoy (Istio too), HAProxy, Traefik, Kong, APISIX, Caddy and API gateways show on the topology with a proxy badge when they send spans.
  • Forwarding headers: when the service writes headers like X-Forwarded-For, X-Real-IP, X-Request-Id and Via on its span (http.request.header.*), the span detail shows the proxy chain. These headers are kept whatever the request/response capture setting is, and masked.

To add the load balancer or proxy to the trace: Add load balancers and proxies to the trace.

External calls

A client span no instrumented service answered is an external call: a third party API, a SaaS or a service without an agent. The External calls page shows them by domain (e.g. example.co.uk) and host, with calls, error rate, p95 and the calling services. Numbers and UUIDs in paths become one placeholder; query strings are not kept. Database and messaging calls do not count as external calls.

Who captures what

Source Messaging External HTTP call Context propagation
Java (OTel agent, Collector) Kafka, RabbitMQ, JMS (IBM MQ, ActiveMQ), SQS, Pulsar; partition, offset and record time Every HTTP client tracecontext, baggage, b3multi
.NET and .NET Framework, IIS (OTel, Collector) Service Bus, Event Hubs, Kafka (Confluent), RabbitMQ (by library version) HttpClient, HttpWebRequest tracecontext, baggage, b3multi
Node.js (OTel, Collector) kafkajs, amqplib, SQS http, fetch, undici tracecontext, baggage, b3multi
Python (OTel, Collector) kafka-python, confluent-kafka, pika, aio-pika, Celery requests, httpx, urllib3 tracecontext, baggage
PHP (OTel, Collector) No Kafka; AMQP by library Guzzle, cURL tracecontext, baggage
Go, Rust, C++, Ruby (SDK) By the instrumentation library you use By library By SDK setting

The Spitfire Collector sets these propagation settings itself on the applications it turns tracing on for; when the application sets the same variable itself, the application's value stays.

Common problems

  • No Linked traces line: the consumer span has no span link. The producer and the consumer must use the same propagation (tracecontext on both). On Kafka, message headers must be on (Kafka 0.11 and later).
  • The trace says "not stored": the linked trace was sampled out, is outside the retention, or its service does not send to Spitfire.
  • A service shows as an external call: that service sends no spans. Connect it too: Connect services with OTel.
  • No delay: there is no producer span and the instrumentation writes no record time.