Preparing an OTLP sender: key, Collector and continuous APM

This page covers what to set up on your side to send your services' traces, metrics and logs to Spitfire's OTLP receiver with OpenTelemetry (OTLP): a key, an optional Collector in between, a personal data filter and continuous APM. The difference from Zabbix: here you send the data, Spitfire does not fetch it. That is why Test in Spitfire waits to see a test request you send. The receiver's details on the Spitfire side: Observability: OTLP receiver.

#What you need

• Spitfire's external address, e.g. https://spitfire.example.com. The receiver lives at /otlp/v1/traces, /otlp/v1/metrics and /otlp/v1/logs under it. Spitfire fills the address in on the key screen.
• HTTPS egress: the service or the Collector can reach that address on 443 (firewall, Kubernetes egress NetworkPolicy, corporate proxy).
• An OTLP receiver key (sfotlp_…). Open one receiver per system or environment: if one leaks, only that one is cut.
• Continuous APM for data outside runs: when no Spitfire run is in progress, incoming data is kept only if continuous APM is on. Details: Turn on continuous APM.

The receiver only takes data: Spitfire never sends a request to your systems along this path.

#Find your version

If you send straight from an SDK, see the version table in your language's guide (Connecting services with OTel). If you use a Collector, find its version:

otelcol-contrib --version          # on a server
kubectl exec deploy/<collector> -- /otelcol-contrib --version   # on Kubernetes

Verified: OpenTelemetry Collector contrib 0.164.0, Spitfire 1.4.7, 2026-11-09.

#Steps by setup

#Get the key

1. Observability → Add → OTLP receiver. Name it after the system or environment (e.g. orders-prod) and Save.
2. The key (sfotlp_…) is shown once. Spitfire keeps only its hash; if you lose it, make a New key from the list.
3. Put the key straight where secrets live: a Kubernetes Secret, a .env file on the server or a vault.

You can run the check command from the key screen right away: Try it before sending.

#Straight from an SDK

Enough for a single service with little traffic: the service sends to Spitfire itself. Follow your language's guide; they all use the same environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=https://spitfire.example.com/otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=authorization=sfotlp_…

Guides: Java, .NET, Python, Node.js, PHP, Go, Ruby, Rust.

If you have several services, or the application must not be affected at all when Spitfire cannot be reached, use a Collector in between.

#Collector inside Kubernetes

This is the recommended path. Services send to the Collector in the cluster, the Collector sends to Spitfire. If Spitfire cannot be reached the application does not wait; batching, compression, the personal data filter and sampling happen in one place.

1. Secret: the key and the address arrive as environment variables; they are not written into a ConfigMap.

   bashCopy

   read -rs SF_TOKEN   # paste the key, Enter; it stays out of the shell history
   kubectl -n <namespace> create secret generic spitfire-otlp \
     --from-literal=SPITFIRE_OTLP_ENDPOINT=https://spitfire.example.com/otlp \
     --from-literal=SPITFIRE_OTLP_TOKEN="$SF_TOKEN"
   unset SF_TOKEN

2. Collector configuration (ConfigMap). memory_limiter stays below the pod limit; when the queue is full data is dropped and the application is not affected; retries are bounded.

   yamlCopy

   extensions:
     health_check:
       endpoint: 0.0.0.0:13133
   receivers:
     otlp:
       protocols:
         http:
           endpoint: 0.0.0.0:4318
   processors:
     memory_limiter:
       check_interval: 1s
       limit_mib: 200        # pod limit 256Mi: about 80% of it
       spike_limit_mib: 50
     batch:
       send_batch_size: 1024
       send_batch_max_size: 2048
       timeout: 5s
   exporters:
     otlp_http/spitfire:     # in older versions: otlphttp/spitfire
       endpoint: ${env:SPITFIRE_OTLP_ENDPOINT}
       headers:
         authorization: ${env:SPITFIRE_OTLP_TOKEN}
       compression: gzip
       sending_queue:
         num_consumers: 2
         queue_size: 50
       retry_on_failure:
         initial_interval: 5s
         max_interval: 30s
         max_elapsed_time: 120s
   service:
     extensions: [health_check]
     pipelines:
       traces:
         receivers: [otlp]
         processors: [memory_limiter, batch]
         exporters: [otlp_http/spitfire]
       metrics:
         receivers: [otlp]
         processors: [memory_limiter, batch]
         exporters: [otlp_http/spitfire]
       logs:
         receivers: [otlp]
         processors: [memory_limiter, batch]
         exporters: [otlp_http/spitfire]

3. Deployment: image otel/opentelemetry-collector-contrib, the spitfire-otlp Secret via envFrom, a memory limit (e.g. 256Mi) and GOMEMLIMIT (a little below the limit, e.g. 200MiB), readiness and liveness probes on :13133/.

4. NetworkPolicy: only the application pods reach the Collector on 4318; the Collector's egress to Spitfire on 443 and DNS stay open.

5. Services: send to OTEL_EXPORTER_OTLP_ENDPOINT=http://<collector-service>.<namespace>:4318 without a key; only the Collector holds the key.

6. Resource attributes: service.name, service.namespace, deployment.environment and the Kubernetes namespace, deployment, pod and node names. Spitfire's service list and map use them. The k8sattributes processor adds the pod details itself.

The authorization header is accepted as Bearer sfotlp_… or as plain sfotlp_…. If a tool mangles the space in an environment variable, drop the word Bearer.

Verified: OpenTelemetry Collector contrib 0.164.0 on Kubernetes, Spitfire 1.4.7, 2026-11-09.

#Collector on a server or in Docker

The configuration is the same as on Kubernetes. The key and the address come from an environment file:

# /etc/otelcol-contrib/spitfire.env (readable by root only: chmod 600)
SPITFIRE_OTLP_ENDPOINT=https://spitfire.example.com/otlp
SPITFIRE_OTLP_TOKEN=sfotlp_…

Docker: docker run --env-file spitfire.env -v $PWD/config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib. systemd: EnvironmentFile=/etc/otelcol-contrib/spitfire.env in the unit file.

#Filter personal data

Traces and logs should not carry user IDs, email addresses, phone numbers, client IPs, query strings, headers or SQL parameters (GDPR, KVKK). Even when the application writes none, a second filter in the Collector is recommended:

processors:
  attributes/pii:
    actions:
      - { key: enduser.id, action: delete }
      - { key: user.email, action: delete }
      - { key: client.address, action: delete }
      - { key: http.request.header.authorization, action: delete }
      - { key: http.request.header.cookie, action: delete }
  transform/pii:
    error_mode: ignore
    trace_statements:
      - replace_pattern(span.attributes["url.full"], "\\?.*", "")
      - replace_all_patterns(span.attributes, "value", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+", "<email>")
    log_statements:
      - replace_pattern(log.body["string"], "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+", "<email>")

Add the processors to each pipeline's processors list after memory_limiter and before batch. Not sending traceparent on requests to outside services also keeps internal IDs away from third parties.

Spitfire itself is a safety net too: it cuts query strings from URLs and reduces database statements to the operation and table. When it sees an email, phone or public IP pattern in incoming attribute values it shows the names of those attributes on the receiver's row. The value itself is never shown or kept. If you see the warning, add those names to this filter.

#Turn on continuous APM

Spitfire writes the data that arrives during a run into that run's window. Monitoring around the clock needs continuous APM; while it is off, data outside runs is dropped with a 200 and a "partial success" answer. In the Collector's log it looks like this:

warn  Partial success ... no Spitfire run in progress: data outside run windows is not kept

1. APM → Services: an installation admin ticks "I approve that traces take space in the database" and picks the retention days (up to what the license allows).
2. A Spitfire on your own server also needs SPITFIRE_CLICKHOUSE_URL.
3. A rough disk estimate: a small system making about 200 spans per 30 seconds keeps a few GB in 30 days.

If you are not an admin, Spitfire says so with a warning box in the receiver window: pass it on to your admin. The dropped outside runs counter on the receiver's row shows how much data is lost.

Verified: Spitfire 1.4.7, 30-day retention, 2026-11-09.

#Behind a front door

If your own Spitfire sits behind a gate such as Cloudflare Access, the Collector's request stops at the gate (302 or 403). Unlike Zabbix, a service token works here: the client is your own Collector and it can add any header.

1. Zero Trust → Access controls → Service credentials → Service Tokens → Create Service Token; put the Client ID and Client Secret into the Collector's Secret.

2. Open a separate application on the Spitfire hostname with Path otlp/; policy Service Auth, Include → Service Token → the token you created. The UI stays under the existing application's policy.

3. Add the headers to the exporter:

   yamlCopy

   exporters:
     otlp_http/spitfire:
       endpoint: ${env:SPITFIRE_OTLP_ENDPOINT}
       headers:
         authorization: ${env:SPITFIRE_OTLP_TOKEN}
         CF-Access-Client-Id: ${env:CF_ACCESS_CLIENT_ID}
         CF-Access-Client-Secret: ${env:CF_ACCESS_CLIENT_SECRET}

If you cannot use a service token, the second option is a Bypass policy for the sending IPs on the same application. On other gates (nginx auth_request, SSO proxies) open only the /otlp/ path the same way; the Spitfire key protects the receiver.

#Try it before sending

Run it on the machine where the service or the Collector runs. The key is read with read -rs first so it stays out of the shell history:

read -rs SF_TOKEN   # paste the key, Enter
curl -s -o /dev/null -w '%{http_code}\n' -X POST https://spitfire.example.com/otlp/v1/traces \
  -H "Authorization: Bearer $SF_TOKEN" -H 'Content-Type: application/json' --data '{}'
unset SF_TOKEN

On Kubernetes, run the same command from a short-lived pod in the Collector's namespace so the NetworkPolicy is tested too.

Verified: curl 8, Spitfire 1.4.7, 200, 2026-11-08.

#In Spitfire

• Test: press Test in the receiver window and run the command above. Spitfire waits for the first request with this key (an empty {} counts) and shows its time and source IP. When the time runs out it says "no request arrived". Requests with a wrong key cannot be tied to any receiver, so only the 401 count of the last 5 minutes is shown, without IPs. The count is the whole installation's, so only an installation admin sees it.

• Receiver row: per signal (traces, metrics, logs) the time of the last data and the count in the last hour, refused requests by reason and the dropped outside runs counter. A receiver with no data for 24 hours turns yellow, one that never got data turns red.

  Note

  The counts (last hour, refused, dropped outside runs, 401) are kept in memory only: they start from zero when Spitfire restarts. Yellow/red comes from the stored last-data time, so it survives a restart.

• Did data arrive: a minute or two after restarting the service, the service name shows up under Towers & metrics → Resources and the first trace in APM. On the Collector:

  bashCopy

  kubectl logs deploy/<collector> --since=1m | grep -c "Partial success"

  Expect 0. If the number is above zero, see the last row of Common errors.

Verified: OpenTelemetry Collector contrib 0.164.0, Spitfire 1.4.7, 0 after the continuous APM consent, 2026-11-09.

#Common errors

#Undoing it

1. Stop sending: the environment variable that turns tracing off in the SDK, or remove the otlp_http/spitfire exporter from the Collector, then redeploy.
2. In Spitfire, disable or delete the receiver; or make a New key so the old one stops working at once.
3. Delete the Secret and the application you opened on the front door.

Data that arrived stays for the retention period; data tied to runs stays with those runs.

#Related pages

• Observability: OTLP receiver
• Connecting services with OTel
• APM screens