Spitfirev0.71.0 · What's new

Version: 0.71.0This documentation is for Spitfire 0.71.0.

Connecting a Python service with OTel

OpenTelemetry Python automatic instrumentation (opentelemetry-instrument) collects traces, metrics and logs without touching the code. Values and general rules: Connecting services with OTel: overview.

Version table

Python opentelemetry-distro opentelemetry-exporter-otlp-proto-http
3.10 and later 0.66b2 1.45.2
3.9 0.62b1 1.41.1
3.8 0.54b1 1.33.1
3.7 and earlier not supported Upgrade to Python 3.8+ or take server metrics only: Collector

Find your version: python --version with the Python the service uses.

The commands below use the 3.10+ versions; if your band is 3.9 or 3.8, write the versions from the table.

Verified: 2026-11-02. Source: changelog (3.8 support removed in 1.34.0, 3.9 support in 1.42.0), PyPI opentelemetry-sdk, Python zero-code documentation.

What you need

  • Spitfire address https://<spitfire> and OTLP receiver token sfotlp_…
  • Tower address http://<tower>:4318 and Collector token sfcol_…
  • Protocol http/protobuf
  • Service name <service-name>, version <service-version>, environment <environment>
  • With a private CA, the CA file /etc/spitfire-otel/ca.pem
  • With a proxy, the Tower and Spitfire in NO_PROXY
  • Sampling parentbased_traceidratio, 0.1

Linux (systemd)

  1. Install the packages into the service's virtual environment (<venv>, e.g. /srv/app/.venv):
bash
sudo <venv>/bin/pip install opentelemetry-distro==0.66b2 opentelemetry-exporter-otlp-proto-http==1.45.2
sudo <venv>/bin/opentelemetry-bootstrap -a install
sudo mkdir -p /opt/spitfire-otel /etc/spitfire-otel
  1. Create the file /etc/spitfire-otel/<service-name>.env and paste this:
bash
IS_OTEL_ENABLED=true
OTEL_SERVICE_NAME=<service-name>
OTEL_RESOURCE_ATTRIBUTES=service.version=<service-version>,deployment.environment=<environment>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<spitfire>/otlp/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=sfotlp_…
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://<spitfire>/otlp/v1/logs
OTEL_EXPORTER_OTLP_LOGS_HEADERS=authorization=sfotlp_…
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<tower>:4318/v1/metrics
OTEL_EXPORTER_OTLP_METRICS_HEADERS=authorization=sfcol_…
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
NO_PROXY=<tower>,<spitfire>
#OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/spitfire-otel/ca.pem
  1. At the end of the same file, add these two lines for Python logs:
bash
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
OTEL_PYTHON_LOG_AUTO_INSTRUMENTATION=true
  1. Restrict its permission: sudo chmod 600 /etc/spitfire-otel/<service-name>.env
  2. Create the file /opt/spitfire-otel/otel-run and paste this:
bash
#!/bin/sh
# IS_OTEL_ENABLED=true: the command starts through opentelemetry-instrument; any other value: without OTel.
if [ "$IS_OTEL_ENABLED" = "true" ]; then
  exec "$(dirname "$(command -v "$1")")/opentelemetry-instrument" "$@"
fi
export OTEL_SDK_DISABLED=true
exec "$@"
  1. Make it executable: sudo chmod 755 /opt/spitfire-otel/otel-run
  2. Find the service's current start command: systemctl cat <unit> | grep ^ExecStart. The first word of the command must be the full path inside the virtual environment (e.g. /srv/app/.venv/bin/gunicorn).
  3. Open sudo systemctl edit <unit> and paste this (on the last line, replace the command after ExecStart= with the command from step 7):
ini
[Service]
EnvironmentFile=/etc/spitfire-otel/<service-name>.env
ExecStart=
ExecStart=/opt/spitfire-otel/otel-run /srv/app/.venv/bin/gunicorn app:app --bind 0.0.0.0:8000
  1. Restart: sudo systemctl daemon-reload && sudo systemctl restart <unit>
  2. See that it runs: in the systemctl status <unit> output, the process started through opentelemetry-instrument.

Docker

  1. Put the otel-run file (Linux step 5) next to the Dockerfile.
  2. Add this to the Dockerfile; /opt/spitfire-otel/otel-run goes at the start of ENTRYPOINT, the rest is your command (the first word a full path):
dockerfile
RUN pip install opentelemetry-distro==0.66b2 opentelemetry-exporter-otlp-proto-http==1.45.2 \
 && opentelemetry-bootstrap -a install
COPY --chmod=755 otel-run /opt/spitfire-otel/otel-run
ENTRYPOINT ["/opt/spitfire-otel/otel-run", "/usr/local/bin/gunicorn", "app:app", "--bind", "0.0.0.0:8000"]
  1. Save the environment file (Linux steps 2 and 3) on the Docker machine as /etc/spitfire-otel/<service-name>.env.
  2. Run: docker run --env-file /etc/spitfire-otel/<service-name>.env … or, in compose, under the service env_file: /etc/spitfire-otel/<service-name>.env.
  3. When the environment file changes, recreate the container (restart is not enough): docker compose up -d --force-recreate <service>

Kubernetes

  1. Prepare the image as in the Docker section.
  2. Save the environment file (Linux steps 2 and 3) as <service-name>.env and make it a Secret:
bash
kubectl create secret generic <service-name>-otel --from-env-file=<service-name>.env \
  --dry-run=client -o yaml | kubectl apply -f -
  1. In the Deployment, add this under the container:
yaml
envFrom:
  - secretRef:
      name: <service-name>-otel
  1. Restart: kubectl rollout restart deployment/<deployment>

Windows (service)

  1. In an administrator PowerShell, install the packages into the service's virtual environment (<venv>, e.g. C:\app\.venv):
powershell
<venv>\Scripts\pip.exe install opentelemetry-distro==0.66b2 opentelemetry-exporter-otlp-proto-http==1.45.2
<venv>\Scripts\opentelemetry-bootstrap.exe -a install
  1. Write the service's name (Name in the Get-Service list) in place of <windows-service> and the virtual environment's folder in place of <venv>, then run:
powershell
$Service = "<windows-service>"
$Venv = "<venv>"
$IS_OTEL_ENABLED = "true"
$Otel = @(
  "IS_OTEL_ENABLED=$IS_OTEL_ENABLED",
  "OTEL_SERVICE_NAME=<service-name>",
  "OTEL_RESOURCE_ATTRIBUTES=service.version=<service-version>,deployment.environment=<environment>",
  "OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf",
  "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<spitfire>/otlp/v1/traces",
  "OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=sfotlp_…",
  "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://<spitfire>/otlp/v1/logs",
  "OTEL_EXPORTER_OTLP_LOGS_HEADERS=authorization=sfotlp_…",
  "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<tower>:4318/v1/metrics",
  "OTEL_EXPORTER_OTLP_METRICS_HEADERS=authorization=sfcol_…",
  "OTEL_TRACES_EXPORTER=otlp",
  "OTEL_METRICS_EXPORTER=otlp",
  "OTEL_LOGS_EXPORTER=otlp",
  "OTEL_TRACES_SAMPLER=parentbased_traceidratio",
  "OTEL_TRACES_SAMPLER_ARG=0.1",
  "NO_PROXY=<tower>,<spitfire>",
  "OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true",
  "OTEL_PYTHON_LOG_AUTO_INSTRUMENTATION=true"
)
# Private CA: $Otel += "OTEL_EXPORTER_OTLP_CERTIFICATE=C:\spitfire-otel\ca.pem"
if ($IS_OTEL_ENABLED -eq "true") {
  $Otel += "PYTHONPATH=$Venv\Lib\site-packages\opentelemetry\instrumentation\auto_instrumentation"
} else {
  $Otel += "OTEL_SDK_DISABLED=true"
}
$Key = "HKLM:\SYSTEM\CurrentControlSet\Services\$Service"
$Keep = @((Get-ItemProperty $Key).Environment | Where-Object { $_ -and $_ -notmatch '^(IS_OTEL_ENABLED|OTEL_|PYTHONPATH|NO_PROXY)=' })
Set-ItemProperty $Key -Name Environment -Type MultiString -Value ($Keep + $Otel)
Restart-Service -Name $Service -Force
  1. Send a few requests to the service and follow the Verifying steps.
Note

PYTHONPATH loads OTel the way opentelemetry-instrument does (source); the service command does not change. If the service already has a PYTHONPATH, add it after a ; on that line.

Turning off and removing

  1. Turning off: set IS_OTEL_ENABLED=false in the environment file (in the script on Windows) and restart the service (Linux step 9, Docker step 5, Kubernetes step 4, or run the Windows script again).
  2. Removing: sudo systemctl revert <unit>, sudo rm -r /opt/spitfire-otel /etc/spitfire-otel, and remove the opentelemetry- packages with <venv>/bin/pip uninstall; in Docker, delete the lines added to the Dockerfile; on Windows, delete the OTel lines from the service's Environment value.

Common problems

Symptom Cause Fix
opentelemetry-instrument: not found The command's first word is outside the virtual environment Write the full path inside the virtual environment in ExecStart (e.g. /srv/app/.venv/bin/python)
pip finds no matching version The Python version is outside the band Write your band's versions from the table
Traces arrive but no library spans opentelemetry-bootstrap did not run Run the second command of step 1 again in the service's virtual environment
No metrics from gunicorn or uWSGI workers Forking servers Apply the post_fork setting from the fork model example
No traces or metrics arrive Address, token or network Follow the Verifying steps