Version: 1.0.0This documentation is for Spitfire 1.0.0.

Connecting a Rust service with OTel

Rust is compiled and has no agent: you add one file and a few lines to the service and build it again. The service's tracing spans and events then go to Spitfire as traces and logs, and its HTTP requests as a metric; the addresses and tokens come from the environment, as in the other guides. Values and general rules: Connecting services with OTel: overview.

Version table

Rust	Crates	Note
1.88 and later	`opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp`, `opentelemetry-http`, `opentelemetry-appender-tracing` 0.33.1; `tracing-opentelemetry` 0.34.0; `tracing` 0.1.44, `tracing-subscriber` 0.3.23	The OTel crates say 1.75, but the current versions of the crates they pull in need Rust 1.88 (Rust 2024 edition, ICU crates)
1.87 and earlier	not supported	Upgrade the toolchain (`rustup update`)

Find your version: rustc --version on the build machine (or rust-version in Cargo.toml).

The examples below are for axum 0.8; with another framework, the middleware in step 1 is written the same way (take traceparent from the request headers, open the span with that parent).

Verified: 2026-11-03. Source: Rust getting started, opentelemetry-otlp (features, environment variables), tracing-opentelemetry, crate versions.

What you need

Spitfire address https://<spitfire> and OTLP receiver token sfotlp_…
Tower address http://<tower>:4318 and Collector token sfcol_…
Protocol http/protobuf
Service name <service-name>, version <service-version>, environment <environment>
With a private CA, the CA certificate in the system trust store of the machine (or image)
With a proxy, the Tower and Spitfire in NO_PROXY
Sampling parentbased_traceidratio, 0.1
The service's source code and the machine it is built on

Linux (systemd)

Create the file src/otel.rs and paste this:

rust

use std::sync::OnceLock;

use axum::{extract::Request, middleware::Next, response::Response};
use opentelemetry::{global, metrics::{Counter, Histogram}, trace::TracerProvider as _, KeyValue};
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_http::HeaderExtractor;
use opentelemetry_otlp::{LogExporter, MetricExporter, SpanExporter};
use opentelemetry_sdk::{logs::SdkLoggerProvider, metrics::SdkMeterProvider, propagation::TraceContextPropagator, trace::SdkTracerProvider, Resource};
use tracing::{level_filters::LevelFilter, Instrument};
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{filter::Targets, layer::SubscriberExt, util::SubscriberInitExt};

/// Sends what is left when dropped: keep it until main returns.
pub struct Otel {
    tp: SdkTracerProvider,
    mp: SdkMeterProvider,
    lp: SdkLoggerProvider,
}

impl Drop for Otel {
    fn drop(&mut self) {
        let _ = self.tp.shutdown();
        let _ = self.mp.shutdown();
        let _ = self.lp.shutdown();
    }
}

/// IS_OTEL_ENABLED=true turns OTel on; any other value leaves it off.
/// Addresses, tokens, sampling and the service name come from the OTEL_*
/// environment values.
pub fn setup() -> Option<Otel> {
    // The exporters' own crates stay out, or every export would log again.
    let filter = Targets::new()
        .with_default(LevelFilter::INFO)
        .with_target("hyper", LevelFilter::OFF)
        .with_target("h2", LevelFilter::OFF)
        .with_target("reqwest", LevelFilter::OFF)
        .with_target("opentelemetry", LevelFilter::OFF);
    if std::env::var("IS_OTEL_ENABLED").as_deref() != Ok("true") {
        tracing_subscriber::registry().with(filter).with(tracing_subscriber::fmt::layer()).init();
        return None;
    }
    let resource = Resource::builder().build();
    let tp = SdkTracerProvider::builder()
        .with_batch_exporter(SpanExporter::builder().with_http().build().expect("OTLP span exporter"))
        .with_resource(resource.clone())
        .build();
    let mp = SdkMeterProvider::builder()
        .with_periodic_exporter(MetricExporter::builder().with_http().build().expect("OTLP metric exporter"))
        .with_resource(resource.clone())
        .build();
    let lp = SdkLoggerProvider::builder()
        .with_batch_exporter(LogExporter::builder().with_http().build().expect("OTLP log exporter"))
        .with_resource(resource)
        .build();
    global::set_text_map_propagator(TraceContextPropagator::new());
    global::set_tracer_provider(tp.clone());
    global::set_meter_provider(mp.clone());
    tracing_subscriber::registry()
        .with(filter)
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer(tp.tracer("otel")))
        .with(OpenTelemetryTracingBridge::new(&lp))
        .init();
    Some(Otel { tp, mp, lp })
}

/// axum middleware: a server span per request, continuing an incoming
/// traceparent (Spitfire's test), and the request count and duration as metrics.
pub async fn trace_http(req: Request, next: Next) -> Response {
    static DURATION: OnceLock<Histogram<f64>> = OnceLock::new();
    static REQUESTS: OnceLock<Counter<u64>> = OnceLock::new();
    let duration = DURATION.get_or_init(|| {
        global::meter("otel").f64_histogram("http.server.request.duration").with_unit("s").build()
    });
    let requests = REQUESTS.get_or_init(|| {
        global::meter("otel").u64_counter("http.server.requests").with_unit("{request}").build()
    });
    let parent = global::get_text_map_propagator(|p| p.extract(&HeaderExtractor(req.headers())));
    let method = req.method().to_string();
    let path = req.uri().path().to_string();
    let span = tracing::info_span!("request", otel.name = %format!("{method} {path}"), otel.kind = "server",
        http.request.method = %method, url.path = %path, http.response.status_code = tracing::field::Empty);
    let _ = span.set_parent(parent);
    let start = std::time::Instant::now();
    let res = next.run(req).instrument(span.clone()).await;
    let status = res.status().as_u16();
    span.record("http.response.status_code", status);
    let attrs = [
        KeyValue::new("http.request.method", method),
        KeyValue::new("http.response.status_code", i64::from(status)),
    ];
    duration.record(start.elapsed().as_secs_f64(), &attrs);
    requests.add(1, &attrs);
    res
}

In src/main.rs, add the module, call setup first and put the middleware on the router:

rust

mod otel;

// at the start of main; keep it until main returns:
let _otel = otel::setup();

// on the router (axum::middleware):
let app = Router::new()
    .route("/", get(handler))
    .layer(axum::middleware::from_fn(otel::trace_http));

If the service already sets up tracing_subscriber, delete that line: setup does it.

In Cargo.toml, add these lines under [dependencies] and build the service again (cargo build --release):

toml

opentelemetry = "=0.33.1"
opentelemetry_sdk = "=0.33.1"
opentelemetry-otlp = { version = "=0.33.1", features = ["reqwest-rustls"] }
opentelemetry-http = "=0.33.1"
opentelemetry-appender-tracing = "=0.33.1"
tracing-opentelemetry = "=0.34.0"
tracing = "=0.1.44"
tracing-subscriber = "=0.3.23"

reqwest-rustls is needed for https:// addresses; it uses the system's trust store.

Create the directory, then the file /etc/spitfire-otel/<service-name>.env, and paste this:

bash

sudo mkdir -p /etc/spitfire-otel

bash

IS_OTEL_ENABLED=true
OTEL_SERVICE_NAME=<service-name>
OTEL_RESOURCE_ATTRIBUTES=service.version=<service-version>,deployment.environment=<environment>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<spitfire>/otlp/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=sfotlp_…
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://<spitfire>/otlp/v1/logs
OTEL_EXPORTER_OTLP_LOGS_HEADERS=authorization=sfotlp_…
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<tower>:4318/v1/metrics
OTEL_EXPORTER_OTLP_METRICS_HEADERS=authorization=sfcol_…
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
NO_PROXY=<tower>,<spitfire>

Restrict its permission: sudo chmod 600 /etc/spitfire-otel/<service-name>.env
Open sudo systemctl edit <unit> and paste this:

ini

[Service]
EnvironmentFile=/etc/spitfire-otel/<service-name>.env

Put the new binary in place and restart: sudo systemctl daemon-reload && sudo systemctl restart <unit>
See that it started: the journalctl -u <unit> -n 50 output must not show OTLP span exporter (the service stops with that message when the exporter cannot be built).

Note

With a private CA, add the CA to the system trust store (sudo cp ca.pem /usr/local/share/ca-certificates/spitfire-ca.crt && sudo update-ca-certificates); OTEL_EXPORTER_OTLP_CERTIFICATE is not read by the Rust exporter.

Docker

Make the code changes (Linux steps 1–3); the Dockerfile needs no new line, the crates are compiled into the binary.
The final image must have CA certificates for https://<spitfire> (on Debian/Ubuntu: RUN apt-get update && apt-get install -y ca-certificates); with a private CA, copy it in and run update-ca-certificates.
Save the environment file (Linux step 4) on the Docker machine as /etc/spitfire-otel/<service-name>.env.
Run: docker run --env-file /etc/spitfire-otel/<service-name>.env … or, in compose, under the service env_file: /etc/spitfire-otel/<service-name>.env.
When the environment file changes, recreate the container (restart is not enough): docker compose up -d --force-recreate <service>

Kubernetes

Prepare the image as in the Docker section.
Save the environment file (Linux step 4) as <service-name>.env and make it a Secret:

bash

kubectl create secret generic <service-name>-otel --from-env-file=<service-name>.env \
  --dry-run=client -o yaml | kubectl apply -f -

In the Deployment, add this under the container:

yaml

envFrom:
  - secretRef:
      name: <service-name>-otel

Restart: kubectl rollout restart deployment/<deployment>

Windows (service)

Make the code changes (Linux steps 1–3) and build the .exe again.
In an administrator PowerShell, write the service's name (Name in the Get-Service list) in place of <windows-service> and run:

powershell

$Service = "<windows-service>"
$IS_OTEL_ENABLED = "true"
$Otel = @(
  "IS_OTEL_ENABLED=$IS_OTEL_ENABLED",
  "OTEL_SERVICE_NAME=<service-name>",
  "OTEL_RESOURCE_ATTRIBUTES=service.version=<service-version>,deployment.environment=<environment>",
  "OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf",
  "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<spitfire>/otlp/v1/traces",
  "OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=sfotlp_…",
  "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://<spitfire>/otlp/v1/logs",
  "OTEL_EXPORTER_OTLP_LOGS_HEADERS=authorization=sfotlp_…",
  "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<tower>:4318/v1/metrics",
  "OTEL_EXPORTER_OTLP_METRICS_HEADERS=authorization=sfcol_…",
  "OTEL_TRACES_SAMPLER=parentbased_traceidratio",
  "OTEL_TRACES_SAMPLER_ARG=0.1",
  "NO_PROXY=<tower>,<spitfire>"
)
$Key = "HKLM:\SYSTEM\CurrentControlSet\Services\$Service"
$Keep = @((Get-ItemProperty $Key).Environment | Where-Object { $_ -and $_ -notmatch '^(IS_OTEL_ENABLED|OTEL_|NO_PROXY)=' })
Set-ItemProperty $Key -Name Environment -Type MultiString -Value ($Keep + $Otel)
Restart-Service -Name $Service -Force

Turning off and removing

Turning off: set IS_OTEL_ENABLED=false in the environment file (in the script on Windows) and restart the service (Linux step 7, Docker step 5, Kubernetes step 4, or run the Windows script again). The code stays; setup then only writes logs to stdout.
Removing: delete src/otel.rs, the mod otel;, setup and layer lines and the crates added to Cargo.toml, and build again; sudo systemctl revert <unit> and sudo rm -r /etc/spitfire-otel; on Windows, delete the OTel lines from the service's Environment value.

Common problems

Symptom	Cause	Fix
`feature edition2024 is required` or `requires rustc 1.88`	The toolchain is too old	`rustup update` (see the version table)
`the trait bound … is not satisfied` between `opentelemetry` types	Two versions of the `opentelemetry` crates in the build	Write every crate with the versions above (`=`); `cargo tree -d` shows duplicates
`invalid peer certificate: UnknownIssuer`	Private CA not in the trust store	Add it as in the Linux note
`a global default trace dispatcher has already been set`	The service sets up `tracing_subscriber` itself too	Delete that line (Linux step 2)
No traces or metrics arrive	Address, token or network	Follow the Verifying steps

Connecting a Rust service with OTel

#Version table

#What you need

#Linux (systemd)

#Docker

#Kubernetes

#Windows (service)

#Turning off and removing

#Common problems

#Related pages

Version table

What you need

Linux (systemd)

Docker

Kubernetes

Windows (service)

Turning off and removing

Common problems

Related pages