SpitfireLoad Testing & APMWhat's new

Offline and restricted networks

In banks and public institutions the test environment often cannot reach the internet. There are three ways: temporary internet access, an internal registry mirror (Harbor, Artifactory, Nexus) or a single-file offline bundle. We suggest the registry mirror, because later updates arrive the same way. Installing behind a proxy is on this page too.

Every path ends with the same check: curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8470/healthz prints 204.

Updating without internet

A controller without internet access cannot learn about a new release itself. When the update check is off (SPITFIRE_UPDATE_CHECK=off) or its last attempt failed, and the installed release is more than 60 days old, admins see a This release is N days old note at the top of the page; the installed release's date is also on the License → Version and updates card. The {{version}} in the commands is the release this help came with; if you downloaded a newer one, use its number.

a) Temporary internet access

  1. For the time of the update, allow HTTPS (443) from the controller server (on Kubernetes, the nodes too) to:
    • spitfire.sh, spitfire.tr: the install script, bundles, releases.json, the Helm chart
    • Docker Hub: registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com, docker-images-prod.6aa30f8b08e16409b46e0173d6de2f56.r2.cloudflarestorage.com (image layers), hub.docker.com (the release check of a Kubernetes install)
  2. Press Check now on the License → Version and updates card; the new release and its notes show up. With the check off, skip this step.
  3. Run the update command the card shows (Updating step by step); the database is dumped first. If you go out through a proxy, run it with HTTPS_PROXY set (Installing behind a proxy).
  4. Close the access again.

b) Internal registry mirror (Harbor, Artifactory, Nexus)

  1. On a machine with internet access, pull the image and push it to your internal registry (or use your registry's Docker Hub proxy cache):

    bash
    docker pull usespitfire/spitfire:{{version}}
    docker tag usespitfire/spitfire:{{version}} harbor.example.local/spitfire/spitfire:{{version}}
    docker push harbor.example.local/spitfire/spitfire:{{version}}
  2. On the controller server, unpack the installer bundle (spitfire-{{version}}.tar.gz, listed at usespitfire.com/install) and install with that image (pick the Docker or the Kubernetes line):

    bash
    tar -xzf spitfire-{{version}}.tar.gz -C ~/spitfire
    ~/spitfire/install.sh docker --image harbor.example.local/spitfire/spitfire:{{version}}
    ~/spitfire/install.sh kubernetes --image harbor.example.local/spitfire/spitfire:{{version}}

With Helm: helm upgrade spitfire ./spitfire-{{version}}.tgz -n spitfire --reuse-values --set image.repository=harbor.example.local/spitfire/spitfire --set image.tag={{version}}. A new installation also needs postgres:16 in the mirror: push it the same way and give --postgres-image harbor.example.local/spitfire/postgres:16 (Helm: --set postgres.image=…); the image given is kept on later runs.

c) Offline bundle (--offline)

Every release publishes one bundle per platform: spitfire-offline-<version>-linux-amd64.tar.gz and …-linux-arm64.tar.gz (with their .sha256 at usespitfire.com/install). It holds the Spitfire and postgres:16 images (docker save), the installer bundle, the Helm chart, the signed releases.json and a signed SHA256SUMS. A first installation works from the bundle alone.

  1. Download the bundle on a machine with internet access and copy it to the controller server the way your organisation moves files in (approved USB media, a file transfer server, scp through a jump host). Keep three times its size free on the disk.

  2. For a first installation, unpack the bundle's installer files (an update skips this):

    bash
    mkdir -p ~/spitfire && tar -xzf spitfire-offline-{{version}}-linux-amd64.tar.gz -C ~/spitfire spitfire-{{version}}.tar.gz
    tar -xzf ~/spitfire/spitfire-{{version}}.tar.gz -C ~/spitfire && rm ~/spitfire/spitfire-{{version}}.tar.gz
  3. Install (the Docker or the Kubernetes line):

    bash
    ~/spitfire/install.sh docker --offline spitfire-offline-{{version}}-linux-amd64.tar.gz
    ~/spitfire/install.sh kubernetes --offline spitfire-offline-{{version}}-linux-amd64.tar.gz --registry harbor.example.local/spitfire
  4. Success check: the /healthz command above prints 204. If the signature does not hold, the script stops without touching anything.

The script never reaches the internet. It checks the SHA256SUMS signature with the release key written into it (OpenSSL 3, Ed25519) and every file against the list, loads the image with docker load, unpacks the installer files (deploy/*/.env is kept) and installs with --image. It loads the Postgres image only when it is not on the machine; when the version changes it dumps the database first, and --rollback works as usual. On Kubernetes the cluster cannot see local images, so --registry is required: the images are pushed as REG/spitfire:<version> and REG/postgres:16 (docker login first). With Helm, the chart is the bundle's helm/spitfire-<version>.tgz: --set image.repository=REG/spitfire --set image.tag=<version> --set postgres.image=REG/postgres:16. Runner servers: install.sh runner --offline <bundle>.

Without OpenSSL 3 on the server (e.g. RHEL 8) the script refuses the bundle; then give the .sha256 value from usespitfire.com/install with --offline-sha256 <value>. You can also verify the bundle by hand on a machine with OpenSSL 3:

bash
mkdir bundle && tar -xzf spitfire-offline-{{version}}-linux-amd64.tar.gz -C bundle && cd bundle
{ printf 'spitfire-offline-v1\n'; cat SHA256SUMS; } > signed.txt
openssl base64 -d -A -in SHA256SUMS.sig -out SHA256SUMS.bin
openssl pkeyutl -verify -rawin -pubin -inkey spitfire-release.pub.pem -in signed.txt -sigfile SHA256SUMS.bin
sha256sum -c SHA256SUMS

spitfire-release.pub.pem is Spitfire's release key (also in the bundle's README.txt):

text
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAE7pSZPzN3PgBdeCyFwRvfMP2EWlKt87gSgRu84qu5Io=
-----END PUBLIC KEY-----
Note

If the installed install.sh does not know --offline, unpack the bundle's spitfire-<version>.tar.gz into ~/spitfire first (step 2), then run the command. Windows (install.ps1) has no --offline: verify the bundle on a Linux machine, load it with docker load -i spitfire-image.tar and docker load -i postgres-image.tar, then run & "$env:USERPROFILE\spitfire\install.ps1" docker -Image usespitfire/spitfire:<version>.

Serving the release list on your network

To keep new releases showing in the UI, put the bundle's releases.json and releases.json.sig on an internal web server and give the controller SPITFIRE_UPDATE_FEED=https://intranet.example.local/spitfire/releases.json (the .sig must be at the same address). The list is signed; if it is altered on the mirror it is refused. Replace both files with every new bundle. Without a mirror, set SPITFIRE_UPDATE_CHECK=off and follow what's new on usespitfire.com/whats-new.

Installing behind a proxy

Run the install command with HTTPS_PROXY (and if needed HTTP_PROXY, NO_PROXY) set:

bash
export HTTPS_PROXY=http://proxy.example.com:3128
export NO_PROXY=.example.local
curl -fsSL https://spitfire.sh | bash -s -- docker

The script passes these to the controller (Docker: deploy/docker/.env; Kubernetes: the spitfire-proxy Secret), which then makes the update check, webhooks, SSO calls and, when on, health data sharing through the proxy. The installation's own names (localhost, postgres, controller…) go into NO_PROXY automatically; add an internal SSO provider and webhook targets yourself. If a later run gives no proxy, the earlier setting is kept.

Common problems

Symptom: The Version and updates card shows Check failed. Cause: The controller cannot reach spitfire.tr. Fix: Behind a proxy, run the install command again with HTTPS_PROXY set; without internet access set SPITFIRE_UPDATE_CHECK=off.