SpitfireLoad Testing & Observabilityv1.2.0 · What's new

Version: 1.2.0This documentation is for Spitfire 1.2.0.

High availability and disaster recovery

This page covers what Spitfire keeps running today and what it does not, the scheduled encrypted backup, the restore drill, the downtime of an update and what happens while the controller is down. The backup commands themselves are in Installation → Backups.

What it is for

So you can write Spitfire into your organisation's continuity plan (RPO/RTO) correctly: which part is a single copy, which one is replaced by a new one, how much data a restore loses and how long it takes.

When to use it

  • For the architecture and continuity sign-off before installing.
  • When you set up the scheduled backup.
  • When you run a restore drill, at least once a year.
  • When you plan a maintenance window for an update.

Current state

Part State
Controller (UI, API, scheduler, synthetic monitoring, runner connections) A single copy. On Kubernetes replicas: 1 with strategy: Recreate: the live state of runs is in the controller's memory, two copies never run at once. If the pod or the server goes down, Kubernetes/Docker restarts it; meanwhile the UI and the API are down.
Runners Stateless and replaceable. Results stream to the controller, nothing stays on the runner. If a runner goes down, start a new one; it connects with the same token. For a runner lost during a run, the test's When a runner is lost option applies (Test editor).
Postgres and ClickHouse that come with the installation A single copy, on the installation's own disk.
Your own Postgres or ClickHouse High availability (replicas, failover, backups) belongs to your database team. Spitfire uses one connection address; it must stay the same after a failover.
Backup install.sh --backup takes an encrypted full backup in one file (Postgres + the installation's keys). You set up the schedule (below); the product has no backup scheduler of its own.
APM store (ClickHouse: traces, logs) Not in the backup. This data is limited by its retention (1–30 days) anyway; after a restore it fills with new data.

An active/passive second controller is planned; it does not exist today.

While the controller is down

What What happens
Running runs A runner that cannot reach the controller for 15 s stops its load by itself ("controller unreachable for more than 15s"): the target never gets unsupervised load. When the controller is back, these runs are Interrupted, reason "controller restarted". A controller that is stopped cleanly (update, docker compose stop) first stops the active runs and saves their results (up to 30 s).
Idle runners They retry at growing intervals (at most 15 s) and connect by themselves once the controller is up.
Scheduled runs They do not start. If the controller was down for less than 10 minutes, the missed run starts at startup; if longer, that run is recorded as missed, is not caught up, and its notification goes out.
Synthetic monitoring Pauses; continues at the next interval after startup. The checks in between are not made later.
UI, API, CI pipelines Down; CI steps that call the API fail.
OTLP receiver (APM) Your services' OpenTelemetry SDKs retry for a short while; data beyond their buffers is lost.

Step by step

1. Scheduled encrypted backup (Docker)

  1. Create the backup folder:

    bash
    mkdir -p -m 700 ~/spitfire-backups

  2. Write the backup passphrase (at least 12 characters) to a file only you can read:

    bash
    install -m 600 /dev/null ~/.spitfire-backup-pass && printf '%s' 'YOUR-PASSPHRASE' > ~/.spitfire-backup-pass

  3. Take one backup by hand and see that it works:

    bash
    SPITFIRE_BACKUP_PASSPHRASE="$(cat ~/.spitfire-backup-pass)" ~/spitfire/install.sh docker --backup ~/spitfire-backups/spitfire-$(date +%F).sfbak

  4. For a backup every night at 03:00 and deleting backups older than 14 days, open crontab -e and add these two lines:

    text
    0 3 * * * SPITFIRE_BACKUP_PASSPHRASE="$(cat $HOME/.spitfire-backup-pass)" $HOME/spitfire/install.sh docker --backup $HOME/spitfire-backups/spitfire-$(date +\%F).sfbak >> $HOME/spitfire-backups/backup.log 2>&1
30 3 * * * find $HOME/spitfire-backups -name '*.sfbak' -mtime +14 -delete

  5. Copy the ~/spitfire-backups folder to another server with your own backup system. Keep the passphrase apart from the backups (in a password vault): a backup does not open without it.

Success check: the next morning today's file is there and the log has no error:

bash
ls -lh ~/spitfire-backups/ && tail -n 5 ~/spitfire-backups/backup.log

The RPO (the most data you can lose) is your backup interval: with a nightly backup, at most the last 24 hours.

2. Scheduled backup (Kubernetes and Helm)

With the Postgres that comes with the installation (the default), the same command runs from an admin machine with kubectl access. The installation package unpacked under ~/spitfire is enough (usespitfire.com/install).

  1. Take a backup by hand:

    bash
    SPITFIRE_BACKUP_PASSPHRASE="$(cat ~/.spitfire-backup-pass)" ~/spitfire/install.sh kubernetes --namespace spitfire --backup ~/spitfire-backups/spitfire-$(date +%F).sfbak

  2. Set up steps 4 and 5 of the Docker part with kubernetes --namespace spitfire in place of docker.

If you use your own database (externalDatabase.existingSecret), your own backup system backs up the database; take only the keys from Spitfire:

bash
kubectl -n spitfire get secret spitfire-secrets -o yaml > ~/spitfire-backups/spitfire-secrets.yaml

Success check: ls -lh ~/spitfire-backups/ shows today's file.

3. Restore drill

Run the drill on a separate trial machine, not on the running installation. The time it takes is your RTO; write that into your continuity plan.

  1. Note the start time:

    bash
    date

  2. Install Spitfire on the trial machine (the same release or a newer one):

    bash
    curl -fsSL https://spitfire.sh | bash -s -- docker

  3. Copy the newest .sfbak file to this machine (e.g. with scp) and restore it:

    bash
    SPITFIRE_BACKUP_PASSPHRASE="$(cat ~/.spitfire-backup-pass)" ~/spitfire/install.sh docker --restore ~/spitfire-backups/spitfire-2026-11-04.sfbak

    On Kubernetes: ~/spitfire/install.sh kubernetes --namespace spitfire --restore ….

  4. See that the controller is ready:

    bash
    curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8470/readyz

  5. Sign in to the UI with a user from the backup; in Runs see the runs up to the backup's day, in Connections see that a connection's Test button works (the secrets can be read).

  6. Note the end time (date). The difference between the two is your RTO.

Success check: step 4 prints 204, and the sign-in in step 5 shows the backup's data.

In a real disaster use the new server in place of the trial machine and point the name (DNS) the runners connect to at the new server. The controller's TLS identity comes back with the backup's key, so the runners' CA pin does not change; the runners connect by themselves.

4. Updating and rolling back

On Docker and install.sh kubernetes installations, an update that changes the version first dumps the database (~/spitfire/backups/*.dump, the last 5 are kept).

  1. Update:

    bash
    curl -fsSL https://spitfire.sh | bash -s -- docker

  2. If something goes wrong, undo the last update (the database returns to its state in that dump and the version to the previous one; changes made after it are lost):

    bash
    ~/spitfire/install.sh docker --rollback

    Kubernetes: ~/spitfire/install.sh kubernetes --rollback.

A Helm installation has no automatic dump. Take the backup from part 2 before upgrading, then upgrade:

bash
helm repo update spitfire && helm upgrade spitfire spitfire/spitfire -n spitfire --reuse-values

To roll back, first return to the old release, then restore the backup you took before the upgrade (the new release may have changed the database schema):

bash
helm rollback spitfire -n spitfire
bash
SPITFIRE_BACKUP_PASSPHRASE="$(cat ~/.spitfire-backup-pass)" ~/spitfire/install.sh kubernetes --namespace spitfire --restore ~/spitfire-backups/spitfire-2026-11-04.sfbak

Downtime: on an update the controller stops once and starts with the new release (a single copy, Recreate). The downtime is the time to stop the active runs (up to 30 s), to start and to migrate the schema, and is in the order of minutes. The dump is taken while the controller runs; with a large database it makes the whole update longer. To measure the downtime on your own installation, start this in a second terminal before updating; lines other than 204 are downtime:

bash
while :; do printf '%s %s\n' "$(date +%T)" "$(curl -s -o /dev/null -m 2 -w '%{http_code}' http://localhost:8470/readyz)"; sleep 1; done

Success check: after the update the lines print 204 again and the Version and updates card on the License page shows the new release.

Warning

Update while no run is going and between scheduled runs: runs going during the update are stopped.

Capacity measurement

The largest verified test (most VUs, sustained requests/s, duration, runner count, CPU/memory per runner, result loss and error rate) will be added once it is measured.

Measure Value
Most real VUs will be added once measured
Sustained requests/s will be added once measured
Duration will be added once measured
Runner count, CPU/memory per runner will be added once measured
Result loss (requests sent vs stored) will be added once measured
Error rate will be added once measured

The measurement is made with Spitfire's own benchmark tool against a target that does almost no work per request: the requests the target counted are compared with those the controller stored, and the runners' CPU and memory are sampled during the run. For planning on your own hardware, use the method in Runners → Capacity.

Common problems

Symptom Cause Fix
No nightly backup, the log says docker: command not found cron's PATH is short Add the line PATH=/usr/local/bin:/usr/bin:/bin at the top of the crontab.
The log says "A passphrase is needed." SPITFIRE_BACKUP_PASSPHRASE came empty Check that ~/.spitfire-backup-pass exists and is not empty.
The restore says "The backup is from a newer release" The Spitfire on the trial machine is older than the one that took the backup Update the trial machine first, then restore.
After a restore, a connection's Test button gives a password/secret error A plain dump was restored instead of the backup, the keys are old Restore with the .sfbak, or set SPITFIRE_SECRET_KEY to the value in your backup.
After the controller comes back, a run is Interrupted, reason "controller restarted" The controller stopped during the run Expected; start the run again.