SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Server monitoring — Linux, Windows, macOS

The Spitfire agent is installed on the server, sends its metrics to a Tower, and the server shows on the Servers In Spitfire: /servers page. Take the Collector token (sfcol_...) from Towers In Spitfire: /towers → Collector.

Linux (systemd)

Quick install

  1. Copy the command, run it in a terminal, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/host.sh" -o /tmp/sf-host.sh && bash /tmp/sf-host.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Download the installer.
bash
curl -fsSL "http://<tower>:4318/v1/agent/dist/install-collector.sh" -o /tmp/sf-install-collector.sh
  1. Install the agent.
bash
sudo bash /tmp/sf-install-collector.sh --tower "http://<tower>:4318" --token "sfcol_..."
  1. Check the report.
bash
curl -fsS -H "Authorization: Bearer sfcol_..." "http://<tower>:4318/v1/agent/host?name=$(hostname -s)" | grep -Eq '"reported": *true' && echo "OK: $(hostname -s) reports to Spitfire"

macOS (launchd)

Quick install

  1. Copy the command, run it in a terminal, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/host.sh" -o /tmp/sf-host.sh && bash /tmp/sf-host.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Download the installer.
bash
curl -fsSL "http://<tower>:4318/v1/agent/dist/install-collector.sh" -o /tmp/sf-install-collector.sh
  1. Install the agent.
bash
sudo bash /tmp/sf-install-collector.sh --tower "http://<tower>:4318" --token "sfcol_..."
  1. Check the report.
bash
curl -fsS -H "Authorization: Bearer sfcol_..." "http://<tower>:4318/v1/agent/host?name=$(hostname -s)" | grep -Eq '"reported": *true' && echo "OK: $(hostname -s) reports to Spitfire"

Windows (service)

Quick install

  1. Open PowerShell as administrator, run the command.
powershell
Invoke-WebRequest -UseBasicParsing "https://<spitfire>/api/v1/setup/ds/host.ps1" -OutFile "$env:TEMP\sf-host.ps1"; powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\sf-host.ps1"
  1. When you see OK: ..., you are done.

Manual install

  1. Download the installer.
powershell
Invoke-WebRequest -UseBasicParsing "http://<tower>:4318/v1/agent/dist/install-collector.ps1" -OutFile "$env:TEMP\sf-install-collector.ps1"
  1. Install the agent.
powershell
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\sf-install-collector.ps1" -Tower "http://<tower>:4318" -Token "sfcol_..."
  1. Check the report.
powershell
if ((Invoke-RestMethod -Headers @{ Authorization = "Bearer sfcol_..." } "http://<tower>:4318/v1/agent/host?name=$env:COMPUTERNAME").data.reported) { "OK: $env:COMPUTERNAME reports to Spitfire" }

Uninstall

  1. Linux and macOS.
bash
sudo bash /tmp/sf-install-collector.sh --uninstall && echo "OK: agent removed"
  1. Windows (administrator PowerShell).
powershell
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\sf-install-collector.ps1" -Uninstall; "OK: agent removed"

What each system sends

Metric Linux Windows macOS
CPU (total, per core, state) yes yes yes
Load (1/5/15 min) yes approximate (from the processor queue) yes
Memory, swap / page file yes yes yes
File systems, inodes yes yes (no inodes) yes
Disk read/write, latency yes yes yes
Network traffic, errors, drops yes yes yes
Top processes yes yes yes
Process states (running, sleeping…) yes no count only
systemd units yes no no
Windows services and counters no yes no
Note

Each server counts as one server in the license. The agent dials out to the Tower only (4318); it opens no port on the server. IIS metrics come from the IIS monitored system; its setup adds a second config to the agent (-ExtraConfig).

Files and service

Linux macOS Windows
Service spitfire-collector (systemd) tr.spitfire.collector (launchd) spitfire-collector
Program /usr/local/bin/spitfire-collector /usr/local/bin/spitfire-collector C:\Program Files\Spitfire\spitfire-collector.exe
Config /etc/spitfire/collector.yaml /usr/local/etc/spitfire/collector.yaml C:\ProgramData\Spitfire\collector.yaml
Log journalctl -u spitfire-collector -n 50 tail -n 50 /var/log/spitfire-collector.log Get-WinEvent -FilterHashtable @{ LogName = 'Application'; ProviderName = 'spitfire-collector' } -MaxEvents 20

Findings

Finding When
No report no metrics for 3 collection intervals ("Unreachable" after 15 min)
Disk full above 85% a warning, above 95% critical (read-only volumes left out)
CPU / memory high above 90% for 5 minutes
Load high the 5-minute load above twice the cores
Swap growing swap over 10% used and grown by more than 5% in 15 minutes
Network errors 10 or more errors or drops on one interface in 15 minutes
Service stopped a Windows service set to start automatically is not running
Unit failed a systemd unit is failed

Agents

The Agents In Spitfire: /agents page lists every agent and Tower in one table: version, status, last report, uptime, points per second, failed and queued points. Only install admins open it; every action goes into the audit log. There is no remote shell: an agent accepts only restart, update and log settings commands.

Change the log level

  1. On the Agents In Spitfire: /agents page, click the agent's row.
  2. In the Log settings card, pick debug, info, warn or error under Log level.
  3. Press Save. debug goes back to error by itself after the fleet's debug time (60 minutes by default); the card shows the time left.
  4. To follow the fleet again, pick Use the fleet default and save.

Live log

  1. On the agent page, press Start in the Live log card.
  2. The agent picks the request up within 15 seconds and the lines start to flow.
  3. Narrow the lines by level and text; keep them with Pause, Copy and Download the last N lines.

Restart and update

  1. On the agent page, press Restart or Update now and confirm.
  2. For several agents, tick their rows in the list and press Restart or Update now above it.
  3. The outcome shows in the Update history table; if the new version does not start, the agent goes back to the old one (rollback).

Auto-update

  1. On the Agents In Spitfire: /agents page, open the Fleet defaults card.
  2. Switch Auto-update on or off; optionally enter an update window (the agent's local time).
  3. Press Save. A single agent can differ in the Log settings card on its page.

Upgrade an older agent

A server marked "unmanaged" in the list runs an older agent; run the installer again, settings are kept.

  1. On Linux or macOS:
bash
curl -fsSL "http://<tower>:4318/v1/agent/dist/install-collector.sh" -o /tmp/sf-install-collector.sh && sudo bash /tmp/sf-install-collector.sh --tower "http://<tower>:4318" --token "sfcol_..."
  1. On Windows, open PowerShell as administrator:
powershell
Invoke-WebRequest -UseBasicParsing "http://<tower>:4318/v1/agent/dist/install-collector.ps1" -OutFile "$env:TEMP\sf-install-collector.ps1"; powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\sf-install-collector.ps1" -Tower "http://<tower>:4318" -Token "sfcol_..."
  1. Within a minute the row's type becomes "agent" and the actions open up.