Spitfire Collector: applications and server metrics
The Spitfire Collector is installed once per server and does two jobs as one service: it sends the server's CPU, memory, disk, network and process metrics to the Servers In Spitfire: /servers page (Server metrics), and it finds the Java, .NET, .NET Framework (Windows only), Node.js, Python and PHP applications on the server and switches their tracing on. You need no separate Server Collector; if the old one is there, the install takes it over and removes it (migration).
It opens no inbound port: it reports to Spitfire every minute and receives your requests in the answer. When Spitfire is unreachable it drops the data without writing it to disk and never holds the application up. Uninstalling it reverts every setting it added to applications. A workspace admin installs it.
Quick install
- On APM → Collectors → App Collectors press Install Collector at the top right. The window opens with your browser's operating system selected and a new receiver key (
sfotlp_). If the server runs another OS, press its button at the top. - Copy the command shown and run it on the server (Linux, Windows, macOS). On Windows and macOS you can download the installer and paste the Connection code instead.
- Leave the window open. Within seconds the panel at the bottom says The Collector reported and lists the applications it found. When the first span of a traced application arrives, the panel links to the service in APM.
The defaults: every runtime, environment prod, sampling 10%, auto-trace on, no restart.
The key is shown only in this window, inside the command and the connection code. Do not close the window before the install is done; a new window makes a new key.
The most common mistake: opening Spitfire in the browser as localhost and copying the command from there. The window warns about it; type the address the server reaches Spitfire at under Advanced settings.
Advanced settings
Before copying you can change these under Advanced settings; the command updates as you type:
- Auto-trace and Restart the applications now (the restart consent).
- Environment, Sampling (
0.1= 10% of requests), Runtimes to watch. - Spitfire address, an existing
sfotlp_key, Tower address andsfcol_key, HTTPS proxy, the full path of the CA file on the server. - Scope: applications to leave out, service names, modules, the approval list (Scope).
Auto-trace
Auto-trace (--auto in the command, "autoTrace": true in collector.json) switches tracing on, once, for every application it finds that runs under systemd, Windows service, IIS, launchd or php-fpm, and gives it a service.name from its name.
- Tracing starts at the application's next restart (a deploy, a reboot). Nothing is restarted without consent. To start at once, tick Restart the applications now (
--restart,"autoRestart": true) or press Restart now in the panel. - Docker, Kubernetes and applications started by hand are never switched on automatically; the Collector's page gives copy-paste steps.
- An application you switch off in Spitfire stays off. With an expired (read-only) license auto-trace makes no change.
The live panel
The panel at the bottom of the install window has a switch per row that turns the application on or off. Trace all switches on the ones that are off. Waiting for restart rows show the command to run on the server and a Restart now button.
Linux
Needs: systemd, root or sudo, curl (or wget), amd64 or arm64.
- Open a terminal on the server as a user with
sudo. - Paste the command from the install window and press Enter. It looks like this:
curl -fsSL https://spitfire.example.com/api/v1/runner-dist/install-apm-collector.sh | \
sudo bash -s -- --from https://spitfire.example.com \
--token sfotlp_... \
--auto- The command checks the prerequisites, installs the Collector, tests the connection and starts the service.
- Success check: this command should print the
OK: connected to ...line and below itServer metrics: on, as "SERVER-NAME", to Spitfire (Servers).(with a Tower:to the Tower ... (Servers)).
sudo /usr/local/bin/spitfire-apm-collector check- The service's status (it should say
active (running)):
sudo systemctl status spitfire-apm-collector --no-pager- Within about 1 minute the server shows on the Servers
In Spitfire: /serverspage.
Files: settings /etc/spitfire-apm/collector.json, server-metrics part /opt/spitfire-apm/host/, OpenTelemetry kits /opt/spitfire-apm/kits, state /var/lib/spitfire-apm, log /var/log/spitfire-apm-collector.log.
Windows
Needs: 64-bit Windows Server 2016+ or Windows 10+, an administrator account, PowerShell 5.1+.
With the installer
- In the install window pick the Installer (msi/exe) tab.
- Download
spitfire-apm-collector-setup.exe(orspitfire-apm-collector.msi) and copy it to the server if needed. - Press the copy button next to Connection code (it starts with
SFAPM1.). - Double-click the exe and approve the admin prompt. The connection code page is filled from the clipboard; if not, paste the code. Finish with Next.
- For the success check open PowerShell with Run as administrator:
& "$env:ProgramFiles\Spitfire\APM\spitfire-apm-collector.exe" check -config "$env:ProgramData\Spitfire\APM\collector.json"; Get-Service SpitfireAPMCollectorYou should see OK: connected to ..., below it Server metrics: on, as "SERVER-NAME", to Spitfire (Servers)., and the service Running. Within about 1 minute the server shows on the Servers In Spitfire: /servers page.
Silent install with the msi for software distribution (for the exe: /S /CODE=SFAPM1....):
msiexec /i spitfire-apm-collector.msi /qn CODE=SFAPM1....With a command
- Open PowerShell with Run as administrator.
- Paste the command from the install window and press Enter. It looks like this:
& ([scriptblock]::Create((Invoke-RestMethod -UseBasicParsing 'https://spitfire.example.com/api/v1/runner-dist/install-apm-collector.ps1'))) `
-From 'https://spitfire.example.com' `
-Token 'sfotlp_...' `
-Auto- The success check is the
checkcommand above.
The service is installed as SpitfireAPMCollector and shows as Spitfire Collector in Services. Settings and log are in C:\ProgramData\Spitfire\APM\ (collector.json, collector.log).
macOS
Needs: macOS 12+, an administrator password, Intel or Apple Silicon.
With the installer
- In the install window pick the Installer (dmg) tab and download
spitfire-apm-collector.dmg. - Press the copy button next to Connection code.
- Open the dmg, double-click Spitfire Collector and press Install. The code is read from the clipboard; if not, it asks, paste it. Finish with your admin password.
- Success check (Terminal): you should see
OK: connected to ...andServer metrics: on, as "SERVER-NAME", to Spitfire (Servers).
sudo /usr/local/bin/spitfire-apm-collector check- Within about 1 minute the server shows on the Servers
In Spitfire: /serverspage.
With a command
The same as on Linux; the install window gives the right one when macOS is picked. The success check is the check command above; for the service, this should print state = running:
sudo launchctl print system/tr.spitfire.apm-collector | grep stateFiles: settings /Library/Spitfire/APM/collector.json, server-metrics part /Library/Spitfire/APM/host/, log /Library/Logs/spitfire-apm-collector.log.
Server metrics
The same service collects CPU, memory, disk, file system, network, service and busiest-process metrics with a separate part. If the part stops, the service starts it again; APM is not affected.
- Without a Tower the metrics go straight to Spitfire with the same
sfotlp_key. This needs Collect and keep metrics on under Towers & metricsIn Spitfire: /towers(Turning metrics on). Your license's server limit applies. - With a Tower add
--tower http://tower:4318 --tower-token sfcol_...to the command (Windows:-Towerand-TowerToken).
Settings are added to the command; a setting you leave out is kept:
| Linux / macOS | Windows | Meaning |
|---|---|---|
--host-name NAME |
-HostName NAME |
The server's name on Servers (default: the computer's name) |
--host-interval SEC |
-HostInterval SEC |
How often it measures, 10–300 seconds (default 30) |
--host-processes N |
-HostProcesses N |
How many of the busiest processes it keeps: 10, 25 or 50 |
--host-binary FILE |
-HostBinary FILE |
Server without internet: install the part from this file instead of downloading it |
--modules host,processes,apm,docker,iis |
-Modules ... |
The modules that run; host and processes are the server metrics |
For APM only give --modules apm (Windows: -Modules apm).
Check: the second line of check gives the state of the metrics.
| Line | Meaning |
|---|---|
Server metrics: on, as "NAME", to Spitfire (Servers). |
Running; the server is on Servers within about 1 minute |
Server metrics: off (modules). |
You switched them off in the modules |
Server metrics: not installed here: ... |
The part is missing in this install: run the install command again |
Server metrics: cannot start (see the log). |
The part does not start: the Collector log says why |
Moving from the old Collectors
Run the install command (or, on Windows, the setup file) as it is. If the old Server Collector is there (the spitfire-collector service, on Linux /etc/spitfire/collector.env), the install takes over its name, Tower, interval and process count and removes the old service once the new one connects. The server is not counted twice on Servers. An APM Collector already installed is upgraded in place with its settings kept. A setting you give in the command (such as --host-name) wins over the old one.
Check (Linux):
systemctl is-active spitfire-collector >/dev/null 2>&1 && echo "The old service is still there" || echo "OK: old Server Collector removed"; sudo /usr/local/bin/spitfire-apm-collector checkWindows (admin PowerShell):
if (Get-Service spitfire-collector -ErrorAction SilentlyContinue) { 'The old service is still there' } else { 'OK: old Server Collector removed' }; Get-Service SpitfireAPMCollectorIf the old service is still there, the new one has not connected yet: look at the NOT CONNECTED line in the check output.
Switching tracing on
Auto-trace already does this for systemd, launchd, Windows service, IIS and PHP-FPM applications. To do it by hand:
- On App Collectors
In Spitfire: /apm/app-collectorsclick the server. - In Discovered applications press Switch tracing on in the application's row.
- service.name comes with a suggested name; change it if needed. Empty environment and sampling take the install values.
- To restart at once tick Restart ... now. Unticked, the state becomes Waiting for restart and the command you need to run appears in the row.
- Press Switch tracing on. The request reaches the Collector within a minute.
- Check: once the application restarts the state is Tracing on and traces arrive on APM → Services → Services
In Spitfire: /apm/services.
To switch off: Change › Switch tracing off in the same row; the application returns to its old state at its next restart. Docker, Kubernetes and manually started processes are listed but the Collector does not touch them; Add a service in the row shows the steps. An application marked Monitored by another tool already has a monitoring tool, and the Collector does not overwrite it.
One service name for every app
Example: an IIS server runs the sites abc, abc-payment and abc-order, and you want them in APM as one service, abc. The Collector sends every application under one service.name and keeps each one's own name in the resource attribute spitfire.app.
On App Collectors
In Spitfire: /apm/app-collectorsclick the server.In the Send every app under one service name card type the name, for example:
abcPress Apply. The card shows Every application sends as abc.
The new name takes effect when the application restarts. With restart consent given at install, the Collector restarts the applications itself; without it, you restart them with the command in each row.
Check: in the Discovered applications table each application shows
→ service.name abcandspitfire.app <its own name>. After the restart, APM → Services → ServicesIn Spitfire: /apm/serviceslistsabc.
Worth knowing:
- A name typed for one application in its own dialog (Change) wins.
- Data already received stays under the old names; Spitfire does not rename it.
- The setting affects this Collector only. To go back, empty the field and press Apply.
- To tell the applications apart in APM, use the Applications (spitfire.app) card on the service page, the spitfire.app list on Top requests, or the
spitfire.app:abc-paymentfilter in trace search.
Naming rules
Use rules when only some applications belong together. Example: the three abc sites become abc, shop-cart-v2 becomes shop-cart, and intranet keeps its own name.
On App Collectors
In Spitfire: /apm/app-collectorsclick the server.In the Send every app under one service name card press Naming rules (0).
Press Add a rule and fill in the row (a glob:
*is any run of characters, it matches the whole name in any case):Pattern:
abc*service.name:
abc
Press Add a rule again, tick regex and fill in the second row (Go RE2;
$1is the first group):Pattern:
^shop-(\w+)-v\d+$service.name:
shop-$1
Look at the Preview table:
abc,abc-paymentandabc-ordershowabcwith Rule 1,shop-cart-v2showsshop-cartwith Rule 2,intranetshows Its own name. A rule with a mistake is marked in red and Apply stays off.Press Apply. The card shows 2 naming rules.
Check: in the Discovered applications table
abc-paymentshows→ service.name abc,shop-cart-v2shows→ service.name shop-cart.
Rules are tried from the top and the first match wins: put the narrow ones first. An application no rule matches uses the card's one service name or its own name. Write ${1}x instead of $1x when letters follow the group. A rule whose result is not a valid service.name (letters, digits, . _ / -, at most 100 characters) is skipped. At most 50 rules per Collector.
Scope: exclusions, modules, the approval list
For each Collector you can choose which applications are never traced, which modules run, and whether it shows you the list for approval before auto-trace. You set these at install time (Advanced settings, the command, collector.json) or later on the Collector's page. Changes are written to the Audit log In Spitfire: /audit.
Exclusions
An application that matches a pattern is never traced; if it is traced, tracing goes at its next restart.
On the App Collectors
In Spitfire: /apm/app-collectorspage click the server.In the Scope: exclusions and modules card press Add pattern.
Type the pattern (glob:
*any run,?one character; any case):*test*In the Preview the applications left out show
excluded (rule 1).Press Save. The card's header says 1 exclude patterns.
Check: in the Discovered applications table rows such as
orders-testare greyed and sayexcluded (rule 1).
A pattern is tried on the application's own name, its name in Spitfire and its service unit; one match is enough. With regex ticked it is Go RE2 (write ^...$ for the whole name). At most 50 patterns per Collector.
Modules
In the same card, a module you untick under Modules does not run:
- With Host metrics and Processes both off, the server is no longer updated on Servers. With Processes off, PIDs, user and command line are not sent; tracing can still be switched on.
- With APM off no application is traced and the traced ones are taken back. With IIS off the IIS application pools' tracing is taken back too.
- With Docker off containers are not listed.
Switching a module back on does not switch applications on; use Switch tracing on or Trace all.
The approval list
With Show me the list first, I will approve it ticked, auto-trace switches nothing on until you approve; applications found later wait for approval too. It is off by default. We suggest turning it on for the first install, especially on crowded servers.
At install time tick the box under Advanced settings (the command gets
--review), or later tick it in the card on the Collector's page and press Save.Once the Collector reports, the panel shows the List awaiting approval.
Untick the ones you do not want traced. Correct the service.name where needed; the same name on several rows makes one service. Example: in the
ordersandcartrows type:shopPress Approve. The ticked applications are switched on; the unticked ones stay excluded for good.
Check: in the Discovered applications table
ordersshows→ service.name shopandspitfire.app orders.
The names you type are kept as naming rules.
Scope in a silent install
For software distribution the scope settings are parameters; a parameter given with a connection code replaces the code's value.
| Setting | Linux / macOS command | PowerShell | msi | exe |
|---|---|---|---|---|
| Exclusions | --exclude |
-Exclude |
EXCLUDE= |
/EXCLUDE= |
| Service names | --names |
-Names |
NAMES= |
/NAMES= |
| Naming rules | --rules |
-Rules |
RULES= |
/RULES= |
| Modules to run | --modules |
-Modules |
MODULES= |
/MODULES= |
| Approval list | --review |
-Review 'true' |
REVIEW=true |
/REVIEW=true |
- Values are comma separated. Start a regex pattern with
re:(re:^tmp-). A regex with a comma cannot be a parameter; set it on the Collector's page or incollector.json. --namestakesapplication=service.namepairs,--rulespattern=service.name(abc*=abc).--moduleslists the modules that run:host,processes,apm,docker,iisorall.noneempties a list (--exclude none).
Linux or macOS:
curl -fsSL https://spitfire.example.com/api/v1/runner-dist/install-apm-collector.sh | \
sudo bash -s -- --from https://spitfire.example.com \
--token sfotlp_... \
--auto \
--exclude '*test*' \
--names 'orders=shop,cart=shop' \
--modules host,apm,docker,iis \
--reviewWindows PowerShell (Run as administrator):
& ([scriptblock]::Create((Invoke-RestMethod -UseBasicParsing 'https://spitfire.example.com/api/v1/runner-dist/install-apm-collector.ps1'))) `
-From 'https://spitfire.example.com' `
-Token 'sfotlp_...' `
-Auto `
-Exclude '*test*' `
-Names 'orders=shop,cart=shop' `
-Modules 'host,apm,docker,iis' `
-Review 'true'Silent install with the msi:
msiexec /i spitfire-apm-collector.msi /qn CODE=SFAPM1.... EXCLUDE="*test*" NAMES="orders=shop,cart=shop" MODULES="host,apm,docker,iis" REVIEW=trueSilent install with the exe (in the folder that holds the exe):
.\spitfire-apm-collector-setup.exe /S /CODE=SFAPM1.... /EXCLUDE=*test* /NAMES=orders=shop,cart=shop /MODULES=host,apm,docker,iis /REVIEW=trueIn collector.json (restart the service after changing it):
{
"exclude": [{ "pattern": "*test*" }, { "pattern": "^tmp-(a|b)$", "regex": true }],
"names": { "orders": "shop", "cart": "shop" },
"rules": [{ "pattern": "abc*", "service": "abc" }],
"modules": { "processes": false },
"review": true
}Spitfire applies the file's settings only when they change; what you change later on the Collector's page stays until the file changes again. Check: the card on the Collector's page shows the from collector.json label.
Managing the Collector
Open a server on the App Collectors In Spitfire: /apm/app-collectors page. The Collector card shows the state from the last report (Running, Stopped, No report), the CPU and memory the Collector itself uses, the log level and the last error in the log. Changes reach the Collector with its next report (while a change waits, the Collector reports every 15 seconds). A workspace admin makes the changes; each one is recorded in the Audit log In Spitfire: /audit.
- Live log: press Start; the log starts flowing within a minute. The lines are kept in memory only (the last 1000) and never written to disk; you can download them as
.log. - Log level: pick
debug, choose when it returns toinfo(15 minutes, 1 hour, 4 hours, 24 hours, or never) and press Apply. The Collector falls back toinfoat that time even if it cannot reach Spitfire. - Stop: the Collector keeps reporting but switches nothing on or off, and Spitfire does not keep the traces, metrics and logs of the applications it switched on. The applications are not restarted. Start undoes it.
An older Collector shows update on its card. Run the install command on that server again; the settings are kept.
Uninstalling
Linux and macOS:
curl -fsSL https://spitfire.example.com/api/v1/runner-dist/install-apm-collector.sh | sudo bash -s -- --uninstallWindows (admin PowerShell):
& ([scriptblock]::Create((Invoke-RestMethod -UseBasicParsing 'https://spitfire.example.com/api/v1/runner-dist/install-apm-collector.ps1'))) -UninstallThe command first reverts every setting added to applications, then deletes the service, the server-metrics part, the settings, the kits and the log; anything left from the old Server Collector goes too. To have applications return at once, add --restart-apps (Windows: -RestartApps). If a revert fails somewhere, the kits and the log are kept: no application breaks, and the log shows why.
If you installed from a setup file: on Windows remove it under Apps & features, or silently with & "$env:ProgramFiles\Spitfire\APM\setup\uninstall-apm-collector.exe" /S for the exe and msiexec /x spitfire-apm-collector.msi /qn (with the .msi you installed from) for the msi. On macOS open the app in the dmg and press Uninstall.
Check (Linux, should print OK):
test ! -e /opt/spitfire-apm && test ! -e /etc/spitfire-apm && test ! -e /usr/local/bin/spitfire-collector && ! systemctl cat spitfire-apm-collector >/dev/null 2>&1 && echo "OK: Spitfire Collector removed"Windows:
if (-not (Get-Service SpitfireAPMCollector, spitfire-collector -ErrorAction SilentlyContinue)) { 'OK: Spitfire Collector removed' }To remove the server from the list press Remove from list on its page. For the other components: Removing what you installed.
Common problems
| Symptom | What to do |
|---|---|
NOT CONNECTED |
Check the Spitfire address, the key and the server's way out to Spitfire (proxy, firewall); then run check again. |
| The server shows No report | Linux: sudo systemctl status spitfire-apm-collector --no-pager; log /var/log/spitfire-apm-collector.log. Windows: Get-Service SpitfireAPMCollector. |
| An application is missing | On the server sudo /usr/local/bin/spitfire-apm-collector discover prints what it finds. |
| State Failed | The error text in the row says why (e.g. the kit could not be downloaded). If a proxy is needed, fill it in under Advanced settings in the install window and rerun the command; the settings are updated. |
| Waiting for restart after the install | Auto-trace without restart consent: the application gets tracing at its next restart. Run the command in the row, or press Restart now. |
Server metrics: not installed here |
Run the install command again (same settings); the part is added and the settings kept. On a server without internet give it with --host-binary. |
| The server is missing on Servers or not updated | Without a Tower, Towers & metrics In Spitfire: /towers → Collect and keep metrics must be on. If the license's server limit is reached, the Servers page says so. If check says cannot start, the Collector log says why. |
| Tracing on but no data | Check that APM data is on in this install (Before you start). If the server shows Stopped, press Start on its page. |