Version: 1.4.9This documentation is for Spitfire 1.4.9.
Built-in policies, groups and notification delivery
The Events → Event policies → Built-in policies tab holds the rules that watch Kubernetes, Docker, Linux and
Windows servers and network devices (SNMP) with the metrics Spitfire collects. They come with the installation, are
on and raise events with the source spitfire.builtin. You do not have to write rules of your own.
How it works
- Every minute Spitfire measures each rule for each server, pod, container or interface it applies to.
- When the value crosses a threshold and stays there for the For time, an event opens (warning or critical).
- Going from warning to critical sends one more notice; when the value recovers the event resolves with a "resolved" notice.
- A subject that can no longer be measured (a deleted pod, a removed disk) resolves after 15 minutes.
Every channel picked in the rule's Channels column (Slack, e-mail, SMS, webhook…) gets its own notice; pick as many as you need. Without a channel the rule only records events (event only). Your event policies apply to these events too.
Change, switch off, back to defaults
- Edit changes the warning and critical thresholds, the for time and the channels for your workspace. A changed rule carries the Changed badge.
- Back to defaults returns the thresholds, for time and channels to the rule's defaults.
- The switch turns a rule off: while off there are no events and no notices, and its open events resolve without a notice. A built-in cannot be deleted, only switched off.
- Changing needs a workspace admin. Every change is in the audit log (
event.builtin.update,event.builtin.enable,event.builtin.disable,event.builtin.reset). - Rules have a version (
v1). When Spitfire updates a rule its version goes up; your threshold and channel changes are kept.
Rules
Thresholds are in the rule's unit: %, minutes, a count or per core. A "condition" rule has no threshold to change;
it fires when the state is seen.
| Rule | Warning | Critical | For |
|---|---|---|---|
linux.host_down, windows.host_down Server not reporting |
5 min | at once | |
linux.cpu_high, windows.cpu_high High CPU |
85% | 95% | 10 min |
linux.load_per_core Load per core |
1.5 | 3 | 10 min |
linux.iowait iowait |
20% | 40% | 10 min |
linux.memory_high, windows.memory_high Memory |
90% | 97% | 10 min |
linux.swap_high Swap |
50% | 80% | 15 min |
linux.disk_full File system use |
85% | 95% | 5 min |
linux.inode_full Inode use |
85% | 95% | 5 min |
windows.disk_free_low Free disk space (below) |
15% | 5% | 5 min |
linux.unit_failed systemd unit failed |
condition | 5 min | |
windows.service_stopped Automatic service stopped |
condition | 5 min | |
k8s.pod_restarts Pod restarts (in 15 min) |
3 | 10 | at once |
k8s.crashloop Container crash looping |
condition | 5 min | |
k8s.pod_pending Pod stuck in Pending |
condition | 10 min | |
k8s.node_not_ready Node not ready |
condition | 2 min | |
k8s.node_memory_pressure Node memory pressure |
condition | 2 min | |
k8s.node_disk_pressure Node disk pressure |
condition | 2 min | |
k8s.replicas_unavailable Replicas unavailable |
1 | 10 min | |
k8s.pvc_full PVC use |
85% | 95% | 5 min |
k8s.oomkilled Container OOMKilled |
condition | at once | |
docker.restarting Container restarts (in 15 min) |
3 | 10 | at once |
docker.exited Container stopped |
condition | 5 min | |
docker.cpu_high Container CPU |
85% | 95% | 10 min |
docker.memory_high Container memory against its limit |
90% | 97% | 5 min |
network.snmp_unreachable Network device unreachable |
condition | 5 min | |
network.if_down Interface down |
condition | 2 min | |
network.if_errors Interface errors + discards |
10 | 100 | 5 min |
network.if_util Interface utilisation |
80% | 95% | 10 min |
network.device_cpu Network device CPU |
85% | 95% | 10 min |
Each rule's description and its "What to do" hint are behind the orange i next to its name.
Source column
The Source column of the Events list shows where an event came from with a logo and a short name: Spitfire's own sources (built-in policy, monitor, APM alert…) with the Spitfire mark, monitoring connections with their own logo, API and webhook inputs with their own icon. On a narrow screen only the logo shows; the name appears on hover. Clicking a source filters the list to it.
Cross-source groups
Problems (warning or worse) of the same host or service that come from different sources (built-in policies,
monitoring connections, webhook inputs) within 10 minutes are put in one group. The group's first event is
notified; the others join its timeline without notices of their own. Host names are compared without case, domain
and port (WEB-1.example.com:9100 and web-1 are the same). Pods group by their workload's service.
- The Events → Groups view shows each group in a bordered card: the host or service, the worst severity, the number of events and the sources' logos in its header, and a stripe in its source's colour on every row.
- The noise figure on top shows how many rows the list is down to from the occurrences that came in (repeats and groups taken out).
- Change the window (1–1440 minutes) or switch the feature off with the Cross-source correlation card on the Event policies page. When a correlate policy groups an event first, it wins.
Stop all alarm notifications
During maintenance, or an incident you already work on, the installation admin cuts every alarm notice with the Stop all alarm notifications switch on the Event policies page, optionally until a given time.
- While the stop holds, a banner shows on top of every page.
- Events are still recorded and stay on Events.
- Notices not sent meanwhile are never sent later; the delivery log shows them as "Not sent".
- Run and report notices are not stopped.
- Switching it on and off is in the audit log (
notifications.pause,notifications.resume).
Notification delivery and retries
Notices go out from a queue, without holding up any service and with bounded concurrency. A failed delivery is retried after 30 seconds, 2 minutes and 10 minutes. When the last attempt fails too:
- The system closes the delivery; the delivery log under Integrations shows "Closed by the system: 3 attempts failed".
- The event stays open; ack and resolve are yours.
- One "channel unreachable" notice is recorded for the channel; the same channel does not warn again and again.