SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

APM alerts and SLOs

Hear about it when a service's p95, error rate or request rate goes bad, or a log search grows; follow the error budget with SLOs. Menu: APM › Alerts and APM › SLOs. Adding rules and SLOs is for workspace admins.

Add an alert

  1. APM › Services › click the service.
  2. Press Add alert at the top.
  3. The Last 24 hours line shows the service's p50/p95/p99 and error rate; the thresholds are filled from it (warning: p95 × 1.5, critical: p95 × 3).
  4. Pick the Metric: p95, p99, p50, average duration, error rate, request rate drop/spike, new or reopened error group.
  5. Change the thresholds if needed. A threshold you typed stays when you change the metric.
  6. For (min): the alert fires when the value stays over the threshold this many minutes in a row.
  7. Optionally pick notification Channels, then press Save.

For another service: APM › Alerts › Add alert, type the service name.

Note

With no channel picked, the alert goes to the channels that chose the apm.alert.firing and apm.alert.resolved events in notification channels. These events are opt-in; they are not selected by default.

Alert on a log search

  1. APM › Logs: type the search, tick the level and service (e.g. card declined, level error).
  2. Press Alert on this search. The dialog opens with the search filled in.
  3. The Last 24 hours line shows the search's records and the busiest window; the thresholds are filled from it (warning: busiest window × 1.5, critical: × 3).
  4. Pick the Metric:
    • Log records: fires when the records matching the search in the Window (1–60 min, default 5) go over the threshold.
    • New log pattern: fires when a pattern of the search shows up in the last 5 minutes that the 7 days before did not have (threshold 1).
  5. Optionally type a Service and Environment: they narrow the search. Leave them empty: the search's services.
  6. Save. Check: in APM › Alerts the rule's scope is the search itself; click it and the Logs page opens with that search.
Note

Log alerts have no "for" period and no minimum of requests; they fire on the first minute over the threshold. The notice names the search and its Open the logs link opens the Logs page over the alert's range; new patterns come with an example line and a link. A new pattern rule counts as no data while the search has no earlier records (or has logs written before 1.4.4, which have no pattern). Log rules count toward the same rule limit as the other alerts.

Alerts

  1. APM › Alerts.
  2. In Firing now each alert has three links: Chart (the service page), Traces (traces slower than the threshold, or failed), Why did it break? (the service page with the root cause panel open). A log alert has one: Open the logs (the search's logs from a window before it started).
  3. The rules table shows the state: OK, Pending (over the threshold, the minutes not up yet), Firing (warning/critical), Resolved.
  4. To silence a rule for an hour press the bell button on its row. Pencil and bin to edit and delete.
  5. History shows the last state changes and whether they were notified (kept 90 days).

Silences and maintenance windows

  1. APM › Alerts › Silences and maintenance windows.
  2. What: all alerts, a rule, an SLO or a service.
  3. Enter the Start (empty: now; a later time: a maintenance window), the Duration and the Reason, press Silence.
  4. To finish one press End on its row (Cancel for one not started).
Note

A silence mutes the notice, not the state. When the silence ends and the alert is still firing, it is notified then.

Add an SLO

  1. Press Add SLO on the service page (or APM › SLOs › Add SLO).
  2. Pick the SLI: Availability (requests without errors) or Latency (requests under a threshold).
  3. For latency pick the Latency threshold (25 ms – 10 s); the suggestion is the first value above p95 × 1.5.
  4. Objective (%): filled with what the last 24 hours would have met, e.g. 99.9.
  5. Window: 7, 28 or 30 days (default 30).
  6. Leave Burn rate alerts on, press Save.

SLOs

  1. APM › SLOs: each SLO's objective, error budget left and burn state.
  2. Click a row: the budget left and burn rate charts, the burn rates now (1 h, 5 m, 6 h, 30 m) and the history open.
  3. Status: learning (the window is not full yet), healthy, low, exhausted. Burn: OK, Burning slowly, Burning fast.
Burn Windows Threshold Meaning
Fast 1 hour and 5 minutes 14.4× a 30-day budget is gone in 2 days
Slow 6 hours and 30 minutes 6× a 30-day budget is gone in 5 days

Notification events: apm.slo.burn and apm.slo.recovered.

An SLO from a capacity run

  1. Open the capacity (breaking point) run.
  2. On the capacity card press Make an SLO from this p95.
  3. Pick the service; the threshold is the p95 of the highest step that held, rounded up. Save.

Plan limits

Plan Alert rules SLOs
Free 3 1
Starter 20 5
Growth and up unlimited unlimited

After a downgrade no rule is deleted; the oldest ones keep being evaluated.

CLI and CI

Check for alerts before a deploy or a load test:

bash
spitfire cloud apm alerts --fail-on-firing
spitfire cloud apm slos --fail-on-firing

The exit code is 99 when a rule fires or an SLO burns fast or has no budget left. REST: /api/v1/apm/alerts, /api/v1/apm/slos (API reference).