Sweeper (site scan)

Sweeper opens your site page by page in a real browser (Chromium) and follows its links. On each page it records:

• the status and the TTFB, load and LCP times,
• console errors and uncaught exceptions,
• failed requests (a script, image, style or XHR that answered 4xx/5xx or did not load),
• broken links (a link on the page that leads to an error),
• blank pages (loaded, but nothing visible) and slow pages (slower than the sweep's limit).

#What it does not do

• In Read mode (the default) it only makes GET requests to open pages (links to other sites, when you ask for them, get a HEAD or GET). It fills in no form, clicks nothing and sends nothing. Interactive mode does click, fill and submit, on staging only.
• Logout-like paths (/logout, /signout, /cikis, …) are always skipped, so a sweep with a session cookie stays signed in.
• It does not crawl other sites: only the start address's host and the extra hosts you list.

#Runner requirement

A sweep runs on a runner with a browser: the usespitfire/spitfire:<version>-browser image, which labels itself browser: chromium. Each runner takes one sweep at a time.

Without a browser runner the Sweeper page shows a warning and Start stays disabled; the API and the CLI answer no_sweep_runner. To add one:

1. Open Runners and click Create key.
2. In the Docker tab tick For browser steps.
3. Copy the command and run it on the machine that will be the runner.
4. Check: the runner appears on the Runners page with the browser label, and the Sweeper page says "1 browser runner connected".

#Creating a sweep

1. Open Sweeper in the menu and click New sweep.
2. Type a Name (e.g. Shop) and the Start address (e.g. https://shop.example.com/).
3. Leave the rest as it is, or set Max pages, Max depth, Pages at a time (1–4) and the Slow page limit.
4. For pages behind a login, fill in Headers (Authorization: Bearer …) or the Cookie. They are stored encrypted and go only to the sweep's own hosts.
5. Under Pass / fail set Max errors (0: any error fails) and, if slow pages should fail the run too, Max slow pages.
6. Click Save, then Start on the sweep's row.
7. Check: the run page opens and the page count grows; at the end the badge says passed or failed.

Errors are every finding except slow pages. Only these paths and Skip these paths take one pattern a line; * matches anything (/blog/*, */print*), a pattern without * is a prefix.

#Reading a run

• The cards at the top count the pages and each kind of finding; click a card to show only that kind.
• The table lists the pages with at least one finding: the address, the page that linked to it, the status, TTFB, load time, LCP and each finding with its detail.
• When the run did not pass, the reason is written under the badge (e.g. 3 errors > 0).
• Stop ends a running sweep; what it found so far is kept.

#Interactive mode

Interactive mode also tries what is on every page, the way someone checks a release by hand: buttons, menu items, tabs, checkboxes and radios, selects (it picks another option and goes back) and forms (filled with your form values and submitted). After each interaction it looks for a new console error or exception, a 4xx/5xx from the sweep's hosts, a blank page and a click that did not reach the element. Each problem is an Interaction failed finding with the element's label and selector, what was done (click, select, submit), the result (POST /api/cart → 500, TypeError: …) and a screenshot when one could be taken.

It clicks, fills and submits, so run it only on a staging or test environment.

What keeps it safe:

• Only the sweep's hosts. Every request to another host (a payment provider, a mail service, an analytics post) and every sign-out request is stopped in the browser and listed on the run as not sent. Images, scripts and styles from other hosts still load, so pages look right.
• The skip list. Elements whose text, aria-label, title or icon class matches a line are never clicked. The default list holds what deletes, revokes, starts or stops something, pays, orders or sends (pay, öde, satın al, checkout, place order, subscribe, send, gönder, …); remove a line to have it tried. Log out buttons and red (danger) buttons are never clicked.
• 400 and 422 after a submit are fine: that is the form's own validation answering. A 5xx is a finding.
• Confirm dialogs are dismissed (Cancel), downloads and new windows are not opened.
• Staging tick. Saving an interactive sweep asks you to tick "… is a staging or test environment" for its hosts. Who ticked it and when is stored on the sweep; the tick holds while the hosts stay the same.
• Never the live site. If a test's environment named prod, production, live, canli, … points at one of the hosts (Environments), the sweep cannot be saved or started in interactive mode. This is checked again at every start, schedules included. A www. host needs a second tick ("… is not the live site") unless a staging or test environment points at it.
• Every interactive start goes to the audit log (sweep.interactive_start, with who and which hosts).

#Setting it up

1. Open the sweep (or New sweep) and pick Interactive (staging only) under Mode.
2. Tick … is a staging or test environment (and, for a www. host, … is not the live site).
3. Under What to try untick what should not be tried.
4. Check the Never click (skip list): one line each, a word or phrase, or a CSS selector (#buy, .danger).
5. Fill in Form values, one a line: match = value. The match is a field's name, id, label, placeholder or input type, or a CSS selector (email = test@example.com, #qty = 2). Fields no line names are sent as they are.
6. Set Interactions per page and Wait per interaction (ms) if you need to, then Save.
7. Check: the sweep's row has the Interactive (staging only) badge; after a run, the run page shows the Interactions card with what was tried, failed and skipped, and the requests not sent.

Interaction failures are errors, so Max errors decides pass or fail as for any other finding. spitfire cloud sweep is the same command with the same exit codes; the mode comes from the sweep.

#Staging gate in GitHub Actions

Deploy to staging, sweep it interactively, and deploy to production only when the sweep passes. The sweep (Shop staging here) is an interactive sweep on the staging address.

# .github/workflows/release.yml
on:
  push:
    branches: [main]
jobs:
  staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh staging          # your staging deploy
  sweep:
    needs: staging
    runs-on: ubuntu-latest
    steps:
      - env:
          SPITFIRE_URL: https://spitfire.example.com
          SPITFIRE_TOKEN: ${{ secrets.SPITFIRE_TOKEN }}
        run: docker run --rm -e SPITFIRE_URL -e SPITFIRE_TOKEN usespitfire/spitfire spitfire cloud sweep "Shop staging" --timeout 30m
  production:
    needs: sweep                          # runs only when the sweep passed (exit code 0)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh production       # your production deploy

Check: the sweep job's log ends with sweep Shop staging: passed — … and an interactions: … tried, … failed line; a failed sweep (exit code 99) stops the production job.

#Sweeping from CI

The command fails the job when the sweep does not pass (exit code 99), so a deploy can be stopped by a broken link or a console error. It works from any folder; SPITFIRE_URL and SPITFIRE_TOKEN are the same variables as in CI/CD.

1. Create the sweep in the UI (above) and note its name, e.g. Shop.

2. Store an API token in the CI's secret variable SPITFIRE_TOKEN.

3. Add this step to the job (Docker, no download):

   bashCopy

   docker run --rm -e SPITFIRE_URL -e SPITFIRE_TOKEN usespitfire/spitfire spitfire cloud sweep "Shop"

   Or, with the CLI from CI/CD installed:

   bashCopy

   spitfire cloud sweep "Shop" --timeout 30m -o "${RUNNER_TEMP:-/tmp}/sweep.json"

4. Check: the job's log ends with sweep Shop: passed — … pages, 0 errors … and the link of the run; the job is green.

Exit codes: 0 passed, 99 the thresholds failed (more errors or slow pages than allowed), 1 any other error (no browser runner, a wrong token, a sweep that does not exist).

The same over the REST API (API reference In Spitfire: /api/docs, tag Sweeper):

curl -fsS -X POST -H "Authorization: Bearer $SPITFIRE_TOKEN" "$SPITFIRE_URL/api/v1/sweeps/Shop/start?trigger=ci"

The answer is the run; ask GET /api/v1/sweep-runs/{id} until done is true, then read passed.

#Schedules and notifications

• With the Sweeper module, a sweep can run on a cron schedule (Schedule in the dialog, e.g. 0 3 * * * = every day at 03:00).
• A run that does not pass (or cannot run) goes to the notification channels subscribed to the sweep.failed event: on Integrations, edit a Slack, Teams, e-mail or webhook channel and tick it (see notification channels). It is off by default and does not go to SMS or issue trackers.

#License

The module comes with the Enterprise plan or as an add-on to any edition. An installation admin can try it once for 7 days from the Sweeper page or the License page.

#Common problems

• "No idle runner with a browser": connect a runner from the -browser image (see Runner requirement), or wait for a running sweep to end.
• Every page is a 401/403: the session is missing; add the header or cookie in the dialog.
• "interactive_not_allowed" on start: a test's production environment now points at one of the hosts, or the hosts changed since the staging tick. Open the sweep, check the address, tick the staging box and save.
• An interactive run tried nothing on a page: the page's elements are all in the skip list, hidden, or over Interactions per page; the Interactions card lists what was skipped and why.
• Too few pages: raise Max depth or Max pages, and check Only these paths; without the module a run stops at 50 pages.