SpitfireLoad Testing & APMWhat's new

Deploy marks: ArgoCD and Jenkins

Send your ArgoCD and Jenkins deploys to Spitfire; each deploy becomes a vertical mark on the APM service charts, so "it broke after the deploy" is visible at a glance. A deploy also:

  • enters Why did it break? on the APM service page: the 30 minutes before the deploy are compared with what came after, and if the error rate or p95 went up the deploy ranks high.
  • lands in the Events list and goes through event policies: a failed deploy opens a major problem, and the next successful deploy of the same service and environment closes it.

Deploy inputs are webhook inputs: each has its own sfhook_ token, shown only once. They need no monitoring events module.

Where the service name comes from

In ArgoCD the application name, in Jenkins the body's service (else the job name) is the APM service name. When the names differ, write one app=service per line in the input's Service map field:

text
# ArgoCD application = APM service
orders-prod=orders
billing-api=billing

Case does not matter; an app without a line uses its own name. A changed map applies to the deploys after it.

Creating the input

  1. Towers & metrics In Spitfire: /towers → Webhook inputs → New input.

  2. A name (e.g. argocd-prod) and the format: ArgoCD or Jenkins. Write the service map if you need one → New input.

  3. The dialog shows the token, the URL on the controller and the sender's setup. The token is shown only here; keep the dialog open until the steps are done.

  4. Copy the A test deploy (curl) command and run it in a terminal.

    Success check: the answer is {"data":{"received":1,"services":["orders"]}} and Last delivery fills in. A 401 means the token or the URL was copied wrong. The test leaves a mark with revision test on the orders service.

ArgoCD

argocd-notifications (shipped with ArgoCD 2.6 and later) sends a webhook through three triggers: on-deployed (sync done, app Healthy), on-sync-failed and on-health-degraded. The commands assume ArgoCD lives in the argocd namespace.

  1. Put the token in the argocd-notifications Secret. The command asks for the token (not shown on screen or kept in shell history); paste the sfhook_… token and press Enter:

    bash
    read -rsp 'sfhook_ token: ' T; echo
    kubectl -n argocd get secret argocd-notifications-secret >/dev/null 2>&1 \
      || kubectl -n argocd create secret generic argocd-notifications-secret
    kubectl -n argocd patch secret argocd-notifications-secret --type merge \
      -p "{\"stringData\":{\"spitfire-token\":\"$T\"}}" && unset T

    Success check: it prints secret/argocd-notifications-secret patched.

  2. Save the YAML below as ~/spitfire-argocd-cm.yaml. Replace https://SPITFIRE_ADDRESS/api/v1/events/in/INPUT_ID with the dialog's URL on the controller (the dialog's argocd-notifications Secret + ConfigMap text has it filled in already).

    yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: argocd-notifications-cm
      namespace: argocd
    data:
      service.webhook.spitfire: |
        url: https://SPITFIRE_ADDRESS/api/v1/events/in/INPUT_ID
        headers:
        - name: Authorization
          value: Bearer $spitfire-token
        - name: Content-Type
          value: application/json
      template.spitfire-deploy-succeeded: |
        webhook:
          spitfire:
            method: POST
            body: |
              {"app":"{{.app.metadata.name}}","status":"succeeded","revision":"{{.app.status.sync.revision}}","environment":"{{.app.spec.destination.namespace}}","url":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}","who":"{{with .app.status.operationState}}{{with .operation.initiatedBy}}{{.username}}{{end}}{{end}}","at":"{{with .app.status.operationState}}{{.finishedAt}}{{end}}"}
      template.spitfire-deploy-failed: |
        webhook:
          spitfire:
            method: POST
            body: |
              {"app":"{{.app.metadata.name}}","status":"failed","revision":"{{.app.status.sync.revision}}","environment":"{{.app.spec.destination.namespace}}","url":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}","who":"{{with .app.status.operationState}}{{with .operation.initiatedBy}}{{.username}}{{end}}{{end}}","at":"{{with .app.status.operationState}}{{.finishedAt}}{{end}}"}
      template.spitfire-deploy-degraded: |
        webhook:
          spitfire:
            method: POST
            body: |
              {"app":"{{.app.metadata.name}}","status":"failed","revision":"{{.app.status.sync.revision}}","environment":"{{.app.spec.destination.namespace}}","url":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}","who":"{{with .app.status.operationState}}{{with .operation.initiatedBy}}{{.username}}{{end}}{{end}}"}
      trigger.on-deployed: |
        - description: Synced and healthy
          oncePer: app.status.sync.revision
          send: [spitfire-deploy-succeeded]
          when: app.status.operationState != nil and app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
      trigger.on-sync-failed: |
        - description: Sync failed
          send: [spitfire-deploy-failed]
          when: app.status.operationState != nil and app.status.operationState.phase in ['Error', 'Failed']
      trigger.on-health-degraded: |
        - description: Health degraded
          send: [spitfire-deploy-degraded]
          when: app.status.health.status == 'Degraded'
      subscriptions: |
        - recipients: [spitfire]
          triggers: [on-deployed, on-sync-failed, on-health-degraded]
  3. Apply it. --server-side leaves your other keys in the ConfigMap alone and only adds these:

    bash
    kubectl apply --server-side --field-manager=spitfire -f ~/spitfire-argocd-cm.yaml

    Success check: it prints configmap/argocd-notifications-cm serverside-applied and this command shows the URL:

    bash
    kubectl -n argocd get cm argocd-notifications-cm -o jsonpath='{.data.service\.webhook\.spitfire}'

    A conflict error means another tool manages the same keys; add the keys to the ConfigMap in that tool.

  4. Sync an application (Sync in the ArgoCD UI, or a new commit).

    Success check: Last delivery updates; in APM the service's charts show a mark and a Deploys list under them. If nothing comes:

    bash
    kubectl -n argocd logs deploy/argocd-notifications-controller --since=15m | grep -i spitfire
Tip

subscriptions subscribes every application. For only some of them, delete that block and annotate each application:

bash
kubectl -n argocd annotate application orders --overwrite \
  notifications.argoproj.io/subscribe.on-deployed.spitfire="" \
  notifications.argoproj.io/subscribe.on-sync-failed.spitfire="" \
  notifications.argoproj.io/subscribe.on-health-degraded.spitfire=""

Jenkins

A curl in the Jenkinsfile's post{} block reports how the deploy ended. It never breaks the build: it waits 10 seconds at most, and the build goes on even when it fails (|| true).

  1. Add the token to Jenkins: Manage Jenkins → Credentials → System → Global credentials → Add Credentials → Kind: Secret text, Secret: the sfhook_… token from the dialog, ID: spitfire-deploy-token → Create.

    Success check: spitfire-deploy-token is in the credentials list.

  2. Add the spitfireDeploy function and the post block to your Jenkinsfile (the dialog's Jenkinsfile text has the URL filled in). Put your service and environment in place of orders and prod, and your deploy steps in place of the echo line.

    groovy
    // Spitfire deploy notice: never fails the build (|| true, 10 s at most)
    def spitfireDeploy(String status) {
      def who = currentBuild.getBuildCauses()[0]?.userId ?: 'jenkins'
      withCredentials([string(credentialsId: 'spitfire-deploy-token', variable: 'SPITFIRE_TOKEN')]) {
        sh """curl -sS --max-time 10 -X POST 'https://SPITFIRE_ADDRESS/api/v1/events/in/INPUT_ID' \\
          -H "Authorization: Bearer \$SPITFIRE_TOKEN" -H 'Content-Type: application/json' \\
          -d '{"service":"orders","environment":"prod","status":"${status}","revision":"${env.GIT_COMMIT ?: ''}","url":"${env.BUILD_URL}","job":"${env.JOB_NAME}","build":"${env.BUILD_NUMBER}","who":"${who}"}' || true"""
      }
    }
    
    pipeline {
      agent any
      stages {
        stage('Deploy') {
          steps {
            script { spitfireDeploy('started') }
            echo 'your deploy steps here'
          }
        }
      }
      post {
        success { script { spitfireDeploy('succeeded') } }
        failure { script { spitfireDeploy('failed') } }
      }
    }
  3. Run the job (Build Now).

    Success check: the build log shows {"data":{"received":1,"services":["orders"]}} twice (started, ended) and the orders charts show a mark. A failed build puts a red mark and opens a problem.

Notification plugin

The Notification plugin cannot add headers, so the token goes in the URL's user info (Spitfire takes a Basic auth password as the token).

  1. Job → Configure → Job Notifications → Add Endpoint: Format JSON, Protocol HTTP, Event All Events, URL:

    text
    https://spitfire:sfhook_…@SPITFIRE_ADDRESS/api/v1/events/in/INPUT_ID

    To keep the token out of sight, enter the URL as a Secret text credential (URL Source: Secret).

  2. Run the job. STARTED is a started deploy, COMPLETED an ended one (SUCCESS succeeded; FAILURE, UNSTABLE, ABORTED failed). The service name is the job name (the map or a SERVICE build parameter changes it), the environment the ENVIRONMENT build parameter.

    Success check: the input's latest events show Deploy succeeded: <job>.

Body format

You can post from your own script too; one object or a list of at most 100:

Field Meaning
service / app / job Service name (the map applies)
status started, succeeded, failed (success, failure, synced, degraded… work too)
revision / commit Version or commit
environment / env Environment; matches the environment picked in APM, empty shows in every environment
url CI/CD link (http/https only)
who Who deployed
at Time (RFC 3339 or Unix seconds); else the time it arrived
message Note