Deploy marks: ArgoCD and Jenkins
Send your ArgoCD and Jenkins deploys to Spitfire; each deploy becomes a vertical mark on the APM service charts, so "it broke after the deploy" is visible at a glance. A deploy also:
- enters Why did it break? on the APM service page: the 30 minutes before the deploy are compared with what came after, and if the error rate or p95 went up the deploy ranks high.
- lands in the Events list and goes through event policies: a failed deploy opens a
majorproblem, and the next successful deploy of the same service and environment closes it.
Deploy inputs are webhook inputs: each has its own sfhook_ token, shown only once. They need no monitoring events module.
Where the service name comes from
In ArgoCD the application name, in Jenkins the body's service (else the job name) is the APM
service name. When the names differ, write one app=service per line in the input's Service
map field:
# ArgoCD application = APM service
orders-prod=orders
billing-api=billingCase does not matter; an app without a line uses its own name. A changed map applies to the deploys after it.
Creating the input
Towers & metrics
In Spitfire: /towers→ Webhook inputs → New input.A name (e.g.
argocd-prod) and the format: ArgoCD or Jenkins. Write the service map if you need one → New input.The dialog shows the token, the URL on the controller and the sender's setup. The token is shown only here; keep the dialog open until the steps are done.
Copy the A test deploy (curl) command and run it in a terminal.
Success check: the answer is
{"data":{"received":1,"services":["orders"]}}and Last delivery fills in. A401means the token or the URL was copied wrong. The test leaves a mark with revisionteston theordersservice.
ArgoCD
argocd-notifications (shipped with ArgoCD 2.6 and later) sends a webhook through three triggers: on-deployed (sync done, app Healthy), on-sync-failed and on-health-degraded. The commands assume ArgoCD lives in the argocd namespace.
Put the token in the argocd-notifications Secret. The command asks for the token (not shown on screen or kept in shell history); paste the
sfhook_…token and press Enter:read -rsp 'sfhook_ token: ' T; echo kubectl -n argocd get secret argocd-notifications-secret >/dev/null 2>&1 \ || kubectl -n argocd create secret generic argocd-notifications-secret kubectl -n argocd patch secret argocd-notifications-secret --type merge \ -p "{\"stringData\":{\"spitfire-token\":\"$T\"}}" && unset TSuccess check: it prints
secret/argocd-notifications-secret patched.Save the YAML below as
~/spitfire-argocd-cm.yaml. Replacehttps://SPITFIRE_ADDRESS/api/v1/events/in/INPUT_IDwith the dialog's URL on the controller (the dialog's argocd-notifications Secret + ConfigMap text has it filled in already).apiVersion: v1 kind: ConfigMap metadata: name: argocd-notifications-cm namespace: argocd data: service.webhook.spitfire: | url: https://SPITFIRE_ADDRESS/api/v1/events/in/INPUT_ID headers: - name: Authorization value: Bearer $spitfire-token - name: Content-Type value: application/json template.spitfire-deploy-succeeded: | webhook: spitfire: method: POST body: | {"app":"{{.app.metadata.name}}","status":"succeeded","revision":"{{.app.status.sync.revision}}","environment":"{{.app.spec.destination.namespace}}","url":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}","who":"{{with .app.status.operationState}}{{with .operation.initiatedBy}}{{.username}}{{end}}{{end}}","at":"{{with .app.status.operationState}}{{.finishedAt}}{{end}}"} template.spitfire-deploy-failed: | webhook: spitfire: method: POST body: | {"app":"{{.app.metadata.name}}","status":"failed","revision":"{{.app.status.sync.revision}}","environment":"{{.app.spec.destination.namespace}}","url":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}","who":"{{with .app.status.operationState}}{{with .operation.initiatedBy}}{{.username}}{{end}}{{end}}","at":"{{with .app.status.operationState}}{{.finishedAt}}{{end}}"} template.spitfire-deploy-degraded: | webhook: spitfire: method: POST body: | {"app":"{{.app.metadata.name}}","status":"failed","revision":"{{.app.status.sync.revision}}","environment":"{{.app.spec.destination.namespace}}","url":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}","who":"{{with .app.status.operationState}}{{with .operation.initiatedBy}}{{.username}}{{end}}{{end}}"} trigger.on-deployed: | - description: Synced and healthy oncePer: app.status.sync.revision send: [spitfire-deploy-succeeded] when: app.status.operationState != nil and app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' trigger.on-sync-failed: | - description: Sync failed send: [spitfire-deploy-failed] when: app.status.operationState != nil and app.status.operationState.phase in ['Error', 'Failed'] trigger.on-health-degraded: | - description: Health degraded send: [spitfire-deploy-degraded] when: app.status.health.status == 'Degraded' subscriptions: | - recipients: [spitfire] triggers: [on-deployed, on-sync-failed, on-health-degraded]Apply it.
--server-sideleaves your other keys in the ConfigMap alone and only adds these:kubectl apply --server-side --field-manager=spitfire -f ~/spitfire-argocd-cm.yamlSuccess check: it prints
configmap/argocd-notifications-cm serverside-appliedand this command shows the URL:kubectl -n argocd get cm argocd-notifications-cm -o jsonpath='{.data.service\.webhook\.spitfire}'A
conflicterror means another tool manages the same keys; add the keys to the ConfigMap in that tool.Sync an application (Sync in the ArgoCD UI, or a new commit).
Success check: Last delivery updates; in APM the service's charts show a mark and a Deploys list under them. If nothing comes:
kubectl -n argocd logs deploy/argocd-notifications-controller --since=15m | grep -i spitfire
subscriptions subscribes every application. For only some of them, delete that block and annotate each application:
kubectl -n argocd annotate application orders --overwrite \
notifications.argoproj.io/subscribe.on-deployed.spitfire="" \
notifications.argoproj.io/subscribe.on-sync-failed.spitfire="" \
notifications.argoproj.io/subscribe.on-health-degraded.spitfire=""Jenkins
A curl in the Jenkinsfile's post{} block reports how the deploy ended. It never breaks the build: it waits 10 seconds at most, and the build goes on even when it fails (|| true).
Add the token to Jenkins: Manage Jenkins → Credentials → System → Global credentials → Add Credentials → Kind: Secret text, Secret: the
sfhook_…token from the dialog, ID:spitfire-deploy-token→ Create.Success check:
spitfire-deploy-tokenis in the credentials list.Add the
spitfireDeployfunction and thepostblock to your Jenkinsfile (the dialog's Jenkinsfile text has the URL filled in). Put your service and environment in place ofordersandprod, and your deploy steps in place of theecholine.// Spitfire deploy notice: never fails the build (|| true, 10 s at most) def spitfireDeploy(String status) { def who = currentBuild.getBuildCauses()[0]?.userId ?: 'jenkins' withCredentials([string(credentialsId: 'spitfire-deploy-token', variable: 'SPITFIRE_TOKEN')]) { sh """curl -sS --max-time 10 -X POST 'https://SPITFIRE_ADDRESS/api/v1/events/in/INPUT_ID' \\ -H "Authorization: Bearer \$SPITFIRE_TOKEN" -H 'Content-Type: application/json' \\ -d '{"service":"orders","environment":"prod","status":"${status}","revision":"${env.GIT_COMMIT ?: ''}","url":"${env.BUILD_URL}","job":"${env.JOB_NAME}","build":"${env.BUILD_NUMBER}","who":"${who}"}' || true""" } } pipeline { agent any stages { stage('Deploy') { steps { script { spitfireDeploy('started') } echo 'your deploy steps here' } } } post { success { script { spitfireDeploy('succeeded') } } failure { script { spitfireDeploy('failed') } } } }Run the job (Build Now).
Success check: the build log shows
{"data":{"received":1,"services":["orders"]}}twice (started, ended) and theorderscharts show a mark. A failed build puts a red mark and opens a problem.
Notification plugin
The Notification plugin cannot add headers, so the token goes in the URL's user info (Spitfire takes a Basic auth password as the token).
Job → Configure → Job Notifications → Add Endpoint: Format JSON, Protocol HTTP, Event All Events, URL:
https://spitfire:sfhook_…@SPITFIRE_ADDRESS/api/v1/events/in/INPUT_IDTo keep the token out of sight, enter the URL as a Secret text credential (URL Source: Secret).
Run the job.
STARTEDis a started deploy,COMPLETEDan ended one (SUCCESSsucceeded;FAILURE,UNSTABLE,ABORTEDfailed). The service name is the job name (the map or aSERVICEbuild parameter changes it), the environment theENVIRONMENTbuild parameter.Success check: the input's latest events show
Deploy succeeded: <job>.
Body format
You can post from your own script too; one object or a list of at most 100:
| Field | Meaning |
|---|---|
service / app / job |
Service name (the map applies) |
status |
started, succeeded, failed (success, failure, synced, degraded… work too) |
revision / commit |
Version or commit |
environment / env |
Environment; matches the environment picked in APM, empty shows in every environment |
url |
CI/CD link (http/https only) |
who |
Who deployed |
at |
Time (RFC 3339 or Unix seconds); else the time it arrived |
message |
Note |