SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Infrastructure — cloud, vCenter, Kubernetes, Docker and MongoDB Atlas

In the license each AWS account, Azure subscription, Google Cloud project, vCenter and Kubernetes cluster counts as one cloud account / cluster; MongoDB Atlas counts as a data source and a Docker host as a server.

AWS, Azure, Google Cloud and vCenter are Beta; what is read, the cost and the interval are behind the orange i on each catalog card.

AWS

Quick install

  1. Copy the command, paste it into the terminal and run it. Answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/aws.sh?t=<token>" -o /tmp/sf-aws.sh && bash /tmp/sf-aws.sh
  1. When you see OK: ..., you are done.

Manual install (access key)

  1. Create the IAM user.
bash
aws iam create-user --user-name spitfire-monitor
  1. Give it the read-only CloudWatch policy.
bash
aws iam put-user-policy --user-name spitfire-monitor --policy-name SpitfireCloudWatchRead --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["cloudwatch:GetMetricData","cloudwatch:ListMetrics"],"Resource":"*"}]}'
  1. Create its access key.
bash
aws iam create-access-key --user-name spitfire-monitor --query 'AccessKey.[AccessKeyId,SecretAccessKey]' --output text
  1. Add it in Infrastructure sources → AWS CloudWatch.
text
Regions: eu-central-1 · Access key ID / Secret access key: the two values above · Test → Save
  1. Check the policy.
bash
aws iam list-user-policies --user-name spitfire-monitor --output text | grep -q SpitfireCloudWatchRead && echo "OK: spitfire-monitor reads CloudWatch"

Manual install (role, another account)

  1. Create the role in the monitored account.
bash
aws iam create-role --role-name spitfire-monitor --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::<tower-account-id>:root"},"Action":"sts:AssumeRole","Condition":{"StringEquals":{"sts:ExternalId":"<external-id>"}}}]}'
  1. Give it the read-only CloudWatch policy.
bash
aws iam put-role-policy --role-name spitfire-monitor --policy-name SpitfireCloudWatchRead --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["cloudwatch:GetMetricData","cloudwatch:ListMetrics"],"Resource":"*"}]}'
  1. Let the Tower's identity assume it (Tower's account).
bash
aws iam put-user-policy --user-name spitfire-monitor --policy-name SpitfireAssumeMonitor --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:iam::<monitored-account-id>:role/spitfire-monitor"}]}'
  1. Add it in Infrastructure sources → AWS CloudWatch.
text
Regions · Role ARN: arn:aws:iam::<monitored-account-id>:role/spitfire-monitor · External ID · Test → Save
  1. Check the role (monitored account).
bash
aws iam list-role-policies --role-name spitfire-monitor --output text | grep -q SpitfireCloudWatchRead && echo "OK: spitfire-monitor role reads CloudWatch"

Azure

Quick install

  1. Copy the command, paste it into the terminal and run it. Answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/azure.sh?t=<token>" -o /tmp/sf-azure.sh && bash /tmp/sf-azure.sh

On Windows, in PowerShell:

powershell
Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/azure.ps1?t=<token>" -OutFile "$env:TEMP\sf-azure.ps1"; powershell -ExecutionPolicy Bypass -File "$env:TEMP\sf-azure.ps1"
  1. When you see OK: ..., you are done.

Manual install

  1. Sign in to Azure.
bash
az login
  1. Create the service principal with Monitoring Reader.
bash
az ad sp create-for-rbac --name spitfire-monitor --role "Monitoring Reader" --scopes /subscriptions/<subscription-id> --query "{tenant:tenant,appId:appId,password:password}" -o table
  1. Add each further subscription.
bash
az role assignment create --assignee <appId> --role "Monitoring Reader" --scope /subscriptions/<subscription-id-2>
  1. Add it in Infrastructure sources → Azure Monitor.
text
Tenant ID: tenant · Client (app) ID: appId · Client secret: password · Subscription IDs · Test → Save
  1. Check the role.
bash
az role assignment list --assignee <appId> --all --query "[?roleDefinitionName=='Monitoring Reader'] | length(@)" -o tsv | grep -qv '^0
  

 && echo "OK: spitfire-monitor is Monitoring Reader"

Google Cloud

Quick install

  1. Copy the command, paste it into the terminal and run it. Answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/gcp.sh?t=<token>" -o /tmp/sf-gcp.sh && bash /tmp/sf-gcp.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Turn on the Cloud Monitoring API.
bash
gcloud services enable monitoring.googleapis.com --project=<project-id>
  1. Create the service account.
bash
gcloud iam service-accounts create spitfire-monitor --project=<project-id> --display-name="Spitfire monitor"
  1. Give it monitoring.viewer (each project).
bash
gcloud projects add-iam-policy-binding <project-id> --member="serviceAccount:spitfire-monitor@<project-id>.iam.gserviceaccount.com" --role="roles/monitoring.viewer" --condition=None
  1. Create its key.
bash
gcloud iam service-accounts keys create "$HOME/spitfire-monitor-key.json" --iam-account=spitfire-monitor@<project-id>.iam.gserviceaccount.com && cat "$HOME/spitfire-monitor-key.json"
  1. Add it in Infrastructure sources → Google Cloud Monitoring.
text
Project IDs · Service account key: the whole JSON above · Test → Save
  1. Delete the key file.
bash
rm "$HOME/spitfire-monitor-key.json"
  1. Check the role.
bash
gcloud projects get-iam-policy <project-id> --flatten="bindings[].members" --filter="bindings.members:spitfire-monitor@" --format="value(bindings.role)" | grep -q monitoring.viewer && echo "OK: spitfire-monitor is monitoring.viewer"

VMware vCenter

Quick install

  1. Copy the command, paste it into the terminal and run it. Answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/vcenter.sh?t=<token>" -o /tmp/sf-vcenter.sh && bash /tmp/sf-vcenter.sh

On Windows, in PowerShell (PowerCLI):

powershell
Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/vcenter.ps1?t=<token>" -OutFile "$env:TEMP\sf-vcenter.ps1"; powershell -ExecutionPolicy Bypass -File "$env:TEMP\sf-vcenter.ps1"
  1. When you see OK: ..., you are done.

Manual install (govc)

  1. Sign govc in as the administrator.
bash
export GOVC_URL='https://vcenter.corp' GOVC_USERNAME='administrator@vsphere.local' GOVC_PASSWORD='<admin-password>' GOVC_INSECURE=1
  1. Create the SSO user.
bash
govc sso.user.create -p '<password>' spitfire-monitor
  1. Give it ReadOnly at the root, propagated.
bash
govc permissions.set -principal spitfire-monitor@vsphere.local -role ReadOnly -propagate=true /
  1. Add it in Infrastructure sources → vCenter.
text
vCenter URL: https://vcenter.corp · User: spitfire-monitor@vsphere.local · Password · Test → Save
  1. Check the permission.
bash
govc permissions.ls / | grep spitfire-monitor | grep -q ReadOnly && echo "OK: spitfire-monitor is ReadOnly"

Manual install (PowerCLI, Windows)

  1. Install PowerCLI.
powershell
Install-Module VMware.PowerCLI, VMware.vSphere.SsoAdmin -Scope CurrentUser
  1. Create the SSO user.
powershell
Connect-SsoAdminServer -Server vcenter.corp -User administrator@vsphere.local -Password '<admin-password>' -SkipCertificateCheck; New-SsoPersonUser -UserName spitfire-monitor -Password '<password>'
  1. Give it ReadOnly at the root, propagated.
powershell
Connect-VIServer -Server vcenter.corp -User administrator@vsphere.local -Password '<admin-password>'; New-VIPermission -Entity (Get-Folder -NoRecursion) -Principal 'VSPHERE.LOCAL\spitfire-monitor' -Role (Get-VIRole -Name ReadOnly) -Propagate:$true
  1. Add it in Infrastructure sources → vCenter.
text
vCenter URL: https://vcenter.corp · User: spitfire-monitor@vsphere.local · Password · Test → Save
  1. Check the permission.
powershell
if (Get-VIPermission -Principal 'VSPHERE.LOCAL\spitfire-monitor' | Where-Object { $_.Role -eq 'ReadOnly' -and $_.Propagate }) { 'OK: spitfire-monitor is ReadOnly' }

Kubernetes

Metrics are collected from inside the cluster: the manifest Spitfire gives puts a Collector on every node (DaemonSet, kubelet_stats) and one for the cluster's state (Deployment, k8s_cluster); the Collectors send to the Tower. Nodes count as hosts. Prerequisites, a Tower inside the cluster, verification and common errors: Connecting Kubernetes.

Quick install

  1. Run the command in a terminal with kubectl access, answer the questions.

    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/kubernetes.sh?t=<token>" -o /tmp/sf-kubernetes.sh && bash /tmp/sf-kubernetes.sh

  2. When you see this line, you are done.

    text
    OK: Kubernetes added to Spitfire, first metrics within ~2 min
Note

In the product (the Quick install tab) the command comes ready with your address and a fresh token; the token is valid for 1 hour and adds one monitored system. The script asks for the kubectl context, the cluster name, the Tower address and the sfcol_ Collector token; it applies the secret and the manifest and waits for the rollout.

Manual install

  1. Infrastructure sources → Add a source → Kubernetes → Manual install.

  2. Enter the Cluster name (e.g. prod-eu) and the Tower address (e.g. http://spitfire-tower.spitfire.svc:4318 with the Tower inside the cluster).

  3. Save.

  4. Copy the secret command with Copy and run it in a terminal with access to the cluster. The sfcol_ token is the Collector token shown when the Tower was added.

  5. Copy the manifest command with Copy and run it in the same terminal. It is one command: kubectl reads the manifest from the command itself, so no file is saved. Its shape:

    bash
    kubectl apply -f - <<'EOF'
# ... the manifest Spitfire gives ...
EOF

  6. Wait until the rollout ends:

    bash
    kubectl -n spitfire rollout status daemonset/spitfire-collector-node --timeout=120s

  7. Test: when it says "Collected from inside the cluster", wait; the first data comes in 1-2 minutes.

Success check: the collector pod is Running here:

bash
kubectl -n spitfire get pods

Cluster view

  1. Infrastructure sources → Dashboard on the Kubernetes source's row.
  2. In Cluster view check the node, namespace and pod counts and the CPU and memory tiles. The three boxes below it sum up the trouble: Workloads not ready (ready/desired pods), Pods not running (phase) and the pods with Most restarts; click a name to open that level.
  3. Pick a namespace from the namespace list.
  4. In Workloads click a Deployment, DaemonSet or StatefulSet name.
  5. In Pods click a pod's arrow to open its containers; click a container name.
  6. Click a finding's name: the view goes to its pod, workload or node.

Success check: the charts show the selected level's CPU, memory, filesystem and network; for containers with limits the CPU and memory charts have a lim line.

Findings:

Finding When
OOMKilled The container last ended OOMKilled and restarted in the window or is not ready
CPU near limit Container CPU use at 90% of its limit or more (throttled)
Memory near limit Container working set above 90% of its limit
DaemonSet missing pods ready_nodes < desired_scheduled_nodes for 5 minutes
StatefulSet / Deployment not full Ready/available pods < desired for 5 minutes
MemoryPressure / DiskPressure The node's condition is True
Note

A pod's workload is found from its name (Deployment pods are <name>-<hash>-<id>, DaemonSet and Job pods <name>-<id>, StatefulSet pods <name>-<ordinal>); no extra RBAC or processor is needed. Values are the last 3 minutes. Each pod makes about 10 series per node and 7 in the cluster; series left without points are deleted daily, so finished pods drop out of the series limit. To see a container's last terminated reason, update the manifest: Copy the manifest command in the source's dialog again and run it.

Drill-down: counts, pods and restarts

From 1.4.8 the summary counts of the cluster view open lists, a list row opens its detail, and a pod or container that restarted says why. The same view is in APM → Kubernetes and on the Kubernetes source's dashboard.

  1. Update the Collector manifests once after updating Spitfire to 1.4.8 (the restart reason comes from a new Collector setting): Infrastructure sources → the Kubernetes source → the Manual install tab → Copy the manifest command and run it in a terminal with kubectl access. Then wait for the rollout:

    bash
    kubectl -n spitfire rollout status daemonset/spitfire-collector-node --timeout=120s

  2. For the Kubernetes events in the panels, install the Kubernetes log collector with Kubernetes events too ticked: Observability → Log collectors → Kubernetes → tick Kubernetes events too (k8sobjects) → Generate configuration, then run its commands (Log collection: Kubernetes). Without it the panels work, only the events list stays empty.

  3. Open APM → Kubernetes (or Dashboard on the source's row) and click a summary count, e.g. Pods not up: the list it stands for opens. Click a row for its detail; Close, all lists goes back.

  4. Click a Deployment (or ReplicaSet): Pods not up lists its pods that are up or not, each with the reason (CrashLoopBackOff, ImagePullBackOff, Pending/Unscheduled, a failing readiness probe…), and Restarted pods and containers lists the ones that restarted.

  5. Click a pod or a container with a reason badge: the Why did it restart? panel shows the reason Kubernetes gave (now and at the last end) with what it means and what to check, the restarts in the last hour by reason, the memory in the 5 minutes before the last restart against the limit (so an OOMKilled is visible), the logs before the restart and the pod's Kubernetes events.

Success check: these two commands show the Collector and (with step 2) the log collector pods Running, and a pod that restarted has a reason badge in the view within 2 minutes:

bash
kubectl -n spitfire get pods
bash
kubectl -n spitfire-logs get pods

Docker (Beta)

The Collector runs as a container on the Docker host and reads the Docker API through /var/run/docker.sock (docker_stats: list, inspect, stats and events of running containers only). Each Docker host counts as one server in the hosts limit.

Note

Socket access is powerful: the full Docker API can start a privileged container, which is root on the host. The :ro mount does not narrow the API; it only stops the file being replaced. The Collector only reads, runs as a non-root user with the socket's group and has no other host access. For strict read-only, put a GET-only socket proxy in front and set Docker API to tcp://proxy:2375.

  1. Infrastructure sources → Add a source → Docker.
  2. Enter the Docker host name (e.g. docker-1; the server Collector's host name to make them one server) and the Tower address (e.g. http://tower.corp:4318).
  3. Save.
  4. Copy the docker run command with Copy; replace <sfcol_…> with the Tower's Collector token.
  5. Run it on the Docker host (Docker Desktop: replace the --group-add value with 0).
  6. Or, with Docker Compose 2.23 or later, copy the second block with Copy and run it on the Docker host instead.
  7. Check that the Collector runs:
bash
docker logs --tail 20 spitfire-docker-collector
  1. Test: when it says metrics are arriving, open Dashboard on the Docker source's row.

Container view

  1. Infrastructure sources → Dashboard on the Docker source's row.
  2. In Containers check the running / total count and the CPU and memory tiles.
  3. Type part of a name in Container name, or pick an Image or a Compose project.
  4. Click a container name: the charts show its CPU, memory (with the lim line), network and block I/O.
  5. Click all containers to chart the whole host again.

Success check: each running container has a row with CPU %, memory and its limit, network, block I/O and restarts; a container you stop shows gone within 2-3 minutes.

Findings:

Finding When
Restarting Docker's restart count rose in the last 15 minutes
Memory near limit Memory above 90% of the limit (the host's memory without a limit)
CPU throttled 25% or more of the CPU periods throttled in 3 minutes (only with a CPU limit)
Stopped The container's data stopped while the host's goes on (exited, stopped or removed)
Note

Docker reports only running containers, so a stopped container's exit code is not known: "Stopped" covers exit 0, a non-zero exit and docker rm. CPU is percent of one core, as docker stats shows it. The docker_stats receiver is alpha in Collector contrib v0.164.1.

MongoDB Atlas

First create an API key: Atlas → Organization → Access Manager → API Keys → Create API Key → Organization Read Only.

Quick install

  1. Run the command on any machine, answer the questions.

    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/mongodbatlas.sh?t=<token>" -o /tmp/sf-mongodbatlas.sh && bash /tmp/sf-mongodbatlas.sh

  2. When you see this line, you are done.

    text
    OK: MongoDB Atlas added to Spitfire, first metrics within ~5 min
Note

In the product (the Quick install tab) the command comes ready with your address and a fresh token; the token is valid for 1 hour and adds one monitored system. The script asks for the public and the private key, tries them on the Atlas Admin API (digest auth) and adds the source.

Manual install

  1. Try the key.

    bash
    curl -fsS --digest -u '<public-key>:<private-key>' -H 'Accept: application/vnd.atlas.2023-01-01+json' https://cloud.mongodb.com/api/atlas/v2/orgs >/dev/null && echo 'OK: the Atlas key works'

  2. Add it in Infrastructure sources → MongoDB Atlas → Manual install.

Success check

  1. Kubernetes: the collector pod is Running in kubectl -n spitfire get pods.
  2. Docker: docker ps --filter name=spitfire-docker-collector shows it Up.
  3. In the sources table Last data is filled (cloud accounts: within one interval, 5 minutes by default, plus up to 2 minutes the provider needs to publish a minute).
  4. Dashboard shows the panels of the services you use; findings appear under the dashboard.