Connecting Kubernetes: a Tower in the cluster, no port opened

This page connects a Kubernetes cluster to Spitfire end to end: the nodes' server metrics (CPU, memory, disk, network), the kubelet's metrics (pods and containers) and the cluster's state (Deployments, DaemonSets, node conditions). The Tower runs inside the cluster and opens one outbound connection to Spitfire: no port is opened to the cluster from outside. The data source's fields and the cluster view: Infrastructure: Kubernetes.

#Prerequisites

• Continuous metrics on: Towers & metrics In Spitfire: /towers → Collect and keep metrics (Turning metrics on). While it is off the Tower takes metrics in and drops them; the manifest applies cleanly and nothing shows up.
• A Tower: this page installs it inside the cluster (A Tower in the cluster). If your network already has a Tower the cluster's pods can reach, you can use that one.
• kubectl access: a user who can create namespaces, ClusterRoles and ClusterRoleBindings (usually cluster-admin). Kubernetes 1.25 or later (Pod Security labels).
• Outbound access: the Tower pod must reach Spitfire's address (HTTPS). Nothing else needs internet access.
• Installation admin: the Towers page is open to the installation admin only.

#What gets installed

All in the spitfire namespace; the ClusterRole and the ClusterRoleBinding are cluster-wide.

Pod Security: with the host Collector on, the namespace is enforce: privileged, because reading the node's root, network and processes (hostPath, hostNetwork, hostPID) exists at no other level. warn and audit stay restricted; every pod other than the host Collector (the Tower, the kubelet and cluster Collectors) meets restricted: a non-root user, RuntimeDefault seccomp, a read-only root file system, every capability dropped. With the host Collector off, the namespace is enforce: restricted.

#A Tower in the cluster

1. Towers & metrics In Spitfire: /towers → Add Tower → a name (e.g. prod-eu-k8s) and an optional zone → Add Tower.

2. The two tokens are shown only in this dialog. Keep it open through the next steps.

3. Choose the Kubernetes (manifest) tab.

4. The Also watch every node as a server box: on, the nodes show up in Servers (the namespace becomes privileged, above). If this Tower carries the mobile push notifications, tick that box too.

5. Copy the 1. The namespace and the Secrets command into a terminal that reaches the cluster. It asks for a token twice: first paste sftower_… from the dialog, then sfcol_…, Enter after each.

6. Save 2. spitfire-tower.yaml in your home folder as spitfire-tower.yaml.

7. Apply it:

   bashCopy

   kubectl apply --dry-run=server -f spitfire-tower.yaml && kubectl apply -f spitfire-tower.yaml

   --dry-run=server asks the server first: a Pod Security or permission refusal shows up here, before anything changes.

8. The Tower turns Connected in the list (a few seconds). The address under the tab (http://spitfire-tower.spitfire.svc:4318) is the data source's Tower address.

The NetworkPolicy lets in 4318 only. With the host Collector off, only pods in the spitfire namespace can send. With it on, the host Collector sends from its node's address (which no pod selector matches), so the port is open to the cluster and the sfcol_ token guards it. A CNI that does not enforce NetworkPolicies ignores the rule.

#Secrets from a hidden prompt

The command asks for the tokens with read -rs: they are not shown and stay out of the shell history and the manifest. They reach kubectl on standard input (--from-file=…=/dev/stdin); --from-literal leaves the value on the command line, in the process list and in the history. The command is one logical line (lines end in \): when a multi-line block is pasted, read takes the next pasted line for the token. To type it yourself:

kubectl create namespace spitfire
printf 'sftower_ token: ' && read -rs SF_TOWER && echo && \
printf 'sfcol_ token: ' && read -rs SF_COL && echo && \
printf '%s' "$SF_TOWER" | kubectl -n spitfire create secret generic spitfire-tower --from-file=SPITFIRE_TOWER_TOKEN=/dev/stdin && \
printf '%s' "$SF_COL" | kubectl -n spitfire create secret generic spitfire-collector --from-file=token=/dev/stdin; \
unset SF_TOWER SF_COL

The manifest holds no token, so it can be kept in a repository (GitOps); the Secrets are managed apart.

#Pinning the image to a digest

The manifest names the image by tag (usespitfire/spitfire:<version>, and <version>-collector for the host Collector); the version must match Spitfire's. For a digest instead of the tag:

docker buildx imagetools inspect usespitfire/spitfire:<version> --format '{{json .Manifest.Digest}}'
docker buildx imagetools inspect usespitfire/spitfire:<version>-collector --format '{{json .Manifest.Digest}}'

and change the manifest's image: lines to usespitfire/spitfire:<version>@sha256:<digest>. When upgrading, take the new manifest from the dialog and compare with kubectl diff -f spitfire-tower.yaml.

#The Kubernetes data source

1. Infrastructure sources → Add a source → Kubernetes. If a prerequisite is missing the dialog says so at the top.

2. The Manual install tab: Cluster name (e.g. prod-eu; the cluster's name in Spitfire) and Tower address http://spitfire-tower.spitfire.svc:4318 → Save.

3. Skip the Secret command: the spitfire-collector Secret was made in the Tower step. (If you send to another Tower in your network, run it; it asks for that Tower's sfcol_ token.)

4. Copy the manifest command with Copy and run it in the same terminal. It is one command (kubectl apply -f - <<'EOF' … EOF); no file is saved. Then wait for the rollout:

   bashCopy

   kubectl -n spitfire rollout status daemonset/spitfire-collector-node --timeout=120s

5. Test: if it says "Collected from inside the cluster", wait; the first data comes in 1-2 minutes.

The Quick install tab does the same: Create the command gives a command with a one-hour, single-use sfsetup_ token; the script asks for the kubectl context, the cluster name, the Tower address and the sfcol_ token, updates an existing Secret and applies the manifest. The token is made only when you press the button; opening and closing the dialog leaves no valid token behind.

#Verification

1. The pods are Running: one spitfire-tower, one spitfire-collector-node on every node (and one spitfire-collector-host with the host Collector on), one spitfire-collector-cluster.

   bashCopy

   kubectl -n spitfire get pods -o wide

2. The Tower is connected: its log has connected to controller.

   bashCopy

   kubectl -n spitfire logs deploy/spitfire-tower | grep -E 'connected|unreachable|not kept'

3. No no such host, connection refused, forbidden or 401 in the Collectors' logs:

   bashCopy

   kubectl -n spitfire logs ds/spitfire-collector-node --tail=50
   kubectl -n spitfire logs deploy/spitfire-collector-cluster --tail=50

4. In Spitfire: in Towers & metrics the Tower is Connected and its traffic grows; in Infrastructure sources the Kubernetes source is green; APM → Kubernetes shows the nodes and namespaces; with the host Collector on, Servers shows the node names.

#Common errors

#Undoing it

1. In Spitfire: delete the Kubernetes source in Infrastructure sources, and Revoke the Tower in Towers & metrics. Both tokens stop working.

2. In the cluster, delete the manifests (the ClusterRole, the ClusterRoleBinding and the namespace too; the Secrets go with the namespace):

   bashCopy

   kubectl delete namespace spitfire
   kubectl delete clusterrolebinding spitfire-collector --ignore-not-found
   kubectl delete clusterrole spitfire-collector --ignore-not-found

   Expected output: namespace "spitfire" deleted, then deleted for the ClusterRole and the ClusterRoleBinding. If you installed into another namespace, write it instead of spitfire.

Past metrics are deleted when their retention runs out.

#Related pages

• Infrastructure: Kubernetes
• Tower and continuous metrics
• Server monitoring
• APM