SpitfireLoad Testing & APMWhat's new

Infrastructure: OpenShift (Beta)

On an OpenShift cluster the Kubernetes data source also shows what only OpenShift has: ClusterVersion and updates, ClusterOperators, routes and their certificates, projects, DeploymentConfigs and builds, MachineConfigPools and the firing alerts of cluster monitoring. The Tower reads the OpenShift API every 60 seconds with a service account token that only has get, list and watch; nothing on the cluster changes.

Note

This view is Beta: it was tested against recorded OpenShift API responses, not yet against a live cluster.

Page: All integrations In Spitfire: /admin/integrations#cloud → Dashboard on your Kubernetes source's row → Cluster view.

Quick install

  1. Run the Kubernetes command with oc or kubectl logged in to the cluster, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/kubernetes.sh?t=<token>" -o /tmp/sf-kubernetes.sh && bash /tmp/sf-kubernetes.sh
  1. The script asks This cluster is OpenShift. Turn on the OpenShift view too?; answer Y (or Enter). It applies the access below, reads the token and the API's CA from the cluster and saves them encrypted.
  2. When you see OK: ..., you are done. The OpenShift part appears in the cluster view within ~1 min.

Manual install

Run steps 1–5 on a machine where oc is logged in to the cluster as a cluster admin.

  1. Check that the cluster is OpenShift.
bash
oc api-resources --api-group=config.openshift.io -o name | grep -q clusterversions && echo 'OK: OpenShift'
  1. Create the read-only access: the spitfire namespace, the ClusterRole spitfire-openshift, the service account and its token Secret, the binding for cluster monitoring's alerts, and the nonroot-v2 SCC for Spitfire's Collector and Tower (they run as user 10001, which the default restricted-v2 SCC refuses).
bash
oc create namespace spitfire --dry-run=client -o yaml | oc apply -f - && oc apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: spitfire-openshift}
rules:
  - apiGroups: [config.openshift.io]
    resources: [clusterversions, clusteroperators]
    verbs: [get, list, watch]
  - apiGroups: [route.openshift.io]
    resources: [routes]
    verbs: [get, list, watch]
  - apiGroups: [project.openshift.io]
    resources: [projects]
    verbs: [get, list, watch]
  - apiGroups: [""]
    resources: [namespaces]
    verbs: [get, list, watch]
  - apiGroups: [apps.openshift.io]
    resources: [deploymentconfigs]
    verbs: [get, list, watch]
  - apiGroups: [build.openshift.io]
    resources: [builds]
    verbs: [get, list, watch]
  - apiGroups: [machineconfiguration.openshift.io]
    resources: [machineconfigpools]
    verbs: [get, list, watch]
  - apiGroups: [discovery.k8s.io]
    resources: [endpointslices]
    verbs: [get, list, watch]
  - apiGroups: [""]
    resources: [services]
    verbs: [get, list, watch]
  - apiGroups: [networking.k8s.io]
    resources: [ingresses]
    verbs: [get, list, watch]
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: spitfire-openshift, namespace: spitfire}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: spitfire-openshift}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: spitfire-openshift}
subjects:
  - {kind: ServiceAccount, name: spitfire-openshift, namespace: spitfire}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: spitfire-openshift-monitoring}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: cluster-monitoring-view}
subjects:
  - {kind: ServiceAccount, name: spitfire-openshift, namespace: spitfire}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: spitfire-scc-nonroot-v2, namespace: spitfire}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: "system:openshift:scc:nonroot-v2"}
subjects:
  - {kind: ServiceAccount, name: spitfire-collector, namespace: spitfire}
  - {kind: ServiceAccount, name: spitfire-tower, namespace: spitfire}
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: spitfire-openshift-token
  namespace: spitfire
  annotations: {kubernetes.io/service-account.name: spitfire-openshift}
EOF
  1. Check the access.
bash
oc auth can-i list clusteroperators.config.openshift.io --as=system:serviceaccount:spitfire:spitfire-openshift | grep -q yes && oc auth can-i delete routes.route.openshift.io --as=system:serviceaccount:spitfire:spitfire-openshift -A | grep -q no && echo 'OK: read-only access'
  1. Print the token (copy it for step 7).
bash
oc -n spitfire get secret spitfire-openshift-token -o jsonpath='{.data.token}' | base64 -d | tee /dev/stderr | grep -q . && echo && echo 'OK: token'
  1. Print the API address and its CA certificate (copy both for step 7).
bash
oc whoami --show-server && oc -n openshift-config-managed get configmap kube-apiserver-server-ca -o jsonpath='{.data.ca-bundle\.crt}' | tee /dev/stderr | grep -q 'BEGIN CERTIFICATE' && echo 'OK: API address and CA'
  1. On the Tower's machine, check that the Tower reaches the API with the token. Put the token and the API address from steps 4 and 5 in place of <token> and <api>. -k is only for this check; Spitfire verifies the certificate with the CA.
bash
curl -fsSk -H 'Authorization: Bearer <token>' '<api>/apis/config.openshift.io/v1/clusterversions/version' | grep -q ClusterVersion && echo 'OK: the Tower reaches the OpenShift API'
  1. Spitfire → All integrations → your Kubernetes source → Edit (or Add a source → Kubernetes for a new cluster).
    • OpenShift API address: the address from step 5, e.g. https://api.ocp.corp:6443
    • OpenShift service account token: the token from step 4
    • API CA certificate: the PEM from step 5 (empty: the system's CAs)
    • Tower: the Tower that reaches the API (step 6)
  2. Test connection → Save. The test says OpenShift API ok when the token reads every part. Within ~1 min the Cluster view shows the OpenShift part.
Warning

Apply the Collector manifest only after step 2: without the nonroot-v2 SCC its pods are refused (unable to validate against any security context constraint). If you turned on Also watch every node as a server, that DaemonSet needs the privileged SCC: oc adm policy add-scc-to-user privileged -z spitfire-collector -n spitfire.

The token in the Secret does not expire; delete the Secret to revoke it. A short-lived token works too (oc -n spitfire create token spitfire-openshift --duration=8760h), but it must be replaced before it ends.

What is read

Part API Without the right
ClusterVersion, ClusterOperators config.openshift.io not shown, the test fails
Routes and their services' endpoints route.openshift.io, discovery.k8s.io not shown, the test fails
Projects project.openshift.io not counted, the test fails
DeploymentConfigs, builds apps.openshift.io, build.openshift.io not shown, the test fails
MachineConfigPools machineconfiguration.openshift.io not shown, the test fails
Alerts route openshift-monitoring/thanos-querier, /api/v1/alerts not shown; optional, the test passes

When the route has no certificate (the router keeps it), the Tower opens a TLS connection to the route's host on 443 once an hour and reads its end date. Each read is bounded (45 s, at most 500 routes and 2000 builds); an unreachable API is read again at the next minute, nothing is buffered.

Findings

Finding When Severity
ClusterOperator not available Available = False critical
ClusterOperator degraded Degraded = True critical
Cluster update failing ClusterVersion Failing = True critical
MachineConfigPool degraded Degraded = True or a degraded machine critical
Route not admitted the router did not admit it (e.g. HostAlreadyClaimed) warning
Route certificate ending under 14 days; under 3 days warning; critical
Build failed the latest build of a BuildConfig is Failed or Error warning

Firing alerts are shown, not turned into findings; route them with event policies.

Common errors

  • Test: auth: the token is wrong or was revoked. Run step 4 again and paste the token.
  • Test: permission … spitfire-openshift: the ClusterRole or its binding is missing. Run step 2 again.
  • Test: tls: the API's certificate is from an internal CA. Paste the CA from step 5.
  • The alerts line says "not read": the spitfire-openshift-monitoring binding is missing, or the Tower does not reach the thanos-querier route's host. The rest works without it.

Undoing it

Only turning off the OpenShift view (Kubernetes metrics go on):

  1. Delete the OpenShift objects. It should print OK: removed.
bash
oc delete clusterrolebinding spitfire-openshift spitfire-openshift-monitoring --ignore-not-found && oc delete clusterrole spitfire-openshift --ignore-not-found && oc -n spitfire delete secret/spitfire-openshift-token serviceaccount/spitfire-openshift rolebinding/spitfire-scc-nonroot-v2 --ignore-not-found && echo 'OK: removed'
  1. On All integrations In Spitfire: /admin/integrations#cloud, open the Kubernetes source, clear the three OpenShift fields and save.

Removing everything:

  1. Run step 1 above.
  2. Remove the Kubernetes source and the Collector as in Connecting Kubernetes → Undoing it (deleting the source in Spitfire included).
  3. Delete the file the script left.
bash
rm -f ~/.spitfire-setup/kubernetes.state && echo 'OK: local files removed'