Version: 1.4.9This documentation is for Spitfire 1.4.9.

Infrastructure — databases and caches

The Infrastructure sources page collects metrics from your databases, queues, search, caches and web servers through a Tower. Each source comes with a built-in dashboard and simple threshold findings. This page covers PostgreSQL, MySQL, SQL Server, Oracle (Beta), InfluxDB (Beta), ClickHouse (Beta), MongoDB, CouchDB, Redis, Couchbase (Beta) and Cassandra (Beta). The others: queues, search, web servers, cloud and Kubernetes.

Page: Infrastructure sources In Spitfire: /data-sources (installation admins only).

Requirements

At least one Tower installed and connected. Its Collector runs with SPITFIRE_COLLECTOR_MANAGED=1 (on in the Tower and Docker compose files).
Metrics collection is on in Towers & metrics.
The Tower machine can reach the database port.

Adding a source

Before the first source the catalog opens directly; after that the added sources are on the list page and the catalog on the Add a source page.

Quick install

Infrastructure sources → Add a source → click the technology's card.
On the Quick install tab, pick your operating system.
Copy the command with Copy, run it on the server and answer the questions.
When OK: ... shows, Waiting for install… turns green; Go to the dashboard.

Note

The script checks the prerequisites, creates the read-only user, tests the connection and adds the source. The admin password is used only in that session and is never sent to Spitfire. Use Download the file to read it first. If it stops halfway, running the same command again is safe.

Manual install

Infrastructure sources → Add a source → the technology's card → the Manual install tab.
Step 1: copy the command with Copy and run it on the database as an administrator. Replace <password> with a password you choose.
Step 2: enter the address (host:port), the user (spitfire_monitor) and the password.
Step 3: Test. The test runs on the Tower; without a Tower, on the Spitfire server.
Step 4: Save. The dashboard opens.

Read-only users

PostgreSQL

Quick install

Run the command on the database server, answer the questions.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/postgresql.sh?t=<token>" -o /tmp/sf-postgresql.sh && bash /tmp/sf-postgresql.sh

When you see this line, you are done.

text

OK: PostgreSQL added to Spitfire, first metrics within ~1 min

Note

In the product (the Quick install tab) the command comes ready with your address and a fresh token; the token is valid for 1 hour and adds one monitored system. If PostgreSQL runs in a Docker container the script asks for its name; when spitfire_monitor exists it only renews its password.

Manual install

Create the read-only user.

bash

psql -h <host> -U postgres -c "CREATE ROLE spitfire_monitor WITH LOGIN PASSWORD '<password>';" -c "GRANT pg_monitor TO spitfire_monitor;"

Test the connection.

bash

psql "host=<host> user=spitfire_monitor dbname=postgres" -c "select 'OK: spitfire_monitor connected';"

Add it in Infrastructure sources → PostgreSQL → Manual install.

MySQL

Quick install

Run the command on the database server, answer the questions.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/mysql.sh?t=<token>" -o /tmp/sf-mysql.sh && bash /tmp/sf-mysql.sh

When you see this line, you are done.

text

OK: MySQL added to Spitfire, first metrics within ~1 min

Note

In the product (the Quick install tab) the command comes ready with your address and a fresh token; the token is valid for 1 hour and adds one monitored system. The same works for MariaDB.

Manual install

Create the read-only user.

bash

mysql -h <host> -u root -p -e "CREATE USER 'spitfire_monitor'@'%' IDENTIFIED BY '<password>'; GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'spitfire_monitor'@'%'; GRANT SELECT ON performance_schema.* TO 'spitfire_monitor'@'%';"

Test the connection.

bash

mysql -h <host> -u spitfire_monitor -p -N -e "SELECT 'OK: spitfire_monitor connected'"

Add it in Infrastructure sources → MySQL → Manual install.

SQL Server

Quick install

Run the command on the SQL Server machine, answer the questions.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/sqlserver.sh?t=<token>" -o /tmp/sf-sqlserver.sh && bash /tmp/sf-sqlserver.sh

On Windows, run this in PowerShell instead.

powershell

[Net.ServicePointManager]::SecurityProtocol = 'Tls12'; irm "https://<spitfire>/api/v1/setup/ds/sqlserver.ps1?t=<token>" -OutFile "$env:TEMP\sf-sqlserver.ps1"; powershell -ExecutionPolicy Bypass -File "$env:TEMP\sf-sqlserver.ps1"

When you see this line, you are done.

text

OK: SQL Server added to Spitfire, first metrics within ~1 min

Note

In the product (the Quick install tab) the command comes ready with your address and a fresh token; the token is valid for 1 hour and adds one monitored system. On Windows the script connects with Windows authentication or an SQL login. Password policy: 8 or more characters with upper case, lower case and digits.

Manual install

Create the read-only login.

bash

sqlcmd -S <host>,1433 -U sa -C -Q "CREATE LOGIN spitfire_monitor WITH PASSWORD = '<password>'; GRANT VIEW SERVER STATE TO spitfire_monitor; GRANT VIEW ANY DATABASE TO spitfire_monitor;"

Test the connection.

bash

sqlcmd -S <host>,1433 -U spitfire_monitor -C -h -1 -Q "SET NOCOUNT ON; SELECT 'OK: spitfire_monitor connected'"

Add it in Infrastructure sources → SQL Server → Manual install.

Oracle (Beta)

Quick install (on the database host, as the oracle user)

Copy the command, paste it into the terminal and run it.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/oracledb.sh?t=<token>" -o /tmp/sf-oracledb.sh && bash /tmp/sf-oracledb.sh

On Windows, run this in PowerShell instead.

powershell

Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/oracledb.ps1?t=<token>" -OutFile "$env:TEMP\sf-oracledb.ps1"; & "$env:TEMP\sf-oracledb.ps1"

When you see OK: oracledb ..., you are done.

Note

In the product UI these lines come ready, with your address and a fresh token. The script asks the PDB, creates spitfire_monitor (an existing one gets the new password), grants the views below and tests the login. The Oracle card has a Beta badge: its receiver is alpha in Collector contrib v0.164.1, so metric names may change.

Manual install

Connect as SYS.
bash
```
sqlplus / as sysdba
```
Switch to your PDB.
sql
```
ALTER SESSION SET CONTAINER=FREEPDB1;
```

Create the read-only user.

sql

CREATE USER spitfire_monitor IDENTIFIED BY "<password>";
GRANT CREATE SESSION TO spitfire_monitor;
GRANT SELECT ON V_$INSTANCE TO spitfire_monitor;
GRANT SELECT ON V_$DATABASE TO spitfire_monitor;
GRANT SELECT ON V_$SESSION TO spitfire_monitor;
GRANT SELECT ON V_$SYSSTAT TO spitfire_monitor;
GRANT SELECT ON V_$RESOURCE_LIMIT TO spitfire_monitor;
GRANT SELECT ON V_$OSSTAT TO spitfire_monitor;
GRANT SELECT ON V_$SGAINFO TO spitfire_monitor;
GRANT SELECT ON V_$PROCESS TO spitfire_monitor;
GRANT SELECT ON V_$PARAMETER TO spitfire_monitor;
GRANT SELECT ON V_$TRANSACTION TO spitfire_monitor;
GRANT SELECT ON V_$DATAFILE TO spitfire_monitor;
GRANT SELECT ON V_$PDBS TO spitfire_monitor;
GRANT SELECT ON V_$CONTAINERS TO spitfire_monitor;
GRANT SELECT ON V_$LOCK TO spitfire_monitor;
GRANT SELECT ON V_$SYSMETRIC TO spitfire_monitor;
GRANT SELECT ON V_$CON_SYSMETRIC TO spitfire_monitor;
GRANT SELECT ON V_$CON_SYSSTAT TO spitfire_monitor;
GRANT SELECT ON V_$ROWCACHE TO spitfire_monitor;
GRANT SELECT ON DBA_TABLESPACES TO spitfire_monitor;
GRANT SELECT ON DBA_DATA_FILES TO spitfire_monitor;
GRANT SELECT ON DBA_TABLESPACE_USAGE_METRICS TO spitfire_monitor;
GRANT SELECT ON DBA_FREE_SPACE TO spitfire_monitor;
GRANT SELECT ON DBA_RECYCLEBIN TO spitfire_monitor;

Grant the top SQL views.

sql

GRANT SELECT ON V_$SQLSTATS TO spitfire_monitor;
GRANT SELECT ON V_$SQL TO spitfire_monitor;
GRANT SELECT ON DBA_OBJECTS TO spitfire_monitor;

Test the login.

bash

echo "SELECT 'OK: spitfire_monitor' FROM dual;" | sqlplus -s -L 'spitfire_monitor/"<password>"@//<host>:1521/FREEPDB1'

Add it under Infrastructure sources → Oracle, address <host>:1521. Service name: FREEPDB1.

Note

Put your PDB instead of FREEPDB1. V_$SQLSTATS and V_$SQL fill the top SQL list of the live view and the Slow queries card; DBA_OBJECTS names the table a lock wait is on. Statements are shown without their values.

InfluxDB (Beta)

The InfluxDB card has a Beta badge: the Collector reads InfluxDB's own /metrics with the prometheus receiver and maps the 2.x and 3.x metrics onto one dashboard. 2.x and 3.x are supported, 1.x is not.

InfluxDB 2.x (port 8086):

On any machine that reaches InfluxDB, create a read-only token (put your operator or all-access token in place of <admin-token> and your organization name in place of <org>):
bash
```
influx auth create --host http://<host>:8086 --token '<admin-token>' --org '<org>' --read-buckets --read-orgs --description spitfire_monitor
```
Copy the token value it prints.
In step 2 enter the address (http://<host>:8086), API token and Organization.
Test: the result should be Connected.

InfluxDB 3.x (port 8181):

Start the server with its health and metrics addresses open (with -e on Docker, as an environment variable in a service file):
text
```
INFLUXDB3_DISABLE_AUTHZ=health,ping,metrics
```

Check that they are open; it should print HTTP 200:

bash

curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://<host>:8181/metrics

In step 2 enter the address (http://<host>:8181) and leave API token empty.
Test: the result should be Connected.

MongoDB

Quick install (on any machine with mongosh)

Copy the command, paste it into the terminal and run it.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/mongodb.sh?t=<token>" -o /tmp/sf-mongodb.sh && bash /tmp/sf-mongodb.sh

On Windows, run this in PowerShell instead.

powershell

Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/mongodb.ps1?t=<token>" -OutFile "$env:TEMP\sf-mongodb.ps1"; & "$env:TEMP\sf-mongodb.ps1"

When you see OK: mongodb ..., you are done.

Note

In the product UI these lines come ready, with your address and a fresh token. The script connects to the PRIMARY, creates spitfire_monitor with clusterMonitor (an existing one gets the new password), optionally turns profiling on in the databases you name, and tests the login.

Manual install

Connect to the PRIMARY as admin.

bash

mongosh "mongodb://<host>:27017/admin?directConnection=true" -u admin -p

Create the read-only user.

javascript

db.getSiblingDB("admin").createUser({user: "spitfire_monitor", pwd: "<password>", roles: [{role: "clusterMonitor", db: "admin"}]})

Optional: turn profiling on.

javascript

db.getSiblingDB("<database>").setProfilingLevel(1, {slowms: 100})

Test the login.

bash

mongosh --quiet "mongodb://<host>:27017/admin?directConnection=true" -u spitfire_monitor -p '<password>' --eval 'print("OK: serverStatus " + db.adminCommand({serverStatus: 1}).ok)'

Add it under Infrastructure sources → MongoDB, address <host>:27017.

Note

clusterMonitor reads the replica set state, the oplog window, running operations and system.profile; it cannot read your collections. Profiling (step 3) fills the profiler list of the live view and the slow operations of the Slow queries card; leave it out if you do not want it.

CouchDB

Quick install

Run the command on a machine that reaches CouchDB, answer the questions.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/couchdb.sh?t=<token>" -o /tmp/sf-couchdb.sh && bash /tmp/sf-couchdb.sh

When you see this line, you are done.

text

OK: CouchDB added to Spitfire, first metrics within ~1 min

Note

In the product (the Quick install tab) the command comes ready with your address and a fresh token; the token is valid for 1 hour and adds one monitored system. CouchDB's statistics endpoint (_node/_local/_stats) needs an admin, so the script defines spitfire_monitor as one.

Manual install

Create the monitor user.

bash

curl -u admin -X PUT 'http://<host>:5984/_node/_local/_config/admins/spitfire_monitor' -d '"<password>"'

Test the connection.

bash

curl -fsS -u spitfire_monitor 'http://<host>:5984/_node/_local/_stats/couchdb/request_time' >/dev/null && echo 'OK: spitfire_monitor connected'

Add it in Infrastructure sources → CouchDB → Manual install.

Redis

Quick install (on any machine with redis-cli)

Copy the command, paste it into the terminal and run it.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/redis.sh?t=<token>" -o /tmp/sf-redis.sh && bash /tmp/sf-redis.sh

When you see OK: redis ..., you are done.

Note

In the product UI these lines come ready, with your address and a fresh token. The script creates the ACL user spitfire_monitor (ACL SETUSER ... reset rewrites an existing one), asks before turning the slowlog on when it is off, and tests the login.

Manual install

Create the read-only ACL user.

bash

redis-cli -h <host> -p 6379 --askpass ACL SETUSER spitfire_monitor reset on '><password>' -@all +info +ping +slowlog

Repeat it on every replica.

bash

redis-cli -h <replica-host> -p 6379 --askpass ACL SETUSER spitfire_monitor reset on '><password>' -@all +info +ping +slowlog

Test the login.

bash

redis-cli -h <host> -p 6379 --user spitfire_monitor --pass '<password>' --no-auth-warning SLOWLOG LEN && echo "OK: spitfire_monitor"

Add it under Infrastructure sources → Redis, address <host>:6379.

Note

+slowlog fills the slowlog list of the live view; commands are shown without keys or values (SET ? ?). ACLs are not replicated, so step 2 is needed on each replica.

Couchbase (Beta)

Quick install (on a Couchbase node)

Copy the command, paste it into the terminal and run it.

bash

curl -fsSL "https://<spitfire>/api/v1/setup/ds/couchbase.sh?t=<token>" -o /tmp/sf-couchbase.sh && bash /tmp/sf-couchbase.sh

On Windows, run this in PowerShell instead.

powershell

Invoke-WebRequest "https://<spitfire>/api/v1/setup/ds/couchbase.ps1?t=<token>" -OutFile "$env:TEMP\sf-couchbase.ps1"; & "$env:TEMP\sf-couchbase.ps1"

When you see OK: couchbase ..., you are done.

Note

In the product UI these lines come ready, with your address and a fresh token. The script creates (or updates) the local user spitfire_monitor with external_stats_reader and checks that it reads /metrics. The Couchbase card has a Beta badge: the metrics come from Couchbase Server's own Prometheus endpoint (7.0 and later), so names may change with Couchbase releases.

Manual install

Create the stats-only user.

bash

/opt/couchbase/bin/couchbase-cli user-manage -c localhost -u Administrator -p '<admin-password>' --set --rbac-username spitfire_monitor --rbac-password '<password>' --roles external_stats_reader --auth-domain local

Test it from the Tower machine.

bash

curl -fsS -o /dev/null -u spitfire_monitor:'<password>' http://<host>:8091/metrics && echo "OK: spitfire_monitor"

Add it under Infrastructure sources → Couchbase, address http://<host>:8091.

Note

external_stats_reader reads only /metrics; it cannot read buckets or documents. With TLS, enter https://<host>:18091 and turn Use TLS on. On Windows the CLI is C:\Program Files\Couchbase\Server\bin\couchbase-cli.exe.

ClickHouse (Beta)

The ClickHouse card has a Beta badge. This source watches your ClickHouse server; it has nothing to do with the ClickHouse where Spitfire keeps its own APM data. Metrics come from ClickHouse's own Prometheus endpoint (port 9363), the slow queries and tables from the HTTP interface (port 8123).

On the ClickHouse server, turn on the Prometheus endpoint:

bash

sudo tee /etc/clickhouse-server/config.d/spitfire-prometheus.xml >/dev/null <<'EOF'
<clickhouse>
    <prometheus>
        <endpoint>/metrics</endpoint>
        <port>9363</port>
        <metrics>true</metrics>
        <events>true</events>
        <asynchronous_metrics>true</asynchronous_metrics>
    </prometheus>
</clickhouse>
EOF
sudo systemctl restart clickhouse-server

On Docker: put the same file in the container's /etc/clickhouse-server/config.d/ and run docker restart <container> (the Docker box on the page gives the ready command).

Check (the number must be above 0):

bash

curl -s http://localhost:9363/metrics | grep -c ClickHouseProfileEvents_

Create the read-only user with clickhouse-client:

sql

CREATE USER IF NOT EXISTS spitfire_monitor IDENTIFIED BY '<password>' SETTINGS readonly = 1;
GRANT SELECT ON system.* TO spitfire_monitor;
GRANT SHOW TABLES ON *.* TO spitfire_monitor;

SHOW TABLES shows table names and part counts only; it cannot read table data.

In step 2, enter the address (http://<host>:8123), the user and the password. With an empty Prometheus endpoint, http://<host>:9363/metrics is used. With empty Databases, all are read.
Test: the result must be Connection successful.

Live view

On Oracle, Redis and MongoDB sources the dashboard has a Live view card. Read now reads, on the Tower, what no metric carries; while the page stays open it reads again every minute.

Technology	Shows
Oracle	Top SQL by total time (`V$SQLSTATS`); by mean time and calls, with lock waits, on the Slow queries card
Redis	Role, master link, replicas with their lag, last save, `latest_fork`, the slowlog
MongoDB	Replica set members and lag, oplog window, WiredTiger cache, operations running now, the profiler (last 15 minutes)

Statements are shown without their values: every literal is replaced by ? at the source.

Cassandra (Beta)

Quick install

Copy the command, run it as root on a Cassandra node, answer the questions.
bash
```
curl -fsSL "https://<spitfire>/api/v1/setup/ds/cassandra.sh?t=<token>" -o /tmp/sf-cassandra.sh && sudo bash /tmp/sf-cassandra.sh
```
(In the product this line comes ready, with the address and a fresh token.)
When you see OK: ..., you are done.

Manual install (on each node in turn, as root)

Create the folder.
bash
```
sudo mkdir -p /opt/jmx-exporter
```

Download the javaagent.

bash

sudo curl -fsSL -o /opt/jmx-exporter/jmx_prometheus_javaagent.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar

Write the rules file.

bash

sudo tee /opt/jmx-exporter/cassandra.yml >/dev/null <<'EOF'
# Spitfire: Cassandra node metrics for the Prometheus JMX exporter javaagent.
lowercaseOutputName: false
includeObjectNames:
  - "org.apache.cassandra.metrics:type=ClientRequest,scope=Read,*"
  - "org.apache.cassandra.metrics:type=ClientRequest,scope=Write,*"
  - "org.apache.cassandra.metrics:type=Compaction,name=PendingTasks"
  - "org.apache.cassandra.metrics:type=ThreadPools,*"
  - "org.apache.cassandra.metrics:type=DroppedMessage,name=Dropped,*"
  - "org.apache.cassandra.metrics:type=Table,name=LiveSSTableCount,*"
  - "org.apache.cassandra.metrics:type=Storage,*"
  - "org.apache.cassandra.net:type=FailureDetector"
  - "java.lang:type=Memory"
  - "java.lang:type=GarbageCollector,*"
rules:
  - pattern: 'org.apache.cassandra.metrics<type=ClientRequest, scope=(Read|Write), name=Latency><>99thPercentile'
    name: cassandra_client_request_latency_p99_ms
    labels:
      op: "$1"
    valueFactor: 0.001
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=ClientRequest, scope=(Read|Write), name=Latency><>Count'
    name: cassandra_client_requests
    labels:
      op: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=ClientRequest, scope=(Read|Write), name=Timeouts><>Count'
    name: cassandra_client_request_timeouts
    labels:
      op: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=ClientRequest, scope=(Read|Write), name=Unavailables><>Count'
    name: cassandra_client_request_unavailables
    labels:
      op: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=Compaction, name=PendingTasks><>Value'
    name: cassandra_compaction_pending_tasks
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=ThreadPools, path=\w+, scope=(MutationStage|ReadStage|Native-Transport-Requests|CompactionExecutor), name=PendingTasks><>Value'
    name: cassandra_threadpool_pending_tasks
    labels:
      pool: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=ThreadPools, path=\w+, scope=(MutationStage|ReadStage|Native-Transport-Requests|CompactionExecutor), name=CurrentlyBlockedTasks><>Count'
    name: cassandra_threadpool_blocked_tasks
    labels:
      pool: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=ThreadPools, path=\w+, scope=(MutationStage|ReadStage|Native-Transport-Requests|CompactionExecutor), name=TotalBlockedTasks><>Count'
    name: cassandra_threadpool_total_blocked_tasks
    labels:
      pool: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=DroppedMessage, scope=(MUTATION|COUNTER_MUTATION|READ|RANGE_SLICE|READ_REPAIR|PAGED_RANGE|REQUEST_RESPONSE|HINT|BATCH_STORE|BATCH_REMOVE|_TRACE), name=Dropped><>Count'
    name: cassandra_dropped_messages
    labels:
      message: "$1"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=Table, keyspace=((?!system)[^,]+), scope=([^,]+), name=LiveSSTableCount><>Value'
    name: cassandra_table_live_sstables
    labels:
      keyspace: "$1"
      table: "$2"
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=Storage, name=Load><>Count'
    name: cassandra_storage_load_bytes
    type: GAUGE
  - pattern: 'org.apache.cassandra.metrics<type=Storage, name=TotalHintsInProgress><>Count'
    name: cassandra_hints_in_progress
    type: GAUGE
  - pattern: 'org.apache.cassandra.net<type=FailureDetector><>DownEndpointCount'
    name: cassandra_gossip_down_endpoints
    type: GAUGE
  - pattern: 'org.apache.cassandra.net<type=FailureDetector><>UpEndpointCount'
    name: cassandra_gossip_up_endpoints
    type: GAUGE
  - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used'
    name: cassandra_jvm_heap_used_bytes
    type: GAUGE
  - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max'
    name: cassandra_jvm_heap_max_bytes
    type: GAUGE
  - pattern: 'java.lang<(?:type=GarbageCollector, name=([^,>]+)|name=([^,>]+), type=GarbageCollector)><>CollectionTime'
    name: cassandra_jvm_gc_time_ms
    labels:
      gc: "$1$2"
    type: GAUGE
EOF

Add the JVM option once.

bash

grep -q jmx_prometheus_javaagent /etc/cassandra/cassandra-env.sh || echo 'JVM_OPTS="$JVM_OPTS -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7070:/opt/jmx-exporter/cassandra.yml"' | sudo tee -a /etc/cassandra/cassandra-env.sh

Restart the node.

bash

nodetool drain && sudo systemctl restart cassandra

Wait until the node is UN.

bash

until nodetool status 2>/dev/null | grep -q '^UN'; do sleep 5; done; echo 'OK: node UN'

Check the exporter.

bash

curl -s http://localhost:7070/metrics | grep -q cassandra_gossip_up_endpoints && echo 'OK: Cassandra exporter'

Start again at step 1 on the next node.
bash
```
nodetool status
```
Add the source in Spitfire: Infrastructure sources → Cassandra, every node's address.
text
```
cass-1.corp:7070, cass-2.corp:7070, cass-3.corp:7070
```
Check access from the Tower machine.

bash

curl -s http://cass-1.corp:7070/metrics | grep -q cassandra_ && echo 'OK: the Tower reaches the exporter'

Test results

Result	What to do
The host name does not resolve	Try `nslookup <host>` on the Tower machine.
Nothing listens on that port	Check the port and the interface the database listens on.
No answer	Allow the Tower to the database port in the firewall.
Wrong user or password	Run step 1 again and re-enter the password.
The user lacks the read-only grant	Run the `GRANT` lines of step 1.
Database or service name not found	On Oracle the Service name is wrong: as SYS, find the right one with `SELECT name FROM v$services;`.
Wrong or missing token	On InfluxDB 2.x make the token of step 1 again and enter it; on 3.x check the `INFLUXDB3_DISABLE_AUTHZ` setting.
The organization, or the /health or /metrics path, is not there	On InfluxDB the Organization name is wrong: find the right one with `influx org list --host http://<host>:8086 --token '<admin-token>'`.
The answer is not InfluxDB 2.x or 3.x	InfluxDB 1.x is not supported; fix the address if it points at another service.
Something answers, but not in the expected format (Cassandra)	The exporter runs without the rules file: check steps 3 and 4 and restart the node.
TLS mismatch	Turn Use TLS on or off to match the database.
This Tower is older than the live view	Update the Tower; the dashboard keeps working meanwhile.

Slow queries and locks

Below the PostgreSQL, MySQL, SQL Server, ClickHouse, Oracle and MongoDB dashboard is the Slow queries table. Queries are read through the Tower when you open it and are not stored in Spitfire. Every value in a query text is replaced by a question mark: no customer data reaches the screen.

Total time / Mean time (on ClickHouse, Oracle and MongoDB also Most frequent) change the order.
Waiting on a lock now: the waiting session, the object it waits on and the session holding it (PostgreSQL, MySQL, SQL Server, Oracle; on MongoDB the operation and the locks it holds or asks).
On MongoDB also Running now: the operations of $currentOp, with their type and plan; a plan that reads the whole collection has a red COLLSCAN badge.
On ClickHouse also the Tables: parts and disk and Replicas tables.
If the Tower is older than 1.4 (1.4.2 for Oracle and MongoDB), an update notice shows instead of the table.

PostgreSQL: pg_stat_statements

With an empty table the page shows the missing step. If the module is not loaded:

See the setting:

bash

sudo -u postgres psql -c "SHOW shared_preload_libraries;"

If empty, set it (if not empty, write the current value followed by ,pg_stat_statements):

bash

sudo -u postgres psql -c "ALTER SYSTEM SET shared_preload_libraries = 'pg_stat_statements';"

Restart:
bash
```
sudo systemctl restart postgresql
```

Create the extension in the watched database (replace postgres with its name):

bash

sudo -u postgres psql -d postgres -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;"

Check: Refresh on the page; the table fills.

MySQL: performance_schema

It is on in MySQL 5.7+ and 8. If it is off:

Write the setting:

bash

printf '[mysqld]\nperformance_schema=ON\n' | sudo tee /etc/mysql/conf.d/spitfire-performance-schema.cnf >/dev/null

Restart:
bash
```
sudo systemctl restart mysql
```
Check: Refresh on the page; the table fills.

SQL Server and ClickHouse

SQL Server reads sys.dm_exec_query_stats and needs nothing more (VIEW SERVER STATE is granted in step 1). ClickHouse reads system.query_log, which is on by default. With an empty table:

Flush the log to disk:

bash

clickhouse-client --query "SYSTEM FLUSH LOGS"

Check (prints 1):

bash

clickhouse-client --query "EXISTS TABLE system.query_log"

Oracle: V$SQLSTATS

The list reads V$SQLSTATS (one row per statement, the totals of its plans) with the schema from V$SQL; Oracle's own schemas and the monitoring user are left out. Lock waits come from V$SESSION (blocking_session) and V$LOCK. If the user may not read them, the page shows the grants with your user and PDB:

Connect as SYS:
bash
```
sqlplus / as sysdba
```

Grant the views (put your PDB instead of FREEPDB1; if the service is not a PDB, leave out the first line):

sql

ALTER SESSION SET CONTAINER=FREEPDB1;
GRANT SELECT ON V_$SQLSTATS TO SPITFIRE_MONITOR;
GRANT SELECT ON V_$SQL TO SPITFIRE_MONITOR;
GRANT SELECT ON V_$SESSION TO SPITFIRE_MONITOR;
GRANT SELECT ON V_$LOCK TO SPITFIRE_MONITOR;
GRANT SELECT ON DBA_OBJECTS TO SPITFIRE_MONITOR;

Check: Refresh on the page; the table fills.

Note

GRANT SELECT_CATALOG_ROLE TO SPITFIRE_MONITOR; gives the same in one line, with more views than needed.

MongoDB: $currentOp and the profiler

Running now and the lock waits read $currentOp; the lists read system.profile of each database (the last 15 minutes, or the databases in the source's databases field). Each operation is shown as its shape: the command, its collection and the field names, every value is ?. The clusterMonitor role is enough; it cannot read your collections.

If the user may not run $currentOp:

Connect to the PRIMARY as admin:

bash

mongosh "mongodb://<host>:27017/admin?directConnection=true" -u admin -p

Give the role:

javascript

db.getSiblingDB("admin").grantRolesToUser("spitfire_monitor", [{role: "clusterMonitor", db: "admin"}])

If the profiler is off, only the running operations are listed and the page says so. To list the slow operations too:

Connect as in step 1 above.
Turn profiling on in the database (operations over 100 ms are recorded):
javascript
```
db.getSiblingDB("<database>").setProfilingLevel(1, {slowms: 100})
```
Check: Refresh on the page; the lists fill as slow operations run.

Cache hit ratio

The tiles above the PostgreSQL and MySQL dashboard show the last 15 minutes' cache hit ratio and its limit. Below 1,000 reads no ratio is shown.

Threshold findings

Technology	Finding
PostgreSQL	Connections above 80% of `max_connections` (warning)
PostgreSQL	Cache hit ratio below 90%, over at least 1,000 reads (warning)
PostgreSQL	Deadlocks occurred (warning)
MySQL	Buffer pool hit ratio below 90%, over at least 1,000 reads (warning)
MySQL	Row lock waits occurred (warning)
SQL Server	Deadlocks occurred (warning)
SQL Server	More than 1 lock wait per second (warning)
ClickHouse	More than 300 active parts in a partition (warning), more than 1,000 (critical)
ClickHouse	A readonly replica (critical)
ClickHouse	Replication delay above 300 seconds (warning)
ClickHouse	Failed queries rose by more than 10 (warning)
Redis	Memory fragmentation ratio above 1.5 (warning)
Redis	Keyspace hit ratio below 80%, over at least 100 lookups (warning)
Oracle	Sessions above 80% of the `sessions` limit (warning)
Oracle	Deadlocks occurred (warning)
InfluxDB	Write or query requests fail (warning)
InfluxDB	Points were rejected (warning)
InfluxDB	A bucket has more than 1,000,000 series: high cardinality (warning, 2.x)
InfluxDB	A shard's disk grew by more than 1 GiB in 15 minutes (warning, 2.x)
Oracle	A tablespace above 85% of its max size (warning), above 95% (critical)
Redis	Keys evicted (warning)
Redis	Memory above 90% of `maxmemory` (warning)
Redis	The replica lost its master link (critical, live view)
Redis	The last RDB save or AOF write failed (critical, live view)
MongoDB	A secondary more than 10 s behind (warning)
MongoDB	A member neither PRIMARY nor SECONDARY, or no PRIMARY (critical, live view)
MongoDB	WiredTiger cache above 95% (warning, live view)
MongoDB	Connections above 80% of the limit (warning, live view)
MongoDB	An operation running longer than 60 s (warning, live view)
Couchbase	The node does not answer (critical)
Couchbase	Bucket memory above 90% of its quota (warning)
Couchbase	Under 10% of active documents in memory (warning)
Couchbase	Temporary out-of-memory errors (warning)
Couchbase	XDCR backlog growing by more than 1000 in 15 minutes (warning)
Cassandra	A node's exporter does not answer, `up` = 0 (critical)
Cassandra	Gossip sees nodes down (critical)
Cassandra	Heap above 85% on a node (warning), above 95% (critical)
Cassandra	Messages were dropped (warning)
Cassandra	More than 100 pending compactions (warning)
Cassandra	Thread pool tasks were blocked (warning)
Cassandra	Read/write timeouts or unavailables (warning)

When a critical finding fires, the warning of the same threshold is not shown again.

All metrics

The dashboard's built-in panels show some of the source's metrics. The All metrics tab lists every metric the source sends; it works for Prometheus endpoints too.

Infrastructure sources → Dashboard on the source's row.
Click the All metrics tab at the top.
Type part of the metric's name in the search box, e.g. commits or http_requests.
Click the metric in the list. The chart is drawn; counters show their increase per second.
Optionally pick a value in the attribute boxes, or use Split by to draw each value as its own line.
Add to dashboard → enter a Title.
For a threshold: Threshold finding → above or below → value → Warning or Critical.
Save with Add to dashboard.
Go back to the Dashboard tab.

Success check: the chart shows below the built-in panels and is still there after a reload. With a threshold set and crossed, the findings list shows a line with the chart's title. To remove the chart: the bin icon at its top right → Remove from dashboard.

License

Each source counts as one monitored system. At the limit the page warns and links to License. More monitored systems come with the +5 monitored systems pack.

Success check

In the sources table, Last test = Connected.
Within 1-2 minutes Last data is filled and the dashboard shows lines.
The All metrics tab lists the source's metrics.

Infrastructure — databases and caches

#Requirements

#Adding a source

#Read-only users

#PostgreSQL

#MySQL

#SQL Server

#Oracle (Beta)

#InfluxDB (Beta)

#MongoDB

#CouchDB

#Redis

#Couchbase (Beta)

#ClickHouse (Beta)

#Live view

#Cassandra (Beta)

#Test results

#Slow queries and locks

#PostgreSQL: pg_stat_statements

#MySQL: performance_schema

#SQL Server and ClickHouse

#Oracle: V$SQLSTATS

#MongoDB: $currentOp and the profiler

#Cache hit ratio

#Threshold findings

#All metrics

#License

#Success check

Requirements

Adding a source

Read-only users

PostgreSQL

MySQL

SQL Server

Oracle (Beta)

InfluxDB (Beta)

MongoDB

CouchDB

Redis

Couchbase (Beta)

ClickHouse (Beta)

Live view

Cassandra (Beta)

Test results

Slow queries and locks

PostgreSQL: pg_stat_statements

MySQL: performance_schema

SQL Server and ClickHouse

Oracle: V$SQLSTATS

MongoDB: $currentOp and the profiler

Cache hit ratio

Threshold findings

All metrics

License

Success check