SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Infrastructure — Elasticsearch and OpenSearch

Elasticsearch

Quick install

  1. Copy the command, paste it into the terminal, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/elasticsearch.sh?t=<token>" -o /tmp/sf-elasticsearch.sh && bash /tmp/sf-elasticsearch.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Create the read-only role.
bash
curl -u elastic -X PUT 'http://<host>:9200/_security/role/spitfire_monitor' -H 'Content-Type: application/json' -d '{"cluster":["monitor"],"indices":[{"names":["*"],"privileges":["monitor"]}]}'
  1. Create the monitor user.
bash
curl -u elastic -X PUT 'http://<host>:9200/_security/user/spitfire_monitor' -H 'Content-Type: application/json' -d '{"password":"<password>","roles":["spitfire_monitor"]}'
  1. Test the connection.
bash
curl -fsS -u 'spitfire_monitor:<password>' 'http://<host>:9200/_cluster/health' >/dev/null && echo 'OK: spitfire_monitor connected'
  1. Add it in Spitfire → Infrastructure sources → Elasticsearch.
text
http://<host>:9200   spitfire_monitor   <password>

OpenSearch (Beta)

Quick install

  1. Copy the command, paste it into the terminal, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/opensearch.sh?t=<token>" -o /tmp/sf-opensearch.sh && bash /tmp/sf-opensearch.sh
  1. When you see OK: ..., you are done.

Manual install

  1. Create the read-only role.
bash
curl -u admin -X PUT 'http://<host>:9200/_plugins/_security/api/roles/spitfire_monitor' -H 'Content-Type: application/json' -d '{"cluster_permissions":["cluster_monitor"],"index_permissions":[{"index_patterns":["*"],"allowed_actions":["indices_monitor"]}]}'
  1. Create the monitor user.
bash
curl -u admin -X PUT 'http://<host>:9200/_plugins/_security/api/internalusers/spitfire_monitor' -H 'Content-Type: application/json' -d '{"password":"<password>"}'
  1. Map the user to the role.
bash
curl -u admin -X PUT 'http://<host>:9200/_plugins/_security/api/rolesmapping/spitfire_monitor' -H 'Content-Type: application/json' -d '{"users":["spitfire_monitor"]}'
  1. Test the connection.
bash
curl -fsS -u 'spitfire_monitor:<password>' 'http://<host>:9200/_cluster/health' >/dev/null && echo 'OK: spitfire_monitor connected'
  1. Add it in Spitfire → Infrastructure sources → OpenSearch.
text
http://<host>:9200   spitfire_monitor   <password>

AWS OpenSearch Service

  1. Turn on Fine-grained access control (internal user database).
text
AWS console → OpenSearch → <domain> → Security configuration → Fine-grained access control: on, internal user database
  1. Run OpenSearch manual steps 1–3 as the master user.
bash
curl -u '<master-user>' -X PUT 'https://search-<domain>.<region>.es.amazonaws.com/_plugins/_security/api/internalusers/spitfire_monitor' -H 'Content-Type: application/json' -d '{"password":"<password>"}'
  1. Test the connection.
bash
curl -fsS -u 'spitfire_monitor:<password>' 'https://search-<domain>.<region>.es.amazonaws.com:443/_cluster/health' >/dev/null && echo 'OK: spitfire_monitor connected'
  1. Add it as OpenSearch with this address.
text
https://search-<domain>.<region>.es.amazonaws.com:443   spitfire_monitor   <password>

Dashboard

Cluster health, unassigned / initializing / relocating shards, pending cluster tasks, JVM heap %, GC time, free disk %, indexing and search rate and latency, thread pool rejections and queue (write, search), circuit breaker trips, query cache hit ratio and the Largest indices table (store size and documents, top 10).

Threshold findings

Finding Severity
Cluster health is yellow Warning
Cluster health is red Critical
Unassigned shards Warning
More than 20 pending cluster tasks Warning
JVM heap above 85% Warning
JVM heap above 95% Critical
Free disk below 15% Warning
Free disk below 10% Critical
Thread pool rejections in the last 15 minutes Warning
Indexing pressure rejected writes Warning
A circuit breaker tripped Warning
Over 30 s of old-generation GC in 15 minutes Warning