Version: 1.4.9This documentation is for Spitfire 1.4.9.
Infrastructure — Java application servers (Beta)
For Tomcat, WildFly, WebLogic, WebSphere Liberty and any JVM (Spring Boot etc.) Spitfire reads heap, GC,
threads, requests, the thread pool, the JDBC pool and sessions. Each server is one monitored system; one monitored
system can take several addresses, comma separated (app1:9404,app2:9404).
Tomcat
Quick install
- Run it on the Tomcat server and answer the questions.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/tomcat.sh?t=<token>&lang=en" -o /tmp/sf-tomcat.sh && bash /tmp/sf-tomcat.sh - When you see
OK: ..., you are done.
Note
The script puts the javaagent and its rules file in /opt/spitfire/jmx, adds one line to
bin/setenv.sh (backed up to setenv.sh.spitfire.bak first) and asks before restarting Tomcat.
The exporter only reads JMX. In the product this line comes ready, with the address and a fresh token.
Manual install
- Download the javaagent.
sudo mkdir -p /opt/spitfire/jmx && sudo curl -fsSL -o /opt/spitfire/jmx/jmx_prometheus_javaagent.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar - Write the rules file.
sudo tee /opt/spitfire/jmx/spitfire-tomcat.yml >/dev/null <<'SPITFIRE_YML' # Spitfire tomcat: Prometheus JMX exporter javaagent 1.0.1 rules (read-only). lowercaseOutputName: false includeObjectNames: - "java.lang:type=Memory" - "java.lang:type=GarbageCollector,*" - "java.lang:type=Threading" - "java.lang:type=ClassLoading" - "java.lang:type=OperatingSystem" - "Catalina:type=GlobalRequestProcessor,*" - "Catalina:type=ThreadPool,*" - "Catalina:type=Manager,*" - "Catalina:type=DataSource,*" - "Tomcat:type=GlobalRequestProcessor,*" - "Tomcat:type=ThreadPool,*" - "Tomcat:type=Manager,*" - "Tomcat:type=DataSource,*" - "com.zaxxer.hikari:*" rules: - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used' name: java_heap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max' name: java_heap_max_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>used' name: java_nonheap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>max' name: java_nonheap_max_bytes type: GAUGE - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionCount' name: java_gc_count type: GAUGE labels: name: "$1" - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionTime' name: java_gc_seconds type: GAUGE valueFactor: 0.001 labels: name: "$1" - pattern: 'java.lang<type=Threading><>ThreadCount' name: java_threads_live type: GAUGE - pattern: 'java.lang<type=Threading><>DaemonThreadCount' name: java_threads_daemon type: GAUGE - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount' name: java_classes_loaded type: GAUGE - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad' name: java_cpu_load type: GAUGE - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>requestCount' name: java_requests type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>errorCount' name: java_request_errors type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>processingTime' name: java_request_seconds type: GAUGE valueFactor: 0.001 labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>currentThreadsBusy' name: java_threadpool_busy type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>maxThreads' name: java_threadpool_max type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>currentThreadCount' name: java_threadpool_size type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=Manager[,>])(?=(?:[^>]*, )?context="?([^",>]+)"?[,>])(?=(?:[^>]*, )?host="?([^",>]+)"?[,>])[^>]*><>activeSessions' name: java_sessions_active type: GAUGE labels: context: "$1" host: "$2" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>numActive' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>numIdle' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>(?:maxTotal|maxActive)' name: java_jdbc_max type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>waitCount' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>ActiveConnections' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>IdleConnections' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>TotalConnections' name: java_jdbc_size type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>ThreadsAwaitingConnection' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=PoolConfig \(([^)]+)\)><>MaximumPoolSize' name: java_jdbc_max type: GAUGE labels: datasource: "$1" SPITFIRE_YML - Add the JVM option to
setenv.sh.grep -q jmx_prometheus_javaagent /opt/tomcat/bin/setenv.sh 2>/dev/null || echo 'CATALINA_OPTS="$CATALINA_OPTS -javaagent:/opt/spitfire/jmx/jmx_prometheus_javaagent.jar=9404:/opt/spitfire/jmx/spitfire-tomcat.yml"' | sudo tee -a /opt/tomcat/bin/setenv.sh - Restart Tomcat.
sudo systemctl restart tomcat - Add the monitored system: Infrastructure sources → Tomcat, address
<host>:9404. - Check the exporter.
curl -fsS http://localhost:9404/metrics | grep -q java_heap_used_bytes && echo "OK: tomcat"
Windows (PowerShell)
- Open PowerShell as Administrator and run it.
Invoke-WebRequest -UseBasicParsing -Uri "https://<spitfire>/api/v1/setup/ds/tomcat.ps1?t=<token>&lang=en" -OutFile "$env:TEMP\sf-tomcat.ps1"; powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\sf-tomcat.ps1" - When you see
OK: ..., you are done.
Manual install (Windows):
- Download the javaagent.
New-Item -ItemType Directory -Force C:\spitfire\jmx | Out-Null; Invoke-WebRequest -UseBasicParsing -Uri https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar -OutFile C:\spitfire\jmx\jmx_prometheus_javaagent.jar - Write the rules file.
$Rules = @' # Spitfire tomcat: Prometheus JMX exporter javaagent 1.0.1 rules (read-only). lowercaseOutputName: false includeObjectNames: - "java.lang:type=Memory" - "java.lang:type=GarbageCollector,*" - "java.lang:type=Threading" - "java.lang:type=ClassLoading" - "java.lang:type=OperatingSystem" - "Catalina:type=GlobalRequestProcessor,*" - "Catalina:type=ThreadPool,*" - "Catalina:type=Manager,*" - "Catalina:type=DataSource,*" - "Tomcat:type=GlobalRequestProcessor,*" - "Tomcat:type=ThreadPool,*" - "Tomcat:type=Manager,*" - "Tomcat:type=DataSource,*" - "com.zaxxer.hikari:*" rules: - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used' name: java_heap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max' name: java_heap_max_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>used' name: java_nonheap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>max' name: java_nonheap_max_bytes type: GAUGE - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionCount' name: java_gc_count type: GAUGE labels: name: "$1" - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionTime' name: java_gc_seconds type: GAUGE valueFactor: 0.001 labels: name: "$1" - pattern: 'java.lang<type=Threading><>ThreadCount' name: java_threads_live type: GAUGE - pattern: 'java.lang<type=Threading><>DaemonThreadCount' name: java_threads_daemon type: GAUGE - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount' name: java_classes_loaded type: GAUGE - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad' name: java_cpu_load type: GAUGE - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>requestCount' name: java_requests type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>errorCount' name: java_request_errors type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>processingTime' name: java_request_seconds type: GAUGE valueFactor: 0.001 labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>currentThreadsBusy' name: java_threadpool_busy type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>maxThreads' name: java_threadpool_max type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>currentThreadCount' name: java_threadpool_size type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=Manager[,>])(?=(?:[^>]*, )?context="?([^",>]+)"?[,>])(?=(?:[^>]*, )?host="?([^",>]+)"?[,>])[^>]*><>activeSessions' name: java_sessions_active type: GAUGE labels: context: "$1" host: "$2" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>numActive' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>numIdle' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>(?:maxTotal|maxActive)' name: java_jdbc_max type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>waitCount' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>ActiveConnections' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>IdleConnections' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>TotalConnections' name: java_jdbc_size type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>ThreadsAwaitingConnection' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=PoolConfig \(([^)]+)\)><>MaximumPoolSize' name: java_jdbc_max type: GAUGE labels: datasource: "$1" '@ [IO.File]::WriteAllText('C:\spitfire\jmx\spitfire-tomcat.yml', $Rules + "`n") - Add the JVM option to the Tomcat service (service name
Tomcat10).& "C:\Program Files\Apache Software Foundation\Tomcat 10.1\bin\tomcat10.exe" //US//Tomcat10 ++JvmOptions "-javaagent:C:\spitfire\jmx\jmx_prometheus_javaagent.jar=9404:C:\spitfire\jmx\spitfire-tomcat.yml" - Restart the service.
Restart-Service Tomcat10 - Check the exporter.
if ((Invoke-WebRequest -UseBasicParsing http://localhost:9404/metrics).Content -match 'java_heap_used_bytes') { 'OK: tomcat' }
WildFly
Quick install
- Run it on the WildFly server, as the user that runs WildFly.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/wildfly.sh?t=<token>&lang=en" -o /tmp/sf-wildfly.sh && bash /tmp/sf-wildfly.sh - When you see
OK: ..., you are done.
Note
WildFly already serves /metrics on its management port (9990); no javaagent is needed.
The script turns on the Undertow and data source statistics and asks before the reload. When the
Tower is on another machine it also offers to bind the management interface to 0.0.0.0; allow 9990
only to the Tower. Undertow's error count counts 5xx responses only.
Manual install
- Turn on the Undertow statistics.
/opt/wildfly/bin/jboss-cli.sh --connect --command='/subsystem=undertow:write-attribute(name=statistics-enabled,value=true)' - Turn on the data source statistics (your own data source instead of
ExampleDS)./opt/wildfly/bin/jboss-cli.sh --connect --command='/subsystem=datasources/data-source=ExampleDS:write-attribute(name=statistics-enabled,value=true)' - Reload.
/opt/wildfly/bin/jboss-cli.sh --connect --command=':reload' - Add the monitored system: Infrastructure sources → WildFly, address
<host>:9990. - Check
/metrics.curl -fsS http://localhost:9990/metrics | grep -q base_memory_usedHeap_bytes && echo "OK: wildfly"
WebLogic
Beta: tested only against emulated WebLogic MBeans, not on a real WebLogic.
Quick install
- Run it on the WebLogic server, once per server (AdminServer, managed servers).
curl -fsSL "https://<spitfire>/api/v1/setup/ds/weblogic.sh?t=<token>&lang=en" -o /tmp/sf-weblogic.sh && bash /tmp/sf-weblogic.sh - When you see
OK: ..., you are done.
Note
The exporter reads the com.bea runtime MBeans from the platform MBean server ("Platform
MBean Server Used" must be on in the domain, the default since 10.3.3). No user or password is asked.
Give each server on one machine its own exporter port (9404, 9405…).
Manual install
- Download the javaagent.
sudo mkdir -p /opt/spitfire/jmx && sudo curl -fsSL -o /opt/spitfire/jmx/jmx_prometheus_javaagent.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar - Write the rules file.
sudo tee /opt/spitfire/jmx/spitfire-weblogic.yml >/dev/null <<'SPITFIRE_YML' # Spitfire weblogic: Prometheus JMX exporter javaagent 1.0.1 rules (read-only). lowercaseOutputName: false includeObjectNames: - "java.lang:type=Memory" - "java.lang:type=GarbageCollector,*" - "java.lang:type=Threading" - "java.lang:type=ClassLoading" - "java.lang:type=OperatingSystem" - "com.bea:Type=ServerRuntime,*" - "com.bea:Type=ThreadPoolRuntime,*" - "com.bea:Type=JDBCDataSourceRuntime,*" - "com.bea:Type=JMSDestinationRuntime,*" - "com.bea:Type=WebAppComponentRuntime,*" rules: - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used' name: java_heap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max' name: java_heap_max_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>used' name: java_nonheap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>max' name: java_nonheap_max_bytes type: GAUGE - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionCount' name: java_gc_count type: GAUGE labels: name: "$1" - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionTime' name: java_gc_seconds type: GAUGE valueFactor: 0.001 labels: name: "$1" - pattern: 'java.lang<type=Threading><>ThreadCount' name: java_threads_live type: GAUGE - pattern: 'java.lang<type=Threading><>DaemonThreadCount' name: java_threads_daemon type: GAUGE - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount' name: java_classes_loaded type: GAUGE - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad' name: java_cpu_load type: GAUGE - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ServerRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>StateVal' name: java_server_state type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ServerRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>State: RUNNING' name: java_server_running type: GAUGE value: 1 labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ServerRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>State: ' name: java_server_running type: GAUGE value: 0 labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>StuckThreadCount' name: java_stuck_threads type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>HoggingThreadCount' name: java_hogging_threads type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>PendingUserRequestCount' name: java_requests_pending type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>CompletedRequestCount' name: java_requests type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>ExecuteThreadTotalCount' name: java_threadpool_size type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>ExecuteThreadIdleCount' name: java_threadpool_idle type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>ActiveConnectionsCurrentCount' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>NumAvailable' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>CurrCapacity' name: java_jdbc_size type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>WaitingForConnectionCurrentCount' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JMSDestinationRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>MessagesCurrentCount' name: java_jms_messages type: GAUGE labels: destination: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JMSDestinationRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>MessagesPendingCount' name: java_jms_pending type: GAUGE labels: destination: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=WebAppComponentRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>OpenSessionsCurrentCount' name: java_sessions_active type: GAUGE labels: app: "$1" SPITFIRE_YML - Add the JVM option for AdminServer to
setUserOverrides.sh.grep -q "spitfire AdminServer" /u01/oracle/user_projects/domains/base_domain/bin/setUserOverrides.sh 2>/dev/null || echo 'if [ "${SERVER_NAME:-}" = "AdminServer" ]; then JAVA_OPTIONS="$JAVA_OPTIONS -javaagent:/opt/spitfire/jmx/jmx_prometheus_javaagent.jar=9404:/opt/spitfire/jmx/spitfire-weblogic.yml"; fi # spitfire AdminServer' >> /u01/oracle/user_projects/domains/base_domain/bin/setUserOverrides.sh - Restart AdminServer.
/u01/oracle/user_projects/domains/base_domain/bin/stopWebLogic.sh; nohup /u01/oracle/user_projects/domains/base_domain/bin/startWebLogic.sh >/dev/null 2>&1 & - Add the monitored system: Infrastructure sources → WebLogic, address
<host>:9404. - Check the exporter.
curl -fsS http://localhost:9404/metrics | grep -q java_server_state && echo "OK: weblogic"
Windows (PowerShell)
- Open PowerShell as Administrator and run it.
Invoke-WebRequest -UseBasicParsing -Uri "https://<spitfire>/api/v1/setup/ds/weblogic.ps1?t=<token>&lang=en" -OutFile "$env:TEMP\sf-weblogic.ps1"; powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\sf-weblogic.ps1" - When you see
OK: ..., you are done.
Manual install (Windows):
- Download the javaagent.
New-Item -ItemType Directory -Force C:\spitfire\jmx | Out-Null; Invoke-WebRequest -UseBasicParsing -Uri https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar -OutFile C:\spitfire\jmx\jmx_prometheus_javaagent.jar - Write the rules file.
$Rules = @' # Spitfire weblogic: Prometheus JMX exporter javaagent 1.0.1 rules (read-only). lowercaseOutputName: false includeObjectNames: - "java.lang:type=Memory" - "java.lang:type=GarbageCollector,*" - "java.lang:type=Threading" - "java.lang:type=ClassLoading" - "java.lang:type=OperatingSystem" - "com.bea:Type=ServerRuntime,*" - "com.bea:Type=ThreadPoolRuntime,*" - "com.bea:Type=JDBCDataSourceRuntime,*" - "com.bea:Type=JMSDestinationRuntime,*" - "com.bea:Type=WebAppComponentRuntime,*" rules: - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used' name: java_heap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max' name: java_heap_max_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>used' name: java_nonheap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>max' name: java_nonheap_max_bytes type: GAUGE - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionCount' name: java_gc_count type: GAUGE labels: name: "$1" - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionTime' name: java_gc_seconds type: GAUGE valueFactor: 0.001 labels: name: "$1" - pattern: 'java.lang<type=Threading><>ThreadCount' name: java_threads_live type: GAUGE - pattern: 'java.lang<type=Threading><>DaemonThreadCount' name: java_threads_daemon type: GAUGE - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount' name: java_classes_loaded type: GAUGE - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad' name: java_cpu_load type: GAUGE - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ServerRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>StateVal' name: java_server_state type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ServerRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>State: RUNNING' name: java_server_running type: GAUGE value: 1 labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ServerRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>State: ' name: java_server_running type: GAUGE value: 0 labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>StuckThreadCount' name: java_stuck_threads type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>HoggingThreadCount' name: java_hogging_threads type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>PendingUserRequestCount' name: java_requests_pending type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>CompletedRequestCount' name: java_requests type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>ExecuteThreadTotalCount' name: java_threadpool_size type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=ThreadPoolRuntime[,>])(?=(?:[^>]*, )?ServerRuntime="?([^",>]+)"?[,>])[^>]*><>ExecuteThreadIdleCount' name: java_threadpool_idle type: GAUGE labels: server: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>ActiveConnectionsCurrentCount' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>NumAvailable' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>CurrCapacity' name: java_jdbc_size type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JDBCDataSourceRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>WaitingForConnectionCurrentCount' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JMSDestinationRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>MessagesCurrentCount' name: java_jms_messages type: GAUGE labels: destination: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=JMSDestinationRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>MessagesPendingCount' name: java_jms_pending type: GAUGE labels: destination: "$1" - pattern: 'com.bea<(?=(?:[^>]*, )?Type=WebAppComponentRuntime[,>])(?=(?:[^>]*, )?Name="?([^",>]+)"?[,>])[^>]*><>OpenSessionsCurrentCount' name: java_sessions_active type: GAUGE labels: app: "$1" '@ [IO.File]::WriteAllText('C:\spitfire\jmx\spitfire-weblogic.yml', $Rules + "`n") - Add the JVM option for AdminServer to
setUserOverrides.cmd.Add-Content -Encoding ASCII C:\Oracle\Middleware\Oracle_Home\user_projects\domains\base_domain\bin\setUserOverrides.cmd 'if "%SERVER_NAME%"=="AdminServer" set "JAVA_OPTIONS=%JAVA_OPTIONS% -javaagent:C:\spitfire\jmx\jmx_prometheus_javaagent.jar=9404:C:\spitfire\jmx\spitfire-weblogic.yml" & rem spitfire AdminServer' - Restart AdminServer, then check the exporter.
if ((Invoke-WebRequest -UseBasicParsing http://localhost:9404/metrics).Content -match 'java_server_state') { 'OK: weblogic' }
WebSphere Liberty
Liberty only (Open Liberty, WebSphere Liberty). Traditional WebSphere Application Server is not supported.
Quick install
- Run it on the Liberty server.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/websphere.sh?t=<token>&lang=en" -o /tmp/sf-websphere.sh && bash /tmp/sf-websphere.sh - When you see
OK: ..., you are done.
Note
The script writes configDropins/overrides/spitfire-metrics.xml; Liberty reads it without a
restart and server.xml does not change. /metrics opens on the HTTP port without authentication;
allow the port only to the Tower. If the mpMetrics feature is not installed: featureUtility installFeature mpMetrics-5.1. Liberty does not publish the thread pool's maximum or deadlocks.
The script picks the mpMetrics version from server.xml: 5.1 for Jakarta EE 10+, 4.0 for EE 9.1, 3.0
for Java EE 8 (javaee-8.0). In the manual install use the same; a mismatch logs CWWKF0033E and
Liberty drops its features.
Manual install
- Write the mpMetrics dropin.
sudo mkdir -p /opt/wlp/usr/servers/defaultServer/configDropins/overrides && sudo tee /opt/wlp/usr/servers/defaultServer/configDropins/overrides/spitfire-metrics.xml >/dev/null <<'SPITFIRE_XML' <?xml version="1.0" encoding="UTF-8"?> <server> <featureManager> <feature>mpMetrics-5.1</feature> </featureManager> <mpMetrics authentication="false"/> </server> SPITFIRE_XML - Add the monitored system: Infrastructure sources → WebSphere Liberty, address
<host>:9080. - Check
/metrics.curl -fsS http://localhost:9080/metrics | grep -q memory_usedHeap_bytes && echo "OK: websphere"
Java / JMX (Spring Boot)
Quick install
- Run it on the app's server.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/jmx.sh?t=<token>&lang=en" -o /tmp/sf-jmx.sh && bash /tmp/sf-jmx.sh - When you see
OK: ..., you are done.
Note
For a systemd service the script adds JAVA_TOOL_OPTIONS in a drop-in and asks before the
restart; for an app that runs any other way it prints the option. In Spring Boot, the embedded Tomcat
and HikariCP metrics need server.tomcat.mbeanregistry.enabled=true and
spring.datasource.hikari.register-mbeans=true.
Manual install
- Download the javaagent.
sudo mkdir -p /opt/spitfire/jmx && sudo curl -fsSL -o /opt/spitfire/jmx/jmx_prometheus_javaagent.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar - Write the rules file.
sudo tee /opt/spitfire/jmx/spitfire-jmx.yml >/dev/null <<'SPITFIRE_YML' # Spitfire jmx: Prometheus JMX exporter javaagent 1.0.1 rules (read-only). lowercaseOutputName: false includeObjectNames: - "java.lang:type=Memory" - "java.lang:type=GarbageCollector,*" - "java.lang:type=Threading" - "java.lang:type=ClassLoading" - "java.lang:type=OperatingSystem" - "Catalina:type=GlobalRequestProcessor,*" - "Catalina:type=ThreadPool,*" - "Catalina:type=Manager,*" - "Catalina:type=DataSource,*" - "Tomcat:type=GlobalRequestProcessor,*" - "Tomcat:type=ThreadPool,*" - "Tomcat:type=Manager,*" - "Tomcat:type=DataSource,*" - "com.zaxxer.hikari:*" rules: - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used' name: java_heap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max' name: java_heap_max_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>used' name: java_nonheap_used_bytes type: GAUGE - pattern: 'java.lang<type=Memory><NonHeapMemoryUsage>max' name: java_nonheap_max_bytes type: GAUGE - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionCount' name: java_gc_count type: GAUGE labels: name: "$1" - pattern: 'java.lang<(?=(?:[^>]*, )?type=GarbageCollector[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>CollectionTime' name: java_gc_seconds type: GAUGE valueFactor: 0.001 labels: name: "$1" - pattern: 'java.lang<type=Threading><>ThreadCount' name: java_threads_live type: GAUGE - pattern: 'java.lang<type=Threading><>DaemonThreadCount' name: java_threads_daemon type: GAUGE - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount' name: java_classes_loaded type: GAUGE - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad' name: java_cpu_load type: GAUGE - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>requestCount' name: java_requests type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>errorCount' name: java_request_errors type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=GlobalRequestProcessor[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>processingTime' name: java_request_seconds type: GAUGE valueFactor: 0.001 labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>currentThreadsBusy' name: java_threadpool_busy type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>maxThreads' name: java_threadpool_max type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=ThreadPool[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>currentThreadCount' name: java_threadpool_size type: GAUGE labels: connector: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=Manager[,>])(?=(?:[^>]*, )?context="?([^",>]+)"?[,>])(?=(?:[^>]*, )?host="?([^",>]+)"?[,>])[^>]*><>activeSessions' name: java_sessions_active type: GAUGE labels: context: "$1" host: "$2" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>numActive' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>numIdle' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>(?:maxTotal|maxActive)' name: java_jdbc_max type: GAUGE labels: datasource: "$1" - pattern: '(?:Catalina|Tomcat)<(?=(?:[^>]*, )?type=DataSource[,>])(?=(?:[^>]*, )?name="?([^",>]+)"?[,>])[^>]*><>waitCount' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>ActiveConnections' name: java_jdbc_active type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>IdleConnections' name: java_jdbc_idle type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>TotalConnections' name: java_jdbc_size type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=Pool \(([^)]+)\)><>ThreadsAwaitingConnection' name: java_jdbc_waiting type: GAUGE labels: datasource: "$1" - pattern: 'com.zaxxer.hikari<type=PoolConfig \(([^)]+)\)><>MaximumPoolSize' name: java_jdbc_max type: GAUGE labels: datasource: "$1" SPITFIRE_YML - Start the app with the javaagent.
java -javaagent:/opt/spitfire/jmx/jmx_prometheus_javaagent.jar=9404:/opt/spitfire/jmx/spitfire-jmx.yml -Dserver.tomcat.mbeanregistry.enabled=true -Dspring.datasource.hikari.register-mbeans=true -jar /opt/app/app.jar - Add the monitored system: Infrastructure sources → Java / JMX, address
<host>:9404. - Check the exporter.
curl -fsS http://localhost:9404/metrics | grep -q java_heap_used_bytes && echo "OK: jmx"
Threshold findings
| Finding | Source | Severity |
|---|---|---|
The target does not answer (up = 0) |
All | Critical |
| Heap above 85% / above 95% | All | Warning / critical |
| Deadlocked threads | Tomcat, WebLogic, Java / JMX | Critical |
| Thread pool more than 90% busy | Tomcat, WildFly, Java / JMX | Warning |
| JDBC pool more than 90% used | Tomcat, WildFly, Java / JMX | Warning |
| Requests waiting for a JDBC connection | Tomcat, WebLogic, Java / JMX, WildFly, Liberty | Warning |
| Request errors went up | Tomcat, WildFly, Liberty, Java / JMX | Warning |
| WebLogic stuck threads | WebLogic | Critical |
| WebLogic server not RUNNING | WebLogic | Critical |
Success check
- On the server the matching
curlcommand printsOK: .... - Last test = Connected, and within 1-2 minutes the dashboard shows heap and thread lines.