SpitfireLoad Testing & APMWhat's new

Infrastructure: IBM WebSphere (traditional), App Connect Enterprise and DataPower (Beta)

The Tower reads the three IBM products of the bank pack, each over its own HTTP interface. No agent, no jar, no Collector. All three are Beta.

Source What the Tower reads
WebSphere Application Server (traditional) 8.5.5 / 9.0 the PerfServlet's PMI XML (perfServletApp)
IBM App Connect Enterprise 12 / 13 the admin REST API /apiv2 and its statistics stream
IBM DataPower Gateway the REST management interface (port 5554)

Every user below is read-only; Spitfire stops, starts or changes nothing. Passwords are stored encrypted and never logged. To add a source: All integrations In Spitfire: /admin/integrations → Add integration → the product's card.

Quick install

  1. Do your product's user and role steps below once. Then run the command on a machine the Tower can reach and answer the questions. Use was.sh, ace.sh or datapower.sh; in the product this line comes filled in with the address and a fresh token.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/was.sh?t=<token>" -o /tmp/sf-was.sh && bash /tmp/sf-was.sh
  1. When you see OK: ..., you are done. The first metrics arrive within about 1 minute (read every 60 s).

WebSphere Application Server (traditional)

The PerfServlet (perfServletApp, shipped with WAS in installableApps/PerfServletApp.ear) is installed on one application server and returns the PMI data of the whole cell as XML through the deployment manager. The user needs only the monitor administrative role.

  1. On the deployment manager's machine, as the WebSphere owner, set the profile, the server that will host perfServletApp, and the passwords. The script of steps 2 and 3 sets PMI to the extended set on every application server at run time, creates the user spitfire in the file-based registry (with LDAP, create it there and the script keeps it), maps it to the monitor role, installs and starts perfServletApp and synchronizes the nodes.
bash
WAS_PROFILE=/opt/IBM/WebSphere/AppServer/profiles/Dmgr01
export SPITFIRE_NODE=AppNode01 SPITFIRE_SERVER=server1
WAS_ADMIN=wasadmin
read -rsp 'WebSphere admin password: ' WAS_ADMIN_PW; echo
read -rsp 'New password for the user spitfire: ' SPITFIRE_PW; echo; export SPITFIRE_PW
[ -x "$WAS_PROFILE/bin/wsadmin.sh" ] && echo 'OK: wsadmin found'
  1. Save the wsadmin script (the product's Steps box shows the same text).
bash
cat > /tmp/spitfire-was.py <<'SPITFIRE_PY'
import java
sep = java.lang.System.getProperty('line.separator')
env = java.lang.System.getenv
node, server, pw = env('SPITFIRE_NODE'), env('SPITFIRE_SERVER'), env('SPITFIRE_PW')
for s in AdminConfig.list('Server').split(sep):
    if not s or AdminConfig.showAttribute(s, 'serverType') != 'APPLICATION_SERVER':
        continue
    AdminConfig.modify(AdminConfig.list('PMIService', s), [['enable', 'true'], ['statisticSet', 'extended']])
    perf = AdminControl.completeObjectName('type=Perf,process=%s,*' % AdminConfig.showAttribute(s, 'name'))
    if perf:
        AdminControl.invoke(perf, 'setStatisticSet', 'extended')
try:
    AdminTask.createUser(['-uid', 'spitfire', '-password', pw, '-confirmPassword', pw, '-cn', 'spitfire', '-sn', 'monitor'])
except:
    print 'user spitfire: exists (kept)'
AdminTask.mapUsersToAdminRole(['-roleName', 'monitor', '-userids', 'spitfire'])
if 'perfServletApp' not in AdminApp.list().split(sep):
    ear = java.lang.System.getProperty('was.install.root') + '/installableApps/PerfServletApp.ear'
    AdminApp.install(ear, ['-appname', 'perfServletApp', '-node', node, '-server', server, '-usedefaultbindings'])
AdminConfig.save()
for n in AdminControl.queryNames('type=NodeSync,*').split(sep):
    if n:
        AdminControl.invoke(n, 'sync')
try:
    am = AdminControl.completeObjectName('type=ApplicationManager,node=%s,process=%s,*' % (node, server))
    AdminControl.invoke(am, 'startApplication', 'perfServletApp')
except:
    print 'perfServletApp: already started'
for agm in AdminControl.queryNames('type=AuthorizationGroupManager,*').split(sep):
    if agm:
        AdminControl.invoke(agm, 'refreshAll')
print 'SPITFIRE-OK'
SPITFIRE_PY
echo 'OK: script saved'
  1. Run it.
bash
"$WAS_PROFILE/bin/wsadmin.sh" -lang jython -user "$WAS_ADMIN" -password "$WAS_ADMIN_PW" -f /tmp/spitfire-was.py | grep -q SPITFIRE-OK && echo 'OK: PMI extended, spitfire in the monitor role, perfServletApp installed'; rm -f /tmp/spitfire-was.py
  1. From the Tower's machine, check the PerfServlet with the new user (9443 is the server's https port, 9080 its http port). If it answers HTTP 403, the user is not in the monitor role: run steps 2 and 3 again or map it in the console (Security → Global security → Administrative user roles).
bash
curl -fsS -k -u "spitfire:$SPITFIRE_PW" 'https://<was-server>:9443/wasPerfTool/servlet/perfservlet?module=jvmRuntimeModule' | grep -q 'responseStatus="success"' && echo 'OK: PerfServlet answers the spitfire user'
  1. Spitfire → Infrastructure sources → WebSphere Application Server (traditional).
    • Endpoint: https://was-server-1.corp:9443 (the server that runs perfServletApp; a second one for failover).
    • User / password: spitfire and its password.
  2. Test connection → Save.

Hung-thread counts need the extended statistic set (steps 1 to 3). JVM GC counts need the -agentlib:pmiJvmtiProfiler JVM argument; without it only the GC panels stay empty.

IBM App Connect Enterprise

The Tower reads /apiv2 (servers, applications, message flows and their state) and the statistics websocket (/apiv2/statistics). Flow rates, error rates and times come from snapshot statistics, which ACE 12 and 13 publish by default every 20 s. The Tower only listens and does not switch on accounting. The user gets a role with read only (read+:write-:execute-).

Warning

Turning on admin security with authorizationEnabled makes every other admin user need a role too. Give your own admin users a role (mqsiwebuseradmin ... -r <admin role>) before you restart, or you lock yourself out of the web UI.

  1. An integration node (admin port 4414): create the role and its user, switch on file-based admin security and restart the node.
bash
ACE_NODE=BANKNODE
read -rsp 'New password for the user spitfire: ' SPITFIRE_PW; echo
mqsichangefileauth "$ACE_NODE" -r spitfire_ro -p read+,write-,execute- && mqsiwebuseradmin "$ACE_NODE" -c -u spitfire -a "$SPITFIRE_PW" -r spitfire_ro && mqsichangeauthmode "$ACE_NODE" -s active -m file && mqsistop "$ACE_NODE" && mqsistart "$ACE_NODE" && echo 'OK: spitfire_ro role and user'
  1. An independent integration server (admin port 7600): add the role to its server.conf.yaml, create the user in its work directory and restart the server.
bash
ACE_WORK=/home/aceuser/ace-server
read -rsp 'New password for the user spitfire: ' SPITFIRE_PW; echo
cat >> "$ACE_WORK/overrides/server.conf.yaml" <<'SPITFIRE_YAML'
RestAdminListener:
  basicAuth: true
  authorizationEnabled: true
  authorizationMode: file
Security:
  Permissions:
    spitfire_ro: 'read+:write-:execute-'
SPITFIRE_YAML
mqsiwebuseradmin -w "$ACE_WORK" -c -u spitfire -a "$SPITFIRE_PW" -r spitfire_ro && echo 'OK: spitfire_ro role and user'
  1. From the Tower's machine, check the API and that snapshot statistics are on (4414 for a node, 7600 for a server). If the second check prints nothing, snapshot statistics were switched off on purpose: flow and server state are still read, the rate/error/time panels stay empty and the statistics stream off finding says so. Spitfire does not switch them on for you.
bash
curl -fsS -u "spitfire:$SPITFIRE_PW" http://<ace-host>:7600/apiv2 | grep -q '"type":"integration' && echo 'OK: ACE admin API with the spitfire user'
curl -fsS -u "spitfire:$SPITFIRE_PW" http://<ace-host>:7600/apiv2/statistics/snapshot | grep -q '"publicationOn":"active"' && echo 'OK: snapshot statistics on'
  1. Spitfire → Infrastructure sources → IBM App Connect Enterprise.
    • Endpoint: http://ace-node-1.corp:4414 (a node: all its servers are read) or http://ace-1.corp:7600.
    • User / password: spitfire and its password.
  2. Test connection → Save. The test also opens the statistics stream once.

IBM DataPower Gateway

From the REST management interface on port 5554 the Tower reads CPU, memory, file systems, TCP, the objects down per domain, the transaction rate and mean time per service (MPGW, WSP, XML firewall, API gateway) and the expiry of the crypto certificates. The user belongs to a group whose access profile reads everything and writes nothing (*/*/*?Access=r).

  1. In the appliance's CLI as an admin (ssh <datapower>, default domain), switch on the REST interface, create the read-only group and the user (replace <password>).
text
configure terminal
rest-mgmt
  admin-state enabled
  port 5554
exit
usergroup spitfire-ro
  summary "Spitfire read-only"
  access-policy */*/*?Access=r
exit
user spitfire
  password <password>
  access-level group-defined
  group spitfire-ro
exit
write memory
  1. Log in to the WebGUI once as spitfire if the appliance asks new users to change their password, and use the new one below.

  2. From the Tower's machine, check the REST interface with the new user.

bash
curl -fsS -k -u 'spitfire:<password>' https://<datapower>:5554/mgmt/status/default/CPUUsage | grep -q oneMinute && echo 'OK: DataPower REST with the spitfire user'
  1. Spitfire → Infrastructure sources → IBM DataPower Gateway.
    • Endpoint: https://dp-1.corp:5554 (one per appliance; several separated by commas).
    • User / password: spitfire and its password.
    • Domains: leave empty for all domains (up to 20), or e.g. default, BANK_PROD.
    • Skip TLS certificate verification: tick it when the appliance uses its self-signed certificate.
  2. Test connection → Save.

Certificate expiry is read every 6 hours (one action per certificate, at most 200). If the access profile does not allow ViewCertificateDetails, only the certificate panel stays empty.

Uninstalling

The quick install scripts change nothing in the product; the manual steps create the user and the role. For each product first delete the source: All integrations In Spitfire: /admin/integrations#observability → the bin on its row → Delete. The Tower stops reading; metrics already collected stay. We use the default names; if you chose another, use it. A run that stopped halfway can leave ~/.spitfire-setup/<product>.state (rm -f ~/.spitfire-setup/was.state).

WebSphere Application Server (traditional)

  1. On the deployment manager's machine, as the WebSphere owner, enter the profile and the admin password.
bash
WAS_PROFILE=/opt/IBM/WebSphere/AppServer/profiles/Dmgr01
WAS_ADMIN=wasadmin
read -rsp 'WebSphere admin password: ' WAS_ADMIN_PW; echo
[ -x "$WAS_PROFILE/bin/wsadmin.sh" ] && echo 'OK: wsadmin found'
  1. Save the undo script. It takes the user out of the monitor role, deletes it from the file-based registry (with LDAP, delete it there), uninstalls perfServletApp and syncs the nodes.
bash
cat > /tmp/spitfire-was-undo.py <<'SPITFIRE_PY'
import java
sep = java.lang.System.getProperty('line.separator')
try:
    AdminTask.removeUsersFromAdminRole(['-roleName', 'monitor', '-userids', 'spitfire'])
except:
    print 'monitor role: spitfire not in it'
for u in AdminTask.searchUsers(['-uid', 'spitfire']).split(sep):
    if u:
        AdminTask.deleteUser(['-uniqueName', u])
if 'perfServletApp' in AdminApp.list().split(sep):
    AdminApp.uninstall('perfServletApp')
AdminConfig.save()
for n in AdminControl.queryNames('type=NodeSync,*').split(sep):
    if n:
        AdminControl.invoke(n, 'sync')
for agm in AdminControl.queryNames('type=AuthorizationGroupManager,*').split(sep):
    if agm:
        AdminControl.invoke(agm, 'refreshAll')
print 'SPITFIRE-OK'
SPITFIRE_PY
echo 'OK: script saved'
  1. Run it. It should print OK: spitfire and perfServletApp removed.
bash
"$WAS_PROFILE/bin/wsadmin.sh" -lang jython -user "$WAS_ADMIN" -password "$WAS_ADMIN_PW" -f /tmp/spitfire-was-undo.py | grep -q SPITFIRE-OK && echo 'OK: spitfire and perfServletApp removed'; rm -f /tmp/spitfire-was-undo.py

PMI stays on the extended set, which is harmless. To set it back, in the console pick Monitoring and Tuning → Performance Monitoring Infrastructure (PMI) → the server → Basic.

IBM App Connect Enterprise

  1. Integration node: delete the user. It should print OK: spitfire removed.
bash
ACE_NODE=BANKNODE
mqsiwebuseradmin "$ACE_NODE" -d -u spitfire && ! mqsiwebuseradmin "$ACE_NODE" -l | grep -qw spitfire && echo 'OK: spitfire removed'
  1. Independent integration server: delete the user from the work directory. It should print OK: spitfire removed.
bash
ACE_WORK=/home/aceuser/ace-server
mqsiwebuseradmin -w "$ACE_WORK" -d -u spitfire && ! mqsiwebuseradmin -w "$ACE_WORK" -l | grep -qw spitfire && echo 'OK: spitfire removed'
  1. Only if you turned on admin security for Spitfire: on an independent server delete the RestAdminListener and Security block appended to server.conf.yaml (the command first keeps a server.conf.yaml.spitfire.bak backup), then restart the server. If the file has another RestAdminListener: line the command does nothing; delete the block by hand. It should print OK: block removed.
bash
F=/home/aceuser/ace-server/overrides/server.conf.yaml; [ "$(grep -c '^RestAdminListener:' "$F")" = 1 ] && sed -i.spitfire.bak '/^RestAdminListener:$/,/^    spitfire_ro: /d' "$F" && ! grep -q spitfire_ro "$F" && echo 'OK: block removed'

On a node the same is mqsichangeauthmode "$ACE_NODE" -s inactive -m file, then mqsistop "$ACE_NODE" && mqsistart "$ACE_NODE".

The spitfire_ro role on the node stays; with no user in it, it gives access to nothing.

IBM DataPower Gateway

  1. In the device CLI as admin (ssh <datapower>, default domain), delete the user and the group.
text
configure terminal
no user spitfire
no usergroup spitfire-ro
write memory
  1. From the Tower's machine, check that the user can no longer sign in. It should print OK: spitfire can no longer sign in.
bash
curl -sk -o /dev/null -w '%{http_code}' -u 'spitfire:<password>' https://<datapower>:5554/mgmt/status/default/CPUUsage | grep -qE '^40[13]$' && echo 'OK: spitfire can no longer sign in'

If you turned on the REST management interface only for Spitfire, turn it off: configure terminal, rest-mgmt, admin-state disabled, exit, write memory.

Findings

The panels show thread pools, JDBC pools, heap, flow rates, certificates and the like per product. The findings and their thresholds:

Finding Threshold Severity
WAS: PerfServlet not answering up = 0 critical
WAS: application server down no PMI data critical
WAS: threads waiting for a JDBC connection > 0 critical
WAS: hung threads > 0 critical
WAS: JDBC pool / thread pool full > 95% warning
WAS: heap usage > 90% warning
WAS: JDBC connection faults, transaction timeouts rose warning
ACE: not answering up = 0 critical
ACE: integration server stopped 0 critical
ACE: message flow stopped 0 warning
ACE: flow error rate > 5% (at least 10 messages) warning
ACE: slow flow > 1000 ms per message warning
ACE: heap usage > 90% warning
ACE: statistics stream off 0 info
DataPower: not answering up = 0 critical
DataPower: service down (MPGW, WSP, …) > 0 critical
DataPower: certificate expired < 0 days critical
DataPower: object down enabled but down warning
DataPower: certificate expiring < 30 days warning
DataPower: CPU / memory / file system > 90% warning

In a load test we look at "threads waiting for a JDBC connection" first: if threads wait for a connection, the bottleneck is the data source's pool size, not the application code.