SpitfireLoad Testing & APMWhat's new

Infrastructure: Apache HBase (Beta)

An HBase cluster (Beta) is read through the Tower: the Tower reads the /jmx servlet on the info port of the Master (16010) and of every RegionServer (16030). No jar, restart or Collector is needed; no table data is read. Each cluster is one monitored system.

Page: All integrations In Spitfire: /admin/integrations → Add integration → Apache HBase. The <spitfire> and <token> in the command come filled in the UI.

Quick install

  1. Run the command on a machine the Tower can reach, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/hbase.sh?t=<token>" -o /tmp/sf-hbase.sh && bash /tmp/sf-hbase.sh
  1. When you see OK: ..., you are done.

Manual install

  1. From the Tower's machine, check the Master's /jmx servlet.
bash
curl -fsS 'http://<hbase-master>:16010/jmx?qry=Hadoop:service=HBase,name=Master,sub=Server' | grep -q numRegionServers && echo 'OK: HBase Master /jmx'
  1. Check a RegionServer's /jmx servlet.
bash
curl -fsS 'http://<hbase-regionserver>:16030/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server' | grep -q regionCount && echo 'OK: HBase RegionServer /jmx'
  1. See the RegionServer names the Master lists.
bash
curl -fsS 'http://<hbase-master>:16010/jmx?qry=Hadoop:service=HBase,name=Master,sub=Server' | grep -o '"tag.liveRegionServers" *: *"[^"]*"' && echo 'OK: RegionServer list'
  1. Spitfire → Infrastructure sources → Apache HBase.
    • Master addresses: http://hbase-master-1.corp:16010, http://hbase-master-2.corp:16010 (up to 3 Masters; the active one is found).
    • RegionServer addresses: leave empty; if the names from step 3 do not resolve from the Tower, enter hbase-rs-1.corp:16030, hbase-rs-2.corp:16030.
    • User / password: only with a basic-auth proxy in front of the info ports.
  2. Test connection → Save. The first metrics arrive within ~1 min.
Note

For https, write the address with https://; with an internal CA tick Skip TLS certificate verification. RegionServers are read with the Master's scheme. Info ports behind Kerberos (SPNEGO) or Apache Knox are not supported; the Tower reports HTTP 401 Negotiate.

The most common case: the Master lists RegionServers by short host name and the Tower cannot resolve them. Then fill RegionServer addresses with full names.

Panels and findings

The panels show the active Master, live/dead RegionServers and RIT (regions in transition) for the cluster, and request rates, block cache, compaction/flush queues, slow* operations, IPC, GC and heap per RegionServer.

Finding Threshold Severity
Master not answering up = 0 critical
No active Master 0 critical
Dead RegionServer > 0 critical
RegionServer unreachable up = 0 warning
RIT stuck ritCountOverThreshold > 0 (HBase's threshold, 60 s by default) warning
Low block cache hit ratio < 80% (at least 1000 requests) warning
High compaction queue > 20 warning
Heap usage > 90% warning
Slow operations slow* counters rose (operations longer than HBase's hbase.ipc.slow.metric.time, 1000 ms by default) warning

Uninstalling

The script changes nothing in HBase.

  1. Delete the source in Spitfire: All integrations In Spitfire: /admin/integrations#observability → the bin on the source's row → Delete. Metrics already collected stay.

If you opened a basic auth user only for Spitfire on the proxy, delete it too. The state file of a run that stopped halfway goes with rm -f ~/.spitfire-setup/hbase.state.