Infrastructure: Apache HBase (Beta)
An HBase cluster (Beta) is read through the Tower: the Tower reads the /jmx servlet
on the info port of the Master (16010) and of every RegionServer (16030). No jar, restart or Collector is needed;
no table data is read. Each cluster is one monitored system.
Page: All integrations In Spitfire: /admin/integrations → Add integration → Apache HBase. The <spitfire> and
<token> in the command come filled in the UI.
Quick install
- Run the command on a machine the Tower can reach, answer the questions.
curl -fsSL "https://<spitfire>/api/v1/setup/ds/hbase.sh?t=<token>" -o /tmp/sf-hbase.sh && bash /tmp/sf-hbase.sh- When you see
OK: ..., you are done.
Manual install
- From the Tower's machine, check the Master's
/jmxservlet.
curl -fsS 'http://<hbase-master>:16010/jmx?qry=Hadoop:service=HBase,name=Master,sub=Server' | grep -q numRegionServers && echo 'OK: HBase Master /jmx'- Check a RegionServer's
/jmxservlet.
curl -fsS 'http://<hbase-regionserver>:16030/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server' | grep -q regionCount && echo 'OK: HBase RegionServer /jmx'- See the RegionServer names the Master lists.
curl -fsS 'http://<hbase-master>:16010/jmx?qry=Hadoop:service=HBase,name=Master,sub=Server' | grep -o '"tag.liveRegionServers" *: *"[^"]*"' && echo 'OK: RegionServer list'- Spitfire → Infrastructure sources → Apache HBase.
- Master addresses:
http://hbase-master-1.corp:16010, http://hbase-master-2.corp:16010(up to 3 Masters; the active one is found). - RegionServer addresses: leave empty; if the names from step 3 do not resolve from the Tower, enter
hbase-rs-1.corp:16030, hbase-rs-2.corp:16030. - User / password: only with a basic-auth proxy in front of the info ports.
- Master addresses:
- Test connection → Save. The first metrics arrive within ~1 min.
For https, write the address with https://; with an internal CA tick Skip TLS certificate
verification. RegionServers are read with the Master's scheme. Info ports behind Kerberos (SPNEGO) or Apache
Knox are not supported; the Tower reports HTTP 401 Negotiate.
The most common case: the Master lists RegionServers by short host name and the Tower cannot resolve them. Then fill RegionServer addresses with full names.
Panels and findings
The panels show the active Master, live/dead RegionServers and RIT (regions in transition) for the cluster, and request rates, block cache, compaction/flush queues, slow* operations, IPC, GC and heap per RegionServer.
| Finding | Threshold | Severity |
|---|---|---|
| Master not answering | up = 0 | critical |
| No active Master | 0 | critical |
| Dead RegionServer | > 0 | critical |
| RegionServer unreachable | up = 0 | warning |
| RIT stuck | ritCountOverThreshold > 0 (HBase's threshold, 60 s by default) | warning |
| Low block cache hit ratio | < 80% (at least 1000 requests) | warning |
| High compaction queue | > 20 | warning |
| Heap usage | > 90% | warning |
| Slow operations | slow* counters rose (operations longer than HBase's hbase.ipc.slow.metric.time, 1000 ms by default) |
warning |
Uninstalling
The script changes nothing in HBase.
- Delete the source in Spitfire: All integrations
In Spitfire: /admin/integrations#observability→ the bin on the source's row → Delete. Metrics already collected stay.
If you opened a basic auth user only for Spitfire on the proxy, delete it too. The state file of a run that
stopped halfway goes with rm -f ~/.spitfire-setup/hbase.state.