Infrastructure — web servers, Memcached, Prometheus

Nginx, Apache, HAProxy, Memcached and Prometheus endpoints are collected through a Tower. They need no user; open the status page to the Tower's IP only. The add steps are the same as for databases.

#Nginx

Quick install

1. Run on the nginx server.
   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/nginx.sh?t=<token>&lang=en" -o /tmp/sf-nginx.sh && bash /tmp/sf-nginx.sh

2. When you see OK: ..., you are done.

Manual install

1. Write the Tower machine's IP.
   bashCopy

   TOWER_IP=10.0.0.5

2. Write the Spitfire config.
   bashCopy

   sudo tee /etc/nginx/conf.d/spitfire.conf >/dev/null <<'EOF'
   log_format spitfire_json escape=json '{"time":"$time_iso8601","status":"$status","request_time":"$request_time",'
       '"upstream":"$proxy_host","upstream_addr":"$upstream_addr","upstream_status":"$upstream_status",'
       '"upstream_response_time":"$upstream_response_time","method":"$request_method"}';
   access_log syslog:server=<tower-ip>:5140,tag=nginx,nohostname spitfire_json;
   
   server {
       listen 8089;
       location = /nginx_status {
           stub_status;
           allow <tower-ip>;
           allow 127.0.0.1;
           deny all;
       }
   }
   EOF
   sudo sed -i "s/<tower-ip>/$TOWER_IP/g" /etc/nginx/conf.d/spitfire.conf

3. Test and reload nginx.
   bashCopy

   sudo nginx -t && sudo systemctl reload nginx

4. Add it: Infrastructure sources → Nginx.
   textCopy

   Address: http://<nginx-host>:8089/nginx_status   Access log port: 5140

5. Check from the Tower machine.
   bashCopy

   curl -fsS http://<nginx-host>:8089/nginx_status | grep -q '^Active connections' && echo 'OK: nginx stub_status reachable'

#Apache

Quick install

1. Run the command on the Apache server, answer the questions.

   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/apache.sh?t=<token>" -o /tmp/sf-apache.sh && bash /tmp/sf-apache.sh

2. When you see this line, you are done.

   textCopy

   OK: Apache added to Spitfire, first metrics within ~1 min

Manual install

1. Add the status page.

   bashCopy

   sudo tee /etc/apache2/conf-available/spitfire-status.conf >/dev/null <<'EOF'
   ExtendedStatus On
   <Location "/server-status">
       SetHandler server-status
       Require ip <tower-ip>
   </Location>
   EOF
   sudo a2enmod status && sudo a2enconf spitfire-status

2. Check the config and reload.

   bashCopy

   sudo apachectl -t && sudo apachectl graceful

3. Try it from the Tower machine.

   bashCopy

   curl -fsS 'http://<host>/server-status?auto' | grep -q '^Total Accesses' && echo 'OK: Apache status page is open'

4. Add it in Infrastructure sources → Apache → Manual install.

#HAProxy

Quick install

1. Run on the HAProxy server.
   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/haproxy.sh?t=<token>&lang=en" -o /tmp/sf-haproxy.sh && bash /tmp/sf-haproxy.sh

2. When you see OK: ..., you are done.

Manual install

1. Write the Tower machine's IP.
   bashCopy

   TOWER_IP=10.0.0.5

2. Back up haproxy.cfg.
   bashCopy

   sudo cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.spitfire.bak

3. Add the stats frontend.
   bashCopy

   sudo tee -a /etc/haproxy/haproxy.cfg >/dev/null <<'EOF'
   
   # BEGIN spitfire
   frontend spitfire_stats
       mode http
       bind *:8404
       acl spitfire_tower src <tower-ip> 127.0.0.1
       http-request deny unless spitfire_tower
       http-request use-service prometheus-exporter if { path /metrics }
       stats enable
       stats uri /stats
       stats refresh 10s
   # END spitfire
   EOF
   sudo sed -i "s/<tower-ip>/$TOWER_IP/g" /etc/haproxy/haproxy.cfg

4. Test and reload HAProxy.
   bashCopy

   sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy

5. Add it: Infrastructure sources → HAProxy.
   textCopy

   Address: http://<haproxy-host>:8404/stats

6. Check from the Tower machine.
   bashCopy

   curl -fsS 'http://<haproxy-host>:8404/stats;csv' | grep -q '^# pxname' && echo 'OK: HAProxy stats reachable'

#Memcached

Quick install (on the Tower machine)

1. Copy the command, paste it into the terminal and run it.

   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/memcached.sh?t=<token>" -o /tmp/sf-memcached.sh && bash /tmp/sf-memcached.sh

2. When you see OK: memcached ..., you are done.

Manual install

1. Read the stats from the Tower machine.

   bashCopy

   printf 'stats\r\nquit\r\n' | nc <host> 11211 | grep -q '^STAT uptime' && echo "OK: memcached"

2. Add it under Infrastructure sources → Memcached, address <host>:11211.

#Live view

The Memcached dashboard has a Live view card. Read now reads stats, stats slabs and stats items on the Tower: connections against max_connections, memory against limit_maxbytes, evictions, reclaimed and expired-unfetched items, and the slab classes by memory. A class with out-of-memory errors is marked red, one that evicts yellow. Counters are compared over the last 15 minutes of reads.

#Prometheus

Quick install

1. Run the command on the machine serving /metrics, answer the questions.

   bashCopy

   curl -fsSL "https://<spitfire>/api/v1/setup/ds/prometheus.sh?t=<token>" -o /tmp/sf-prometheus.sh && bash /tmp/sf-prometheus.sh

2. When you see this line, you are done.

   textCopy

   OK: Metrics endpoint added to Spitfire, first metrics within ~1 min

Manual install

1. Try the endpoint from the Tower machine.

   bashCopy

   curl -fsS 'http://<host>:9100/metrics' | grep -q '^# TYPE' && echo 'OK: metrics endpoint answered'

2. Add it in Infrastructure sources → Prometheus → Manual install.

3. Every metric of the endpoint is on the dashboard's All metrics tab: All metrics.

#Threshold findings

#Success check

1. On the Tower machine curl -s http://<host>:8089/nginx_status (or the address in question) answers.
2. Last test = Connected, and within 1-2 minutes the dashboard shows lines.