SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Infrastructure — network devices (SNMP, Beta)

Switches, routers, firewalls, load balancers and Linux servers are read over SNMP through a Tower. It only reads; nothing changes on the device. Each device counts as one server (the license's server limit). Devices are listed one by one as host[:161]; no ranges, CIDR or network scans.

Quick install

  1. Copy the command, paste it into the terminal, answer the questions.
bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/snmp.sh?t=<token>" -o /tmp/sf-snmp.sh && bash /tmp/sf-snmp.sh
  1. When you see OK: ..., you are done.

The file asks for a mode: linux adds a read-only v3 user to this machine's net-snmp, device only tests network gear set up already.

Manual install

First add the read-only v3 user with your device family's commands below. Then:

  1. Test from the Tower machine.
bash
snmpget -v3 -l authPriv -u spitfire_monitor -a SHA -A '<auth-password>' -x AES -X '<priv-password>' -t 5 -r 1 udp:<device>:161 1.3.6.1.2.1.1.5.0 1.3.6.1.2.1.1.3.0 && echo "OK: SNMP v3 answers"
  1. Spitfire → Infrastructure sources → SNMP → enter the devices and the user.

Pick v3. v2c only for a device without v3: the community travels as cleartext.

net-snmp (Linux)

  1. Install the packages.
bash
sudo apt-get install -y snmpd snmp libsnmp-dev || sudo dnf install -y net-snmp net-snmp-utils net-snmp-devel
  1. Stop snmpd.
bash
sudo systemctl stop snmpd
  1. Add the read-only v3 user.
bash
sudo net-snmp-create-v3-user -ro -a SHA-256 -A '<auth-password>' -x AES -X '<priv-password>' spitfire_monitor
  1. Listen on all addresses (backed up first).
bash
sudo sed -i.spitfire.bak 's/^agentaddress .*/agentaddress udp:161/' /etc/snmp/snmpd.conf
  1. Start snmpd.
bash
sudo systemctl start snmpd
  1. Test it.
bash
snmpget -v3 -l authPriv -u spitfire_monitor -a SHA-256 -A '<auth-password>' -x AES -X '<priv-password>' udp:127.0.0.1:161 1.3.6.1.2.1.1.3.0 && echo "OK: snmpd answers"

Cisco IOS and IOS XE

  1. Add the view, group and user, open to the Tower.
text
configure terminal
ip access-list standard SPITFIRE-SNMP
 permit <tower-ip>
 exit
snmp-server view SPITFIRE-RO iso included
snmp-server group SPITFIRE v3 priv read SPITFIRE-RO access SPITFIRE-SNMP
snmp-server user spitfire_monitor SPITFIRE v3 auth sha <auth-password> priv aes 128 <priv-password>
end
write memory
  1. Show the user.
text
show snmp user spitfire_monitor

On NX-OS: snmp-server user spitfire_monitor network-operator auth sha <auth-password> priv aes-128 <priv-password>.

Juniper Junos

  1. Add the user and the read-only view.
text
configure
set snmp v3 usm local-engine user spitfire_monitor authentication-sha authentication-password "<auth-password>"
set snmp v3 usm local-engine user spitfire_monitor privacy-aes128 privacy-password "<priv-password>"
set snmp view SPITFIRE-RO oid .1 include
set snmp v3 vacm security-to-group security-model usm security-name spitfire_monitor group SPITFIRE
set snmp v3 vacm access group SPITFIRE default-context-prefix security-model usm security-level privacy read-view SPITFIRE-RO
commit

Arista EOS

  1. Add the view, group and user.
text
configure
snmp-server view SPITFIRE-RO iso included
snmp-server group SPITFIRE v3 priv read SPITFIRE-RO
snmp-server user spitfire_monitor SPITFIRE v3 auth sha <auth-password> priv aes <priv-password>
write memory

HPE Aruba (ArubaOS-CX, ProCurve)

  1. On ArubaOS-CX, add the user.
text
configure terminal
snmp-server vrf default
snmpv3 user spitfire_monitor auth sha auth-pass plaintext <auth-password> priv aes priv-pass plaintext <priv-password>
snmpv3 context default vrf default
write memory
  1. On ProCurve / ArubaOS-Switch, add the user.
text
configure
snmpv3 enable
snmpv3 user spitfire_monitor auth sha <auth-password> priv aes <priv-password>
snmpv3 group operator user spitfire_monitor sec-model ver3
write memory

MikroTik RouterOS

  1. Add the read-only v3 user, open to the Tower.
text
/snmp community add name=spitfire_monitor addresses=<tower-ip>/32 security=private read-access=yes write-access=no authentication-protocol=SHA1 authentication-password=<auth-password> encryption-protocol=AES encryption-password=<priv-password>
/snmp set enabled=yes

FortiGate

  1. Add the v3 user, open to the Tower.
text
config system snmp sysinfo
    set status enable
end
config system snmp user
    edit "spitfire_monitor"
        set notify-hosts <tower-ip>
        set security-level auth-priv
        set auth-proto sha
        set auth-pwd <auth-password>
        set priv-proto aes
        set priv-pwd <priv-password>
        set queries enable
    next
end
  1. Allow SNMP on the interface facing the Tower.
text
config system interface
    edit "<interface>"
        append allowaccess snmp
    next
end

Palo Alto (PAN-OS)

  1. Add the v3 user and the view.
text
configure
set deviceconfig system snmp-setting access-setting version v3 views SPITFIRE-RO view all oid 1.3.6.1 option include
set deviceconfig system snmp-setting access-setting version v3 users spitfire_monitor view SPITFIRE-RO authpwd <auth-password> privpwd <priv-password>
commit
  1. Enable SNMP on the management interface.
text
set deviceconfig system service disable-snmp no
commit

Check Point (Gaia)

  1. Add the v3 user, read only.
text
set snmp agent on
set snmp agent-version v3-Only
add snmp usm user spitfire_monitor security-level authPriv auth-pass-type SHA1 auth-pass-phrase <auth-password> privacy-pass-type AES128 privacy-pass-phrase <priv-password>
save config

F5 BIG-IP

  1. Allow the Tower and add the v3 user.
bash
tmsh modify sys snmp allowed-addresses add { <tower-ip> } users add { spitfire_monitor { username spitfire_monitor auth-protocol sha auth-password '<auth-password>' privacy-protocol aes privacy-password '<priv-password>' oid-subset .1 access ro } } && tmsh save sys config

pfSense and OPNsense

  1. Enable the SNMP service.
text
pfSense: Services → SNMP → Enable, Bind interface = the interface facing the Tower
OPNsense: System → Firmware → Plugins → install os-net-snmp, Services → Net-SNMP → Enable
  1. Add the v3 user (OPNsense: Services → Net-SNMP → Users, Read only).
text
User: spitfire_monitor, Auth: SHA, Encryption: AES, Read only

pfSense's built-in SNMP speaks v2c: limit the community to the Tower's IP.

Ubiquiti (UniFi, EdgeOS)

  1. On EdgeOS, add the v3 user.
text
configure
set service snmp v3 user spitfire_monitor auth type sha
set service snmp v3 user spitfire_monitor auth plaintext-password <auth-password>
set service snmp v3 user spitfire_monitor privacy type aes
set service snmp v3 user spitfire_monitor privacy plaintext-password <priv-password>
set service snmp v3 user spitfire_monitor mode ro
commit; save
  1. On UniFi: Network → Settings → System → SNMP → v3, user and password.
text
Version: SNMPv3, Username: spitfire_monitor, Password: <auth-password>

Windows

The Windows SNMP service speaks v1/v2c only and is deprecated by Microsoft; there is no setup file for Windows. Add Windows servers with host monitoring. If nothing else works, v2c:

  1. Install the SNMP service.
powershell
Add-WindowsCapability -Online -Name "SNMP.Client~~~~0.0.1.0"
  1. Open the community read only, to the Tower only.
powershell
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\SNMP\Parameters\ValidCommunities" -Name "<community>" -Value 4 -PropertyType DWord -Force; New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\SNMP\Parameters\PermittedManagers" -Name "1" -Value "<tower-ip>" -PropertyType String -Force; Restart-Service SNMP; Write-Output "OK: SNMP v2c on"

What is read

Group Contents
System sysName, sysDescr, sysUpTime, sysObjectID
IF-MIB Interface name, status, speed, traffic, errors and discards
HOST-RESOURCES, UCD CPU, memory, storage
ENTITY-SENSOR Temperature, fans, power supplies
BGP4, OSPF Neighbor states
Product profiles Picked by sysObjectID: Cisco, Juniper, Arista, HPE/Aruba, MikroTik, FortiGate, Palo Alto, Check Point, F5, pfSense/OPNsense, Ubiquiti, net-snmp

Traps are counted on the Tower; the device view shows the last 24 hours' trap count.

Threshold findings

Finding Severity
The device does not answer critical
The device restarted (sysUpTime reset) warning
An interface is down while admin up warning; critical on an uplink
An interface above 90% used warning
Errors or discards rising on an interface warning
CPU above 90% for 15 minutes warning
Storage or memory above 90% used warning
A sensor reports a failure / is out of its normal range critical / warning
A BGP neighbor is not established critical
An OSPF neighbor is not full critical or warning
A VPN tunnel is down (FortiGate) critical
The HA state changed / a member is out of sync warning
An F5 pool / member / virtual server is down critical / warning / warning
The session table above 80% used warning

Uplink: the interface names in the Uplinks field, or those with "uplink" in their alias.

Test results

Code Meaning
ok The device answers; its sysName and profile are shown
timeout No answer: in v2c the community may be wrong; otherwise the ACL, the firewall (UDP 161) or the agent is off
auth Wrong v3 user, password or protocol
dns The device name does not resolve
unreachable The device or its network cannot be reached
permission The user's view does not include the system group

Coming soon

  • NetFlow, sFlow and IPFIX flow data.
  • Cloud-managed devices (through the cloud controller's API).