SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Infrastructure — NetScaler and Citrix (Beta)

NetScaler (Citrix ADC) and Citrix Virtual Apps and Desktops / Citrix DaaS are read by the Tower itself over their REST APIs; no Collector or agent is installed. Each NetScaler (or HA pair) and each Citrix site counts as one monitored system. Both are Beta.

NetScaler

The Tower reads the NITRO API every 30 seconds: CPU (packet engine and management), memory, throughput, HTTP, SSL, interfaces, and every load balancing vserver with its members.

Read-only user

  1. Open the NetScaler CLI (on an HA pair, the primary; it syncs).

    bash
    ssh nsroot@<nsip>

  2. Create a command policy that allows only stat and show of what is read.

    text
    add system cmdPolicy spitfire_readonly ALLOW "(^stat\s+(system|ns|protocol\s+http|ssl|interface|lb\s+vserver)(\s.*)?$)|(^show\s+(lb\s+vserver|service|serviceGroup|ns\s+version)(\s.*)?$)"

  3. Create the API-only local user.

    text
    add system user spitfire_monitor '<password>' -externalAuth DISABLED -allowedManagementInterface API

  4. Bind the policy and save.

    text
    bind system user spitfire_monitor spitfire_readonly 100
save ns config

  5. Test a read from the Tower's machine.

    bash
    curl -fsS -k -H 'X-NITRO-USER: spitfire_monitor' -H 'X-NITRO-PASS: <password>' 'https://<nsip>/nitro/v1/stat/lbvserver' | grep -q '"errorcode": *0' && echo OK

  6. Add it under Infrastructure sources → NetScaler: address https://<nsip>, user spitfire_monitor.

Note

If the connection test answers permission (older firmware words some commands differently), bind the built-in read-only policy instead: bind system user spitfire_monitor read-only 100. The user cannot log in to the GUI or the shell. Tick Skip TLS certificate verification only for the appliance's own self-signed certificate.

vserver table

The NetScaler dashboard has a table of the load balancing vservers: state, health, members UP and DOWN, requests per second, client and server connections and response time. Troubled vservers come first. A second table lists every DOWN member as vserver / service ip:port.

Note

NITRO has no 4xx/5xx counters. The HTTP errors panel shows server busy and incomplete requests or responses; status codes need the NetScaler logs or AppFlow.

Citrix Virtual Apps and Desktops

The Tower reads the Monitor Service OData API (the data behind Director) every 60 seconds: active and disconnected sessions, machines per delivery group (registered, unregistered, maintenance, faulted, load index), logon duration by phase and connection failures. On-premises uses NTLM to a Delivery Controller; Citrix DaaS uses a Citrix Cloud API client.

On-premises: read-only administrator

  1. Create the AD account CORP\spitfire_monitor (a normal domain user).

  2. On a Delivery Controller, open PowerShell as a Full Administrator of the site.

    powershell
    Start-Process powershell -Verb RunAs

  3. Give the account the built-in Read Only Administrator role.

    powershell
    Add-PSSnapin Citrix*
New-AdminAdministrator -Name 'CORP\spitfire_monitor'
Add-AdminRight -Administrator 'CORP\spitfire_monitor' -Role 'Read Only Administrator' -Scope 'All'

  4. Test a read from the Tower's machine.

    bash
    curl -fsS --ntlm -u 'CORP\spitfire_monitor:<password>' 'https://ddc1.corp/Citrix/Monitor/OData/v4/Data/Sessions?$top=1' | grep -q '"value"' && echo OK

  5. Add it under Infrastructure sources → Citrix: mode On-premises, address https://ddc1.corp, domain CORP, user spitfire_monitor.

Citrix DaaS: API client

  1. In Citrix Cloud open Identity and Access Management → API access → Secure clients and create a client with the access policy Read Only Administrator, All.

  2. Copy the customer ID, the client ID and the secret.

  3. Test the client.

    bash
    curl -fsS -d grant_type=client_credentials -d client_id='<client-id>' -d client_secret='<client-secret>' 'https://api.cloud.com/cctrustoauth2/root/tokens/clients' | grep -q access_token && echo OK

  4. Add it under Infrastructure sources → Citrix: mode Citrix DaaS, the customer ID, the client ID and the secret; region Japan only for api.citrixcloud.jp.

Note

The Read Only Administrator role sees the whole site but cannot change it. NTLM is sent only over HTTPS. Use HTTP to a controller only in a lab; the Tower then refuses to send the password.

Delivery group table

The Citrix dashboard has a table of delivery groups: registered, unregistered, maintenance and faulted machines, sessions and the average and highest load index (0–10000). Groups with unregistered machines come first, with up to 50 machine names in the finding.

Findings

Technology Finding
NetScaler NITRO does not answer (critical)
NetScaler Packet engine CPU, management CPU or memory above 90% (warning)
NetScaler "Server busy" errors or interface errors growing in 15 minutes (warning)
NetScaler A vserver is DOWN (critical)
NetScaler Members DOWN behind a vserver that is up (warning)
NetScaler Members went DOWN during the window (critical)
NetScaler Response time rises with load: the busiest third of the window has 1.5 times the requests and twice the response time of the quietest third, above 50 ms (warning)
Citrix The Monitor API does not answer (critical)
Citrix Connection failures in the last 5 minutes (warning)
Citrix Unregistered or faulted machines (warning)
Citrix A delivery group's load index above 8000 (warning)
Citrix Logon duration above 60 s (warning)
Citrix Logon duration rises with sessions: 1.5 times as slow in the busiest third, at least 20 s (warning)

The CPU, memory, busy and failure rules look at the last 15 minutes. The tables and the other findings look at the last 15 minutes too, or at a test run's window with the API: GET /api/v1/datasources/<id>/dashboard?run=<run-id> (from 1 minute before the run starts to 1 minute after it ends, at most 6 hours).

Success check

  1. Last test = Connected.
  2. On the NetScaler dashboard NITRO answers is 1 and the vserver table is filled.
  3. On the Citrix dashboard Monitor API answers is 1 and the delivery group table is filled.