SpitfireLoad Testing & Observabilityv1.4.9 · What's new

Version: 1.4.9This documentation is for Spitfire 1.4.9.

Log collection (syslog, NetScaler, files, Fluent Bit, Splunk, journald, Windows, Kubernetes)

The install and test steps are below. For how log records show on the Backend tab see Observability → Logs.

Before you start

  1. Settings → General → Public address is set.
    text
    https://spitfire.example.com
  2. Add an OTLP receiver and copy its sfotlp_ token.
    text
    Observability → Add → OTLP receiver → Save → Copy the token
  3. On the Log collectors card pick the source and press Generate configuration; the commands below come ready there with your address.
    text
    Observability → Log collectors → <source> → Generate configuration

One-line install

Linux (Docker or the otelcol-contrib package; the token is asked hidden):

bash
curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=syslog bash /tmp/sf-logs.sh

Windows (open PowerShell as administrator):

powershell
$env:SF_SOURCE='windowseventlog'; irm "https://<spitfire>/api/v1/setup/ds/logs.ps1" | iex

SF_SOURCE values: syslog, netscaler, filelog, fluentforward, splunk_hec, journald, logstash, vector (Linux); windowseventlog, filelog (Windows).

To remove it:

bash
SF_ACTION=uninstall SF_SOURCE=syslog bash /tmp/sf-logs.sh

Success check

  1. Send a test record (syslog example).
    bash
    logger --rfc5424 -d -n 127.0.0.1 -P 5514 -p user.err -t spitfire-test "spitfire-test ERROR log collection works"
  2. On the card the First records row turns green: 1 records arrived and spitfire-test.
    text
    Observability → Log collectors → First records
  3. If nothing comes, the Collector's log:
    bash
    sudo docker logs spitfire-logs-syslog 2>&1 | tail -n 30 || sudo journalctl -u otelcol-contrib -n 30 --no-pager

Syslog

  1. Install:
    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=syslog SF_PORT=5514 SF_TRANSPORT=udp SF_PROTOCOL=rfc5424 bash /tmp/sf-logs.sh
  2. Point rsyslog at the Collector:
    bash
    echo '*.* @<collector-host>:5514;RSYSLOG_SyslogProtocol23Format' | sudo tee /etc/rsyslog.d/90-spitfire.conf && sudo systemctl restart rsyslog
  3. Check: logger -p user.err -t spitfire-test "spitfire-test ERROR" → First records.

NetScaler (Citrix ADC)

  1. Install (NetScaler sends RFC 3164):
    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=netscaler SF_PORT=5514 SF_TRANSPORT=udp SF_PROTOCOL=rfc3164 bash /tmp/sf-logs.sh
  2. NetScaler CLI: the syslog action, policy and global bind.
    text
    add audit syslogAction sf_syslog <collector-ip> -serverPort 5514 -transport UDP -logLevel ALL -userDefinedAuditlog YES
add audit syslogPolicy sf_syslog_pol true sf_syslog
bind audit syslogGlobal -policyName sf_syslog_pol -priority 100
  3. A line per request (client IP, vserver, method, URL, HTTP status):
    text
    add audit messageaction sf_http INFORMATIONAL "\"client=\" + CLIENT.IP.SRC + \" vserver=\" + HTTP.REQ.LB_VSERVER.NAME + \" method=\" + HTTP.REQ.METHOD + \" url=\" + HTTP.REQ.URL.PATH + \" status=\" + HTTP.RES.STATUS"
add rewrite policy sf_http_pol true NOREWRITE -logAction sf_http
bind lb vserver <vserver-name> -policyName sf_http_pol -priority 900 -gotoPriorityExpression NEXT -type RESPONSE
save ns config
  4. Check (on the Collector host):
    bash
    printf '<131>%s 10.0.0.10 0-PPE-0 : default AAA Message 1 0 :  "client=203.0.113.7 vserver=vs_web method=GET url=/api/orders status=503 rt_ms=1840"\n' "$(date '+%b %e %H:%M:%S')" | nc -u -w1 127.0.0.1 5514

Parsed fields:

Source Fields
Native format (Source, Vserver, Client_ip) client.address, netscaler.vserver
CEF / audit (AppFW: src=, request=, method=, act=) client.address, url.path, http.request.method, netscaler.module, netscaler.event, netscaler.action; CEF severity 7+ is an error
Message action (client= vserver= method= url= status= rt_ms=) client.address, netscaler.vserver, http.request.method, url.path, http.response.status_code (5xx error, 4xx warn), http.server.request.duration (s)
Note

For the response time use an expression that adds rt_ms=<milliseconds> to the line (your ADC version's transaction time expression); without it the field stays empty and the others are parsed.

Note

AppFlow/IPFIX: the contrib Collector's netflow receiver (alpha) takes IPFIX but cannot decode NetScaler's AppFlow HTTP templates (URL, status, response time). Use the syslog path above for the HTTP fields. NetScaler metrics (NITRO API) are in Infrastructure sources.

Log files

  1. Install (nginx example; under Docker the files' folder is mounted read-only):
    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=filelog SF_PATHS='/var/log/nginx/access.log' SF_PARSER=nginx bash /tmp/sf-logs.sh
  2. Check:
    bash
    echo "$(date -u +%FT%TZ) ERROR spitfire-test log collection works" | sudo tee -a /var/log/app/app.log

Line formats: none, json, nginx, apache (combined; 5xx error, 4xx warn), regex (named groups; on the card only, the script uses none).

Fluentd / Fluent Bit

  1. Install:
    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=fluentforward SF_PORT=24224 bash /tmp/sf-logs.sh
  2. Fluent Bit output (fluent-bit.conf):
    text
    [OUTPUT]
    Name  forward
    Match *
    Host  <collector-host>
    Port  24224
  3. Fluentd output:
    text
    <match **>
  @type forward
  <server>
    host <collector-host>
    port 24224
  </server>
</match>

Splunk

Splunk connects two ways:

Way What Where
Splunk HEC (push) an app, Fluent Bit, Vector or OTel sending to HEC → Collector :8088 → Spitfire Log collectors → Splunk HEC
Splunk search API (pull) Spitfire → Splunk management port :8089, /services/search/jobs/export, read-only Connections → Log stores → Splunk

HEC install:

  1. bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=splunk_hec SF_PORT=8088 bash /tmp/sf-logs.sh
  2. Point the sender's HEC address at the Collector:
    text
    http://<collector-host>:8088/services/collector/event
  3. Check:
    bash
    curl -sS http://127.0.0.1:8088/services/collector/event -H 'Authorization: Splunk any' -d '{"event":"spitfire-test ERROR log collection works","source":"spitfire-test"}'

Search API connection:

  1. Create a token with the search capability in Splunk.
    text
    Settings → Tokens → New Token
  2. Add the connection in Spitfire.
    text
    Observability → Add → Splunk   URL: https://splunk.corp:8089   Auth: Splunk token (Bearer)   Index: main
  3. Test connection → the Splunk version shows.
Note

The Splunk connection comes with the monitoring events add-on.

journald

  1. bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=journald SF_UNITS=nginx.service,api.service bash /tmp/sf-logs.sh
  2. Check:
    bash
    logger -p user.err -t spitfire-test "spitfire-test ERROR log collection works"
Note

The journald receiver is alpha in contrib; the script always installs the package (it needs journalctl).

Windows event logs

  1. Open PowerShell as administrator:
    powershell
    $env:SF_SOURCE='windowseventlog'; $env:SF_CHANNELS='Application,System'; irm "https://<spitfire>/api/v1/setup/ds/logs.ps1" | iex
  2. Check (the script writes one too):
    powershell
    Write-EventLog -LogName Application -Source 'spitfire-test' -EventId 1 -EntryType Error -Message 'spitfire-test ERROR log collection works'
Note

The token is in C:\ProgramData\Spitfire\otlp-token (only SYSTEM and Administrators read it). The windowseventlog receiver is alpha in contrib.

Kubernetes

  1. The namespace and the token secret:
    bash
    kubectl create namespace spitfire-logs --dry-run=client -o yaml | kubectl apply -f -
read -rsp "OTLP token (sfotlp_...): " SF_T && echo && kubectl -n spitfire-logs create secret generic spitfire-otlp --from-literal=token="$SF_T" --dry-run=client -o yaml | kubectl apply -f - && unset SF_T
  2. The Helm values and the install:
    bash
    curl -fsSL "https://<spitfire>/api/v1/setup/logs/config?source=kubernetes&events=1&format=yaml" -o /tmp/spitfire-logs-values.yaml
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts && helm repo update open-telemetry
helm upgrade --install spitfire-logs open-telemetry/opentelemetry-collector -n spitfire-logs -f /tmp/spitfire-logs-values.yaml
  3. Check:
    bash
    kubectl -n spitfire-logs rollout status daemonset -l app.kubernetes.io/instance=spitfire-logs --timeout=120s
kubectl run spitfire-test --rm -i --restart=Never --image=busybox:1.36 -- sh -c 'echo "ERROR spitfire-test log collection works"'

Logstash and Vector

  1. bash
    curl -fsSL "https://<spitfire>/api/v1/setup/ds/logs.sh" -o /tmp/sf-logs.sh && SF_SOURCE=logstash SF_PORT=5170 bash /tmp/sf-logs.sh
  2. Logstash output (spitfire.conf):
    text
    output { tcp { host => "<collector-host>" port => 5170 codec => json_lines } }
  3. Vector sink (spitfire.yaml; SF_SOURCE=vector):
    yaml
    sinks:
  spitfire:
    type: socket
    inputs: ["*"]
    address: "<collector-host>:5170"
    mode: tcp
    encoding:
      codec: json
    framing:
      method: newline_delimited
  4. Check:
    bash
    echo '{"level":"error","message":"spitfire-test ERROR log collection works","service.name":"spitfire-test"}' | nc -w1 127.0.0.1 5170

Reading from log stores (read-only)

Store URL Auth Read
Elasticsearch https://es.corp:9200 API key or user (read on the index) _search + date_histogram, 50 error examples
OpenSearch https://os.corp:9200 User (read on the index) _search + date_histogram, 50 error examples
Graylog https://graylog.corp:9000 Access token or user (Reader) /api/search/universal/absolute, at most 1000 messages
Splunk https://splunk.corp:8089 Token or user (search) /services/search/jobs/export: stats + error examples
Datadog site API key + Application key (logs_read_data) Logs aggregate + events search
New Relic data center User key NRQL FROM Log … FACET … TIMESERIES
  1. text
    Observability → Add → Elasticsearch | OpenSearch | Graylog | Splunk
  2. On a Datadog or New Relic connection: Read the run window's logs too → On.
  3. Test connection → Logs and load on the next run's Backend tab.