Monitoring Suricata detection
The Suricata IDS is an extremely nice piece of software with multiple deployment scenarios, including inline and with mirroring or taps. The process itself usually runs without any problems, but as with any setup with multiple moving parts there are things that can go wrong.
Monitoring and making sure the Suricata processes itself is easy enough. Systemd can make efforts to keep a process running, and tools like Monit can assist in difficult cases. If something goes really wrong there are many monitoring and alerting suites available, for instance Icinga and Naemon.
While the process itself may run as it’s supposed to, something may happen to its surrounding environment. Hence, what we need to monitor is “aliveness”, i.e. making sure Suricata receives and parses the information it should see. Just like a full functionality test of an email server requires sending an email and making sure it passes through the whole delivery chain, a proper approach to verify IDS functionality is to submit some identifiable traffic and check whether the IDS sees it – basically a heartbeat check. I’ve implemented this using a simple cron job and an accompanying Icinga2 check.
The cron job
The cron job runs every minute, using hping3 to craft an ICMP packet with a customized payload. While a regular ping packet (ICMP echo request) sent from a Linux system usually carries a payload of special characters and some number (!”#$%&‘()*+,-./01234567), the detection packet must have a different and identifiable payload.
# cat /etc/cron.d/suricata-heartbeat-ping
* * * * * root /usr/sbin/hping3 --count 1 --icmp --file /root/payload.txt --data 14 some.IP.address 1>/dev/null 2>&1
# cat /root/payload.txt
heartbeat_ping
In my case this cron job runs on an internal system pinging a host on the Internet. The probes can easily be set up the other way, from the outside to a host protected by the IDS.
The IDS detection rule
The IDS must be told how to detect the crafted payload ping packet. This rather simple rule defines that an ICMP echo request packet (code 0, type 8) containing the payload “heartbeat_ping” should trigger an alert:
alert icmp $HOME_NET any -> $EXTERNAL_NET any (msg:"PROTOCOL-ICMP Heartbeat ping"; icode:0; itype:8; content:"heartbeat_ping"; classtype:misc-activity;sid:5100000;rev:1;)
Whenever Suricata sees the customized ping, it logs an alert. Where it logs may differ between installations, but in this setup alerts are explicitly logged to /var/log/suricata/fast.log. Thus, every minute Suricata will add a log entry looking like this:
[**] [1:5100000:1] PROTOCOL-ICMP Heartbeat ping [**] [Classification: Misc activity] [Priority: 3] {ICMP} internal.IP:8 -> some.IP.address:0
The Icinga check
To detect the presence or absence of the heartbeat packets, a short shell script involving logtail is used. Logtail keeps track of how much of a file has already been read, and shows us only what has happened since. Logtail handles log rotation brilliantly so there’s no false positives during log cleanup.
The following scripts looks for and counts new occurrences of “PROTOCOL-ICMP Heartbeat ping“, which is what Suricata logs when observing the heartbeat packet. If there are no occurrences the script will exit with a status code Icinga considers critical, otherwise the script will exit normally.
#!/bin/bash
filename=$(basename $0)
COUNT=$(/usr/sbin/logtail -f /var/log/suricata/fast.log -o /var/tmp/${filename}.offset | \
grep "PROTOCOL-ICMP Heartbeat ping" | \
wc -l)
if [ "$COUNT" -gt 0 ]; then
echo "OK: $COUNT heartbeat alerts observed"
exit 0
else
echo "CRITICAL: No heartbeat alerts observed"
exit 2
fi
The script needs to run with sudo permissions. Below is the command definition used in Icinga2.
object CheckCommand "check_suricata_heartbeat" {
import "plugin-check-command"
command = [ "/usr/bin/sudo", "/usr/local/lib/icinga2/check_suricata_heartbeat"]
}
Finally the Icinga2 service definition, associated with the IDS host:
object Service "check_suricata_heartbeat" {
import "generic-service"
display_name = "Suricata alert detections"
host_name = "IDS_host"
check_command = "check_suricata_heartbeat"
command_endpoint = "IDS_host"
}
