Making working from home even lazier

Working from home encourages home office optimization. During the COVID-19 period, with way more video conferences than usual, certain improvements were found necessary.

Clipart downloaded from freepik. The following link is for attribution:
Business vector created by stories –

I guess everyone that’s been in a video meeting where more than one participant used a regular mic and regular speakers has experienced the wonders of audio feedback. That prompted me to acquire a comfortable gaming headset with a microphone, and I got myself a HyperX Cloud Flight wireless headset at a reasonable price.

I’m using GNOME on my Linux desktop. I find it quite efficient for my work, but the task of changing the sound output and input back and forth between my mic headset and the docking station’s line out and my web camera’s microphone requires just a bit too much point and click for one who prefers keyboard shortcuts for efficiency. To remedy this, I wrote a short bash script for toggling between the different audio components:


# Note: Only two inputs to toggle between.
WEBCAMMIC="HP Webcam HD 4310 Analog Stereo"
HEADSETMIC="HyperX Cloud Flight Wireless Headset Mono"

# Note: Only two outputs to toggle between.
LINEOUT="USB Audio Line Out"
HEADSET="HyperX Cloud Flight Wireless Headset Digital Stereo"

# Change input
INPUTSOURCE=$(pacmd list-sources 2>&1 | egrep 'index:|device\.description' | \
	egrep -B1 "($WEBCAMMIC|$HEADSETMIC)" | \
	grep '  index:' | \
	awk '{print $NF}' | \
	sed "s/'//")
pacmd set-default-source $INPUTSOURCE

# Change output
OUTPUTSINK=$(pacmd list-sinks 2>&1 | egrep 'index:|device\.description' | \
	egrep -B1 "($LINEOUT|$HEADSET)" | \
	grep '  index:' | \
	awk '{print $NF}' | \
	sed "s/'//")
pacmd set-default-sink $OUTPUTSINK

pacmd list-sink-inputs | grep index: | awk '{print $NF}' | while read INPUT; do
  pacmd move-sink-input $INPUT $OUTPUTSINK

In the script I’ve predefined two input sources and two output sinks (that’s the PulseAudio nomenclature) I’m toggling between. For video conferencing I switch input and output to my headset, and when the meeting is over I switch back to listening to music through my docking station’s line out connected to a speaker/subwoofer setup.

The script works by grepping through the results from various uses of the pacmd command, a PulseAudio command line tool. The index of an active input source or output sink is prefixed with an asterisk, so the active component will show up as something like * index: 3 while the other will show up as for instance index: 7 (there are two leading spaces before “index”). For both the input and output, the script identifies the index of the two allowed components and then activates the inactive one.

Now that I had a command line script that toggles my headset in and out for video conferencing, I mapped it to the keyboard shortcut ctrl+F1. At this point, I no longer had to use the mouse to change the sound preferences. But wait, there’s more!

To make the toggling even easier (or lazier), I acquired a USB triple foot pedal from eBay. In Linux, such foot pedals identify as programmable USB keyboards. Subsequently I programmed one of the pedals to send the key combination ctrl+F1, which again triggered the audio toggling script.


And the remaining two pedals, you may ask? I mapped the second pedal to mute the microphone, and when pressing the third my Squeezebox player skips to the next song 🙂

Blocking coronavirus scam mails in Postfix

As always, scammers and phishers use newsworthy events to their advantage. The coronavirus pandemic is no exception. All over the worlds, security researchers observe phishing and scam attempts. Samples for studying and for awareness training are collected at various sites, including

A large number of security researchers have joined forces to establish a cyber threat intelligence site at, providing free IT resources to combat cyber criminals seeking to exploit the COVID-19 situation. The site provides vetted and unvetted lists of IP addresses, domains, URLs and IOC hashes found in corona scams, both as downloadable text files and through Open Threat Exchange pulses.

If you’re already using OTX in your security infrastructure you might want to join the group through which you’ll get their pulses. If not, here’s a short bash script that creates a domain blacklist for use with the Postfix mail server. The script may be run in cron, but please be considerate and don’t run it too often.

TMP=mktemp /tmp/corona.XXXXXXXX
/usr/bin/wget -q -O "${TMP}"
/usr/bin/dos2unix "${TMP}" >/dev/null 2>&1
/bin/grep -v '^#' ${TMP} | /bin/sed 's/$/\tREJECT\tCorona scam/' > "${TMP}.new"
[ -s "${TMP}.new" ] && \
  /bin/mv "${TMP}.new" /etc/postfix/corona_access && \
  /usr/sbin/postmap /etc/postfix/corona_access
/bin/rm ${TMP}

You’ll also need a corresponding entry in your Postfix configuration. Add a check_sender_access check under smtpd_sender_restrictions, something like this:

smtpd_sender_restrictions =
  check_sender_access hash:/etc/postfix/corona_access

SMTP honeypots: Extracting events and decoding MIME headers with Logstash

One of my honeypots runs INetSim which, among many other services, emulates an SMTP server. The honeypot is frequently used by spammers who think they’ve found a mail server with easily guessed usernames and passwords. Obviously I’m logging the intruders’ activities, so I’m shipping the logs to Elasticsearch using Filebeat.

Shipping the regular INetSim activity logs is easy, but extracting the SMTP headers from the mail server honeypot was more of a challenge. INetSim logs each mail’s full content (header and body) in a file, by default on a Debian based system this file is /var/lib/inetsim/smtp/smtp.mbox. Every mail has a header and a body. The start of a header block is indicated by a first line starting with “From”, and the header ends with an empty linefeed followed by the mail body. After the mail’s body there’s a new empty linefeed, separating that mail from the next.

Spam, spam, wonderful spam
(Clip art courtesy of Clipart Library)

Extracting body content from an endless number of spam emails has limited value, but the headers are interesting. Information extracted from these can for instance be used to prime and improve spam filters as well as to identify and report compromised systems.

The following Filebeat multiline configuration extracts the interesting parts of each email:

        - "/var/lib/inetsim/smtp/smtp.mbox"
      type: log
      scan_frequency: 10s
      tail_files: true
      include_lines: ['^From [\w.+=:-]+(@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:[\.](?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)?(?:<.*>)? ']
      multiline.pattern: '^$'
      multiline.negate: true
      multiline.match: after

What happens in this config section is that every block of text between empty linefeeds is concatenated into single lines. That leaves us with one long line for the header and one long line for the body for every logged email. Then, because of the include_lines stanza, the config only cares about the lines starting with From followed by an email address, shipping only those to the Logstash server configured to receive logs from the honeypot.

Spammer activity on a slow day.

On the Logstash side, I’m first extracting what was the first of the header lines, i.e. the From address and the email’s timestamp, then I’m storing the rest of the content in a variable named smtpheaders.

Then I’m parsing the email’s timestamp, using this as the authoritative timestamp for this event.

After some cleanup, the contents of the smtpheaders variable, converted by Filebeat into one single line with the linefeed character \n to separate each header line, are extracted using the key/value (kv) plugin. This plugin puts each whitelisted header into a nested field under the smtp parent field, i.e. the email’s subject header is stored in Elasticsearch as [smtp][subject]. When all headers are mapped, the smtpheaders variable is no longer useful so it’s deleted.

filter {
  # Extract the (fake) sender address, the timestamp,
  # and all the other headers
  grok {
    # Note two spaces in From line
    match => { "message" => [
      "^From %{EMAILADDRESS:smtp.mail_from}\s+%{MBOXTIMESTAMP:mboxtimestamp}\n%{GREEDYDATA:smtpheaders}",
      "^From %{USERNAME:smtp.mail_from}\s+%{MBOXTIMESTAMP:mboxtimestamp}\n%{GREEDYDATA:smtpheaders}",
      "^From %{GREEDYDATA:smtp.mail_from}\s+%{MBOXTIMESTAMP:mboxtimestamp}\n%{GREEDYDATA:smtpheaders}"
    pattern_definitions => {
    remove_field => [ "message" ]
  # Extract the date from the email's header
  date {
    match => [ "mboxtimestamp",
               "EEE MMM dd HH:mm:ss yyyy",
               "EEE MMM  d HH:mm:ss yyyy" ]
  # Replace indentation in Received: header and possibly others
  mutate {
    gsub => [ "smtpheaders", "\n(\t|\s+)", " " ]
  # Map interesting header variables into an object named "smtp"
  kv {
    source => "smtpheaders"
    value_split => ':'
    field_split => '\n'
    target => "smtp"
    transform_key => "lowercase"
    trim_value => " "
    include_keys => [
    remove_field => ["smtpheaders"]

With the above configuration, INetSim provides a steady stream of events – one for every spam or phishing mail the intruders submit. Obviously the mail goes right into /dev/null so their efforts are completely wasted, while I’m getting nice Kibana dashboards telling how much spam my honeypot has caught (and deleted) today.

There’s one more thing, though. Each email usually has two From addresses; one often referred to as envelope-from (this is the first header line), and one called the header-from (the difference is explained here). While the envelope-from is a simple email address, the header-from can be almost anything including all kinds of international characters. Because an email header only allows a limited set of character, the header-from is often MIME encoded. For instance, an email from me would look like this:

From: Bjørn

But in the email header it would say:

From: =?UTF-8?Q?Bj=c3=b8rn?=

Since these headers aren’t decoded automatically, and since Logstash has no native decoding function for MIME, a small Logstash filter trick needs to be implemented. Logstash allows inline Ruby code, and the Ruby codebase provided with Logstash includes the mail gem, which in turn provides a MIME decode function. So I added the following to the Logstash filter:

    if [smtp][from] {
        ruby {
            init => "require 'mail'"
            code => "event.set('[smtp][from_decoded]',

The code first makes sure the mail gem is loaded, then it creates a new nested field [smtp][from_decoded] and inserts the value of [smtp][from] after applying Ruby’s Mail::Encodings.value_decode function. Equal decoding also happens on a few other headers, including the Subject header.

Some things are better left encoded.

With this in place, my honeypot Kibana dashboard displays all the headers properly, and I can read spam subjects like “Cîalis Professional 20ϻg 60 pȋlls” instead of wondering what “=?UTF-8?Q?C=C3=AEalis?= Professional =?UTF-8?Q?20=CF=BBg?= 60 =?UTF-8?Q?p?= =?UTF-8?Q?=C8=8Blls?=” really means.

A series of unfortunate events

A customer of my employer Redpill Linpro was recently the target of a DDoS attack. While investigating the attack, we found a large number of HTTP requests with the User-Agent named CITRIXRECEIVER. The clients performed GET requests to multiple URLs on the customer’s web site at the rate of several thousand packets per second. The originating IP addresses were mostly Norwegian and even registered on major companies and organizations, but we also registered multiple requests from south-east Asian countries like Singapore, Thailand, and Vietnam.

DDoS icon from

At first we didn’t think those were a part of the ongoing DDoS attack, but the frequencies and rates were alarming. The fact that they seemed to hammer the same set of URLs over and over again was also somewhat concerning, since we’ve previously experienced similar scripted attacks from botnets of hijacked browsers running in zombie mode without the computer owner’s knowledge.

Tracking down and analyzing the requests we found that this was probably just something as simple as misconfigured Citrix clients, set to poll a defined beacon as described here. When contacting the owner of one of the more aggressive IP ranges, they confirmed that the URL of the customer’s website had indeed been configured as an “am I alive or not” connectivity check for no less than 25 000 (!) client systems, which absolutely explained the insane polling rate. Those 25 000 clients were associated with a single company – additional requests poured in from other organizations as well.

However, the poll was designed to be HEAD requests which are supposed to be non-intrusive and harmless. So why did they come across as multi-URL GET requests, causing increased load on the customer’s systems?

DDoS icon from

The first part of the explanation is that Varnish, which is used extensively at the customer’s web site, will by default convert a HEAD request to a GET request before responding. In many cases this makes sense, while for plain polling purposes HEAD requests are often configured to shortcut the processing and return a simple response.

The second part has to do with Varnish’ advanced capacity of Edge Side Includes (ESI), which allows a mainly static web site to include one or more dynamic components. The customer’s front page URL includes certain uncacheable, per-visitor components, so when someone accesses the front page multiple dynamic components are processed and included before returning the content.

Since HEAD requests weren’t specifically cut short, the simple and theoretically harmless beacon polling from thousands of Citrix clients turned out to have quite some impact after all.

PS: Not surprisingly, sysadmins configuring the CITRIXRECEIVER beacon polling seem to select well-known and/or short URLs for their connectivity check. Used with BBC’s web site, CITRIXRECEIVER has been reported as the second most popular User-Agent.

Perfectly synchronized dual portscanning

The other day while reviewing my fireplot graphs, I noticed (yet) another portscan. They’re not unusual. This one took around four and a half hour to complete, and covered a lot of TCP ports on one IPv4 address. That’s not unusual either. The curved graph shown below is caused by the plot’s logarithmic Y axis, where approximately linear activity will be presented as a curve. The scanning speed changed above TCP port 10000, when the scanner started increasing the interval between ports hence the “elbow” in the graph. Such behaviour is seen less often.

Port scans in general are mostly harmless and not worth pursuing, but I often check the source IP address anyway. What caught my attention this time was that each TCP port probe was performed from the same two IP addresses at the exact same time.

Now and again, more than one source IP address inevitably hits the same destination address at the same time. In this case, however, the two source IP addresses, from Verizon and China Telecom respectively, probed each and every destination port in the port scan simultaneously. The second source IP address usually did one additional probe after about a second before they both moved on to the next port. Here’s a short log extract:

 06:18:02 - -> my.ip:444
06:18:02 - -> my.ip:444
06:18:03 - -> my.ip:444

06:19:25 - -> my.ip:445
06:19:25 - -> my.ip:445
06:19:26 - -> my.ip:445

06:19:55 - -> my.ip:465
06:19:55 - -> my.ip:465
06:19:55 - -> my.ip:465

06:21:42 - -> my.ip:502
06:21:42 - -> my.ip:502
06:21:43 - -> my.ip:502

06:22:15 - -> my.ip:503
06:22:15 - -> my.ip:503
06:22:16 - -> my.ip:503

06:23:17 - -> my.ip:515
06:23:17 - -> my.ip:515
06:23:18 - -> my.ip:515

06:25:00 - -> my.ip:523
06:25:00 - -> my.ip:523
06:25:01 - -> my.ip:523

06:26:34 - -> my.ip:548
06:26:34 - -> my.ip:548
06:26:35 - -> my.ip:548

06:27:27 - -> my.ip:554
06:27:27 - -> my.ip:554
06:27:27 - -> my.ip:554

06:28:09 - -> my.ip:587
06:28:09 - -> my.ip:587
06:28:10 - -> my.ip:587

06:30:25 - -> my.ip:631
06:30:25 - -> my.ip:631
06:30:26 - -> my.ip:631

06:31:20 - -> my.ip:636
06:31:20 - -> my.ip:636

06:32:04 - -> my.ip:666
06:32:04 - -> my.ip:666
06:32:05 - -> my.ip:666

06:33:15 - -> my.ip:774
06:33:15 - -> my.ip:774
06:33:16 - -> my.ip:774

06:34:31 - -> my.ip:789
06:34:31 - -> my.ip:789
06:34:32 - -> my.ip:789

06:36:05 - -> my.ip:873
06:36:05 - -> my.ip:873
06:36:05 - -> my.ip:873

06:36:52 - -> my.ip:902
06:36:52 - -> my.ip:902
06:36:52 - -> my.ip:902

As the log shows, the two source IP addresses, most likely unknowingly parts of a botnet, were working in perfect synchronization. The timing is too precise to be two individual scanning processes on opposite sides of the globe just started and left to run, so this scan was likely to have been under external control.

Honeypot intruders’ HTTP activity

One of my Cowrie honeypots has been configured to intercept various outbound connections, redirecting them into an INetSim honeypot offering corresponding services. When intruders think they’re making an outbound HTTPS connection, they only reach the INetSim server, where their attempts are registered and logged.

When someone successfully logs in to the Cowrie honeypot, be it bot or a real person, they often check their network location by polling some “check my IP” URL. This is particularly useful for automated bots who call home to report where they’ve gained a foothold.

Then we have the bots who use their login shell as a bouncer towards external services. Quite a few bots think they’ve found an open SMTP relay and spew out large amounts of spam or phishing mail (all going into the /dev/null sink of INetSim).

Others again use the shell to bruteforce web services for logging in to existing accounts with compromised credentials, or to create new users. The latter is particularly common with social media botnets. There are also some attempts made towards Amazon’s “address change” URL, probably to redirect deliveries.

Without further ado, below is top 30 from the last 40 or so days of URL gathering, focusing on bots that use the HTTP POST method to submit data. I’ve made some aggregations for services using load distributing hostnames (in this extract Omegle and Uber). I suspect the “winner” is the victim of some gift card scheme.


Nagios or Icinga plugin for Mikrotik software and firmware version

When upgrading the software (RouterOS) on Mikrotik devices, you should usually also make sure the firmware (RouterBoot) is upgraded to the same level.

In the devices’ various management interfaces including command line, the OS will tell you that there are outstanding firmware patches if you ask it, like this:

/system routerboard print 
routerboard: yes
current-firmware: 3.24
upgrade-firmware: 5.2.1

Or, if you’ve configured the unit for automatic firmware upgrade after a software upgrade, you will be greeted by a message like this at login time:

Firmware upgraded successfully, please reboot for changes to take effect!

Previously there were no logical way to compare matching versions between the OS software and the boot firmware, but some time ago the vendor started aligning the two components’ version numbers and that made today’s small endeavor much easier. When it’s this easy to check whether the firmware upgrade was forgotten after a software upgrade, it took just a few minutes to write a shell script to be used in Icinga or Nagios.

The script takes a couple of arguments: -H for hostname, -c for entering the SNMP community, and -v for versions 1 or 2c. Sorry, I haven’t made it SNMPv3 compatible yet. If you’ve forgotten a firmware upgrade, the script will issue a WARNING text with a corresponding exit value 1, but it also accepts a -C argument to return a CRITICAL state and exit value 2.

while getopts "H:c:v:C" opt; do
case $opt in
echo "Invalid option: -$OPTARG" >&2
FW=$(snmpwalk -Ov -On -Oq -Cc -c $COMMUNITY -v $VERSION $HOST .
SW=$(snmpwalk -Ov -On -Oq -Cc -c $COMMUNITY -v $VERSION $HOST .
if [ "$FW" == "$SW" ]; then
echo "OK: Software and firmware versions match ($SW)"
exit 0
if [ $CRITICAL -gt 0 ]; then
echo "CRITICAL: Software version $SW does not match firmware version $FW"
exit 2
echo "WARNING: Software version $SW does not match firmware version $FW"
exit 1

A test run from the shell provides useful output:

./check_mikrotik_sw_fw -H devicename -c snmpsecret
WARNING: Software version 6.44 does not match firmware version 6.43.12

After configuring Icinga2 to check the Mikrotik devices, I got a nice overview of outstanding tasks:

Status at blog time. They are all upgraded now 😉

Coming up: Write a plugin to warn about units that don’t use the most recent versions. But that’ll be for another blog entry!

Updating wordlists from Elasticsearch

Among the many benefits of running a honeypot is gathering the credentials intruders try in order to log in. As explained in some earlier blog posts, my Cowrie honeypots are redirecting secondary connections to another honeypot running INetSim. For example, an intruder logged in to a Cowrie honeypot may use the established foothold to make further attempts towards other services. INetSim regularly logs various attempts to create fake Facebook profiles, log in to various mail accounts, and submit product reviews.


Top 15 hostnames that honeypot intruders try to submit data to


INetSim activity is obviously tracked as well, which means that login credentials used by Cowrie intruders to gain further access elsewhere will also be stored. I’m logging all honeypot activity to Elasticsearch for easy analysis and for making nice visualizations.


Most recent usernames and passwords used by intruders


Real passwords are always nice to have for populating wordlists used for e.g. password quality assurance, as dictionary attacks are often more efficient than bruteforcing. For this purpose I’m maintaining a local password list extracted from Elasticsearch. With the recent addition of the SQL interface, this extraction process was easy to script.


TODAY=$(date +%Y.%m.%d)

echo "select \"cowrie.password\" from \"logstash-${TODAY}\" \
 where \"cowrie.password\" is not null;" \
 | /usr/share/elasticsearch/bin/elasticsearch-sql-cli 2>&1 \
 | tail -n +7 | head -n -1 | sort -u \
 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
 | while read p; do
   grep -qax "${p}" ${PASSFILE} || echo "$p" | tee -a ${PASSFILE}

echo "select \"password\" from \"logstash-${TODAY}\" \
 WHERE \"service\" IS NOT NULL AND \"password\" IS NOT NULL\
 AND MATCH(tags, 'inetsim');" \
 | /usr/share/elasticsearch/bin/elasticsearch-sql-cli 2>&1 \
 | tail -n +7 | head -n -1 | sort -u \
 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
 | while read p; do
   grep -qax "${p}" ${PASSFILE} || echo "$p" | tee -a ${PASSFILE}


This script (although with some more pipes and filters) is regularly run by cron, continuously adding more fresh passwords to the local wordlist.

X-Forwarded-For DDoS

A discussion forum of one of Redpill Linpro‘s customers has been under attack lately, through a number of DoS and DDoS variants. Today’s attack strain was of the rather interesting kind, as one of its very distinctive identifiers was a suspicious, not to say ridiculous, amount of IP addresses in the incoming X-Forwarded-For HTTP header. The X-Forwarded-For IP addresses included both IPv4 and IPv6 addresses.

The longest X-F-F header observed contained no less than 20 IP addresses that the HTTP request had allegedly been forwarded through on its way to the forum. If we are to believe the headers, this particular request has been following this route: United States → United States → South Africa → United States → United States → Mexico → Uruguay → China → Germany → United States → United States → South Africa → United States → United States → Mexico → Uruguay → China → Germany → Costa Rica → Norway.

This short animation (click to play) illustrates a few of the the alleged routes:


Whether the HTTP requests have indeed been proxied through all these relays is difficult to confirm. By their reverse DNS lookup, quite a few of the IP addresses identify themselves as proxy servers. Checking a sample of the listed IP addresses did not reveal any open proxies or other kinds of relays, neither were they listed on random open relay blacklists. The HTTP headers included the “Via:” header as well, indicating that the request did pass through some HTTP proxies. But as we know, incoming headers can’t be trusted and should never be treated as if they could.

For the purpose of blocking the DDoS attack, it’s not really interesting whether the intermediate IP addresses are real or just faked. We simply reconfigured Varnish to check each incoming HTTP request for two things:

  • Does the X-Forwarded-For header have more than five IP addresses?
  • Is the request destined for the forum currently under siege?

All requests matching the above criteria were then efficiently rejected with the well-known, all-purpose 418 I’m a teapot HTTP response. After a minute or two of serving 418 responses, the attack stopped abruptly.

Control code usernames in telnet honeypot

By running a Cowrie honeypot, I’m gathering interesting information about various kinds of exploits, vulnerabilities, and botnets. Upon a discovery of a new Linux-based vulnerability – often targeting network routers, IoT devices, and lately many IP camera products – the botnets will usually come in waves, testing the new exploits.

The honeypot logs everything the intruders to. In addition to extracting and submitting useful indicators to threat intelligence resources like VirusTotal and AlienVault’s Open Threat Exchange, I’m processing the logs in an Elastic stack for graphing and trending. As shown below, there’s a section of the Kibana dashboard that details activity by time range and geolocation, and I’m also listing the top 10 usernames and passwords used by intruders trying to gain access.

Parts of my Cowrie dashboard in Kibana

Parts of my Cowrie dashboard in Kibana

This morning I briefly checked the top 10 usernames tag cloud when something unusual caught my eye.

KIbana username tag cloud

It wasn’t the UTF rectangle added to the “shell” and the “enable” user names, these are really shell\u0000, enable\u0000, sh\u0000 and are appearing quite frequently nowadays. What caught my eye was this tiny, two-character username, looking like an upside-down version of the astrological sign for Leo and a zigzag arrow.

Weird username from tag cloud

Upon closer inspection, the username is actually \u0014\t\t\u0012 – “Device Control 4”, two TABs, and “Device Control 2”.

One of the passwords used with this username was \u0002\u0003\u0000\u0007\u0013 – visualized in Kibana as follows:

Other passwords from the same IPs also include different control codes, beautifully visualized by Kibana as shown below:

From the Cowrie logs, the first occurrences in my honeynet were 2017-12-16. Exactly what kind of vulnerability these control codes are targeting is not known to me yet, but I am sure we will find out over the next few days.