Tutorials Jan 24th 2015

Unexpected DDOS: Blocking China with ipset and iptables

When the Great Firewall of China starts hosing your server with unexpected and unrelated traffic, how do you deal with it?

Discovering a problem

Three times in the last week I've had email reports from my Linode's automatic warning system, informing me that the server had exceeded an average 8Mb/s output for a two hour period. Each time I logged on the traffic had gone right back down, and my website analytics never showed unusual traffic. By the third occurrence I wanted to get to the bottom of it, and I already had suspicions.

Those spikes are not normal.

Earlier in the day I'd stumbled across Craig Hockenberry's post Fear China, where he was seeing a similar (but larger) problem over a longer period than I was. I looked into my access logs… and discovered I did indeed have the same problem, though it looks like I caught it earlier., or it was less severe.

Being DDOS'd via the Great Firewall of China

Distributed Denial of Service attacks flood a server with pointless requests from many computers all at once.

My logs showed requests for services and URLs that had nothing to do with my server, including an awful lot of BitTorrent URLs. Checking the geolocation of the requesting IPs showed they were all inside China. As Craig's post covered – it looks a lot like there's a mis-configuration with China's state controlled firewall, and people's normal traffic is sometimes being sent to entirely the wrong servers.

I wondered how bad my server was getting hit, as it didn't seem to be in the same league as Craig's:

Almost 27Mb/s out is roughly 95 times greater than normal for that server – close to two orders of magnitude increase, and I didn't like that – I could imagine this getting worse rapidly.

Blocking China

As Craig discusses, there's really no option but to block everyone from China. Unfortunately for me, I wasn't using ipfw as a firewall so I couldn't follow his advice. Having finally figured out how to do this I thought I'd write a step-by-step guide assuming you've not got a firewall already set up.

Note; this all assumes you run Debian.

Set up iptables

iptables is a firewall application for Linux and it's already installed on Debian systems.

If you already have iptables set up and in use, skip this section and go straight to the ipset section.

Create a file where we can declare some rules to use:

sudo nano /etc/iptables.firewall.rules

Inside there you'll want to paste the following:

*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT

Save that. Next, we need to apply those rules – this is just a text file, and we need to instruct iptables to actually use it.

sudo iptables-restore < /etc/iptables.firewall.rules

That should have loaded the rules and applied them; you can check by

iptables -L

The output of that command ought to look like

Chain INPUT (policy ACCEPT)
   target     prot opt source               destination
   ACCEPT     all  --  anywhere             anywhere
   REJECT     all  --  anywhere             127.0.0.0/8          reject-with icmp-port-unreachable
   ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
   ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
   ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
   ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
   ACCEPT     icmp --  anywhere             anywhere
   LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
   DROP       all  --  anywhere             anywhere

   Chain FORWARD (policy ACCEPT)
   target     prot opt source               destination
   DROP       all  --  anywhere             anywhere

   Chain OUTPUT (policy ACCEPT)
   target     prot opt source               destination
   ACCEPT     all  --  anywhere             anywhere

Great, it's working! But if you reboot the server it won't be. So lets fix that by creating a file which will run at boot.

sudo nano /etc/network/if-pre-up.d/firewall

Inside that file paste:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.firewall.rules

Save it. Now we must make sure it's allowed to execute:

sudo chmod +x /etc/network/if-pre-up.d/firewall

Done. The firewall is now running with those rules applied and those rules will be re-applied every time the server reboots. But it's not blocking China yet; it's only blocking anything not on port 80 or 443 (http and https).

Using ipset to block China

You can't manually add a few thousand IP addresses to your iptables, and even doing it automatically is a bad idea because it can cause a lot of CPU load (or so I've read). Instead we can use ipset which is designed for this sort of thing. ipset handles big lists of ip addresses; you just create a list and then tell iptables to use that list in a rule.

Note; I assume that the entirety of the following is done as root. Adjust accordingly if your system is based on sudo.

apt-get install ipset

Next, I wrote a small Bash script to do all the work, which you should be able to understand from the comments in it. Create a file:

nano /etc/block-china.sh

Here's what you want to paste into it:

# Create the ipset list
ipset -N china hash:net

# remove any old list that might exist from previous runs of this script
rm cn.zone

# Pull the latest IP set for China
wget -P . http://www.ipdeny.com/ipblocks/data/countries/cn.zone

# Add each IP address from the downloaded list into the ipset 'china'
for i in $(cat /etc/cn.zone ); do ipset -A china $i; done

# Restore iptables
/sbin/iptables-restore < /etc/iptables.firewall.rules

Save the file. Make it executable:

chmod +x /etc/block-china.sh

This hasn't done anything yet, but it will in a minute when we run the script. First, we need to add a rule into iptables that refers to this new ipset list the script above defines:

nano /etc/iptables.firewall.rules

Add the following line:

-A INPUT -p tcp -m set --match-set china src -j DROP

Save the file. To be clear, my full iptables.firewall.rules now looks like this:

*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Block anything from China
# These rules are pulled from ipset's china list
# The source file is at /etc/cn.zone (which in turn is generated by a shell script at /etc/block-china.sh )
-A INPUT -p tcp -m set --match-set china src -j DROP

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT

Right now, nothing has changed with the server because no new rules have been applied; to do so, run the block-china.sh script:

/etc/block-china.sh

This should show some output as it pulls a fresh list of Chinese based IPs and then, after a few seconds or so, it will complete and drop you back to a command prompt.

To test if it worked, run:

iptables -L

You should now see a new rule blocking China – the output ought to look like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             loopback/8           reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
DROP       tcp  --  anywhere             anywhere             match-set china src
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
DROP       all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Almost done! This works, and will continue to work on re-boots. But, IP addresses change and that list will grow stale over time. If you want to pull and apply an updated list of IPs you can just run the block-china.sh script again.

We can also set the machine to do that automatically via a cron job:

crontab -e

Add a line such as this:

* 5 * * * /etc/block-china.sh

This will run /etc/block-china.sh at 5am every day. The user running the script will need to be root or have root privileges.

Going Further

I don't think there's much more a server admin can do to protect themselves against DDOS. The next thing to try would be a service such as CloudFlare, which has clever and automated protection from this sort of stuff.

References

As is often the case, this post is cobbled together from other sources as I figured out what needed doing. The ever reliable Linode documentation is responsible for the initial iptables part, and the gist of the ipset was from dr0u. My thanks to both!