Locking Down Your Linux Server with APF + BFD

If you allow SSH access to your server, there are some simple steps you can take to restrict access and protect yourself from brute force attacks. Two of my favorite scripts to do this are Advanced Policy Firewall coupled with Brute Force Detection, both by R-FX Networks.

Note: this article is long, but that’s only because I’m trying to explain in some degree of detail, and call out specific gotchas that you may run into. I promise you that setting these two up is incredibly easy, and shouldn’t take you more than a twenty minutes or so to have both up and running.

Advanced Policy Firewall (APF) is an iptables based firewall system that’s easy to set up and administer, and works hand in hand with Brute Force Detection (BFD).

BFD is a modular shell script for parsing application logs and checking for authentication failures. It does this using a rules system where application specific options are stored including regular expressions for each unique auth format.

Together, they provide a simple but effective way to handle locking out brute force login attempts. Using APF, you could actually take it a step further and deny ALL SHH requests except those originating from a set of whitelisted IP addresses. This may not be feasible – or a good idea – if you do not have access to a static IP address, however, since you could end up locked out of your own box. We get into restricted whitelisting a little further down the page.

The basic gist is this: Someone tries to brute force their way into your server via SSH. Since they do not actually have a valid username+password combination, the login attempt will fail, assuming you don’t use shitty passwords that can be easily guessed, in which case they login successfully, and you’re pwned. After x failed attempts (where you define x in the configuration file), BFD will automagically tell APF to add the IP address of the offending attacker to the APF blacklist for a certain amount of time (also configurable in the config file). All services will be denied to that IP address, so they will no longer even be able to see your website.

The purpose of this is pretty obvious, but (for those of you who took the short bus in) one of the primary benefits is the ability to easily mitigate automated brute force attacks on your server, where a script is being used to try various combinations of usernames and passwords until they successfully login.

If you think your server is too insignificant for an attacker to bother with, you’re wrong. If you have an IP address that is visible to the rest of the world, you will end up being brute-forced at some point. Whether or not the attack is successful is up to you.

There are other firewall+brute-force-detection combinations out there, including the very popular Fail2ban, that also work very well. I’m not endorsing one over the other, I’m just more familiar with APF+BFD.

Anyway – let’s get to the good stuff. Note that you will need root/sudo access to your server in order to continue.

Setting Up Advanced Policy Firewall (APF)

Before moving forward, it should be noted that you are installing an iptables-based firewall. This means that if you screw something up, you could lock yourself out of the server, deny all traffic to your server resulting in a downed website, etc. Be careful, and don’t make these kinds of configurations on a production machine during peak site traffic hours. To test that your configuration is working properly, you’ll want to have access to SSH from another IP address that you can safely lock out without limiting your ability to administer the server.

1. [root@server]# cd /root/downloads (or any other temporary folder)
2. [root@server]# wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz
3. [root@server]# tar -xvzf apf-current.tar.gz
4. [root@server]# cd apf-9.7-1 (or whatever the latest version is)
5. Run the install file: [root@server]# ./install.sh
6. You will receive a message saying it has been installed.
   Installing APF 9.7-1: Completed.

Installation Details:
Install path: /etc/apf/
Config path: /etc/apf/conf.apf
Executable path: /usr/local/sbin/apf
DShield Client Parser: /etc/apf/extras/dshield/

Now configure the firewall: [root@server]# vi /etc/apf/conf.apf

Make sure DEVEL_MODE="1" is set until you’ve gotten everything working. This will allow you to get back into your server if you cock something up and get locked out, as it tells the script to clear the cron settings every 5 minutes. Once you’ve got APF tested and working as expected, set DEVEL_MODE="0" here.

The majority of the default options in the config can (and should) be left alone unless you know what you’re doing. As you go further into the config file, you’ll see stuff like this:

##
# [Remote Rule Imports] ##
# Project Honey Pot is the first and only distributed system for identifying
# spammers and the spambots they use to scrape addresses from your website.
# This aggregate list combines Harvesters, Spammers and SMTP Dictionary attacks
# from the PHP IP Data at: http://www.projecthoneypot.org/list_of_ips.php
DLIST_PHP="0"

DLIST_PHP_URL="rfxn.com/downloads/php_list"
DLIST_PHP_URL_PROT="http"

# The Spamhaus Don't Route Or Peer List (DROP) is an advisory "drop all
# traffic" list, consisting of stolen 'zombie' netblocks and netblocks
# controlled entirely by professional spammers. For more information please
# see http://www.spamhaus.org/drop/.
DLIST_SPAMHAUS="0"

DLIST_SPAMHAUS_URL="www.spamhaus.org/drop/drop.lasso"
DLIST_SPAMHAUS_URL_PROT="http"

# DShield collects data about malicious activity from across the Internet.
# This data is cataloged, summarized and can be used to discover trends in
# activity, confirm widespread attacks, or assist in preparing better firewall
# rules. This is a list of top networks that have exhibited suspicious activity.
DLIST_DSHIELD="0"

DLIST_DSHIELD_URL="feeds.dshield.org/top10-2.txt"
DLIST_DSHIELD_URL_PROT="http"

All of the above are optional and allow you to implement additional resources such as DShield and Spamhaus to block known spammy or suspicious IPs from being able to access your server. You can leave them off if you’d like (they are off by default) or turn them on for additional protection. (You’ll need to install the DShield scripts, but I’ll get to that in a moment.)

Configuring Ports:

The APF config file will come with some default ports pre-set, but you’ll want to check and make sure everything you need is covered. You will also want to determine whether or not you’re using any uncommon port numbers (for example, for a hosting control panel) that should be added to the configuration file. Please don’t ask me what port numbers your specific hosting control panel uses. I don’t know, but I’m sure Google does.

# Common inbound (ingress) TCP ports
#IG_TCP_CPORTS="22,80,443"
IG_TCP_CPORTS="21,22,25,53,80,443,110,143"

# Common outbound (egress) TCP ports
EG_TCP_CPORTS="21,25,80,443,43"

# Common outbound (egress) UDP ports
EG_UDP_CPORTS="20,21,53"

If you restart the firewall and something is down but no errors are thrown, there’s a good chance you missed a port number here. Make sure to account for SSL ports (443) if you’re running an SSL certificate, etc.

Once you’ve made all of your tweaks, save the config file and start the firewall:
/usr/local/sbin/apf -s

If you’re satisfied that everything looks okay and all services are responding as they should, go back into the APF config and change DEVEL_MODE="1" to DEVEL_MODE="0" and flush the firewall: /usr/local/sbin/apf -f

Common APF Commands

Start: /usr/local/sbin/apf -s
Restart (flush and load): /usr/local/sbin/apf -r
Flush: /usr/local/sbin/apf -f
List Chain Rules: /usr/local/sbin/apf -l
Status: /usr/local/sbin/apf -st

Manually Whitelisting/Blacklisting IP Addresses

For the commands below, replace HOST with an IP or FQDN (Fully Qualified Domain Name) and COMMENT with your comments (no spaces) as to why you’re manually allowing or blocking an IP.

Add to allowed hosts (whitelist) and load new rule: /usr/local/sbin/apf -a HOST COMMENT
Add to denied hosts (blacklist) and load new rule: /usr/local/sbin/apf -d HOST COMMENT

To autostart apf on reboot, run this:
[root@server]# chkconfig --level 2345 apf on

To remove it from autostart, run this:
[root@server]# chkconfig --del apf

Using DShield

If you’re interested in using DShield with APF, you will need to install it first from the extras directory:

[root@server]# cd /etc/apf/extras/dshield
[root@server dshield]# ./install
Installation completed.
Binary: /usr/local/sbin/dshield
Config: /usr/local/dshield/dshieldpy.conf
Cronjob: /etc/cron.daily/ds

Warning: Running the binary from command line will send reports to dshield.org;
repeated execution may result in your IP being banned from the service.

Now you can edit the DShield configuration file, including turning on email alerts, database logging and other stuff. Again, leave this alone (or leave it uninstalled) if you’re not sure what you’re doing. Your APF will function just fine without it:
[root@server]# vi /usr/local/dshield/dshieldpy.conf

Setting Up Brute Force Detection (BFD)

First things first, you MUST have APF installed. BFD was written specifically to work with APF, so you have to start with APF and then install BFD.

1. [root@server]# cd /root/downloads (or any other temporary folder)
2. [root@server]# wget http://www.rfxnetworks.com/downloads/bfd-current.tar.gz
3. tar -xvzf bfd-current.tar.gz
4. [root@server]# cd bfd-1.4 (or whatever the current version is)
5. Run the install file: [root@server]# ./install.sh
6. You will receive a message saying it has been installed
   .: BFD installed
Install path: /usr/local/bfd
Config path: /usr/local/bfd/conf.bfd
Executable path: /usr/local/sbin/bfd

Now let’s take a look at the configuration file:
[root@server]# vi /usr/local/bfd/conf.bfd

What you’ll see is a short file that starts like this:

# how many failure events must an address have before being blocked?
# you can override this on a per rule basis in /usr/local/bfd/rules/
TRIG="10"

# send email alerts for all events [0 = off; 1 = on] EMAIL_ALERTS="1"

# local user or email address alerts are sent to (separate multiple with comma)
EMAIL_ADDRESS="you@yourdomain.com"

# subject of email alerts
EMAIL_SUBJECT="Brute Force Warning for $HOSTNAME"

# executable command to block attacking hosts
BAN_COMMAND="/etc/apf/apf -d $ATTACK_HOST {bfd.$MOD}"


These options are pretty straightforward. TRIG is the number of tries a user is allowed before they trip the BFD deny trigger. For PCI compliance or other strict environments, this number is usually pretty low – but it’s important to keep things practical. Security must always be a balance between making things safe and keeping them useable by the people who need to use them. If you had a lockout policy that after one try, a user is locked out for a day, odds are excellent that you’d be crippling your admins/devs. Who hasn’t fatfingered a password? These measures should be as unobtrusive to the users who legitimately need to be there as possible.

The very concept of a brute force password attack is one where the attacker doesn’t have either a valid username, a valid password or both. The odds of an attacker randomly guessing a username and password combination within 10 tries, or 20 tries, or even 100 tries is pretty low. Brute force attacks generally exploit things like default admin passwords, very common passwords (like ‘123456’ or ‘password’ or ‘fuckyou’), or they are a more prolonged attack consisting of thousands and thousands of random login attempts. The attacker is literally trying to brute force their way in, since they have no other means by which to access your server that way.

Making the tolerance threshold very low doesn’t keep you safer from a brute force attack and will only serve to frustrate your users and create more work for yourself, since you’ll have to manually release the lock once they’ve boned their password a few times. So keep this number reasonable, and remember what it’s there for, or you’ll be making yourself and everyone who needs to access your server miserable.

Enable Email Alerts

You may or may not want to be alerted when someone has tripped the brute force detection script and has been added to the APF deny rules. If you’re on a frequently hit server, these emails could be overwhelming (or could even arguably help create a denial of service situation) but in general, I find it helpful to leave these on. I have filters set up in my email so they don’t flood my inbox. If you’re using a log analyzer/alert system like Splunk, you probably don’t need to turn on email alerts, but that’s up to you.

Find: ALERT_USR="0" CHANGE TO: ALERT_USR="1"
Find: EMAIL_USR="root" CHANGE TO: EMAIL_USR="your@yourdomain.com"

VERY IMPORTANT! Prevent locking yourself out!

You will want to make sure you’ve whitelisted your own trusted IP addresses pretty early on in this process. If your office has a static IP address or range of IP addresses, you’ll want to add these right away. By whitelisting these IPs, you prevent the possibility of locking yourself out of your own server by fatfingering your own password.

To add IPs to the ignored host list:

[root@server]# vi /usr/local/bfd/ignore.hosts

… and add your own trusted IPs, one per line.

Once you’ve got BFD configured to your liking, start it up!
[root@server]# /usr/local/sbin/bfd -s

Test the System

Once you think you’ve got everything working, try logging in from a non-whitelisted IP. If you have another server with it’s own IP address, for example, you could SSH into that server, and from that server SSH into your now-hardened server, using a username and password combination that you know is not valid.

While doing that, tail the APF logs, so make sure the attempts are being logged and the lockout works as expected:

[root@server]# tail -f /var/log/apf_log

Once you pass the number of attempts specified in the BFD config file, you should see the apf_log record that the offending IP address has been added to the denied hosts file.

Allowing Only Whitelisted IPs to Access SSH

If you’ve got static IP addresses and you want to lock your server down even more, you can skip BFD and simply deny ALL SSH requests coming from unknown IP addresses. This is easy to do, but also easy to forget additional IPs that legitimately require access (remote backup systems, managed hosting company support, etc) so be sure to think through everything that legitimate needs access, and be prepared to tweak the IP list if you discover things you broke.

1. Open the allowed hosts file: [root@server]# vi /etc/apf/allow_hosts.rules
2. Scroll down until after the last comment in the file with the ##
3. Add the following:
   tcp:in:d=22:s=YOURHOMEIPHERE
out:d=22:d=YOURHOMEIPHERE

   The d=22 is the port, since you’re specifically addressing SSH which usually runs on port 22. You can repeat for other services as well to limit other connections by port if you like.

4. Open the denied hosts file: [root@server]# vi /etc/apf/deny_hosts.rules
5. Scroll down until the last default comment ## then below it add the following:

   tcp:in:d=22:s=0/0
   out:d=22:d=0/0

6. Restart APF: [root@server]# /usr/local/sbin/apf -r

You wouldn’t use IP whitelisting restrictions in combination with BFD, since the process of whitelisting your internal IPs will override the BFD protection. In other words, with whitelisting restrictions, any user who isn’t on an authorized IP address won’t even be

Test the System

Testing this one should be pretty easy. Simply try to connect via SSH from any IP address that isn’t one that you whitelisted in step 3 above. What you should see is a connection attempt timeout or connection refusal. Try a new SSH connection from a whitelisted IP and you should get the SSH password prompt.

Two-Factor Authentication

If you really want to lock things down, you may want to consider adding two-factor authentication to your login. SSH keys would be something you have – plus a password as something you know – but for some reason it’s still not possible to require a password with SSH keys (to my knowledge – please correct me if I’m wrong). So instead of two-factor, you end up with a different one-factor (something you have instead of something you know).

Years ago, setting yourself up with true two-factor authentication was prohibitively expensive, so not a lot of smaller folks were doing it. These days, Citrix key fobs are being replaced by a new generation of more affordable and practical tokenless two-factor authentication systems, such as PhoneFactor and DuoSecurity.

Both of these options are pretty cool, and reasonably easy to implement. I’m in the process of setting up a few of our boxes with DuoSecurity (can’t beat the price), with the help of this fantastic tutorial by Jeremy L. Gaddis over at EvilRouters.