Introducing fail2ban, and first steps towards sharing attacker’s IP
by Arturo ‘Buanzo’ Busleiman
Fail2ban is a lovely python-based tool written by Cyril Jaquier that
monitors different logfiles for lines matching regular expressions. From
those lines it extracts the attackers IP address, and runs a command passing
that as a parameter.
In more simple terms, it detects when your SSH (or other service) is
“attacked”, and then proceeds to firewall the attacker.
We could start discussing about false positives here, specially for UDP
based services or log-injection vulnerable daemons, but I’d like to focus on
a special usage scenario: what if I manage multiple servers? Should I wait
for the same attacker to target each of my fail2ban-monitored servers?
Maybe I could share the attacker’s IP, collected from a brute-force attempt
to SSH on server1, with the rest of the servers I manage. Or with my
friends’ servers. Or maybe with everybody through twitter? Well, maybe
everybody is too much and possibly a bad idea if fail2ban is detecting false
positives, or spoofed IP addresses. But it could also help out if you manage
your own small or medium sized group of servers.
If you think this might interest you, then keep reading.
Architecturally, fail2ban is one main process (the fail2ban-server). It
needs a so-called “jail” definition (a group of filter+action+logfile
configuration items), a filter definition (regular expressions for a certain
kind of logfile, such as sshd or apache error log), and an action
definition (what to do when a filter detects an attacker, in that certain
jail).
This is what the standard ssh jail looks like:
[ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/daemon.log
maxretry = 3
As you can see, it defines if the jail is enabled or not (active or not),
what destionation port[s] it will block for the attacker, what filter
(regex) is to be used to determine (or ignore!) attackers, and which
logfile[s] it will monitor. Also, the number of times the same attacker must
be detected in a pre-defined time period for the action to be executed.
What does that mean? Easy: If, by any reason, you or another user mistypes
the password and is trying to access the server from a non-whitelisted IP
address, instead of trying more than 3 times in a row, it should wait a
couple minutes and try again. Having a detection threshold helps avoiding
false positives.
Fail2ban has the ability to ignore certain IP addresses or netblocks, or
have different actions for each jail. This level of flexibility is what
interest us for the purpose of this article.
Standard fail2ban configuration contains 20+ actions, spanning
iptables, ipfw, shorewall, etc. More than 30 filter definitions, including
regexes for apache, sshd, php-url-fopen, proftpd, pam and more.
For a detailed fail2ban-configuration document, I recommend you just
download the package and read its filters, actions and jails. It’s very self
explanatory. A knowledge of regular expressions will come in handy,
especially if your POSIX-compatible system has differences in the logfile
format for the services you’re trying to protect.
* How to share the attacker’s details
Lets imagine we own a small web-hosting company, providing ssh access to our
limited number of customers. Or a big web app load-balanced and with
failover protection spanning 10 geographically-distant dedicated servers.
Also, let’s assume we have a standard ssh jail configured to run iptables,
matching the sshd filter.
If server1′s fail2ban detects, for instance, that an attacker with IP address
203.0.113.20 is attempting to brute-force our ssh service
(note: that address is part of the TEST-NET-3 netblock, allocated by
RFC 5737 for usage in documentation), then the attacker will be firewalled
using iptables.
But only on server1.
Should I manually filter the attacker on the other servers? Well, it’ll take
a long time. Or I could write a script to do it. Or maybe I could try and
work with what fail2ban gives me: lots of flexibility.
The first time I started thinking about this “fail2ban-cluster” tool, I
thought I should design a new “action”, one that would not only run iptables
but also notify the other servers.
But that’d be a waste of resources, not to mention TIME.
So I decided to use a different approach. Fail2ban has a lovely log file,
that tells us when a certain attacker is banned, and then un-banned after a
configurable amount of time.
Maybe I could write a filter to read fail2ban’s log itself, with an action
to spread that IP address, possibly using lynx, or curl, to as many other
servers as I wanted, by means of some authenticated http service. And I’d
run that on top of my inter-server virtual private network, for good
measure.
This is how fail2ban’s log file looks like:
2011-05-01 15:23:56,380 fail2ban.actions: WARNING [ssh] Ban 203.0.113.20
2011-05-01 15:38:57,641 fail2ban.actions: WARNING [ssh] Unban 203.0.113.20
As you can see, we have entries for Ban and for Unban events. But I do not
want to notify unban events, because different servers could have different
how-long-to-ban-the-attacker policies. I just want to notify BANs.
So maybe a simple regular expression like: ‘\ Ban\ <HOST>$’ would take care
of detecting BANs only. And it does. Fail2ban provides the fail2ban-regex
tool, to test regular expressions against a sample log line or file.
I ran it like this: fail2ban-regex /var/log/fail2ban.log ‘\ Ban\ <HOST>$’
It yielded the expected results. The special <HOST> tag in the regex is what
tells fail2ban how to extract the IP address from there. Currently, it
supports IPv4 but the fail2ban team and contributors are working on IPv6
support as I write.
So, now I can write a fail2ban filter configuration file, such as this:
*** contents of /etc/fail2ban/filter.d/f2bcluster.conf ***
[Definition]
failregex = \ Ban\ <HOST>$
ignoreregex =
*** end ***
But we have no action. We can’t write a jail definition without an action!
An action file contains definitions of commands to run: once at the start of
fail2ban, once at the stop, once before each ban event, and the ban action
itself, and the unban one. It also includes a default name, port and
protocol definition, in case the jail config file does not define them.
Inside each action definition you can use special tags, similar to the
<HOST> one used in filters. For actions, we have <ip> (holds the attacker’s
IP obtained from <HOST>), <name> and <protocol> (used for some iptables
parameters).
When the commands are run, each of those special tags will be replaced by
the values we expect.
As an exercise to the reader, or you can wait for the next article in this
series, I’d like you to write an actioban definition, with these hints:
a) the actionban should be a simple bash script
b) the bash script will use lynx/curl to send an authenticated http request
c) the request will go the server2, server3, etc. As many times as required.
d) the request will be received by a simple python-based (or php…) script
that REQUIRES http authentication. It will receive the attacker’s IP from
the QUERY STRING.
e) The http python/php/whatever script will NOT run iptables, but write to a
log file.
As you probably guess, the log file will be monitored by that server’s own
fail2ban instance, by means of a more classic filter definition.
And remember: security is a state of mind.
About me
I’ve been using GNU/Linux non-stop since September 1995, when I
was 13 years old. Currently working in the IT security area as a consultant,
sysadmin and forensics specialist. I’m an OWASP Project Leader (check out
Enigform and mod_openpgp for Apache). I play the guitar, and currently
experiencing with electronic music and livecoding with fluxus. I’m a geek.
