Setup a central logging instance (and use it to block traffic using fail2ban)

A very light-weight approach for storing logs centralized is by just using rsyslog. My virtual machines all use rsyslog. That rsyslog sends it’s logs to another internal virtual machine which runs rsyslog as well. A fail2ban instance is checking all these logs and sending a block command to the firewalls. Here is how.

For the following article I assume that you have installed rsyslog on your central log server as well as rsyslog on your clients. Later I will assume that you have fail2ban installed in the central logserver. Yes, instead of doing so you might just forward auth.* logs to a fail2ban instance 🙂

Receiving rsyslog

First of all the module for input by UDP needs to be loaded:

module(load="imudp")

Then the input needs to be defined. We also add a ruleset to this input which will allow to separate local syslog and remote syslog:

input(type="imudp" port="514" ruleset="remote")

Finally the ruleset needs to be defined:

ruleset(name="remote") {
}

within this ruleset I define a template called remote-logs which will send logs to /var/log/remote/HOSTNAME/SYSLOGFACILITY.log:

$template remote-logs, "/var/log/remote/%HOSTNAME%/%syslogfacility-text%.log"

Finally all logs are sent / use this defined template:

*.* ?remote-logs

All in all this looks like this (/etc/rsyslog.d/remote.conf)

module(load="imudp")
input(type="imudp" port="514" ruleset="remote")

ruleset(name="remote") {
  $template remote-logs, "/var/log/remote/%HOSTNAME%/%syslogfacility-text%.log"
  *.* ?remote-logs
}

Alternatively (using a fail2ban instance)

In the introduction of this article I wrote that you could just forward the auth.* logs to a fail2ban instance, and yes: Instead of using /var/log/remote/%HOSTNAME%/%syslogfacility-text%.log you could just go by /var/log/feed-fail2ban.log. This would put all your systems logs into one big log file. And either here or in the sending rsyslog systems you would instead of using *.* just send the auth-logs (auth.*) and other logs you need.

It might still be advisable to at least go by: /var/log/remote/%HOSTNAME%.log so that your logfiles won’t be too big. You may also mix the rules by using multiple templates. Let me know, if you do cool stuff with this.

Sending rsyslog

Now this is really simple because you only need to define where to send the logs to and it will still locally write logs. So just add /etc/rsyslog.d/remote.conf and add:

*.* @IP:PORT

This will send logs to IP:PORT using UDP. The result is:

root@syslog:/var/log/remote# ls
de-fc-mio1

root@syslog:/var/log/remote# cd de-fc-mio1/

root@syslog:/var/log/remote/de-fc-mio1# ls
auth.log  authpriv.log  cron.log  daemon.log

You can see that the system de-fc-mio1 sends its logs (auth, authpriv, cron, daemon) to the syslog system.

Add logrotate

I made it simple for myself. I just went into /etc/logrotate.d/ and copied the rsyslog configuration to remotelog. And then I just used /var/log/remote/*/*.log in the configuration, e.g.:

root@syslog:/etc/logrotate.d# cat remotelog 
/var/log/remote/*/*.log
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
}

You can test the functionality by issuing logrotate -f /etc/logrotate.d/remotelog. This should rotate your log files in /var/log/remote/…

root@syslog:/var/log/remote/de-fc-mio1# ls
auth.log.1  authpriv.log.1  cron.log.1  daemon.log.1

Fail2ban

Now that all logs are received by this system.. The next steps are pretty easy. Install fail2ban:

apt-get install fail2ban

This syslog system is only internal. So it does not need to block anything. Fail2ban is just running on this system to check the logs. Hence in /etc/fail2ban/jail.d move defaults-debian.conf to remote.conf and edit the file as follows:

[sshd]
enabled = true
logpath = /var/log/remote/*/auth.log

Now when you restart fail2ban just with this small change you will see in /var/log/fail2ban.log (well, you should see) something like:

2023-11-17 19:17:07,792 fail2ban.filter [767]: INFO Added logfile: '/var/log/remote/de-fc-mio1/auth.log' (pos = 0, hash = 73ae599bc8f68a30332f53efeed1c27384db2f45)

So the jail configuration works. Bruteforce against ssh is detected; the logs are read by fail2ban. The next step is to create our own actions on a match. In the beginning of this article I wrote that I want my firewall to ban this. My firewall is a linux virtual machine – however if you’re using OPNSense or VyOS (both excellent firewalls) you should be able to use their APIs.

Similarly to how you would call an API I call my firewall to ban and unban IPs. For this I just teach fail2ban to contact my firewall by adding a new action into /etc/fail2ban/action.d called… very creative… firewall.conf.

[Definition]
actionstart =
actionstop = 
actioncheck = 
actionban = curl -s "http://10.0.0.1/fw/ban/<ip>"
actionunban = curl -s "http://10.0.0.1/fw/unban/<ip>"

The above will send a request containing the ip to be blocked or unblocked to my firewall which will then block the IP in a central way (this means, that IP can’t reach ANY of my virtual machines after this block – Hence I need to be very careful, to not block myself by accident).

However, now you need to define this as an action for the jail:

root@syslog:/etc/fail2ban/jail.d# cat remote.conf
[sshd]
enabled = true
logpath = /var/log/remote/*/auth.log
action = firewall

Now you should see in /var/log/fail2ban.log and in your Firewall that IPs are getting blocked.

A few last words on this one

It would be possible to run fail2ban within my Firewall. But then I would need to send all my logs to the Firewall. This is not what I want. Hence I was using an instance (a central log server) between my clients and my firewall.

Running fail2ban within a central log server still does not sound correct; I could use logstash or other ways to separate this. Alternatively I could (like I wrote before) send the auth-logs to a fail2ban instance (from all clients – or from the central log server)… logstash and other software might also allow me to filter and modify the logs a bit before giving them to a firewall or fail2ban instance…

For now it works like this: all my virtual machines send their logs to my central log server. My central log server uses fail2ban to trigger bans and unbans in my firewall. My firewall then blocks and unblocks accordingly.

In my case my firewall transparently bridges the host interface with my virtual machines so that I can transparently filter / look into the traffic. I’m by the way using nftables for blocking. Here is the result of a few minutes with just a handful of virtual machines sending their logs to the central log server:

table bridge t { # handle 2
   chain c { # handle 1
      type filter hook forward priority 0; policy accept;
      iif "ens3" ip saddr 218.x.x.x counter packets 50 bytes 4096 drop # handle 6
      iif "ens3" ip saddr 218.x.x.x counter packets 42 bytes 3470 drop # handle 7
      iif "ens3" ip saddr 103.x.x.x counter packets 17 bytes 948 drop # handle 8
      iif "ens3" ip saddr 134.x.x.x counter packets 4 bytes 240 drop # handle 9
   }
}