A very light-weight and efficient approach for consolidating logs centrally is by using rsyslog. My virtual machines all use rsyslog to forward their logs to a dedicated internal virtual machine, which acts as the central log hub. A fail2ban instance on this hub checks all incoming logs and sends a block command to an external firewall—a process helpful for automated security.
Architectural Decision: Centralized Analysis
Instead of running individual `fail2ban` instances on each client (which would be inefficient and create distributed administration overhead), I chose to consolidate all relevant log streams on a single server. This central point becomes the heart of the Incident Response system, supplying `fail2ban` with all necessary data for analysis.
Part I: Configuring the Central Log Hub (rsyslog)
1. Receiving Logs (Hub Configuration)
The hub requires configuring rsyslog to listen for incoming UDP connections on port 514 and processing them using a custom ruleset.
# /etc/rsyslog.d/remote.conf
module(load="imudp")
input(type="imudp" port="514" ruleset="remote")
ruleset(name="remote") {
# Template creates separate files per client host and syslog facility (auth, cron, etc.)
$template remote-logs, "/var/log/remote/%HOSTNAME%/%syslogfacility-text%.log"
*.* ?remote-logs
}
This structured logging ensures we can quickly audit specific services (auth.log) for individual clients, which is key for Observability.
2. Sending Logs (Client Configuration)
The clients require only a minimal configuration to forward all log messages to the hub.
# /etc/rsyslog.d/remote.conf (Client configuration)
*.* @HUB_IP:514Critical Note on Log Transport (UDP vs. RELP/TCP)
The choice of using UDP for log transport here is a deliberate architectural trade-off in the context of high network load and potential denial-of-service (DoS) scenarios. While RELP (Reliable Event Logging Protocol, a TCP-based standard) is the industry standard for Data Integrity because it guarantees message delivery, I have prioritized minimizing resource overhead on the client machines. In an environment under attack or severe network strain, TCP handshakes and connection overhead can consume precious CPU cycles and network resources, potentially hindering the application or preventing the immediate processing of other security events.
However, the consequence of this choice must be understood from an SRE perspective: By choosing UDP (“fire and forget”), I consciously risk losing critical log events (e.g., the fifth failed login attempt) if the log hub becomes overloaded and the kernel buffer starts dropping packets silently. This break in the log chain could compromise the Incident Response system by preventing Fail2ban from triggering the necessary block action. For maximum Production Reliability and Auditability, a migration to RELP is the essential next step to ensure guaranteed delivery of all security events.
3. Log Rotation
A custom logrotate configuration is essential for managing the growing volume of centralized logs. I simplify maintenance by using a wildcard path (/var/log/remote/*/*.log).
/var/log/remote/*/*.log
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
Part II: Automated Defense with Fail2ban
The centralized server runs fail2ban to analyze all inbound logs for malicious activity.
1. Fail2ban Setup
The jail configuration must point fail2ban to the aggregated log files. Since all auth.log files are stored under the /var/log/remote/ hierarchy, a wildcard path is used.
# /etc/fail2ban/jail.d/remote.conf
[sshd]
enabled = true
logpath = /var/log/remote/*/auth.log
This single path allows fail2ban to simultaneously monitor SSH brute-force attempts across all virtual machines reporting to the hub.
2. External Firewall Integration (The Action Layer)
Since the log hub is isolated and not the firewall itself, fail2ban must be taught how to communicate the ban/unban commands to the external firewall. This is achieved via an API integration action.
I define a custom action file (/etc/fail2ban/action.d/firewall.conf) that uses curl to send commands to a dedicated Linux firewall VM.
[Definition]
actionstart =
actionstop =
actioncheck =
actionban = curl -s "http://10.0.0.1/fw/ban/<ip>"
actionunban = curl -s "http://10.0.0.1/fw/unban/<ip>"
This is the core of the automated Incident Response process. The final step is linking this action to the jail configuration: action = firewall.
Critical Note: Because this external firewall blocks the IP across ALL my virtual machines, extreme caution must be exercised to avoid self-blocking.
Part III: Firewall Architecture and Log Evidence
My firewall operates as a transparent bridge between the host interface and the virtual machines, allowing it to inspect traffic and perform layer-2/3 filtering using nftables.
The logs clearly show the external firewall successfully dropping packets from banned IPs, proving the automation chain is working:
table bridge t { # handle 2
chain c { # handle 1
type filter hook forward priority 0; policy accept;
iif "ens3" ip saddr 218.x.x.x counter packets 50 bytes 4096 drop # Example of banned IP drop
iif "ens3" ip saddr 103.x.x.x counter packets 17 bytes 948 drop
}
}
Conclusion
This architecture effectively separates the concerns of log collection, security analysis, and traffic enforcement. My centralized log hub uses fail2ban for security intelligence and automatically triggers bans on the external nftables firewall. This ensures a comprehensive, automated defense mechanism across the entire infrastructure.
Sources / See Also
- rsyslog Documentation. Official documentation on Modules (imudp, imtcp) and Rulesets.
https://www.rsyslog.com/doc/v8-stable/ - Fail2ban Documentation. Action Files and Configuration Syntax.
https://www.fail2ban.org/wiki/index.php/MANUAL_0_9#Action_files - ISC Knowledge Base. Linux connection tracking and DNS.
https://www.isc.org/connection-tracking-and-dns/ - ISC BIND. Best practices draft: Restarting BIND intelligently.
- GNU Project. Manpage: logrotate.conf (Configuration and postrotate scripts).
https://manpages.debian.org/testing/logrotate/logrotate.conf.5.en.html