How to create a syslogs server

Nov 2021

If one of your machines is compromised one of the first things that a smart adversary will do is modify the system’s logs and remove the evidence that they were there. To prevent that from happening you may choose to run a special monitoring machine that collects system logs. Running this machine means that even if the logs on the compromised box are modified, unless the syslogs machine is compromised, you will still be able to view them as they were.

Here I will walk through how to configure a machine like that and how to configure other machines to send their logs to it.

The syslogs server

First, open up port 514 for both TCP and UDP connections on the machine that you would like to collect logs on. If you are using a cloud provider you might do this via their console or if you are firewalling on the machine you might use iptables.

With those ports open edit /etc/rsyslog.conf to have the following lines:

# Enable UDP syslog reception.
module(load="imudp")
input(type="imudp" port="514")

# Enable TCP syslog reception.
module(load="imtcp")
input(type="imtcp" port="514")

#
# Only allow logs to be sent from our instances.
#
$AllowedSender UDP, <IP LIST>
$AllowedSender TCP, <IP LIST>

#
# Split logs into files by sender
#
$template Split,"/var/log/%HOSTNAME%/auth.log"
auth.*,authpriv.* ?Split

In the above example please replace <IP LIST> with a comma separated list of IP addresses that you will accept logs from. This list will likely contain the address of all of your other machines. For example:

$AllowedSender UDP, 127.0.0.1, 100.0.2.13
$AllowedSender TCP, 127.0.0.1, 100.0.2.13

These edits enable the reception of remote syslogs, specify what addresses we will accept those logs from, and then provide a template for where we will store those logs. The template line says to place the logs for each machine in /var/log/<ADDRESS OF MACHINE>/auth.log.

Note that here we have configured the system to only accept logs from auth.log. auth.log contains information about logins to the machine and in this case that is what we’re interested in.

Having made these edits restart your rsyslog service so that they take effect:

sudo systemctl restart rsyslog

Finally, you may want to add an A record to a domain you control which will point towards this machine. This will make sending logs from other machines easier as they can specify syslogs.example.com as the target instead of an IP address.

Sending logs to the server

Sending logs to the server is remarkably easy. On the machine you would like to send logs to edit /etc/rsyslog.conf and make the following edit:

 #
 # First some standard log files.  Log by facility.
 #
-auth,authpriv.*                        -/var/log/auth.log
+auth,authpriv.*                        @@<YOUR DOMAIN / ADDRESS>:514
 *.*;auth,authpriv.none         -/var/log/syslog
 #cron.*                                /var/log/cron.log
 daemon.*                       -/var/log/daemon.log

If you would like to send logs via UDP you can replace @@<YOUR DOMAIN / ADDRESS>:514 with @<YOUR DOMAIN / ADDRESS>:514. Also please replace <YOUR DOMAIN / ADDRESS> with the domain or address of the machine configured above. Here are some examples of what that might look like:

auth,authpriv.*                        @@syslogs.example.com:514

auth,authpriv.*                        @@100.13.0.1:514

Finally, restart rsyslog on the machine.

sudo systemctl restart rsyslog

Having done this you should start seeing logs in /var/log/<ip address>/auth.log for your configured machine! For an example of making those logs actionable please see How to get Discord messages on SSH logins.