How to Build a Centralized CentOS 7 Syslog Server

Today we are going to learn how to build a centralized Syslog server on CentOS 7. Syslog is a mechanism which allows the logging of messages from all different software/hardware platforms in a standardized syntax. The Syslog spec was documented in RFC 3164 by the IETF(Internet Engineering Task Force in August of 2001 and was later standardized by RFC 5424 in March of 2009.

Syslog offers the flexability to completely separate message generation, message storage, and message analysis. This separation is fundamental to the security and usability of log data generated by various systems. Not only is log data useful to Systems and Network Engineers, it is also useful to Security Engineers. In fact, modern Security Information and Event Management (SIEM) platforms leverage centralized Syslog data alongside Deep Packet Inspection (DPI) to correlate security events on corporate networks. This in turn allows Security Operations Center (SOC) Analysts and Systems Engineers to identify and mitigate network and security related issues in realtime.

Now that you have a bit of background on Syslog, lets dig in to the tutorial.

Step 1) Patch Your System and Ensure Rsyslog is Installed

In this tutorial I will be using the Rsyslog package from the CentOS Base repository as our Syslog . Rsyslog is the "rocket-fast system for log processing" and is the standard Syslog daemon installed on Red Hat Enterprise Linux (RHEL) based operation systems.

Refresh yum repository cache, perform update, and install rsyslog dependencies
$ yum makecache fast -y
$ yum update -y
$ yum install rsyslog rsyslog-doc rsyslog-relp

Step 2) Configure the Rsyslog Service

Next we will be configuring the Rsyslog service to listen for incoming log streams on TCP and UDP port 514 as well as enabling the Reliable Event Log Protocol (RELP) listener on port 2514. RELP isn't as common as TCP and UDP Syslog streams as it requires more network overhead than both and is typically only used in environments that require message delivery such as financial and government industries.

We will not be enabling TLS encrypted RELP logging in this tutorial as it is an advanced feature and requires setting up a Certificate Authority CA in order generate the necessary encryption keys and certificates.

Add the following lines to a new file named /etc/rsyslog.d/server.conf

Note: $PreserveFQDN on directive can remain "off" or commented out unless you have systems with identical hostnames sending log messages to this Syslog server

# Load UDP event Log protocol Module
$ModLoad imudp
# Listen for UDP log streams on port 514
$UDPServerRun 514

# Load TCP event Log Protocol Module
$ModLoad imtcp
# Listen for TCP log streams on port 514
$InputTCPServerRun 514

# Load Reliable Event Log Protocol (RELP) Module
$ModLoad imrelp
# Listen for RELP log streams on port 2514
$InputRELPServerRun 2514

# Preserve the FQDN on each message
#$PreserveFQDN on

Step 3) Add SELinux Policy Exceptions for the 'syslog-tcp (TCP/514)' and 'syslog-relp (TCP/2514)' Ports.

If using a system with SELinux enabled, exceptions must be made to allow Rsyslog to listen on non-standard ports (ie: TCP/514 and TCP/2514). These exceptions can be made by using the utilities from the 'policycoreutils' and 'setroubleshoot' packages.

Install the policycoreutils and setroubleshoot package
$ yum install -y policycoreutils setroubleshoot
Use semanage to find the ports that are currently part of the syslogd SELinux policy

Note: TCP/514 and TCP/2514 are not present

$ semanage port -l | grep syslog
syslog_tls_port_t              tcp      6514, 10514
syslog_tls_port_t              udp      6514, 10514
syslogd_port_t                 tcp      601, 20514
syslogd_port_t                 udp      514, 601, 20514
Add the non-standard ports
$ semanage port -m -t syslogd_port_t -p tcp 514
$ semanage port -a -t syslogd_port_t -p tcp 2514
$ semanage port -l | grep syslog
syslog_tls_port_t              tcp      6514, 10514
syslog_tls_port_t              udp      6514, 10514
syslogd_port_t                 tcp      2514, 514, 601, 20514
syslogd_port_t                 udp      514, 601, 20514

Step 4) Configure Firewalld to Allow Incoming Log Streams

Next we will be configuring firewalld (a frontend management solution for iptables) on our CentOS 7 host to allow incoming log messages on TCP/514, UDP/514, and TCP/2514.

Check the current firewalld active zone

Running the command firewall-cmd --list-all will display all attributes of the current active zone ("public" in our case).

Note: Take note of the 'services' field below. This field will change after we are done configuring firewalld

$ firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources:
  services: ssh dhcpv6-client
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
Create a new service definition file for UDP Syslog traffic.

/usr/lib/firewalld/services/syslog-udp.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>syslog-udp</short>
  <description>Syslog is a client/server protocol: a logging application transmits a text message to the syslog receiver. The receiver is commonly called syslogd, syslog daemon or syslog server.</description>
  <port protocol="udp" port="514"/>
</service>
Create a new service definition file for TCP Syslog traffic.

/usr/lib/firewalld/services/syslog-tcp.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>syslog-tcp</short>
  <description>Syslog is a client/server protocol: a logging application transmits a text message to the syslog receiver. The receiver is commonly called syslogd, syslog daemon or syslog server.</description>
  <port protocol="tcp" port="514"/>
</service>
Create a new service definition file for TCP Syslog traffic.

/usr/lib/firewalld/services/syslog-relp.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>syslog-relp</short>
  <description>Reliable Event Log Protocol (RELP) traffic over TCP port 2514.</description>
  <port protocol="tcp" port="2514"/>
</service>
Reload firewalld via the firewall-cmd command in order to enumerate the newly create firewalld services.
$ firewall-cmd --reload
success
Add the syslog-udp, syslog-tcp, and syslog-relp services to the public zone for both runtime and permenant configurations

Note: The first command does NOT contain the --permanent flag. This means that the service has been effectively added to the runtime configuration. The second command which DOES contain the --permanant flag will store our service to the firewalld configuration file which will then be loaded as a runtime configuration during service reload/restarts and at boot time.

$ firewall-cmd --zone=public --add-service syslog-udp --add-service syslog-tcp --add-service syslog-relp
success
$ firewall-cmd --zone=public --add-service syslog-udp --add-service syslog-tcp --add-service syslog-relp --permanent
success
$ firewall-cmd --list-services
ssh dhcpv6-client syslog-udp syslog-tcp syslog-relp
Verify the active zone configuration

Note: Notice that the syslog-udp, syslog-tcp, and syslog-relp services have been appended to the 'services' field.

$ firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources:
  services: ssh dhcpv6-client syslog-udp syslog-tcp syslog-relp
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Step 5) Enable and start the Rsyslog service

Now we will start and enable the Rsyslog service via systemd.

Enable the service

Use systemctl to enable the Rsyslog service.

$ systemctl enable rsyslog.service
Created symlink from /etc/systemd/system/multi-user.target.wants/rsyslog.service to /usr/lib/systemd/system/rsyslog.service.
Start the service

Use systemctl to start and check the status of the Rsyslog service.

$ systemctl start rsyslog.service
$ systemctl status rsyslog.service
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-05-31 13:21:46 EDT; 7min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 6120 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─6120 /usr/sbin/rsyslogd -n

May 31 13:21:46 host.domain.tld systemd[1]: Starting System Logging Service...
May 31 13:21:46 host.domain.tld rsyslogd[6120]:  [origin software="rsyslogd" swVersion="8.24.0-34.el7" x-pid="6120" x-info="http://w...] start
May 31 13:21:46 host.domain.tld systemd[1]: Started System Logging Service.
Hint: Some lines were ellipsized, use -l to show in full.
Verify that the TCP network ports are open
$  ss -lnt -4
State      Recv-Q Send-Q              Local Address:Port                             Peer Address:Port
LISTEN     0      55                              *:2514                                        *:*
LISTEN     0      25                              *:514                                         *:*
Verify that the UDP network ports are open
$ ss -lun -4
State      Recv-Q Send-Q              Local Address:Port                             Peer Address:Port
UNCONN     0      0                               *:514                                         *:*

Step 6) Configure Other CentOS 7 Hosts to Send Logs to the New Syslog Server

Now we will configure other CentOS 7 hosts on our network to send log message streams to the freshly built Syslog server.

Configure the /etc/rsyslog.conf file to send data to our Syslog server via UDP port 514
  • Uncomment the line #*.* @@remote-host:514 in /etc/rsyslog.conf
  • Remove 1 of the '@' symbols
  • Specify the correct hostname of the centralized Syslog server.
*.* @host.domain.tld:514

OR

Configure the /etc/rsyslog.conf file to send data to our Syslog server via TCP port 514
  • Uncomment the line #*.* @@remote-host:514 in /etc/rsyslog.conf
  • Specify the correct hostname of the centralized Syslog server.
*.* @@host.domain.tld:514

OR

Configure the /etc/rsyslog.conf file to send data to our Syslog server via RELP over TCP port 2514
  • Replace the line #*.* @@remote-host:514 in /etc/rsyslog.conf with *.* :omrelp:host.domain.tld:2514
  • Specify the correct hostname of the centralized Syslog server.
*.* :omrelp:host.domain.tld:2514
Enable and start/restart the Rsyslog service

Use systemctl to start and check the status of the Rsyslog service.

$ systemctl enable rsyslog.service
Created symlink from /etc/systemd/system/multi-user.target.wants/rsyslog.service to /usr/lib/systemd/system/rsyslog.service.
$ systemctl systemctl restart rsyslog.service
$ systemctl status rsyslog.service

Step 7) Verify Logs are Flowing to the Syslog Server

You can test to make sure logs are flowing from the client to the log server by tailing the /var/log/messages file on the log server and executing a logger command on the client.

Tail the /var/log/messages file on the log server
$ tail -f /var/log/messages
Generate test logs on the client using the logger command
$ logger test123
Show Comments