Syslog (System Logging Protocol) is a standard protocol used by network devices, operating systems, and applications to generate and send diagnostic messages. These logs contain timestamps, severity levels, and specific data about system events, security warnings, configuration changes, or service failures.Tools like rsyslog (an enhanced, modular version of syslog) act as the delivery service. They process those event messages locally and can be configured to forward them over the network. In this post I'll cover the installation and setup of rsyslog. Some platforms such as OpenObserve have deprecated support for rsyslog in favor of syslog-ng. I'll cover that in a separate post.
Why Configure Proxmox to Send Rsyslog Data to a Central Server?
By default, Proxmox VE (which is based on Debian) manages system logs locally via systemd-journald. However, installing rsyslog and forwarding those logs to a centralized syslog server or log aggregator (such as Kiwi Syslog, or Splunk) offers critical operational benefits:
Post-Crash Troubleshooting and Diagnostics: If a Proxmox host experiences a severe hardware failure, kernel panic, or sudden lockup, the local logs often become inaccessible or get corrupted. Offloading logs in real-time ensures you can review the exact moments leading up to the crash from a separate machine.
Tamper-Resistant Security and Auditing: If a Proxmox node is compromised, a malicious actor's first move is often to clear or modify local log files to hide their tracks. Centralized logging preserves an immutable audit trail outside of the attacker's reach.
Centralized Infrastructure Monitoring: Managing logs across a Proxmox cluster (or alongside other firewalls, switches, and servers) is tedious if you have to log into each machine individually. Centralization brings all data under one dashboard.
Proactive Alerting and Long-Term Storage: Central log aggregators allow you to write advanced rules to trigger immediate alerts (e.g., failed SSH logins, hard drive smart errors, or high cluster fencing events). They also let you store massive historical backlogs without consuming valuable local storage on your hypervisor boot drives.
Here are the setup steps for installing and activating rsyslog on each of your Proxmox nodes.
Go to a System prompt:
apt update
apt install --no-install-recommends rsyslog -y
systemctl enable --now rsyslog
##Set (uncomment by removing # and change ForwardToSyslog=yes
nano /etc/systemd/journald.conf
ForwardToSyslog=yes
Ctrl+o and enter to save, Ctrl+x to exit nano
systemctl restart systemd-journald
##Verify /var/log/syslog is growing
tail -f /var/log/syslog
##Create the forwarding configuration
nano /etc/rsyslog.d/10-syslogserver.conf
# Kiwi syslog forwarding
*.* action(
type="omfwd"
target="<COLLECTOR_IP>"
port="<PORT>"
protocol="tcp" ##(or "udp" depending on syslog server requirements)
template="RSYSLOG_SyslogProtocol23Format"
OR
template=RSYSLOG_TraditionalForwardFormat"
##do not use both template lines - use the one required by your syslog server. Kiwi requires Traditional (RFC 3164). Solutions like Adlumin require Protocol23 (RFC 5424).##
queue.type="LinkedList"
queue.filename="kiwi_fwd"
queue.maxdiskspace="1g"
queue.saveonshutdown="on"
action.resumeRetryCount="-1"
)
##Ctrl+o and enter to save, Ctrl+x to exit nano editor
## Validate and restart:
rsyslogd -N1
systemctl restart rsyslog
systemctl status rsyslog
##Test:
logger -t kiwi-test "proxmox-node1 syslog forwarding test $(date)"
systemctl status rsyslog
journalctl -u rsyslog -n 50 --no-pager
ls -la /var/spool/rsyslog
##this should send data to your syslog server. If your Proxmox firewall is enabled, don't forget to create an outbound exception for the IP and Port to your syslog server.##
pveproxy web-UI access logs live in /var/log/pveproxy/access.log, not the journal, so they won't be picked up by the configuration above. If you want your syslog server to track auth/UI access, add an imfile input:
nano /etc/rsyslog.d/10-syslogserver.conf
##place the following text at the top of the 10-syslogserver.conf file, above the action block, and restart rsyslog##
module(load="imfile")
input(type="imfile"
File="/var/log/pveproxy/access.log"
Tag="pveproxy"
Severity="info"
Facility="local6")
Ctrl+o and enter to save, Ctrl+x to exit nano.
systemctl restart rsyslog
systemctl status rsyslog
Why Configure Proxmox to Send Rsyslog Data to a Central Server?
By default, Proxmox VE (which is based on Debian) manages system logs locally via systemd-journald. However, installing rsyslog and forwarding those logs to a centralized syslog server or log aggregator (such as Kiwi Syslog, or Splunk) offers critical operational benefits:
Post-Crash Troubleshooting and Diagnostics: If a Proxmox host experiences a severe hardware failure, kernel panic, or sudden lockup, the local logs often become inaccessible or get corrupted. Offloading logs in real-time ensures you can review the exact moments leading up to the crash from a separate machine.
Tamper-Resistant Security and Auditing: If a Proxmox node is compromised, a malicious actor's first move is often to clear or modify local log files to hide their tracks. Centralized logging preserves an immutable audit trail outside of the attacker's reach.
Centralized Infrastructure Monitoring: Managing logs across a Proxmox cluster (or alongside other firewalls, switches, and servers) is tedious if you have to log into each machine individually. Centralization brings all data under one dashboard.
Proactive Alerting and Long-Term Storage: Central log aggregators allow you to write advanced rules to trigger immediate alerts (e.g., failed SSH logins, hard drive smart errors, or high cluster fencing events). They also let you store massive historical backlogs without consuming valuable local storage on your hypervisor boot drives.
Here are the setup steps for installing and activating rsyslog on each of your Proxmox nodes.
Go to a System prompt:
apt update
apt install --no-install-recommends rsyslog -y
systemctl enable --now rsyslog
##Set (uncomment by removing # and change ForwardToSyslog=yes
nano /etc/systemd/journald.conf
ForwardToSyslog=yes
Ctrl+o and enter to save, Ctrl+x to exit nano
systemctl restart systemd-journald
##Verify /var/log/syslog is growing
tail -f /var/log/syslog
##Create the forwarding configuration
nano /etc/rsyslog.d/10-syslogserver.conf
# Kiwi syslog forwarding
*.* action(
type="omfwd"
target="<COLLECTOR_IP>"
port="<PORT>"
protocol="tcp" ##(or "udp" depending on syslog server requirements)
template="RSYSLOG_SyslogProtocol23Format"
OR
template=RSYSLOG_TraditionalForwardFormat"
##do not use both template lines - use the one required by your syslog server. Kiwi requires Traditional (RFC 3164). Solutions like Adlumin require Protocol23 (RFC 5424).##
queue.type="LinkedList"
queue.filename="kiwi_fwd"
queue.maxdiskspace="1g"
queue.saveonshutdown="on"
action.resumeRetryCount="-1"
)
##Ctrl+o and enter to save, Ctrl+x to exit nano editor
## Validate and restart:
rsyslogd -N1
systemctl restart rsyslog
systemctl status rsyslog
##Test:
logger -t kiwi-test "proxmox-node1 syslog forwarding test $(date)"
systemctl status rsyslog
journalctl -u rsyslog -n 50 --no-pager
ls -la /var/spool/rsyslog
##this should send data to your syslog server. If your Proxmox firewall is enabled, don't forget to create an outbound exception for the IP and Port to your syslog server.##
pveproxy web-UI access logs live in /var/log/pveproxy/access.log, not the journal, so they won't be picked up by the configuration above. If you want your syslog server to track auth/UI access, add an imfile input:
nano /etc/rsyslog.d/10-syslogserver.conf
##place the following text at the top of the 10-syslogserver.conf file, above the action block, and restart rsyslog##
module(load="imfile")
input(type="imfile"
File="/var/log/pveproxy/access.log"
Tag="pveproxy"
Severity="info"
Facility="local6")
Ctrl+o and enter to save, Ctrl+x to exit nano.
systemctl restart rsyslog
systemctl status rsyslog
RSS Feed