<< Back to article

Logging/Journal

Logging is the process of recording events and messages from applications and the operating system. Effective logging is essential for debugging, auditing, compliance, and monitoring system health.

Traditional Syslog

The traditional logging architecture on Linux uses the syslog protocol and daemon (e.g., rsyslog or syslog-ng).

Syslog categorises messages by facility (source) and priority (severity):

Facilities: auth, cron, daemon, kernel, mail, syslog, user, local0-local7.

Priorities: emerg, alert, crit, err, warning, notice, info, debug.

Logs are typically stored in /var/log/:

  • /var/log/syslog or /var/log/messages: General system log.
  • /var/log/auth.log: Authentication events.
  • /var/log/kern.log: Kernel messages.
  • /var/log/dmesg: Kernel ring buffer at boot.

systemd-journald

Modern systems using systemd use systemd-journald as the central logging service. It collects logs from the kernel, systemd services, and other sources.

Key features:

  • Structured logging: Supports JSON and key-value pairs.
  • Binary storage: Logs are stored in /var/log/journal/ (persistent) or /run/log/journal/ (volatile).
  • Correlation: Correlates logs from different services using monotonic timestamps.
  • Forwarding: Can forward logs to syslog, the kernel, or the console.

Viewing Logs with journalctl

The journalctl command queries the journal:

journalctl                        # View all logs
journalctl -u sshd                # Logs for a specific unit
journalctl -b                     # Logs since last boot
journalctl -f                     # Follow logs in real-time
journalctl -p err                 # Priority level and above
journalctl --since "1 hour ago"   # Time-based filtering
journalctl -o json-pretty         # Structured JSON output

Persistent vs Volatile Journal

Enable persistent storage:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

Log Rotation

Traditional syslog daemons use logrotate to rotate logs, preventing disk exhaustion:

# /etc/logrotate.d/rsyslog
/var/log/syslog
{
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
}

Systemd-journald handles rotation automatically, limiting journal size via SystemMaxUse, SystemKeepFree, and MaxRetentionSec in /etc/systemd/journald.conf.

Structured Logging

Applications can emit structured logs in JSON format for easier parsing and analysis:

{"timestamp": "2024-01-15T10:30:00Z", "level": "error", "service": "api", "message": "Database connection failed", "error_code": 1234}