Log Management & Monitoring
Understand system logs, journalctl, and monitoring tools.
journalctl (systemd logs)
$ journalctl # All logs
$ journalctl -f # Follow logs (like tail -f)
$ journalctl -u nginx # Logs for specific service
$ journalctl --since "1 hour ago"
$ journalctl -p err # Only errors and above
$ journalctl -f # Follow logs (like tail -f)
$ journalctl -u nginx # Logs for specific service
$ journalctl --since "1 hour ago"
$ journalctl -p err # Only errors and above
Key Log Files
/var/log/syslog- General system logs (Debian/Ubuntu)/var/log/messages- General system logs (RHEL/CentOS)/var/log/auth.log- Authentication logs/var/log/nginx/- Nginx access and error logs
Understanding journald output
journald stores structured metadata (unit name, PID, UID, boot ID) in addition to message text.
This makes filtering far more reliable than grepping raw files.
# Filter by boot, priority, and time window
$ journalctl -b # current boot
$ journalctl -b -1 # previous boot
$ journalctl -p warning..emerg # only warnings and above
$ journalctl --since \"2025-12-20 10:00\" --until \"2025-12-20 11:00\"
# Useful output formats
$ journalctl -u nginx -o short-iso -n 50
$ journalctl -u nginx -o json-pretty -n 3
$ journalctl -b # current boot
$ journalctl -b -1 # previous boot
$ journalctl -p warning..emerg # only warnings and above
$ journalctl --since \"2025-12-20 10:00\" --until \"2025-12-20 11:00\"
# Useful output formats
$ journalctl -u nginx -o short-iso -n 50
$ journalctl -u nginx -o json-pretty -n 3
Persistent logs (don’t lose logs on reboot)
Some distros keep the journal in memory by default. For servers, you typically want persistent storage.
# Enable persistent journald storage
$ sudo mkdir -p /var/log/journal
$ sudo systemctl restart systemd-journald
# Verify disk usage
$ journalctl --disk-usage
$ sudo mkdir -p /var/log/journal
$ sudo systemctl restart systemd-journald
# Verify disk usage
$ journalctl --disk-usage
Log rotation (logrotate)
Logs grow forever unless rotated. Most distros use logrotate to compress and prune old logs.
You should understand retention (how many days/files) and whether the service needs reload after rotation.
$ ls /etc/logrotate.d | head
$ cat /etc/logrotate.d/nginx | sed -n '1,80p'
# Dry run / debug
$ sudo logrotate -d /etc/logrotate.conf
$ sudo logrotate -f /etc/logrotate.conf
$ cat /etc/logrotate.d/nginx | sed -n '1,80p'
# Dry run / debug
$ sudo logrotate -d /etc/logrotate.conf
$ sudo logrotate -f /etc/logrotate.conf
Monitoring patterns
In production, you want to detect problems early. Typical approaches:
- •Local triage:
journalctl -u service -f+tail -f+ dashboards - •Centralized logging: ship logs to a system like ELK/OpenSearch, Loki, or a SIEM
- •Alerts: trigger on error rate spikes, authentication failures, disk full, service restarts
✅ Practice (15 minutes)
- Pick a service (e.g.,
ssh) and view its last 200 log lines withjournalctl -u. - Check total journal size with
journalctl --disk-usageand set a retention cap in/etc/systemd/journald.conf(read-only if learning). - Find your nginx/apache logrotate config and understand how many rotated files are kept.