Logs and journalctl: Read and understand logs to debug like a ninja

Understanding Linux Logging Architecture

Linux systems generate extensive logs that are crucial for system monitoring, troubleshooting, and security analysis. Modern distributions use systemd’s journal alongside traditional syslog for comprehensive logging.

Key Logging Components

systemd-journald: The primary logging daemon that collects logs from kernel, system services, and applications. It stores logs in a binary format and provides powerful querying capabilities.

rsyslog/syslog: Traditional text-based logging system that works alongside journald, often storing logs in /var/log/ as human-readable files.

Log Levels: Both systems use standard severity levels (emergency, alert, critical, error, warning, notice, info, debug) to categorize message importance.

Essential journalctl Commands

Basic Log Viewing

View all journal entries:

1

journalctl

Show recent logs (last 10 lines):

1

journalctl -n 10

Follow logs in real-time (like tail -f):

1

journalctl -f

Time-Based Filtering

Show logs from today:

1

journalctl --since today

Show logs from the last hour:

1

journalctl --since "1 hour ago"

Show logs between specific times:

1

journalctl --since "2024-01-01 09:00:00" --until "2024-01-01 17:00:00"

Show logs from last boot:

1

journalctl -b

List all boots and show logs from specific boot:

1
2

journalctl --list-boots
journalctl -b -1  # Previous boot

Service and Unit Filtering

Show logs for specific service:

1
2

journalctl -u nginx
journalctl -u ssh.service

Show logs for multiple services:

1

journalctl -u nginx -u apache2

Follow specific service logs:

1

journalctl -u mysql -f

Priority and Message Filtering

Show only error and critical messages:

1

journalctl -p err

Show warning level and above:

1

journalctl -p warning

Search for specific text:

1
2

journalctl | grep "failed"
journalctl -u ssh | grep "authentication failure"

Advanced Filtering Options

Show logs from specific user:

1

journalctl _UID=1000

Show kernel messages:

1

journalctl -k

Show logs from specific process:

1

journalctl _PID=1234

Show logs with specific field:

1

journalctl _SYSTEMD_UNIT=nginx.service

Traditional Log Files Location

Understanding /var/log/ directory structure is essential for comprehensive log analysis:

Critical System Logs

/var/log/syslog: General system activity log containing kernel messages, system services, and application logs.

/var/log/auth.log: Authentication attempts, sudo usage, and security-related events. Critical for security monitoring.

/var/log/kern.log: Kernel messages including hardware detection, driver issues, and system-level errors.

/var/log/dmesg: Boot-time kernel messages, hardware initialization, and early system startup information.

Service-Specific Logs

/var/log/apache2/ or /var/log/httpd/: Web server access and error logs.

/var/log/nginx/: Nginx web server logs.

/var/log/mysql/: Database server logs including slow queries and errors.

/var/log/mail.log: Email server logs for postfix, sendmail, or other mail services.

Log Analysis Techniques

Using Traditional Tools

Tail recent entries:

1
2

tail -f /var/log/syslog
tail -n 50 /var/log/auth.log

Search with grep:

1
2

grep "ERROR" /var/log/syslog
grep -i "failed" /var/log/auth.log

Analyze patterns with awk:

1

awk '/failed/ {print $1, $2, $3, $NF}' /var/log/auth.log

Count occurrences:

1

grep "connection refused" /var/log/syslog | wc -l

Advanced journalctl Analysis

Show unique field values:

1
2

journalctl -F _SYSTEMD_UNIT
journalctl -F PRIORITY

Export logs in different formats:

1
2
3

journalctl -o json
journalctl -o json-pretty
journalctl -o verbose

Show log statistics:

1
2

journalctl --disk-usage
journalctl --verify

Debugging Common Issues

System Boot Problems

Check boot process:

1
2

journalctl -b | grep -i error
journalctl -b | grep -i failed

Analyze specific boot:

1
2

journalctl --list-boots
journalctl -b -2 | grep -i error

Service Failures

Investigate failed service:

1
2

systemctl status service-name
journalctl -u service-name --since "10 minutes ago"

Check service dependencies:

1
2

systemctl list-dependencies service-name
journalctl -u service-name -u dependency-service

Performance Issues

Monitor resource usage patterns:

1
2

journalctl | grep -i "out of memory"
journalctl | grep -i "killed process"

Check disk space issues:

1
2

journalctl | grep -i "no space left"
journalctl | grep -i "disk full"

Security Investigations

Authentication failures:

1
2

journalctl | grep -i "authentication failure"
grep "Failed password" /var/log/auth.log

Successful logins:

1
2

grep "Accepted" /var/log/auth.log
journalctl | grep -i "session opened"

Sudo usage:

1
2

grep "sudo:" /var/log/auth.log
journalctl | grep -i sudo

Log Management Best Practices

Journal Configuration

Configure journal retention in /etc/systemd/journald.conf:

1
2
3

SystemMaxUse=500M
MaxRetentionSec=1month
MaxFileSec=1week

Log Rotation

Traditional logs rotate via logrotate configuration in /etc/logrotate.conf and /etc/logrotate.d/.

Manually rotate logs:

1

sudo logrotate -f /etc/logrotate.conf

Persistent Journals

Make journal persistent across reboots:

1
2

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Pro Tips for Log Ninja Skills

Combine multiple sources: Cross-reference journalctl output with traditional log files for complete picture.

Use time correlation: When investigating issues, check logs from all relevant services during the same time window.

Create custom filters: Save complex journalctl commands as aliases or scripts for repeated use.

Monitor in real-time: Use journalctl -f with specific filters during troubleshooting sessions.

Leverage structured data: Use journalctl’s field filtering capabilities to quickly isolate relevant information.

Regular log review: Establish routine log monitoring to catch issues before they become critical.

Modern Observability: Beyond Traditional Logging

The rise of containers, microservices, and cloud-native applications has transformed how we approach logging and monitoring. Traditional log analysis now forms the foundation layer in a comprehensive observability stack.

The Modern Observability Stack

Metrics Collection: Prometheus scrapes metrics from applications and infrastructure, providing time-series data for performance monitoring.

Log Aggregation: Tools like ELK Stack (Elasticsearch, Logstash, Kibana), Loki, or Fluentd collect and centralize logs from multiple sources.

Visualization: Grafana creates dashboards combining metrics, logs, and traces for unified observability.

Distributed Tracing: Jaeger or Zipkin track requests across microservices to identify bottlenecks.

Container and Kubernetes Logging

Container logs require different approaches since containers are ephemeral:

View Docker container logs:

1
2
3

docker logs container-name
docker logs -f --tail 50 container-name
journalctl -u docker.service

Kubernetes pod logs:

1
2
3

kubectl logs pod-name
kubectl logs -f pod-name -c container-name
kubectl logs --previous pod-name  # Previous container instance

Integration Strategies

Foundation Layer: Traditional journalctl and syslog remain essential for:

  • Host-level debugging when containers fail
  • System service issues affecting container runtime
  • Security investigations requiring host-level audit trails
  • Boot and kernel issues that prevent containers from starting

Application Layer: Modern tools handle:

  • Distributed application logs across multiple services
  • Metrics correlation with log events
  • Real-time alerting and anomaly detection
  • Long-term log retention and analytics

Hybrid Debugging Workflow

Step 1 - Start with Observability Dashboards: Check Grafana dashboards for high-level service health and identify affected components.

Step 2 - Drill Down with Log Aggregation: Use Kibana or Grafana’s log panels to examine application-specific logs and error patterns.

Step 3 - Investigate Infrastructure: When application logs don’t reveal the root cause, use traditional tools:

1
2

journalctl -u docker.service --since "1 hour ago"
journalctl -u kubelet.service -f

Step 4 - Correlate Timeline: Cross-reference timestamps between modern dashboards and system logs to identify infrastructure issues affecting applications.

Practical Modern Scenarios

Kubernetes Pod Crashes:

1
2
3
4
5
6
7

# Modern approach
kubectl describe pod failing-pod
kubectl logs failing-pod --previous

# Traditional backup
journalctl -u kubelet --since "30 minutes ago" | grep failing-pod
journalctl -u docker --since "30 minutes ago" | grep container-id

Container Network Issues:

1
2
3
4
5
6
7

# Check container networking
docker network ls
kubectl get networkpolicies

# Check host networking
journalctl -u systemd-networkd
journalctl | grep -i "network\|interface"

Resource Exhaustion Investigation:

1
2
3
4
5
6
7

# Modern metrics (if available)
# Check Grafana dashboards for CPU/Memory trends

# Traditional fallback
journalctl | grep -i "out of memory"
journalctl | grep -i "killed process"
dmesg | grep -i "memory"

Log Pipeline Architecture

Collection: Fluent Bit or Filebeat agents collect logs from containers and forward to centralized systems.

Processing: Logstash or Vector process, filter, and enrich logs before storage.

Storage: Elasticsearch, Loki, or cloud logging services store processed logs.

Analysis: Kibana, Grafana, or custom dashboards provide search and visualization.

Alerting: Prometheus Alertmanager or similar tools trigger notifications based on log patterns.

When to Use Which Approach

Use Traditional Methods When:

  • Container runtime is failing
  • Host system issues prevent modern tools from working
  • Network connectivity problems affect log aggregation
  • Investigating security incidents requiring audit trails
  • Debugging system boot or kernel issues

Use Modern Tools When:

  • Analyzing distributed application behavior
  • Correlating metrics with log events
  • Setting up proactive monitoring and alerting
  • Investigating application performance issues
  • Managing logs at scale across multiple environments

Bridging the Gap

Export journalctl to modern systems:

1

journalctl -o json | curl -X POST elasticsearch:9200/system-logs/_doc -H "Content-Type: application/json" -d @-

Configure log forwarding:

1
2
3
4
5
6
7
8
9

# Fluent Bit configuration to forward journalctl
[INPUT]
    Name systemd
    Path /var/log/journal
    
[OUTPUT]
    Name elasticsearch
    Host elasticsearch
    Port 9200

Create unified dashboards that combine traditional system metrics with application observability for complete visibility.

Understanding both traditional logging and modern observability tools gives you comprehensive insight into system behavior across the entire stack. Master these complementary approaches, and you’ll debug complex distributed systems with ninja-like precision and efficiency.