Log Aggregation and Analysis: Making Sense of Distributed Systems

When systems are distributed across multiple services, containers, or nodes, logs become scattered and fragmented. The ability to correlate events from different sources into a unified timeline is essential for debugging, monitoring, and operational visibility. This process requires structured ingestion pipelines that can handle high-volume, real-time log streams while preserving context and enabling efficient querying.

Ingesting Logs from Multiple Sources

To collect logs from running processes, use journalctl with the --output=json flag to produce structured output:

journalctl --output=json --since="10 minutes ago" > /tmp/logs.json

For containerized workloads, capture logs using docker logs with timestamp formatting:

docker logs --since=1h --until=now <container_id> | sed 's/^/timestamp="'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"/' > /tmp/container.log

When collecting from Kubernetes pods, stream logs using kubectl logs with a JSON-formatted output:

kubectl logs --since=10m <pod_name> -c <container_name> | jq -R '{"message": .}' > /tmp/pod.json

Structuring Log Data for Analysis

Logs must be structured to enable filtering and aggregation. When writing application logs, include a consistent timestamp field:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "level": "INFO",
  "service": "payment-service",
  "message": "User 12345 initiated payment processing"
}

Use the --format option with logging libraries to enforce structured output. When using Python's logging module, set up a JSON formatter:

import json
import logging

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "service": "my-service",
            "message": record.getMessage()
        }
        return json.dumps(log_entry)

Filtering and Querying Log Streams

Use jq to filter logs by field values:

cat /tmp/logs.json | jq '. | select(.level == "ERROR")'

For timestamp-based filtering, use the --date flag with grep:

grep -E '2023-10-15T[0-9]{2}:[0-9]{2}:[0-9]{2}' /tmp/logs.json

To search for error messages across multiple log files, combine find with grep:

find /var/log -name "*.log" -exec grep -l "ERROR\|FATAL" {} \;

Aggregating Logs Across Services

Create a central log aggregation pipeline using rsyslog or similar tools. Configure rsyslog to forward logs over UDP:

# /etc/rsyslog.conf
*.* @<aggregator-host>:514

In the receiving instance, configure rsyslog to parse and store logs in structured format:

# /etc/rsyslog.d/01-forward.conf
module(load="imudp")
input(type="imudp" port="514")

$ActionFileDefaultTemplate RSYSLOG_FileFormat

Correlating Logs by Trace Context

When logs are part of a distributed trace, include a correlation ID in each log entry:

{
  "timestamp": "2023-10-15T14:30:45.123Z",
  "trace_id": "abc123def456",
  "span_id": "xyz789uvw012",
  "level": "INFO",
  "message": "Payment validated"
}

Use grep to find all logs from a specific trace:

grep "abc123def456" /var/log/*.log

Handling Log Volume and Retention

Configure log rotation using logrotate with retention policies:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    delaycompress
    copytruncate
}

For high-volume logs, enable compression at the source:

journalctl --output=json --since="1 hour ago" | gzip > /tmp/logs.json.gz

Detecting and Alerting on Log Patterns

Set up alerts using awk to detect recurring errors:

tail -f /var/log/app.log | awk '/ERROR/ { print strftime(), $0 }' | grep -E 'ERROR.*timeout'

To detect log lines that exceed a size threshold:

grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}' /var/log/app.log | awk 'length > 1000'

Troubleshooting Common Log Issues

Logs may be dropped silently if the receiving system is overwhelmed. Monitor rsyslog queue sizes with:

grep -i queue /var/log/rsyslog.log

When logs contain malformed JSON, filter out invalid entries:

cat /tmp/logs.json | jq -c 'select(.timestamp and .level and .message)' > /tmp/cleaned.json

Timestamp drift between systems can cause misalignment. Use timedatectl to verify clock synchronization:

timedatectl status

Querying Aggregated Logs

Use grep with regular expressions to search across multiple log files:

find /var/log -name "*.log" -exec grep -l "failed\|error" {} \; | xargs grep -n "failed"

For structured log analysis, use jq with filters:

cat /tmp/aggregated.json | jq '. | select(.level == "ERROR") | .timestamp, .message'

Managing Log Data Lifecycle

Configure automatic cleanup of old logs:

find /var/log -name "*.log" -mtime +30 -delete

Set up log archival to long-term storage:

tar -czf /backup/logs-$(date +%Y-%m-%d).tar.gz /var/log/app/

Identifying Log Ingestion Failures

Monitor for dropped log entries by comparing expected and received counts:

echo "Expected: $(wc -l < /tmp/input.log)" > /tmp/counts.txt
echo "Actual: $(jq -r 'length' /tmp/output.json)" >> /tmp/counts.txt

When logs are missing from aggregation, check for network issues with:

ping -c 3 <aggregator-host>

Handling Log Encoding and Character Sets

Log files may contain non-ASCII characters. Use iconv to convert encodings:

iconv -f utf-8 -t utf-8 //TRANSLIT /tmp/utf8.log > /tmp/conversion.log

Monitoring Log Pipeline Health

Check for log pipeline failures using system monitoring tools:

systemctl status rsyslog
journalctl -u rsyslog --since="10 minutes ago"

Use stat to verify log file integrity:

stat /var/log/app.log

Debugging Log Format Inconsistencies

When logs from different services have varying formats, standardize using sed or awk:

cat /tmp/inconsistent.log | sed 's/^\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}\) \(.*\)/{"timestamp":"\1","message":"\2"}/'

Managing Log Storage and Performance

Monitor disk usage with:

df -h /var/log

When logs grow too large, implement partitioning by date or service:

mkdir -p /var/log/app/$(date +%Y-%m)
mv /var/log/app/*.log /var/log/app/$(date +%Y-%m)/

Configuring Log Forwarding

Set up a log forwarding agent to send logs to an external system:

# Example: forward logs to HTTP endpoint
curl -X POST -H "Content-Type: application/json" --data-binary @/tmp/logs.json http://log-collector.example.com/logs

Handling Log Rotation and File Management

When log files rotate, ensure that the ingestion system can handle file transitions:

tail -F /var/log/app.log | while read line; do echo "$line"; done

This approach ensures that when a log file is rotated, tail continues reading from the new file without interruption.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.