Build a centralized logging system. Covers structured logging, log aggregation with ELK/Loki, log levels, retention policies, and searching logs effectively in production.
Log Management & Centralized Logging is the practice of collecting, storing, querying, and analyzing logs from multiple systems—services, hosts, containers, databases, and external APIs—into a single, unified system. It matters when you have more than two services, when services run across multiple environments, or when troubleshooting issues that span several components. It becomes critical when a user reports a failure and you must trace logs across a dozen microservices, each emitting structured and unstructured data in different formats, across different time zones.
Logs arrive from diverse origins: application code, system daemons, container runtimes, and external services. To centralize them, you must define a consistent ingestion pipeline. The core workflow involves agents running on each host or in each container, or a sidecar or dedicated service that collects logs from stdin, files, or structured outputs.
The simplest ingestion is a file-based approach using rsyslog or systemd-journald. On a Linux host, journalctl exposes logs via a structured JSON format:
journalctl --output=json --follow --since="2024-05-10T08:00:00"Each entry includes __REALTIME_TIMESTAMP, PRIORITY, MESSAGE, and a SYSLOG_IDENTIFIER. To forward these to a centralized system, configure rsyslog to send messages over UDP:
*.* @192.168.1.100:514;RSYSLOG_SyslogProtocol23This sends all logs from the host to a central server at 192.168.1.100 on port 514, using the Syslog Protocol 23 (RFC 5424). The receiver must be configured to parse the structured fields and accept the SYSLOG-MSG format.
For containerized environments, fluentd or Vector are commonly used. A fluentd configuration for collecting logs from Docker containers looks like:
<!-- /fluentd/etc/fluent.conf -->
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match **>
@type file
path /var/log/fluentd/central-logs
flush_interval 10s
compress gzip
</match>
<match app.**>
@type elasticsearch
host 192.168.1.100
port 9200
index_name app-logs-${time_slice}
time_key time
time_format %Y-%m-%d
</match>The forward source listens on TCP port 24224, and each Docker container emits logs via fluentd as a sidecar. The docker-compose.yml for a service includes:
services:
app:
image: myapp:v1.2
volumes:
- /var/log/app:/var/log/app
command: ["sh", "-c", "tail -f /var/log/app/app.log | fluent-cat app.logs"]The fluent-cat utility sends each line from the app’s app.log file into Fluentd’s input stream with a tag app.logs. Fluentd then enriches, filters, and forwards logs to Elasticsearch or another backend.
Raw logs are often unstructured or semi-structured. Centralized logging requires a consistent format. The most effective approach is to use JSON for structured logs, especially when logs contain context beyond the message.
An application emitting structured logs using json format in Go might write:
log.Printf(`{"level":"info","message":"User login succeeded","user_id":123,"duration_ms":34,"timestamp":"2024-05-10T08:01:12.456Z"}`)This output is consumed by a log agent and stored in a time-series database. The key is the consistent schema: timestamp, level, message, and structured fields like user_id, duration_ms. The absence of a timestamp field in a log line causes a drift in time alignment across services.
For systems that emit logs in text format, use a parser to extract structured fields. Fluentd’s parser plugin can split key-value pairs in a key=value format:
<parse>
@type multiline
multiline_start_regexp /^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z)/
format /(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z) (?<level>\w+) (?<message>.*)/
</parse>This parses timestamps and levels from lines like:
2024-05-10T08:01:12.456Z INFO User login succeededThe parser extracts timestamp, level, and message, and can add additional fields via regex capture groups.
Once logs are stored, the next task is querying and searching across thousands of log entries. The most common query interface is a time-range filter with keyword and field-based search.
A typical query in a centralized logging system might look like:
SELECT * FROM logs
WHERE timestamp >= '2024-05-10T08:00:00Z'
AND level = 'error'
AND service = 'auth'
AND 'user_id' = 123
ORDER BY timestamp DESC
LIMIT 100This query retrieves all error-level logs from the auth service for user ID 123, starting from 8:00 AM, sorted by time, returning the 100 most recent entries.
When logs are stored in Elasticsearch, such queries are executed via the query DSL. A full search request might be:
POST /logs/_search
{
"query": {
"bool": {
"must": [
{ "term": { "level": "error" } },
{ "term": { "service": "auth" } },
{ "range": { "timestamp": { "gte": "2024-05-10T08:00:00Z" } } }
],
"should": [
{ "match": { "message": "authentication failed" } },
{ "exists": { "field": "trace_id" } }
]
}
},
"sort": [
{ "timestamp": { "order": "desc" } }
],
"size": 100
}The must clauses require all conditions, while should clauses boost relevance. The exists clause is crucial when tracing distributed requests: a log entry with trace_id implies it belongs to a single request across services.
Failure modes in querying include:
2024-05-10T08:00:00Z, but a query uses 2024-05-10T08:00:00 (no Z) and misses entries.user_id in one service is userId in another, and the query fails to match.term query on a text field performs poorly when the field is not mapped as keyword.When logs are centralized, the next step is correlating events across services. The key enabler is a common trace ID that links requests across microservices.
A service generates a trace_id and propagates it through the request lifecycle. This ID is included in every log entry, and often in HTTP headers. The X-Trace-ID header is widely adopted:
GET /api/v1/users/123 HTTP/1.1
Host: auth.example.com
X-Trace-ID: 7f3a1d8c-b3b0-4d1e-9a6f-7b2c4d5e6f7aEach service logs its part of the request, including the trace_id. A query for a specific trace ID returns all logs in a single request flow:
[
{
"service": "gateway",
"trace_id": "7f3a1d8c-b3b0-4d1e-9a6f-7b2c4d5e6f7a",
"level": "info",
"message": "Request received",
"timestamp": "2024-05-10T08:00:00.123Z"
},
{
"service": "auth",
"trace_id": "7f3a1d8c-b3b0-4d1e-9a6f-7b2c4d5e6f7a",
"level": "error",
"message": "Authentication failed",
"error": "Invalid token",
"timestamp": "2024-05-10T08:00:01.456Z"
},
{
"service": "user",
"trace_id": "7f3a1d8c-b3b0-4d1e-9a6f-7b2c4d5e6f7a",
"level": "info",
"message": "User retrieved",
"user_id": 123,
"timestamp": "2024-05-10T08:00:02.001Z"
}
]The trace ID allows visualizing the entire request flow. A failure in the auth service can be linked to the gateway’s request and the user service’s response, revealing the full context of the error.
With logs centralized, you can set up alerts based on patterns in the log stream. The most common approach is to define alerting rules over log queries.
An alert rule might be:
alert:
name: "High Error Rate in Auth Service"
query: |
SELECT
count(*) as error_count,
avg(duration_ms) as avg_duration
FROM logs
WHERE
service = 'auth'
AND level = 'error'
AND timestamp >= now() - 5m
GROUP BY
time(1m)
condition: |
error_count > 10 OR avg_duration > 500
notification:
email: ["[email protected]", "[email protected]"]
slack:
webhook_url: "https://hooks.slack.com/services/abc/123/xyz"
channel: "#alerts"This rule runs every minute, checks the last five minutes of logs, and triggers an alert if there are more than 10 errors or an average duration over 500ms. The alert is sent via email and Slack.
The alert system must handle alert deduplication. If an error occurs every 30 seconds, the alert system must merge consecutive alerts into a single incident. A common failure is alert storms: a single issue generates 50 alerts in a minute, each with a unique trace_id and timestamp, but the alerting system does not group them by root cause.
Another sharp edge is alert threshold tuning. A threshold of error_count > 10 over 5 minutes works well for steady traffic but fails during peak load. A spike of 150 errors in 30 seconds triggers an alert, but the system does not account for baseline traffic.
Logs grow rapidly. A single service might generate 1 GB of logs per day. Centralized logging systems must manage retention policies and storage costs.
A common configuration uses tiered storage: hot data in fast storage (SSD), warm data in object storage (S3), and cold data in archival storage (tape or Glacier).
Retention is configured by a retention policy:
{
"index_patterns": ["app-logs-*"],
"retention": [
{
"days": 7,
"storage": "hot"
},
{
"days": 30,
"storage": "warm"
},
{
"days": 365,
"storage": "cold"
}
]
}This policy moves logs from hot to warm after 7 days, and to cold after 30 days. The system must trigger a migration task—such as a reindex or rollover—to move data across tiers.
A common performance pitfall is log ingestion lag. When logs arrive faster than they can be processed, a backlog builds. The fluentd agent may push logs to Elasticsearch, but the Elasticsearch cluster becomes overloaded. The logstash pipeline might fail to keep up with incoming logs, leading to:
The solution is to monitor ingestion metrics: queue size, processing latency, and throughput. A healthy system maintains a queue size of less than 10,000 events and a processing latency under 2 seconds.
user_id but the log schema expects userId, causing the field to be ignored./var/log/app.log, but logrotate is misconfigured and does not compress rotated files.fluentd daemon crashes and misses 10 minutes of logs, which are never recovered.client_ip is added to logs, but the central system does not recognize it and stores it as a raw string.LIKE query on message performs poorly when the field is not indexed, and the query planner chooses a sequential scan instead of an index seek.These failures are silent but costly: logs are collected, but the data is inconsistent, incomplete, or inaccessible.
Centralized logging is not a one-time setup. It requires ongoing maintenance: monitoring ingestion, tuning queries, managing retention, and responding to alert storms. The most effective teams treat logs not as a record of past events, but as a real-time operational dashboard.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy