How to make a running system explainable from the outside: what metrics, logs, traces and wide events each answer, why cardinality is the cost driver, how sampling and context propagation actually work, and the failure modes that turn an observability stack into expensive noise.
Observability is the practice of making system behavior visible through structured data collection, analysis, and alerting. It matters when you need to understand what your systems are doing in production without direct access to their internals, particularly during incidents or performance degradation.
Applications emit logs as structured JSON or plain text output to standard error or standard out. For containers running on Kubernetes, logs are collected by the container runtime and stored in the node's filesystem.
Example of a log line:
{"timestamp":"2023-10-15T14:30:45.123Z","level":"ERROR","message":"Database connection failed","error":"connection timeout after 30s"}To collect logs from a pod in Kubernetes:
kubectl logs <pod-name> -n <namespace>Logs are usually timestamped and stored in chronological order. When investigating an incident, you may need to search within a time window.
Example command to get logs from the last hour:
kubectl logs --since=1h <pod-name> -n <namespace>If your system uses a log aggregation platform (like Elasticsearch or Splunk), you can query logs using their respective query languages.
Most log systems support filtering by level, message content, or metadata fields. For example, in a system where logs are structured as JSON, you might filter by error type:
kubectl logs <pod-name> -n <namespace> | grep '"error":"connection timeout"'Metrics are numeric values that represent system state over time. They are typically collected by agents running on each host or in containers.
Example metric line:
http_requests_total{method="GET",endpoint="/api/users",status="200"} 1234Prometheus scrapes metrics endpoints exposed by applications. The scrape configuration is defined in a YAML file:
scrape_configs:
- job_name: 'my-app'
static_configs:
- targets: ['localhost:8080']To add a metric to your application, you can use the Prometheus client library:
import "github.com/prometheus/client_golang/prometheus"
var httpRequestCount = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests",
},
[]string{"method", "endpoint", "status"},
)
func main() {
prometheus.MustRegister(httpRequestCount)
// ...
}Once collected, metrics are queried using PromQL (Prometheus Query Language). To find total requests in the last 5 minutes:
rate(http_requests_total[5m])Alerts are defined by threshold conditions that trigger when metric values cross certain boundaries. For Prometheus, alert rules are written in YAML and stored in a file:
groups:
- name: app-alerts
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status!="200"}[5m]) > 0.1
for: 2m
labels:
severity: page
annotations:
summary: "High error rate detected"Some systems support alerting based on log patterns. For example, if a particular error message appears more than once per minute:
kubectl logs <pod-name> -n <namespace> --since=1m | grep -c "Database connection failed"If the count is greater than 1, this could trigger an alert.
When you see a log message indicating an error, you should correlate it with metrics from that time. If your application logs a request failure, you want to check if there's a corresponding spike in error rates or latency.
Example workflow:
If your system supports distributed tracing (like OpenTelemetry), you can correlate logs with trace IDs:
kubectl logs <pod-name> -n <namespace> | grep 'trace_id=12345'Alerts may be suppressed during known maintenance windows or when other alerts are already active. This is typically handled by the alerting system configuration.
Example of a suppression rule:
- name: maintenance_window
conditions:
- type: time_window
start: "02:00"
end: "06:00"An alert fires when its condition is met for the specified duration (for clause in PromQL). It resolves when the condition is no longer true.
A common silent failure occurs when logs or metrics are not being collected. This can happen due to:
To detect missing data, you should monitor the ingestion rate of your observability tools.
High cardinality in metrics (many unique label combinations) can cause performance problems or make queries slow. For example, a metric with labels like:
http_requests_total{method="GET",endpoint="/api/users/12345",status="200"}is less efficient than:
http_requests_total{method="GET",endpoint="/api/users",status="200"}If logs are not being collected, the system might silently drop them or continue running without visibility. You can detect this by monitoring log volume or using a health check endpoint.
Example command to check if logs are flowing:
kubectl logs <pod-name> -n <namespace> | tail -n 10If no output appears, or if you see errors like "EOF" or "connection reset", it indicates the log collection is failing.
Alerts should not just react to absolute thresholds but also detect unusual patterns. For instance, a sudden change in request latency distribution might indicate an issue even if the average hasn't crossed a threshold.
Example pattern detection:
kubectl logs <pod-name> -n <namespace> | grep 'latency' | awk '{print $NF}' | sort -n | head -10This will show the 10 lowest latency values, which might reveal an unexpected performance degradation.
If you cannot see logs or metrics from a specific application:
Example check for agent status:
kubectl get pods -n monitoringThe alerting system itself can fail. You should monitor:
Example command to verify Prometheus is scraping:
curl http://prometheus:9090/api/v1/targetsIf targets are down, you'll see status "down" in the response.
Too many log messages can overwhelm systems and make relevant information harder to find. Log at appropriate levels:
All applications should use consistent log formats. This allows easy parsing and searching across systems.
Example of a consistent format:
{"timestamp":"2023-10-15T14:30:45.123Z","level":"ERROR","service":"auth-service","message":"Database connection failed","error":"connection timeout after 30s"}The observability system is critical for understanding your systems. You should monitor:
This creates a feedback loop where you can detect failures in your observability infrastructure.
Start with basic alerts and add complexity only when needed. Complex alerting logic is harder to debug and maintain.
Example of a simple alert:
rate(http_requests_total{status!="200"}[5m]) > 0.1Example of a complex alert:
rate(http_requests_total{status!="200"}[5m]) > 0.1 and
abs(rate(http_requests_total[5m]) - avg(rate(http_requests_total[5m])) by (job)) > 0.5The complex version adds a comparison against average rates, which can be useful but also harder to understand when it triggers.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy