Build an observability stack that helps you find the root cause of production issues in minutes, not hours. Covers the three pillars, structured logging, metric design, distributed tracing, correlation IDs, and the alert design that prevents notification fatigue.
This page describes how to use logs, metrics, and traces in ways that help you debug real problems, not just show pretty graphs. It matters when you need to understand what went wrong in a system that's already running, especially when the dashboard view is misleading or unavailable.
You want to see all logs from a request that failed, but your logs are scattered across services. Use grep with timestamps and trace IDs:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "trace-id-abc123"If you're using structured logging, filter by field:
kubectl logs deployment/my-app -l app=my-app --since=1h | jq '.trace_id == "abc123"'Failure mode: If your log lines are not timestamped or do not include trace identifiers, grep will miss relevant entries. Logs without consistent fields will make filtering unreliable.
You see a metric spike in your dashboard but don't know what caused it. Use kubectl top to find the pod with the highest resource usage:
kubectl top pods --sort-by=cpuThen get detailed metrics from Prometheus:
curl -G \
--data-urlencode 'query=rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])' \
http://prometheus:9090/api/v1/queryIf your metric shows a sudden increase, you can also query for the last few minutes to see if it's a transient issue:
curl -G \
--data-urlencode 'query=rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[1m])' \
http://prometheus:9090/api/v1/queryFailure mode: Prometheus may not have the full history if it's configured with short retention periods or if a node was down during part of the spike. You might need to query a cluster-wide view rather than just one metric.
You have a trace ID but want to see what happened in each service along the path. Use kubectl get pods to find services involved in the trace, then examine logs for specific spans:
kubectl logs deployment/service-a -l app=service-a --since=10m | grep "span-id-xyz789"If you're using OpenTelemetry, look for spans with matching trace IDs in your collector's output or log file:
kubectl logs deployment/otel-collector -n observability | grep "trace-id-def456"Failure mode: If the trace ID is not propagated through all services, some spans will appear missing. You can miss intermediate services if they don't implement tracing correctly.
You want to understand why a specific pod started failing or restarted. Check both metrics and logs:
kubectl describe pod my-pod-789abc
kubectl logs my-pod-789abc --previousUse kubectl get events for system-level issues:
kubectl get events --sort-by=.metadata.creationTimestamp | grep "my-pod"Failure mode: Kubernetes events may not contain the exact error messages from your application. You must manually correlate timestamps and log lines to build a timeline.
Sometimes dashboards hide what's in your raw logs. Use kubectl logs without any aggregation:
kubectl logs deployment/my-app -l app=my-app --since=30m --tail=1000To get logs from multiple containers in a pod:
kubectl logs deployment/my-app -l app=my-app --since=30m --all-containers=trueFailure mode: If your log output is too verbose or if you don't have enough tail, you may miss recent errors. Some tools compress logs and require --timestamps to see timestamps.
You're looking at a crash dump or restart and want to understand the state before it happened. Use:
kubectl logs my-pod --previousIf your container uses structured logging, extract key fields:
kubectl logs my-pod --previous | jq '.message'Failure mode: Previous logs are only available if the pod was terminated gracefully. If the container crashed without cleanup, you may lose some of the most recent log lines.
You want to know why a request took longer than expected. Look for slow operations:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "duration.*ms"If you have structured logging, filter by duration thresholds:
kubectl logs deployment/my-app -l app=my-app --since=1h | jq '.duration > 500'Failure mode: If your application doesn't log durations or uses a different time unit (e.g., seconds instead of milliseconds), you'll get false positives.
You suspect a pod is being throttled or killed by the system. Look for OOMKilled events:
kubectl describe pod my-pod | grep "OOMKilled"Check logs for memory warnings:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep -i "memory\|oom\|gc"Failure mode: If your application is not logging memory usage or GC events, you may miss the root cause of throttling.
You're trying to understand why a service seems to be hanging. Use kubectl top and then check logs:
kubectl top pods --sort-by=cpu
kubectl logs deployment/my-service -l app=my-service --since=10m | grep "waiting\|blocked"Failure mode: If your service is using a custom health check, the readiness probe might not match what's actually happening in the application logic.
You're debugging a rollout that didn't go as expected. Use:
kubectl rollout status deployment/my-app
kubectl describe deployment my-app
kubectl get pods -l app=my-appCheck logs for each new pod:
kubectl logs deployment/my-app -l app=my-app --since=5m | grep "starting\|init"Failure mode: If the rollout is stuck, you might not see errors in the deployment status until a timeout occurs. You need to look at individual pod logs and events.
You're seeing missing or zero metrics in your dashboard. Use:
kubectl get pods -l app=my-app
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "metric\|exporter"If you're using Prometheus, query for metric labels:
curl -G \
--data-urlencode 'query=up' \
http://prometheus:9090/api/v1/queryFailure mode: If your metric exporter is misconfigured or fails silently, you may get no data at all. You must manually check the endpoint and verify that metrics are being collected.
You're troubleshooting authentication failures. Look for specific error patterns:
kubectl logs deployment/auth-service -l app=auth-service --since=1h | grep "auth\|token\|invalid"Failure mode: If your service doesn't log all failed attempts or if tokens are handled outside the main application, you may miss important clues.
You're debugging a complex workflow involving multiple services. Start with one service and trace its logs:
kubectl logs deployment/service-a -l app=service-a --since=1h | grep "request-id-xyz123"Then find related logs from downstream services:
kubectl logs deployment/service-b -l app=service-b --since=1h | grep "request-id-xyz123"Failure mode: If your request IDs are not passed through the entire chain, you'll lose context. You must ensure that all components log and propagate trace identifiers.
You're investigating a job that might be silently failing. Use:
kubectl logs job/my-job --previousIf your job has multiple containers, check each one:
kubectl logs job/my-job --previous --all-containers=trueFailure mode: If the job uses a different logging format or if it doesn't log its status to stdout, you may miss what went wrong. You must also check the job's events.
You want to ensure logs are structured correctly for parsing:
kubectl logs deployment/my-app -l app=my-app --since=5m | head -10If logs are JSON, validate them:
kubectl logs deployment/my-app -l app=my-app --since=5m | jq .Failure mode: If your log format changes unexpectedly, or if you have mixed formats in a single service, parsing tools might fail. Always test your log parsing logic with sample data.
Use kubectl logs with --follow to watch for real-time errors:
kubectl logs deployment/my-app -l app=my-app --since=1m --followFailure mode: If the pod crashes or restarts during your observation, you may miss critical information from the previous run. Always use --previous when appropriate.
You've added custom metrics to help debug issues but want to validate they're working:
kubectl get pods -l app=my-app
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "custom_metric"Failure mode: If your metric isn't being exported correctly or if there's a delay in aggregation, you'll see stale data. You must check both the application and the metrics endpoint.
You're using tracing but want to confirm it's working end-to-end:
kubectl logs deployment/otel-collector -n observability --since=10m | grep "trace\|span"Failure mode: If your collector is dropping spans due to high cardinality or misconfiguration, you'll miss traces. You must validate that spans are being received and processed correctly.
You suspect a service isn't working because of missing dependencies. Look for connection errors:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "connection\|timeout\|connect"Failure mode: If your application doesn't log all connection attempts or retries, you may miss the actual failure point. You must also check for DNS resolution or service discovery issues.
You suspect memory issues in a service:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "memory\|gc\|heap"Failure mode: If your application doesn't expose memory metrics or doesn't log garbage collection events, you'll need to rely on external tools or manual inspection.
You want to enable debug logging temporarily without redeploying:
kubectl exec deployment/my-app -it -- sh -c 'echo "debug" > /tmp/log-level'Then watch the logs:
kubectl logs deployment/my-app -l app=my-app --since=5m | grep "debug"Failure mode: If your application doesn't support dynamic log level changes, you'll need to redeploy or restart the service. You must verify that your logging system supports runtime configuration.
Use grep and awk to find common errors:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "error" | awk '{print $NF}' | sort | uniq -cFailure mode: If your error messages are inconsistent or if you're filtering too broadly, you may miss the root cause of a problem.
You want to see how long requests take across different services:
kubectl logs deployment/service-a -l app=service-a --since=1h | grep "duration" | awk '{print $NF}' | sort -nFailure mode: If your latency measurements are inconsistent or if you're missing spans, you'll get incomplete timing data.
You want to see how services interact in time:
kubectl logs deployment/service-a -l app=service-a --since=10m | grep "request-id-xyz123"
kubectl logs deployment/service-b -l app=service-b --since=10m | grep "request-id-xyz123"Failure mode: If timestamps are not synchronized across services, you'll get a misleading timeline. You must ensure all services use consistent time sources.
You're unsure if tracing is working properly:
kubectl logs deployment/otel-collector -n observability --since=5m | grep "export\|span"Failure mode: If the collector is dropping traces due to high cardinality, rate limits, or configuration errors, you'll get incomplete trace data. You must monitor the collector's own metrics and logs.
You want to isolate issues to a specific service:
kubectl logs deployment/my-service -l app=my-service --since=30m | grep "error\|exception"Failure mode: If your service uses multiple logging frameworks or outputs to different streams, you may miss important error messages.
You suspect a misconfiguration is causing problems:
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "config\|setting\|option"Failure mode: If your application doesn't log configuration changes or if it doesn't validate settings, you'll miss issues that are not obvious from the code.
You're debugging why a deployment failed:
kubectl describe deployment my-app
kubectl get events --sort-by=.metadata.creationTimestamp | grep "my-app"
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "init\|setup"Failure mode: If the deployment uses init containers or pre-stop hooks, you must check those as well. You may miss issues if you only look at the main application logs.
You're troubleshooting why logs aren't reaching their destination:
kubectl logs deployment/logging-agent -n logging --since=1h | grep "error\|retry"Failure mode: If your logging agent has its own configuration issues or if it's dropping logs due to rate limits, you'll get incomplete log data. You must validate the agent's health and metrics.
You're debugging a rollback that didn't work as expected:
kubectl rollout history deployment/my-app
kubectl logs deployment/my-app -l app=my-app --since=10m | grep "rollback\|revert"Failure mode: If the rollback process itself failed, you may get no logs or misleading information. You must check both the deployment and the application logs.
You're looking for signs that a service ran out of resources:
kubectl top pods --sort-by=memory
kubectl logs deployment/my-app -l app=my-app --since=1h | grep "memory\|oom\|limit"Failure mode: If your application doesn't log resource usage or if it's not configured to handle resource limits gracefully, you'll miss early warning signs.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy