Debugging Production Guide

Comprehensive guide to debugging production guide covering architecture, implementation, testing, and operational patterns for production engineering teams.

When a production system misbehaves, the pressure to resolve it quickly often conflicts with the need to preserve data integrity and service continuity. This guide maps the practical workflow of identifying, isolating, and resolving live failures without causing further damage. It is intended for engineers who already have operational access and familiarity with their stack’s core tools, but need a structured approach when the usual development workflow breaks down.

The guide is organized by what you are trying to achieve at the moment, not by technology taxonomy. Each section provides the exact commands, flags, and error text you will type, using standard, shipping tools that exist in virtually every modern production environment.

Finding the Entry Point

The first step in any production debug is narrowing the scope of the failure. You need to answer three questions: is it a single host, a service, a network path, or the entire platform? Do not skip this step; proceeding without it almost always results in wasted time and potential collateral damage.

Checking Service Health

Most production systems expose a health endpoint. Use curl with verbose output to see exactly what the server returns, including HTTP status codes, response headers, and any embedded error messages.

curl -v https://your-service.example.com/health

Typical output when a service is unhealthy includes 503 Service Unavailable, 502 Bad Gateway, or a JSON body like {"status":"degraded","reason":"timeout"}. If the connection is refused entirely, the issue is likely not the application but the network or the host.

Reading Recent Logs

Logs are the primary source of truth for most runtime failures. The method for retrieving them depends on your deployment model.

For containerized workloads, use docker or podman:

docker logs -f <container-id>

The -f flag tails the output, which is essential for observing intermittent errors in real time. Common patterns you will see include repeated stack traces, connection pool exhaustion messages, or graceful shutdown signals.

For Kubernetes clusters, kubectl is the standard interface:

kubectl logs -n production deployment/api --tail=100

You can filter by container within a pod if multiple containers share the pod:

kubectl logs -n production pod/api-abc123 -c api-container

If the pod is restarting repeatedly, kubectl get pod will show RestartCount climbing, and kubectl describe pod will include events such as BackOff pulling image or CrashLoopBackOff.

For bare-metal or VM deployments, system logs are the equivalent:

journalctl -u your-service --no-pager -n 50

or, on systems using rsyslog or syslog-ng:

sudo tail -f /var/log/your-service.log

Verifying Node and Process State

Sometimes the application process is running but in a bad state. Use standard introspection tools:

ps aux | grep your-service

Look for zombie processes, processes stuck in D (uninterruptible sleep) state, or multiple instances competing for the same port. top or htop will show CPU and memory pressure; a process consuming 100% CPU without making progress often indicates a deadlock or infinite loop.

Isolating the Failure Domain

Once you have a general sense of where the problem is, the next goal is to reduce the surface area. This is where many debug cycles stall—engineers chase errors that are actually symptoms of a deeper issue elsewhere in the stack.

Database Connectivity

Application failures frequently stem from database issues. If your service reports could not connect to server or connection timeout, verify the database is reachable and accepting connections.

psql -h db.example.com -U app -d production -c "SELECT 1;"

If that succeeds, the issue is client-side: check connection pool settings, TLS certificates, or firewall rules. If the psql command fails with FATAL: database system is shut down or FATAL: password authentication failed for user "app", the database itself may be down, misconfigured, or experiencing replication lag.

Network Path Verification

Use curl with timing and resolution flags to test the full path:

curl -w "\nTime spent: %{time_total}s\n" https://api.example.com/resource

If the request hangs, use traceroute or mtr to see where packets are being dropped:

mtr -r -c 5 api.example.com

DNS resolution problems manifest as Could not resolve host errors. Verify the DNS entry is correct and that the service is listening on the expected port inside its host.

Dependency Health

Modern applications rely on caches, message queues, and external APIs. Isolate each dependency:

Drilling Into Runtime Details

When the high-level signals don’t reveal the root cause, you need to inspect what the running system is actually doing. This section covers the concrete commands and expected output formats you will encounter.

Examining Open File Descriptors and Sockets

A common silent failure mode is running out of file descriptors, causing the server to reject new connections silently.

ulimit -n

If the output is 1024 and your load is higher than that, you have found the limit. Increase it temporarily for debugging:

ulimit -n 4096

Also check which processes have many sockets open:

ss -tlnp | head -20

or

netstat -tlnp

Look for TIME_WAIT sockets accumulating; this often indicates that connections are not being closed properly, either by the application or by a load balancer upstream.

Inspecting HTTP Requests and Responses

When an API endpoint returns unexpected data, you need to see the full request and response cycle. curl with -trace-ascii writes everything to a file:

curl -trace-ascii /tmp/curl-debug.txt https://api.example.com/v1/resource

The resulting file contains the exact HTTP headers and body, which is invaluable when debugging signature verification failures, encoding issues, or middleware injection.

For Kubernetes ingress or load balancer troubleshooting, kubectl get ingress and kubectl describe ingress will show which backend pods are receiving traffic and whether they are marked as ready.

Database Query Inspection

If slow queries are suspected, capture the actual SQL being sent. In PostgreSQL, enable logging of slow statements:

SET log_min_duration_statement = 100;  -- log queries taking >100ms

Then run the operation causing slowness and check the resulting log file. You will see the exact SELECT, UPDATE, or DELETE statement, along with execution time and row counts.

In MySQL, enable the slow query log in my.cnf:

[mysqld]
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1

After reproducing the issue, examine /var/log/mysql/slow.log for the offending queries.

Thread and Coroutine Analysis

For languages using async runtimes (Node.js, Python asyncio, Go), inspect the internal state of the event loop or goroutine scheduler.

Node.js:

node --inspect-brk=9229 server.js

Then open chrome://inspect in Chrome to view active promises, pending timers, and heap snapshots.

Go:

go tool pprof -http=localhost:6060 http://localhost:8080/debug/pprof/profile?seconds=30

This launches a web UI showing CPU profiles, goroutine stacks, and block traces. A common pattern is finding goroutines blocked on channel receives or mutexes, which points to a deadlock or contention issue.

Python (asyncio):

python -c "import asyncio; import sys; print(asyncio.all_tasks())"

or use sys.getcheckinterval() and sys.gettotalrefcount() for debugging reference cycles.

Reproducing and Verifying Fixes

Once you have identified a likely cause, the next phase is to reproduce the issue in a controlled manner and verify that your fix resolves it without side effects.

Git Bisect for Configuration

If a recent deployment coincides with the failure, use git bisect to pinpoint the exact commit that introduced the regression:

git bisect start
git bisect bad      # current state is broken
git bisect good     # known good commit hash

Git will checkout intermediate commits automatically. Test each one with your production smoke test script until the bisect identifies the culprit.

Feature Flag Toggling

Many platforms support runtime feature flags. If your system uses a flagging service (e.g., LaunchDarkly, Unleash, or a custom database table), you can flip a flag off without a deploy:

curl -X POST https://flags.example.com/api/toggle \
  -H "Authorization: Bearer $API_TOKEN" \
  -d '{"key":"new-endpoint-flag","enabled":false}'

Verify the service behavior changes immediately. This is safer than a full rollback and allows you to confirm the flag is the source of the anomaly.

Canary and Rollback

If the issue is a code change, a canary deployment lets you route a small percentage of traffic to the new version while monitoring:

kubectl set env deployment/api NEW_VERSION=v2.3.1 -n production
kubectl rollout status deployment/api -n production --timeout=300s

If the canary shows increased error rates, roll back immediately:

kubectl rollout undo deployment/api -n production

Database Migration Verification

When a schema change is suspected, always check migration logs. Most migration tools (Flyway, Alembic, goose) leave a schema history table:

SELECT * FROM schema_migrations;

If a migration is listed as pending or failed, that is your immediate target. Apply it manually in a staging environment first, or use the tool’s repair command:

flyway repair
flyway migrate

Sharp Edges and Lessons Learned

Silent Failures That Catch You Off Guard

Confused Options and Common Mistakes

What You Only Learn After It Breaks in Production

This guide provides the concrete tools and commands you need to start debugging production issues immediately. The sharp edges listed here are the specific gotchas that consume the most debug time; bookmark them, and the next time the pager goes off, you will spend your time resolving the actual failure rather than chasing ghosts.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.