DNS for Engineers Who Keep Breaking Things: From Resolution to Debugging

DNS for Engineers Who Keep Breaking Things: From Resolution to Debugging is a practical guide to understanding, diagnosing, and fixing DNS issues in production. It targets engineers who have already deployed services, only to watch them fail silently—requests timing out, clients connecting to wrong environments, or configuration changes taking hours to propagate. This page covers the full lifecycle from query to response, with tools, commands, and patterns that work across platforms, not tied to any single vendor’s implementation.

Understanding DNS Resolution: The Full Path of a Query

Every DNS query begins with a client sending a DNS Question packet to a resolver. The query contains a QNAME, QTYPE, and QCLASS. The most common QTYPE is A (IPv4 address), AAAA (IPv6), and CNAME (alias). The QCLASS is almost always IN (Internet).

The client typically starts with a local resolver (often systemd-resolved, dnsmasq, or stub resolver in the OS). The resolver may have cached results, but ultimately forwards the query to one or more upstream resolvers.

The Recursive Query Chain

When a resolver receives a query, it performs a recursive query:

1. It sends a DNS Query with the RD (Recursion Desired) bit set.
2. The upstream server may answer immediately (if it has the answer) or recursively resolve the name.
3. If the upstream server doesn’t have the answer, it traverses the DNS hierarchy:

• Start at the root servers (a.root-servers.net through m.root-servers.net).
• Query the root for the TLD (e.g., com).
• Query the TLD nameserver for the authoritative server for example.com.
• Query the authoritative server for www.example.com.

The final response includes the answer, additional records (like A records for nameservers), and the AD (Additional Data) section.

Example of a dig command showing full resolution:

dig +trace www.example.com

This outputs:

; <<>> DiG 9.18.12 <<>> +trace www.example.com
;; global options: +cmd
.                       518400  IN      NS      a.root-servers.net.
[...]
com.                    172800  IN      NS      a.gtld-servers.net.
[...]
example.com.            172800  IN      NS      ns1.example.com.
example.com.            172800  IN      NS      ns2.example.com.
[...]
www.example.com.        300     IN      A       93.184.216.34

The +trace flag shows the full chain, from root to authoritative server.

Debugging DNS in Production: Tools and Patterns

When a service fails to resolve a hostname, engineers reach for dig, nslookup, host, and dig with flags. But these tools are underused, and their outputs are often misinterpreted.

The Power of dig with Real Flags

dig @8.8.8.8 -t A www.example.com +short +time=1 +retry=3

• @8.8.8.8: query Google’s public DNS.
• -t A: request only A records.
• +short: concise output.
• +time=1: set timeout to 1 second.
• +retry=3: retry up to 3 times on timeout.

This command returns:

93.184.216.34

A common mistake: using dig without +time and +retry, leading to silent timeouts in high-latency environments.

Common Failure Modes and Their Signatures

1. Timeouts and Retries

When dig times out, the response lacks AD and RA flags. Check dig output:

dig @1.1.1.1 www.example.com +noall +answer +stats

Output includes:

;; Query time: 1000 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Apr 10 12:00:00 UTC 2023
;; MSG SIZE  rcvd: 52

A query time of 1000ms with a small MSG SIZE indicates a timeout or slow resolver. If Query time is near timeout, the issue is likely network latency or a misconfigured resolver.

2. CNAME Chains That Break

A CNAME chain like:

www.example.com → alias.example.com → example.com → A record

can break when one link is missing or has incorrect TTL.

Use dig with +all to see the full response:

dig +all www.example.com

Look for:

• CNAME records in the ANSWER section.
• A records for the canonical name in AUTHORITY and ADDITIONAL.
• TTL values indicating propagation delays.

A common error: a CNAME has TTL 300, but the target A record has TTL 60. When the A record changes, clients see the old A record for 300 seconds, but the CNAME is updated quickly—leading to inconsistencies.

3. Missing or Misconfigured NS Records

When a domain fails to resolve, check that its nameservers are properly configured:

dig NS example.com +short

Expected output:

ns1.example.com.
ns2.example.com.

But if the nameservers are ns1.example.net, ns2.example.net, and example.com is not delegating to them, the query fails.

Use dig to validate delegation:

dig example.com NS +additional

This ensures the NS records are included in the ADDITIONAL section, and the A records for the nameservers are present.

Configuring DNS for Reliability: From TTL to Caching

TTL (Time-To-Live) is not just a number—it’s a strategy for propagation, failover, and consistency.

Setting TTL for Different Use Cases

• Fast failover: use TTLs of 60 seconds or less. A change to an A record propagates within a minute.
• Stable infrastructure: 3600 seconds (1 hour) is common for static services.
• High-traffic services: TTL of 300 seconds (5 minutes) balances freshness and load.

A common mistake: setting TTL to 3600 on a load balancer’s A record, but the LB’s health check is every 30 seconds. When a server fails, clients keep using the old IP for 60 minutes.

Resolver Caching and Its Pitfalls

The resolver caches answers based on TTL, but it may not respect the TTL field from the DNS response.

Check resolver cache with:

systemd-resolve --status

Look for:

• Current DNS Server: 192.168.1.1
• DNS Servers: 192.168.1.1, 8.8.8.8
• Cache: yes
• DNSSEC: yes

To force a resolver to flush cache:

systemd-resolve --flush-caches

To check cache contents:

systemd-resolve --query=www.example.com

This returns:

www.example.com: 93.184.216.34 (ttl=300s, from cache)

If from cache is yes, the resolver is using the cached answer. If ttl is less than expected, the resolver is not reading the full TTL from the response.

The Role of DNSSEC and Its Common Failures

DNSSEC adds cryptographic signatures to DNS records. The resolver validates the chain from the root to the answer.

To check DNSSEC status:

dig +dnssec www.example.com

Look for:

• RRSIG records for the answer.
• DNSKEY records in the ADDITIONAL section.
• RRSIG for the DNSKEY records.

A common failure: DNSSEC validation failed, but the resolver lacks the DNSKEY for the zone.

Use dig to validate the chain:

dig +dnssec www.example.com +trace

This shows the entire chain: from root to the answer, with RRSIGs and DNSKEYs at every level.

A silent failure: the resolver validates DNSSEC but returns AD (Authenticated Data) flag, yet the client is not using it. The AD flag is set only if the entire chain is validated.

Monitoring and Alerting on DNS

Health Checks with curl and dig

Use curl to validate DNS-based service health:

curl -s -f -H "Host: www.example.com" http://192.168.1.10:8080/

This checks that:

• The A record is correct.
• The server responds to the correct host header.
• The service is up.

Combine with dig:

#!/bin/bash
set -e

DOMAIN="www.example.com"
IP=$(dig +short $DOMAIN | head -1)

if ! ping -c 1 -W 2 $IP >/dev/null; then
    echo "DNS resolution failed: $DOMAIN -> $IP"
    exit 1
fi

if ! curl -s -f -H "Host: $DOMAIN" http://$IP >/dev/null; then
    echo "HTTP service failed for $DOMAIN"
    exit 1
fi

This script runs as a health check in CI or as a systemd service.

Using dnsmasq for Local Caching and Debugging

Deploy dnsmasq as a local resolver:

listen-address=127.0.0.1
dhcp-range=192.168.1.100,192.168.1.200,12h
server=8.8.8.8
server=1.1.1.1
log-queries
log-dhcp

Enable logging:

systemctl enable dnsmasq
systemctl start dnsmasq
journalctl -u dnsmasq -f

Now, every DNS query is logged, showing:

• Client IP.
• Query type.
• Response time.
• Whether the answer was cached.

Use dnsmasq to simulate slow or failing upstream resolvers:

dnsmasq --server=192.168.1.100 --log-queries --log-dhcp

This allows testing DNS changes in a controlled environment.

When DNS Breaks: A Checklist for Engineers

When a service fails to resolve a hostname, follow this checklist:

1. Verify the domain name:

   dig +short www.example.com

Is the answer correct?

1. Check the TTL:

   dig +short www.example.com

Is the TTL too high? Did it change recently?

1. Validate delegation:

   dig NS example.com +additional

Are the nameservers correct?

1. Check DNSSEC:

   dig +dnssec www.example.com

Is AD set? Are RRSIG and DNSKEY present?

1. Test from multiple locations:

   dig @8.8.8.8 www.example.com
   dig @1.1.1.1 www.example.com

Does the answer differ?

1. Inspect resolver cache:

   systemd-resolve --query=www.example.com
   systemd-resolve --flush-caches

1. Check for CNAME chains:

   dig +all www.example.com

Are all links in the chain correct?

1. Validate TTL propagation:

Run dig every 5 minutes. Watch TTLs update.

1. Monitor DNS resolution time:

Use dig +time=1 +retry=3 to detect latency.

1. Add health checks:

Use curl and dig together to verify both DNS and service availability.

This checklist becomes the foundation of DNS reliability—reusable, repeatable, and effective across teams and environments.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.