Service Discovery Patterns for Distributed Systems

How services find and communicate with each other in dynamic environments — covering DNS-based, registry-based, and mesh-based service discovery patterns.

Service Discovery Patterns for Distributed Systems

Service discovery is the set of patterns that let components of a distributed system find and communicate with one another without hard‑coding addresses. It matters whenever workloads are ephemeral, scaled horizontally, or distributed across hosts, networks, or data centres—scenarios where addresses change, multiply, or vanish between requests. This reference page organizes the problem by what you’re trying to accomplish: locating a service by name, routing only to healthy instances, spanning boundaries, and debugging when things go quiet.

Locating a Service by Name Within a Single Cluster

In many deployments, the orchestration layer exposes a stable DNS domain that is set at bootstrap and remains static for the life of the cluster. A service registered under an identifier is addressable as <identifier>.<namespace>.<domain>. Clients can resolve that name without needing to know the underlying pod IPs or their churn.

To inspect what the cluster has registered, the kubectl CLI provides a direct view:

kubectl get services --namespace=production

The output lists each service’s name, cluster IP (if virtual‑IP backed), and age. If the service is headless, the clusterIP field shows None.

To see the concrete endpoints that back a given service, query the endpoint resource:

kubectl get endpoints production-api --output=json

The returned JSON includes a subsets array. Each entry contains an addresses field listing the pod IPs currently accepted for the service, and a conditions field indicating whether the subset is Ready or NotReady.

From any process that can reach the cluster’s DNS resolver, the service is reachable via its composed name. For example, probing a health endpoint with curl:

curl -sS http://production-api.production.<cluster-domain>/health

Replace <cluster-domain> with the domain suffix your deployment uses (commonly cluster.local, my.cluster.local, or a custom zone). If the service is headless, the DNS response returns the set of pod IPs, and the client cycles through them. If it is virtual‑IP backed, the resolver returns a single stable address that the cluster’s proxy load‑balances across all ready endpoints.

If the name is not registered, kubectl get endpoints returns Error: endpoints "production-api" not found and curl produces curl: (6) Could not resolve host. These error patterns point immediately to registration vs. resolution failures.

Routing Traffic to Healthy Instances

When a client must receive traffic only from fit instances, the system relies on health probes and endpoint management. The orchestration layer runs liveness and readiness probes defined per pod; an endpoint is marked Ready only after the readiness probe returns a success code.

To check a pod’s current readiness status:

kubectl describe pod web-7c9b8d -n production | grep -A5 Readiness

Alternatively, the endpoint resource itself carries the information:

kubectl get endpoints production-api -o yaml

The YAML output lists conditions under each subset. A Ready condition with status True means the subset’s probes have succeeded; False or absent means the subset is excluded from routing.

Clients that honor these marks—such as ingress controllers, service proxies, or sidecar meshes—automatically remove NotReady endpoints from the load‑balance pool. If a client contacts a pod IP directly, it must implement its own probe. A quick check can be issued with curl:

curl -sS -o /dev/null -w "%{http_code}" http://10.244.1.42:8080/ready

A returned HTTP code of 200 indicates the pod is accepting requests; 000 or any non‑2xx code signals unavailability.

From the infrastructure side, an ingress controller or load balancer may return 502 Bad Gateway when all backend endpoints are marked NotReady, and downstream logs will include entries such as health check failed or upstream prematurely closed connection.

Discovering Services Across Boundaries

When services span namespaces, clusters, or data centres, a single name must resolve to the correct location without hard‑coding addresses. Three common patterns emerge: a shared DNS zone, a central service registry, and environment‑variable injection at container start.

Within a single cluster, listing all registered services is straightforward:

kubectl get services --all-namespaces

Filtering for a specific service across namespaces can be done with a pipe:

kubectl get services --all-namespaces | grep production-api

For cross‑cluster discovery, many teams operate a replicated registry that stores a mapping of the form <service-name>: <host>:<port>. The registry may be backed by a key‑value store, a DNS zone, or a configuration management system. If the registry exposes an HTTP API, a reader can fetch the current mapping as:

curl -sS http://registry-prod:8080/v1/services/production-api

The response typically contains host, port, and optional metadata fields such as weight or region. The reader’s client then uses those values to construct the connection address.

In containers that are started with service‑discovery variables injected by the runtime, the process environment includes SERVICE_HOST and SERVICE_PORT (often accompanied by SERVICE_NAME and SERVICE_NAMESPACE). These can be read directly in code:

The runtime typically composes the SERVICE_HOST value from the resolved DNS name or registry entry at pod start, so the internal process sees a stable address even as underlying pod IPs change.

Debugging When Discovery Breaks

When a service cannot be located or reached, the reader should systematically check resolution, registry state, and connectivity. The following steps cover the most common failure points.

First, verify that the DNS name resolves from within a pod. If the cluster provides a debugging pod with standard tools:

kubectl exec -ti <dns-pod> -- nslookup production-api.production.<cluster-domain>

If nslookup is not available, use the GNU getent wrapper:

kubectl exec -ti <dns-pod> -- getent hosts production-api.production.<cluster-domain>

A response with an IP address confirms that the resolver can locate the name. An NXDOMAIN reply or a timeout indicates a registration or forwarder issue.

Second, inspect the registry’s view of the service from the control plane:

kubectl get services production-api --namespace=production

If the output shows clusterIP: None and an empty subsets list, the service may be headless or simply unregistered. If a clusterIP is present but subsets are empty, the endpoints have not yet been populated—often a timing issue during a rolling update.

Third, test direct connectivity to a known pod IP, bypassing the name layer:

curl -v http://10.244.1.37:8080/health

curl output will include the HTTP status code and, for failures, a connection error such as Connection refused or HTTP/1.1 503. A successful response confirms that the pod is running and listening, narrowing the problem to the discovery layer.

Common error patterns and what they signal:

Each error points to a different layer: endpoint registration, health‑probe state, or DNS configuration.

Sharp Edges: What Silently Breaks

Certain behaviours catch operators off guard because they do not generate immediate errors but manifest as intermittent or silent failures.

DNS negative caching. A resolver may cache an NXDOMAIN response past the configured TTL. If a service is created after a failed lookup, the client continues to receive “name not found” until the cache entry expires. This is especially prevalent in environments where the DNS cache is shared across pods or nodes.

Endpoint staleness. After a rolling update, the registry may retain old pod IPs for a window that depends on the controller’s sync interval and the service’s TTL. During that window, some clients route to terminated pods, producing 502 or connection reset errors for the last few requests in a batch.

Environment variable explosion. When many services inject SERVICE_HOST and SERVICE_PORT, the container’s environment list grows. Some runtimes (particularly minimal containers or embedded interpreters) enforce a limit on environment size or ignore variables beyond a certain index. The result is a service name that resolves in one container but not another, with no error logged at the discovery layer.

Headless vs virtual‑IP confusion. A headless service (clusterIP: None) returns all pod IPs in the DNS response, placing the load‑balancing responsibility on the client. A virtual‑IP service (clusterIP set to a cluster‑internal address) routes through the cluster’s proxy, which may apply session affinity, rate‑limiting, or external‑IP translation. Operators who treat clusterIP: None as “disable DNS” are surprised when the name still resolves—to multiple IPs, not to nothing.

Load‑balancer provisioning lag. In cloud environments, an external loadBalancer IP is provisioned asynchronously. During the minutes‑long provisioning window, curl to the service name may time out even though the underlying endpoints are healthy, because the load‑balancer has not yet begun forwarding traffic.

Production breakage lessons. After a service‑mesh rollout, several teams observed that a rolling update faster than the endpoint controller’s sync cycle caused some requests to hit terminated pods, because the client’s DNS cache or the proxy’s endpoint list was stale. In a multi‑cluster setup, replication lag in the shared registry caused service names to resolve to addresses in the wrong region, producing latency spikes and occasional timeouts with no error code flagging the mismatch. Firewall rules that allow inbound traffic to a service name but block the proxy’s dynamic port range produced intermittent failures: curl succeeded from some pods and failed from others, appearing as flaky connectivity rather than a configuration gap.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.