Service Discovery Patterns | The Garnet Wiki

Service discovery enables services to dynamically locate and communicate with each other in a distributed system. It matters when services scale beyond a single host, when they deploy independently, and when network topology changes frequently—common in cloud-native, microservices architectures. The core challenge is: how does a service know where to send its requests when other services come and go?

Service Registration and Discovery

A service registers itself with a discovery system upon startup, publishing its network address, health status, and metadata. Subscribers—clients or other services—query the discovery system to resolve a service name to one or more network endpoints.

The most common registration pattern uses a gossip-based heartbeat. A service sends periodic POST /register requests to a discovery server. Each request includes the service’s name, IP address, port, and optional metadata.

POST /register HTTP/1.1
Host: discovery.example.com
Content-Type: application/json

{
  "service": "user-service",
  "address": "10.0.4.21",
  "port": 8080,
  "health": "healthy",
  "labels": {
    "region": "us-west-2",
    "env": "production"
  }
}

The discovery server stores this information in a registry and responds with a 201 Created. The service continues sending heartbeats every 10 seconds to keep its entry alive. A heartbeat is a simple PATCH /heartbeat with the same payload.

PATCH /heartbeat HTTP/1.1
Host: discovery.example.com
Content-Type: application/json

{
  "service": "user-service",
  "address": "10.0.4.21",
  "port": 8080
}

The discovery server may fail to process a heartbeat. If a service misses three consecutive heartbeats (30 seconds), the server marks it as unhealthy and removes it from the active pool after another 15 seconds.

Clients query the discovery system using GET /services/{name}. The response is a list of active endpoints:

GET /services/user-service HTTP/1.1
Host: discovery.example.com

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "address": "10.0.4.21",
    "port": 8080,
    "health": "healthy",
    "labels": {
      "region": "us-west-2",
      "env": "production"
    }
  },
  {
    "address": "10.0.4.22",
    "port": 8080,
    "health": "healthy",
    "labels": {
      "region": "us-west-2",
      "env": "staging"
    }
  }
]

Clients may cache this response and refresh it periodically. A common pattern is to use a refresh interval and cache invalidation via ETag or Last-Modified headers. If the cache expires, the client fetches the updated list and updates its local service map.

Load Balancing Strategies

Once a service discovers multiple instances, it must decide how to distribute requests. The default is round-robin, but other strategies provide better performance under varying workloads.

Round-robin is implemented by rotating through the list of endpoints in order. A client maintains a pointer to the current instance and advances it on each request.

Weighted round-robin allows instances to have different weights. A service with higher capacity is assigned more requests.

[
  {
    "address": "10.0.4.21",
    "port": 8080,
    "weight": 3,
    "health": "healthy"
  },
  {
    "address": "10.0.4.22",
    "port": 8080,
    "weight": 1,
    "health": "healthy"
  }
]

The client assigns 3 slots to the first instance and 1 to the second. On request, it selects an instance by picking a random number from 1 to 4. If the number is 1 or 2 or 3, it picks the first; if 4, the second. This distributes traffic in a 3:1 ratio.

Least connections keeps track of the number of active requests per instance. The client selects the instance with the fewest concurrent requests.

Random with weighted selection picks an instance based on its weight. The client generates a random number between 0 and the sum of all weights. It then iterates through the list, accumulating weights until it exceeds the random number.

A common failure mode: stale endpoint lists. If a service scales up from 2 to 4 instances but the discovery server does not propagate updates quickly, clients may continue using only the original 2 endpoints. This leads to overloading some instances while others sit idle.

To mitigate this, clients should refresh their endpoint list at least every 30 seconds. They may also implement dynamic load balancing, where each request includes the current instance's load, and the discovery server adjusts weights based on historical load.

Service Health Monitoring

Health monitoring ensures that only healthy instances receive traffic. The discovery server tracks health via heartbeats and can automatically mark instances as unhealthy when heartbeats fail.

Clients may perform active health checks using a GET /health endpoint on each instance. The discovery server polls each endpoint every 5 seconds and updates the health status.

GET /health HTTP/1.1
Host: user-service.example.com

A healthy service returns 200 OK with a JSON body:

{
  "status": "healthy",
  "uptime": 123456,
  "memory": {
    "used": 150000000,
    "total": 250000000
  },
  "db": {
    "connected": true,
    "latency_ms": 12
  }
}

If the response code is not 200, or if the status field is "unhealthy", or if the db.connected is false, the service is marked as unhealthy.

The discovery server may use dual health checks: a liveness probe (is the service running?) and a readiness probe (is the service ready to serve requests?).

A liveness probe returns 200 OK if the service is alive and responding to health checks. A readiness probe returns 200 OK only if the service is fully initialized—e.g., loaded configuration, connected to databases, warmed up.

When a service is marked unhealthy, the discovery server removes it from the available pool. Clients that had cached the list may keep using it for a grace period—typically 10 to 30 seconds—before falling back to the discovery system.

Service Discovery with Metadata Filtering

Services often have multiple instances deployed across regions, environments, or versions. Clients need to filter the list of instances based on metadata.

Metadata is a set of key-value pairs attached to a service instance. A client can filter instances by one or more labels.

A request to /services/user-service may include query parameters:

GET /services/user-service?region=us-west-2&env=production HTTP/1.1
Host: discovery.example.com

The discovery server returns only instances matching all criteria.

Clients may also use tag-based filtering. Each service instance can be tagged with one or more labels, and clients can request instances with specific tags.

GET /services/user-service?tag=primary HTTP/1.1
Host: discovery.example.com

The response includes only instances tagged as primary.

Metadata filtering fails silently when a service instance has a label that does not exist in the query. For example, a service instance has "region": "us-west-2" but the client filters for "region": "us-east-1". The instance is not returned, even though it is healthy and available.

Clients must also handle mismatched label values. A service instance may have "env": "prod" but the client filters for "env": "production". The discovery server must match exact strings, so no case-insensitive or fuzzy matching is assumed unless explicitly specified.

Service Discovery with DNS

In environments where DNS is the primary service registry, service discovery is implemented using DNS-based discovery.

Each service is assigned a DNS name, such as user-service.example.com. The DNS server returns multiple A records for this name.

user-service.example.com. IN A 10.0.4.21
user-service.example.com. IN A 10.0.4.22
user-service.example.com. IN A 10.0.4.23

Clients resolve the name using a standard DNS lookup (dig user-service.example.com or nslookup user-service.example.com). The DNS server may use round-robin DNS, where each A record is rotated on each query.

To support health, the discovery system updates the DNS records dynamically. A service instance that becomes unhealthy is removed from the DNS zone. The DNS server may use TTL (Time-To-Live) to control how long clients cache the record. A TTL of 30 seconds means clients use the same set of IP addresses for 30 seconds before refreshing.

A failure mode: DNS caching in the OS. A client may cache DNS records for several minutes, even if the TTL is set to 30 seconds. This causes delays in propagating changes—e.g., a new instance appears, but clients take 2–5 minutes to start using it.

To address this, the discovery system may use DNS with SRV records. An SRV record specifies the service name, priority, weight, port, and target host.

_user-service._tcp.example.com. IN SRV 10 50 8080 user-service-1.example.com.
_user-service._tcp.example.com. IN SRV 10 50 8080 user-service-2.example.com.
_user-service._tcp.example.com. IN SRV 20 30 8080 user-service-3.example.com.

The client queries SRV _user-service._tcp.example.com and receives a list of service instances with their priority and weight. The client sorts by priority (lower is better), then applies weighted round-robin within each priority group.

Service Discovery in Practice: A Complete Example

A user service starts on a Kubernetes pod with IP 10.0.4.21 and port 8080. It registers itself:

POST /register HTTP/1.1
Host: discovery.example.com
Content-Type: application/json

{
  "service": "user-service",
  "address": "10.0.4.21",
  "port": 8080,
  "health": "healthy",
  "labels": {
    "region": "us-west-2",
    "env": "production",
    "version": "v1.2.3"
  }
}

It then sends heartbeats every 10 seconds:

PATCH /heartbeat HTTP/1.1
Host: discovery.example.com
Content-Type: application/json

{
  "service": "user-service",
  "address": "10.0.4.21",
  "port": 8080
}

A frontend service, running in another pod, needs to call the user service. It queries the discovery system:

GET /services/user-service?region=us-west-2&env=production HTTP/1.1
Host: discovery.example.com

The discovery server returns:

[
  {
    "address": "10.0.4.21",
    "port": 8080,
    "health": "healthy",
    "labels": {
      "region": "us-west-2",
      "env": "production",
      "version": "v1.2.3"
    }
  },
  {
    "address": "10.0.4.22",
    "port": 8080,
    "health": "healthy",
    "labels": {
      "region": "us-west-2",
      "env": "production",
      "version": "v1.2.3"
    }
  }
]

The frontend selects the first instance via round-robin and sends a request:

GET /users/123 HTTP/1.1
Host: 10.0.4.21:8080

After 20 seconds, the discovery server receives a heartbeat from 10.0.4.22 but misses the next two. It marks the instance as unhealthy. The frontend, polling the discovery system every 30 seconds, discovers this change on its next refresh.

The frontend now receives a new list:

[
  {
    "address": "10.0.4.21",
    "port": 8080,
    "health": "healthy",
    "labels": {
      "region": "us-west-2",
      "env": "production",
      "version": "v1.2.3"
    }
  },
  {
    "address": "10.0.4.22",
    "port": 8080,
    "health": "unhealthy",
    "labels": {
      "region": "us-west-2",
      "env": "production",
      "version": "v1.2.3"
    }
  }
]

The frontend stops sending requests to 10.0.4.22 and begins distributing traffic primarily to 10.0.4.21.

Common Pitfalls and Sharp Edges

Heartbeat timing: If a service starts, registers, and sends its first heartbeat 3 seconds after startup, but the discovery server expects a heartbeat every 10 seconds, the service is considered healthy only after 10 seconds, introducing a 7-second delay in availability.
TTL in DNS: A client may cache DNS records for longer than the TTL, especially if the OS or network stack uses a default TTL of 60 seconds.
Metadata mismatch: A service instance with "env": "prod" is not returned when filtering for "env": "production"—no automatic mapping or case-insensitive matching.
Load balancing with stale data: A service instance becomes unhealthy but the discovery server does not update the list until the next heartbeat, causing a window of unavailability.
Multiple discovery systems: Clients may use both a central discovery server and DNS. If the two systems diverge—e.g., DNS includes a new instance but the central registry does not—the client must reconcile both sources, leading to inconsistent routing.

Service discovery is not just about resolving names. It is a continuous process of registration, health checking, filtering, and load balancing—each step with subtle failure modes that silently degrade system performance.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.