Design systems that stay up. Covers active-active, active-passive, failover strategies, health checks, circuit breakers, bulkheads, and achieving 99.99% uptime.
High availability architecture exists to keep services functional when individual components fail. It matters whenever a system must continue operating despite hardware faults, software crashes, network partitions, or human error. If your SLA requires more than a few minutes of downtime per year, if users span time zones, or if data loss is unacceptable, HA patterns become part of the design mandate rather than an afterthought. This page catalogs the recurring mechanisms that deliver resilience, organized by the problems they solve rather than by product taxonomy.
Before selecting patterns, clarify the failure budget. Availability is usually expressed as a percentage (e.g., 99.9% monthly uptime), but percentages obscure the real question: which components, if they disappear, must not take the whole service down? Start by mapping every dependency that, if absent, causes an immediate outage. Typical categories include:
A useful exercise is to list every process, service, or network segment that currently represents a single point of failure, then rank them by likelihood of failure and impact on users. This inventory becomes the backlog for HA work.
Once the gap is identified, the next step is to decide which patterns address it. The following sections are ordered by the reader’s likely objective: first isolate where failures can occur, then make those failures detectable, then automate recovery, and finally keep state consistent across recoveries.
The foundational HA technique is fault domain separation. The goal is to ensure that a failure in one domain cannot cascade to others. In practice this means physical or logical separation across machines, availability zones, or containers.
Compute isolation If you are running containers, avoid scheduling two replicas of the same critical service on the same host. Docker’s --distribution flag or a scheduler’s placement constraints can enforce this. For example, a Docker Compose file that spreads replicas across different Docker daemon endpoints, or a Kubernetes pod anti-affinity rule that prevents two pods of the same deployment from co-locating:
kubectl set env deployment/my-service AFFINITY_POLICY=requiredIf you are managing virtual machines, use separate hypervisors, distinct power circuits, or different cloud availability zones. The pattern is the same: replicate the service, then constrain where those replicas may run.
Network isolation Separate traffic paths so that a cable pull, VLAN misconfiguration, or DNS outage in one zone does not starve others. This often means multi-homed instances, separate load balancer front-ends, or DNS round-robin with short TTLs. A common pattern is to assign each replica a distinct virtual IP or DNS entry, and route client queries through a resolver that can fail over automatically.
Storage isolation Local disks fail. If your service depends on a single filesystem, replication or distributed storage is required. The pattern is to attach storage that can be mounted by multiple nodes simultaneously, or to replicate state to a remote store that survives node loss.
A system cannot fail over if it does not know a component is broken. Health checking is the mechanism that bridges detection and action. The pattern is lightweight, idempotent probes that answer “am I alive and able to serve requests?” without side effects.
HTTP health endpoints Most services expose a /healthz or /status endpoint that returns 200 OK when the process is running and 503 Service Unavailable when it is not. A external watchdog or orchestrator polls this endpoint at a configurable interval. For example, using curl from a monitoring host:
while true; do
if ! curl -fsS http://localhost:8080/healthz > /dev/null; then
echo "Health check failed at $(date)" >> /var/log/healthcheck.log
systemctl restart my-service
fi
sleep 15s
doneThe -fsS flags tell curl to fail silently on HTTP errors, show no progress meter, and use a secure connection if the endpoint is over HTTPS. The script above is a common pattern; the interval, threshold, and restart command are tuned to the service’s startup time and SLA.
Docker HEALTHCHECK When running a container, the HEALTHCHECK instruction in a Dockerfile defines a probe that the Docker daemon tracks. A real example:
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -fsS http://localhost:8080/healthz || exit 1Docker marks the container as unhealthy after the specified number of consecutive failures, and orchestrators (Docker Swarm, Kubernetes) can react by restarting or rescheduling the container.
Readiness vs liveness Two distinct probe concepts are commonly confused. A liveness probe determines whether the process should be restarted; a readiness probe determines whether it should receive traffic. A service may be live (running) but not ready (still loading data, warming caches). Using both probes prevents traffic from being sent to a process that cannot yet fulfill requests, which is a frequent source of cascading failures during deployments.
Once health detection is in place, the next objective is to shift traffic away from a failed component without manual intervention. The pattern depends on the layer at which failure occurs.
DNS-based failover If your service is addressed by a hostname, short TTL records (60–300 seconds) allow a DNS resolver to stop returning the IP of a failed host. Clients that cache the record will eventually re-resolve and obtain a healthy endpoint. This pattern is simple but has a convergence window equal to the TTL plus client cache time. It works well for globally distributed services where a small subset of users experiencing the old IP is acceptable.
Load balancer failover At the traffic distribution layer, a load balancer with health‑checked backends can remove a failed node from its pool and route new connections to remaining nodes. Many load balancers also support graceful connection draining, where existing connections are allowed to complete before the node is taken offline. A typical configuration pattern looks like:
backend http-servers
mode http
balance roundrobin
option httpchk GET /healthz
server s1 10.0.0.1:80 check
server s2 10.0.0.2:80 checkIf s1 stops responding to /healthz, the load balancer stops sending new requests to it. Existing connections drain according to the drain timeout setting, and once s1 is marked down, all traffic flows to s2.
Stateful service failover For services that maintain session or configuration state, failover often requires a standby node that can take over instantly. The pattern is a primary‑standby pair with synchronized state, plus a fencing mechanism that prevents both nodes from writing simultaneously. In practice this looks like a primary database accepting writes, a replica that mirrors WAL or transaction logs, and a watchdog that, upon detecting primary loss, promotes the replica and updates connection strings for downstream clients. The exact implementation varies by database engine, but the logical pattern is: detect primary loss → promote replica → update routing → resume traffic.
Redundancy is meaningless if state does not survive a failure. The patterns below ensure that a failed node can be replaced and continue from where it left off.
Database replication Most relational databases offer asynchronous or synchronous replication. Asynchronous replication allows the primary to acknowledge writes quickly, at the cost of a small window where a primary loss could lose recent transactions. Synchronous replication requires every write to be confirmed by a quorum of replicas before the primary acknowledges, reducing the loss window but increasing write latency. A common concrete command pattern for PostgreSQL replication status:
psql -c "SELECT * FROM pg_stat_replication;"This query shows replica lag, connection status, and whether the replica is in catch‑up or streaming mode. Monitoring this output over time reveals replication health before a failure occurs.
Key‑value store consensus Systems like etcd or Consensus-based stores use a quorum algorithm to agree on the state of keys. Clients write to any node, and the cluster ensures that a majority has the new value before confirming success. If a node disappears, the remaining nodes continue operating with the last agreed state, and a new node can join and sync from the cluster. An etcdctl command to check cluster health:
etcdctl endpoint health --cluster-endpoints=https://etcd-1:2379,https://etcd-2:2379A healthy response indicates that a quorum is reachable; if one endpoint is unhealthy, the cluster may still operate if a majority remains.
Distributed file systems When a service depends on a shared filesystem, patterns such as GlusterFS, Ceph, or HDFS replicate blocks across multiple bricks or nodes. A node loss triggers rebalancing, during which the system reconstructs missing replicas from existing ones. The operational pattern is to monitor “missing replica” alerts and initiate rebalancing before a second node fails, which could otherwise lead to data unavailability.
Even well‑designed HA setups have blind spots. Being aware of them before production is the difference between a planned recovery and an outage.
Health check false negatives A health endpoint may return 200 OK while the service is unable to actual work—e.g., the process is alive but a downstream database connection pool is exhausted, or a required environment variable is missing. Probes that only check process liveness miss these cases. The pattern that works is to include dependency checks in the health endpoint itself, or to run a separate “readiness” probe that validates the full request path.
Liveness/readiness confusion Many operators set a single health check and treat it as both liveness and readiness. This leads to two failure modes: the orchestrator restarts a process that is actually healthy but not yet ready to serve, or it keeps sending traffic to a process that has already crashed because the probe never detected the crash. Deploying two distinct probe types, with different intervals and thresholds, eliminates this class of error.
DNS TTL and client caching Short TTLs reduce failover time but increase DNS query volume. Some clients (especially mobile apps or browsers) cache DNS records beyond the TTL, meaning a subset of users may still resolve to the failed endpoint for minutes after the record has been updated. Mitigation includes combining DNS failover with a secondary layer (load balancer or service mesh) that can redirect traffic without relying on client re‑resolution.
Split‑brain in replicated state When a network partition isolates two nodes that both believe they are primary, both may accept writes, leading to divergent state that must be merged later. Fencing mechanisms—such as a quorum vote, a leasing system, or a physical power switch—prevent both sides from accepting writes simultaneously. The lesson is that replication without coordination is not HA; it is concurrency risk.
Graceful drain vs abrupt kill Load balancers and orchestrator restart policies often kill a pod or instance immediately when a health check fails. If the workload includes long‑running requests, database transactions, or in‑flight API calls, an abrupt kill corrupts state or leaves users with hanging connections. Configuring a drain window—time allowed for existing connections to complete before the instance is terminated—is a concrete step that prevents this silent data loss.
Single‑point orchestrator If the HA pattern itself relies on a single control plane (e.g., one Kubernetes API server, one etcd member), that component becomes the new single point of failure. Deploying multiple control plane instances, using an odd number for quorum, and placing them across failure domains is essential. The pattern is: replicate the control plane, then ensure the client library or DNS can discover available instances.
High availability is not a single configuration or tool; it is a set of layered patterns—fault domain separation, health detection, automated failover, and state synchronization—each addressing a specific failure vector. The concrete commands shown (docker healthcheck flags, curl‑based probes, psql replication queries, etcdctl health checks) are representative of how these patterns are operationalized in real systems. The mechanisms—HTTP probes, quorum consensus, TTL‑based DNS, replica promotion—are portable across platforms; you can map them to your own orchestrator, database, or networking stack without relying on vendor‑specific internals. When you understand the failure modes each pattern mitigates, and the sharp edges that still exist, you can design a resilient architecture that survives the inevitable component loss without user‑visible downtime.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy