Platform Scaling Patterns | The Garnet Wiki

Platform scaling patterns describe the repeatable, mechanism-level ways that an internal platform adds capacity, handles growth, and avoids collapse under load. This page matters when your service latency creeps up, your deployment queue backs up, or your database connections saturate and you need to extend capacity without a full redesign. The patterns here are not vendor-specific roadmaps; they are the concrete knobs, commands, and configuration levers that platform engineers actually turn when load increases.

Adding Compute Capacity to a Running Service

When a single instance no longer handles incoming request volume, the first instinct is to add replicas. In Kubernetes, this is a one‑line kubectl command:

kubectl scale deployment my‑service --replicas=5

If you need the scaling to react automatically, the Horizontal Pod Autoscaler captures the intent in a single declarative object and a single controller loop:

kubectl autoscale deployment my‑service \
  --min=2 --max=10 \
  --cpu-percent=70

Docker Swarm users reach for the same semantic, but with a different CLI surface:

docker service scale my‑service=5

Sharp edge: kubectl scale and docker service scale both increase replica count immediately, but they do not adjust Pod Disruption Budgets, Pod anti‑affinity, or node‑level resource quotas. If a node runs out of memory, the kernel OOM killer may terminate pods before the autoscaler can react, and the newly‑scheduled pod lands on an already‑stressed node. Always pair scaling with kubectl autoscale or manual resource definitions (requests/limits) and verify node capacity with kubectl describe node.

Distributing Load Across Instances

Adding replicas is useless if traffic isn’t routed to them. In most cloud‑native platforms, a Service object automatically creates a kube-proxy ruleset (iptables or nftables) and an Ingress controller (NGINX, Envoy, HAProxy) distributes HTTP(S) traffic. The Ingress can be patched to increase maxConcurrentRequests or adjust sessionAffinity:

kubectl patch ingress my‑ingress -p '{"spec":{"rules":[{"http":{"paths":[{"pathType":"Prefix","backend":{"service":{"name":"my‑service","port":{"number":8080}}}}]}}]}'

For TCP‑level load balancing, kubectl expose creates a ClusterIP Service, and kubectl get svc shows the internal cluster IP. If you run an external load balancer (e.g., HAProxy), its config is updated by a Helm chart or operator; a typical reload command is:

systemctl reload haproxy

Sharp edge: Session affinity (client.ip) is often confused with session.cookie. Enabling cookie‑based affinity without a secret‑signed cookie key causes every request to be routed to a different pod, breaking stateful APIs. Additionally, increasing maxConcurrentRequests on an Ingress controller can trigger connection queue buildup if the underlying worker pool isn’t sized accordingly—symptom: 502 Bad Gateway with "connection refused" in the controller logs.

Scaling Stateful Data Stores

Compute scaling hits a wall when the bottleneck is stateful storage. PostgreSQL deployments at scale commonly use two patterns: read replicas and connection pooling.

Read replicas

A new replica is provisioned with pg_basebackup, which streams the primary’s WAL and materializes a consistent backup:

pg_basebackup -h primary.example.com -D /var/lib/postgresql/replica -Fp -Xs -P -R

After the backup completes, the replica is started with pg_ctl:

pg_ctl -D /var/lib/postgresql/replica -l /var/log/postgresql/replica.log start

The primary is then configured to stream WAL:

ALTER SYSTEM SET wal_level = 'logical';
SELECT pg_reload_conf();

Connection pooling

Direct client connections at scale quickly exhaust OS file descriptors. Pgbouncer sits between application servers and PostgreSQL, pooling sessions:

pgbouncer -c /etc/pgbouncer/pgbouncer.ini

The pool size is configured in pgbouncer.ini:

pool_mode = transaction
max_client_conn = 2000
default_pool_size = 100

If you need to adjust the pool at runtime without a restart:

SET max_client_conn = 2500;
SELECT pg_reload_conf();

Sharp edge: Replica lag is invisible until a failover occurs. If the primary promotes a stale replica, applications may read stale data. Connection poolers introduce a “stale‑connection” risk: if a transaction is abandoned mid‑flight, Pgbouncer may return a reused server connection that still holds an open transaction, causing ERROR: current transaction is aborted, commands ignored until end of transaction block. Always wrap application logic in TRY/COMMIT or use pgbouncer’s reset_query to issue ROLLBACK.

Scaling CI/CD Pipeline Throughput

When the build queue grows, the platform’s CI/CD runners become the bottleneck. Git repository hygiene, artifact caching, and container build strategy directly affect throughput.

Repository cleanup

git gc removes unreachable objects and compresses the local store:

git gc --prune=now

Running this on a mirrored bare repository used by CI agents reduces clone latency by 30‑40 % in repositories with >10 GB of history.

Build cache reuse

Docker build cache is leveraged with --cache-from:

docker build --cache-from registry.example.com/my‑app:latest -t my‑app:ci‑$(date +%s) .

For environments where the base image changes frequently, Kaniko builds a new root filesystem without Docker daemon privileges:

kaniko --dockerfile=Dockerfile --context=dir:. --destination=registry.example.com/my‑app:ci‑$(date +%s)

Sharp edge: Cache staleness is the most common failure mode. If a FROM base image is updated but the tag isn’t bumped, Kaniko and Docker both reuse the stale layer, producing an image that doesn’t reflect the new base. Additionally, git gc --prune=now on a shared bare repository while CI agents are cloning can cause fatal: unlink of '.git/objects/pack/pack‑*.pack': Resource temporarily unavailable, halting the pipeline until the lock clears.

Scaling Platform‑Wide Resource Limits

Every platform runs against hard limits: node CPU, memory, disk, and API server QPS. Making these limits visible and adjustable prevents silent failures.

Kubernetes resource quotas

A ResourceQuota object enforces namespace‑wide totals:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute‑quota
spec:
  hard:
    requests.cpu: "200"
    requests.memory: "500Mi"
    limits.cpu: "500"
    limits.memory: "1Gi"
    pods: "50"
    persistentvolumeclaims: "20"

Applied with:

kubectl apply -f quota.yaml

System‑level limits

On bare‑metal or VM hosts, systemctl can adjust service‑level weights:

systemctl set-property platform‑service CPUWeight=2000

This influences the CFS scheduler’s time‑slice allocation without touching cgroup definitions directly.

Sharp edge: Resource quotas are evaluated at pod creation time; if a pod’s requests exceed the quota, it remains in Pending indefinitely. The scheduler does not back‑off or retry—it just keeps the pod waiting. Similarly, systemctl set-property only affects newly‑started service instances; existing processes retain their original cgroup settings until restarted. A common production surprise is a quiet “no more capacity” situation where kubectl get pods -o wide shows 0/1 nodes but kubectl describe quota reveals use: 200/200 cpu, and no one noticed because the quota was never monitored.

Sharp Edges — What You Only Learn After It Breaks

Pattern	Silent Failure	Confused Options	Production Lesson
HPA	Metrics server not deployed → `HPA` stays at `0%` utilization, never scales out	`--cpu-percent` vs `--metrics-resolution`	Always `kubectl get metrics` before relying on autoscaling
Read replicas	Replica promoted during failover reads stale data; application logs show “version mismatch”	`pool_size` vs `max_connections` in Pgbouncer	Set `autofail.mode` and test failover in a staging cluster
Git gc	Lock contention → pipeline jobs hang with `fatal: unlink` error	`--prune=now` vs `--prune=2.weeks.ago`	Run gc on a quiescent window or use a lock‑free mirror
Ingress affinity	Every request hits a different pod → `500 Internal Server Error` for stateful endpoints	`client.ip` vs `cookie` affinity	Prefer cookie‑based affinity with a signed, rotating key
Resource quota	Pods stay `Pending` forever; no error surfaced in UI	`requests` vs `limits` enforcement	Monitor `kubectl describe quota`; set up alerts on `use/` hard limits

When any of these edges trips, the platform doesn’t always emit a loud alarm. The failure mode is often “nothing happens” — a pod stays pending, a connection pool caps at its limit, or a build stalls behind a gc lock. The pattern is always the same: verify the mechanism’s current state before assuming it will act on your intent.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.