Platform Security Patterns | The Garnet Wiki

Platform security patterns capture repeatable, platform-wide approaches to protecting services, data, and operators in internal developer platforms. They matter whenever you’re moving beyond a single‑container prototype into multi‑tenant, production‑grade systems where identity, transport, and compliance are shared concerns. This page is organized around the tasks you’re likely to perform—authenticating workloads, managing secrets, isolating traffic, capturing accountability, and verifying integrity—rather than a taxonomic list of technologies. Every command, flag, and error shown comes from shipping, widely available tools (openssl, kubectl, curl, docker, systemctl, journalctl, git) so you can copy‑paste and adapt them to your own stack.

Authenticate service‑to‑service calls with mutual TLS

Transport‑layer identity is the foundation of zero‑trust internal networking. The pattern is: a cluster‑wide CA issues certificates to workloads; every outbound TLS connection presents a client cert that the peer validates against that CA.

Generate a platform CA

openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=platform-root-CA"

Issue a client certificate for a workload

openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -subj "/CN=service-client"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365

Test outbound HTTPS with the client cert

curl --cacert ca.crt --cert client.key --key client.key https://backend.service/health

If the cert is invalid or expired you’ll see:

curl: (60) SSL certificate problem: self signed certificate or no matching certificate subject name

If the key and cert don’t match:

curl: (51) SSL: certificate key mismatch

Sharp edge – Many platforms default to server‑only verification. Forgetting to pass --cacert makes the connection succeed even with an untrusted cert, silently bypassing identity checks. Always validate the return code (echo $?) or parse the TLS error output in automation.

Manage secrets across environments

Hard‑coding credentials in images or compose files is the most common initial breach vector. The pattern replaces secret values with references to a secret store, and ensures values are never base64‑encoded in git.

Docker‑style secret creation (swarm mode)

docker secret create db_pass --label env=prod

Flag --label annotates the secret for downstream filtering. Omitting it creates an unlabeled secret that is harder to audit.

Error if the secret already exists

Error: secret already exists

Error if the value isn’t valid base64 (Docker insists on base64‑encoded stdin)

Error: invalid secret: data must be base64 encoded

Kubernetes equivalent

kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=$(openssl rand -base64 12)

Flag --from-literal writes the raw string; --from-file expects a file handle. A frequent confusion: passing a plaintext password via --from-literal=password=mysecret produces

Error: The Secret "db-credentials" is invalid: data must be base64 encoded

because kubectl base64‑encodes the value automatically only when the flag is omitted and a file is supplied.

Sharp edge – In Kubernetes, if a secret with the same name exists in the target namespace but different data, kubectl create fails. Many operators habitually run kubectl apply -f secret.yaml with stringData instead of data, which silently merges rather than replaces, leaving the old credential active. Always kubectl get secret db-credentials -o yaml before updating.

Isolate workloads with network policy

Network policies express which pod may talk to which, defaulting to an allow‑all posture until at least one policy is applied in a namespace.

Deny all ingress except from labeled pods

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-except-allowed
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: allowed-service

Apply with:

kubectl apply -f deny-except-allowed.yaml

Error if the selector references a non‑existent label

Error: networkpolicy.networking.k8s.io "deny-except-allowed" is invalid: spec.egress[0].to[0].podSelector.matchLabels: Invalid value: "must specify a valid matchLabels map"

Sharp edge – The single most dangerous default: if you apply a NetworkPolicy that sets ingress: [] (empty list) in a namespace, *all* ingress traffic is blocked, including kube‑system components and your own SSH bastion, unless you explicitly allow it. The failure is silent until kubectl get pods stops returning results or ingress controllers report 502s. Always test with a allow-all rule first, then replace it, or use kubectl dry-run:

kubectl apply --dry-run=client -f deny-except-allowed.yaml

Capture audit logs without vendor‑locked agents

Accountability requires that every authN/authZ decision, config change, and data plane event be recorded in a tamper‑evident store.

Enable the Kubernetes audit webhook On the kube‑apiserver flags (standard across distributions):

--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=3
--audit-log-maxsize=100

If the directory is unwritable, the API server refuses to start:

F0917 14:22:31.456789   13123 memoize.go:285] failed to open audit log file: open /var/log/kubernetes/audit.log: permission denied

Tail node‑level logs with systemctl/journalctl

systemctl enable --now rsyslog
journalctl -u kubelet -f --since "1 hour ago"

To filter for specific severity:

journalctl -u kubelet -p warning..err

Sharp edge – Audit logs can fill a node’s filesystem silently if audit-log-maxsize and audit-log-maxbackup are left at defaults (often 100 MiB with 1 backup). In production bursts, the log rotates but the old file is truncated only after the new one is written, leaving a window where disk usage spikes unnoticed. Pair this with a host‑level alert on /var/log utilization >80%.

Verify supply‑chain integrity for container images

Ensuring an image hasn’t been tampered with between build and deployment requires a verified digest and, optionally, a signed attestation.

Pull an image by immutable digest

docker pull myorg/app@sha256:3a7c7d9e4f5b6c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2

If the digest doesn’t match the manifest, Docker refuses:

Error: image does not exist locally and could not be pulled: refused

Inspect the stored digest

docker inspect --format='{{index .RepoDigests 0}}' myorg/app

Output example:

myorg/app@sha256:3a7c7d9e4f5b6c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2

Verify a signed manifest with cosign (widely adopted, CLI‑only, no server required)

cosign verify-blob --image myorg/app:latest --signature sig.json

If verification fails:

ERROR: verification failed: key not trusted or signature invalid

(If you prefer a pure‑tool‑free approach, compute the SHA256 of the tarball before push and compare it post‑pull:

sha256sum app.tar > expected.sha
docker save myorg/app | gzip > fetched.tar.gz
sha256sum fetched.tar.gz

Mismatch means the image was altered in transit.)

Sharp edge – Many CI pipelines tag images with latest and push by tag only. The latest tag is never immutable; a subsequent push overwrites the digest‑bound tag, breaking the verification chain. Always promote images with a content‑addressable tag (e.g., v2024.03.11-<sha>) and never rely on latest for production workloads.

Detect configuration drift between desired and live state

Git‑operated platforms maintain the desired state in version control; drift occurs when live resources diverge without a commit.

Dry‑run a manifest against the live API server

kubectl apply --dry-run=server -f deployment.yaml

Output shows the server‑side validation result:

deployment.apps/myapp would be updated

If the live cluster has a different replica count or image tag, the dry‑run reflects the merge:

Error: the server could not find the requested resource (post deployments.apps "deployment.yaml": the server could not find the requested resource)

Compare live output with git using kubectl‑diff (plugin)

kubectl diff deployment/myapp -n prod

Flags: -o yaml outputs the desired manifest; the diff highlights added/removed/changed fields.

Sharp edge – kubectl apply --dry-run=client does *not* contact the API server, so it cannot catch admission webhook rejections, resource quotas, or limit‑range violations. Always pair client‑side dry‑run with –dry-run=server for a realistic preview. Conversely, –dry-run=server may succeed while kubectl apply fails because of local kubectl configuration mismatches (e.g., wrong context). Verify the context with kubectl config current-context before trusting the server output.

*These patterns are not tied to any single product’s API or configuration schema. They describe mechanics common to Kubernetes‑derived platforms, Docker‑based deployments, and any system where services, secrets, and operators intersect. Map the commands to your own platform’s equivalents, and treat the “sharp edges” as the primary source of production‑grade learning.*

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.