Architecture Fitness Functions

Production engineering guide for architecture fitness functions covering patterns, implementation strategies, and operational best practices.

Architecture Fitness Functions are measurable, executable tests that validate whether a cloud architecture meets its non-functional requirements. They are run continuously—on every deployment, at regular intervals, or triggered by events—to ensure that architectural principles like resilience, scalability, and observability are not just documented but actively maintained. They matter when teams must verify that systems evolve without degrading core qualities, especially in large, distributed environments where manual inspection fails.

Validating Resilience

Resilience is the ability of a system to recover from failures without manual intervention. To validate it, use a fitness function that injects faults and verifies recovery.

Start with a synthetic failure: a network partition between services. Use curl to simulate a 500ms delay in the order-service via a load balancer:

curl -X POST http://loadbalancer/api/v1/trigger-latency \
     -d '{"service": "order-service", "delay_ms": 500, "duration_s": 60}'

The load balancer configures a rate-limited proxy to delay all requests to order-service for one minute. The fitness function then verifies that the system recovers within a defined window.

The test script runs:

#!/bin/bash


set -e

# Wait for 60 seconds after failure injection
sleep 60

# Check that the order-service is responsive
response=$(curl -s -f -H "Accept: application/json" \
    http://order-service/api/v1/status)

# Verify that 95% of requests in the last 5 minutes succeeded
query="SELECT count(*) as total, sum(case when status = 200 then 1 else 0 end) as success
       FROM request_logs
       WHERE service = 'order-service'
         AND timestamp >= now() - interval '5 minutes'"

result=$(psql -t -A -c "$query" -d monitoring)

total=$(echo "$result" | awk '{print $1}')
success=$(echo "$result" | awk '{print $2}')
success_rate=$(echo "scale=2; $success / $total" | bc -l)

if (( $(echo "$success_rate < 0.95" | bc -l) )); then
    echo "Resilience failure: $success_rate success rate, expected 0.95"
    exit 1
fi

echo "Resilience: $success_rate success rate, all checks passed"
exit 0

This script checks the service’s health and validates the success rate of recent requests. The function fails if the success rate drops below 95% during the failure window.

The most common silent failure is a mismatch between the time the failure is injected and when the test starts checking. A 60-second delay is injected, but the test starts checking immediately—missing the full recovery window. To fix this, add a wait-for-availability step:

# Wait until the order-service is fully available
while true; do
    status=$(curl -s -o /dev/null -w "%{http_code}" http://order-service/api/v1/status)
    if [ "$status" -eq 200 ]; then
        break
    fi
    sleep 5
done

This ensures the test begins only after the service has stabilized post-failure.

Validating Scalability

Scalability is the ability to handle increasing loads with proportional resource usage. A fitness function measures how well the system scales under load and maintains performance.

Use wrk to generate a sustained load of 5,000 requests per second for 10 minutes:

wrk -t12 -c256 -d10m -R5000 \
    --latency \
    --timeout 10s \
    http://api-gateway/api/v1/products

The test writes latency, throughput, and error data to stdout. The fitness function parses this output and validates that:

The validation script:

#!/bin/bash
# check-scalability.sh

set -e

# Parse wrk output
while IFS= read -r line; do
    case "$line" in
        *"Latency"*)
            latency=$(echo "$line" | awk '{print $3}')
            ;;
        *"Requests/sec"*)
            throughput=$(echo "$line" | awk '{print $3}')
            ;;
        *"Error"*)
            error_rate=$(echo "$line" | awk '{print $3}')
            ;;
    esac
done < <(wrk -t12 -c256 -d10m -R5000 --latency --timeout 10s http://api-gateway/api/v1/products)

# Validate thresholds
if (( $(echo "$latency > 300" | bc -l) )); then
    echo "Scalability failure: latency $latency ms, expected < 300"
    exit 1
fi

if (( $(echo "$throughput < 4500" | bc -l) )); then
    echo "Scalability failure: throughput $throughput RPS, expected >= 4500"
    exit 1
fi

if (( $(echo "$error_rate > 0.01" | bc -l) )); then
    echo "Scalability failure: error rate $error_rate, expected <= 0.01"
    exit 1
fi

echo "Scalability: latency $latency ms, throughput $throughput RPS, error rate $error_rate"
exit 0

The biggest pitfall is not accounting for warm-up time. The first 30 seconds of the test often show poor metrics due to cold caches, slow startup, and load balancing delays. To correct this, the function should discard the first 30 seconds of data.

The second pitfall is the choice between measuring over a fixed time window versus a fixed number of requests. Running for 10 minutes is good, but if the system is under variable load, the results can vary wildly. Use a fixed number of requests instead:

wrk -t12 -c256 -R5000 \
    --latency \
    --timeout 10s \
    --duration 10m \
    --rampup 30s \
    http://api-gateway/api/v1/products

The --rampup 30s flag ensures the system stabilizes before recording metrics, making the test more reliable.

Validating Observability

Observability is the ability to infer the internal state of a system from external outputs. A fitness function validates that logs, metrics, and traces are available and reliable.

The function checks that:

The test script:

#!/bin/bash
# check-observability.sh

set -e

# Check JSON logs in a central log store
logs=$(curl -s http://log-store/api/v1/search?service=order-service&from=now-1h&format=json)

# Verify at least 100 log lines and all entries have "timestamp" and "level"
count=$(echo "$logs" | jq -c 'length')
if [ "$count" -lt 100 ]; then
    echo "Observability failure: only $count logs from order-service in last hour"
    exit 1
fi

if ! echo "$logs" | jq -e 'all(.[], .timestamp and .level)'; then
    echo "Observability failure: missing timestamp or level in logs"
    exit 1
fi

# Check Prometheus metrics
metrics=$(curl -s http://prometheus/api/v1/query?query=up{job="order-service"})

# Ensure the service is up and has recent scrapes
if ! echo "$metrics" | jq -e '.data.result[0].value[1] | . > 0'; then
    echo "Observability failure: order-service not scraped in last 5 minutes"
    exit 1
fi

# Check distributed tracing
traces=$(curl -s http://tracing/api/v1/trace?service=order-service&duration=1h)

# Validate that 95% of requests have a trace ID
total=$(echo "$traces" | jq -c 'length')
traced=$(echo "$traces" | jq -c 'count(.[] | select(.trace_id != null))')

rate=$(echo "scale=2; $traced / $total" | bc -l)

if (( $(echo "$rate < 0.95" | bc -l) )); then
    echo "Observability failure: $rate trace coverage, expected 0.95"
    exit 1
fi

echo "Observability: $count logs, $traced traced requests, all metrics available"
exit 0

The most subtle failure is missing or inconsistent trace IDs. A service may emit spans, but the root span lacks a trace_id field. The function must validate that the root span contains trace_id, and that every span includes span_id and parent_span_id. A common mistake is using traceId instead of trace_id, or sending the ID as a UUID string without standardizing the format.

Another failure mode is missing or misconfigured metric labels. The function must validate that metrics include the expected labels, such as job, instance, and service. A missing job label can cause the service to appear as “unknown” in dashboards.

Validating Deployment Reliability

Deployment reliability is the consistency and predictability of releasing new code. A fitness function verifies that each deployment achieves a defined state.

The function checks that:

The test:

#!/bin/bash
# check-deployment-reliability.sh

set -e

# Wait for 60 seconds after deployment
sleep 60

# Verify service health
curl -f http://api-gateway/api/v1/health

# Check that deployment is recorded in registry
deployment_id=$(curl -s http://deployment-registry/api/v1/deployments/latest | jq -r '.id')

# Verify that deployment has correct metadata
if ! curl -s http://deployment-registry/api/v1/deployments/$deployment_id \
    | jq -e '.service == "order-service"
            and .version == "v2.3.1"
            and .status == "deployed"'; then
    echo "Deployment reliability failure: deployment metadata mismatch"
    exit 1
fi

# Check that required config files are present
config_files=(
    "/etc/order-service/config.yaml"
    "/etc/order-service/logging.json"
    "/etc/order-service/feature-flags.json"
)

for file in "${config_files[@]}"; do
    if ! docker exec order-service-container [ -f "$file" ]; then
        echo "Deployment reliability failure: missing config file $file"
        exit 1
    fi
done

# Check that deployment event is logged
event=$(curl -s http://event-stream/api/v1/events?topic=deployment&deployment_id=$deployment_id)

if [ -z "$event" ]; then
    echo "Deployment reliability failure: no event logged for deployment"
    exit 1
fi

echo "Deployment reliability: all checks passed"
exit 0

The most common failure is the deployment not being registered at all. The function must validate that the deployment is not only created but also updated with a status of “deployed” and that the correct version is recorded. A deployment may be created but remain in “pending” status—causing the function to pass despite the service not being live.

Another silent failure is the order of checks. The function waits 60 seconds, then checks health, but the deployment may have started earlier and the service may have been ready before the 60-second mark. To fix this, use a polling loop that checks for health until the service is ready:

# Poll until service is healthy
timeout=60
while [ $timeout -gt 0 ]; do
    if curl -f http://api-gateway/api/v1/health; then
        break
    fi
    sleep 5
    timeout=$((timeout - 5))
done

if [ $timeout -eq 0 ]; then
    echo "Deployment reliability failure: service not healthy within 60 seconds"
    exit 1
fi

This ensures the deployment is not just *after* 60 seconds but *within* 60 seconds of being ready.

Running Fitness Functions in CI/CD

Fitness functions are most effective when run in a CI/CD pipeline. Each function is a step in the pipeline, and the entire architecture is validated with every code change.

A typical ci.yml step:

- name: Run Architecture Fitness Functions
  run: |
    ./scripts/check-resilience.sh
    ./scripts/check-scalability.sh
    ./scripts/check-observability.sh
    ./scripts/check-deployment-reliability.sh
  env:
    API_GATEWAY_URL: https://api.example.com
    PROMETHEUS_URL: https://prometheus.example.com
    LOG_STORE_URL: https://logs.example.com
    TRACING_URL: https://tracing.example.com

The pipeline fails if any function returns a non-zero exit code. The results are recorded in a shared report, and failed functions are annotated with output logs and error details.

The key insight is that fitness functions are not just tests—they are active guardians of the architecture. They do not merely report on the state of the system; they define it.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.