Well Architected Framework Review

Production-grade guide to well architected framework review covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

A Well Architected Framework Review is a structured assessment of cloud-based systems against a set of best practices across six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability, which was added as a sixth pillar in 2021. It matters when a team is onboarding a new cloud service, preparing for a major release, or diagnosing persistent instability in a production environment. It is the standard diagnostic lens for cloud architecture maturity, used to identify gaps, prioritize technical debt, and align engineering work with enterprise goals.

Preparing for a Review

Before conducting a review, ensure the system under evaluation is instrumented with the following:

Run the following command to validate the state of the environment:

terraform plan -out=tfplan -var-file=prod.tfvars

Check for uncommitted changes, unapplied plans, and drift between infrastructure and code. A review is invalidated if the live system does not match the documented IaC state.

Review Checklist

Use this checklist to guide the review process. Each item should be evaluated against the actual system, not just documentation.

Conducting the Review

Pillar 1: Operational Excellence

Evaluate how well the system supports ongoing operations. Focus on automation, monitoring, and incident response.

Automated Deployment Pipelines

Ensure every service has a ci.yml file in .github/workflows or azure-pipelines.yml in the root. Validate that the pipeline includes:

Check for rollback triggers in the pipeline configuration. A common failure is a timeout on health check during deployment, causing the pipeline to mark a deployment as successful when the service is actually down.

Example failure from a deployment log:

{
  "error": "Deployment failed",
  "timestamp": "2025-04-05T12:34:56Z",
  "deploymentId": "d1234",
  "service": "user-service",
  "step": "wait-for-health",
  "status": "failed",
  "details": {
    "url": "http://user-service.staging.example.com/health",
    "timeout": 30,
    "actualResponse": {
      "status": 503,
      "body": "Service is down",
      "headers": {
        "Content-Type": "application/json"
      }
    }
  }
}

Incident Management

Review incident response by checking:

slos:
  - name: "API Latency"
    service: "order-api"
    target: 99.9
    window: "24h"
    errorBudget: "6m"
    metrics:
      - name: "p95_latency"
        query: "avg by (service) (rate(http_request_duration_seconds_bucket{job=\"order-api\"}[5m]))"
        threshold: 500

An incident is considered SLI-based if it triggers on a SLO violation. A common oversight is using SLA (Service Level Agreement) as a trigger without defining the underlying SLI.

Pillar 2: Security

Evaluate the system’s ability to protect data, systems, and access.

Identity and Access Management

Verify that all services authenticate users via OAuth 2.0 or OpenID Connect.

Check that each service has a client_id and client_secret in a shared configuration file, such as config.json.

{
  "auth": {
    "issuer": "https://auth.example.com",
    "audience": "https://api.example.com",
    "client_id": "api-service-client",
    "client_secret": "s3cr3t-1234567890",
    "jwks_url": "https://auth.example.com/.well-known/jwks.json"
  }
}

Validate that JWT tokens are validated on every request. A typical failure occurs when the token is not validated for audience (aud claim), allowing a client to access the wrong service.

Example JWT payload:

{
  "iss": "https://auth.example.com",
  "sub": "user-123",
  "aud": "https://api.example.com",
  "exp": 1680748800,
  "iat": 1680745200,
  "scope": "read write"
}

If aud is missing or incorrect, the service returns 401 Unauthorized with:

{
  "error": "invalid_audience",
  "error_description": "The token is valid but not intended for this service.",
  "expected_audience": "https://api.example.com"
}

Data Protection

Ensure all data at rest and in transit is encrypted.

Verify HSTS header:

HTTP/1.1 200 OK
Content-Type: application/json
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

A silent failure: HSTS is configured but not preloaded, so clients do not automatically enforce HTTPS on first visit.

Pillar 3: Reliability

Assess the system’s ability to recover from failures and maintain consistent performance.

Fault Tolerance and Recovery

Check that services are designed for failure as a first-class citizen.

Example circuit breaker state:

{
  "name": "user-service",
  "state": "OPEN",
  "failureCount": 7,
  "lastFailure": "2025-04-05T12:35:00Z",
  "resetTimer": 28
}

A service is considered unavailable if the health check returns 503 Service Unavailable and the circuit breaker is in OPEN state.

Backup and Recovery

Evaluate backup strategy:

Run the following command to verify backup integrity:

pg_dump -h test-db.example.com -U backup_user -d production_db --format=custom --file=backup-2025-04-04.dump
psql -h test-db.example.com -U validator -d production_db -c "SELECT COUNT(*) FROM users;"

A silent failure: backups are taken but not tested. The system assumes backups are valid, but a recent backup is corrupted and undetected.

Pillar 4: Performance Efficiency

Assess how well the system uses resources to meet performance goals.

Scalability and Load Handling

Check that the system scales effectively under load.

Use curl to simulate load and monitor metrics:

for i in {1..100}; do
  curl -s http://api.example.com/orders -X POST -d '{"product": "laptop"}' -H "Content-Type: application/json"
done

Monitor cloudwatch get-metric-statistics for CPU and request latency.

A common failure: scaling is too slow, causing request backlog during peak traffic. The system scales after the peak has passed.

Caching and Content Delivery

Verify that caching is used at multiple levels:

Check that the API returns Cache-Control headers:

Cache-Control: public, max-age=3600, s-maxage=7200

A failure occurs when cache keys are inconsistent, leading to stale data. A query parameter change (?region=us-west) is not reflected in the cache key.

Pillar 5: Cost Optimization

Evaluate how well the system minimizes resource costs without sacrificing quality.

Resource Utilization

Check that compute resources are sized appropriately.

Use ps aux to verify processes on a spot instance:

ps aux | grep -E "(batch-worker|etl-job)"

A silent failure: spot instance interruptions are not handled. The instance is terminated but not replaced, leading to job loss.

Cost Tracking and Reporting

Ensure cost data is collected and reported.

Example cost report:

{
  "month": "2025-04",
  "services": [
    {
      "name": "EC2",
      "actual": 12840.50,
      "budget": 12000.00,
      "deviation": 840.50,
      "tags": [
        { "key": "Environment", "value": "production" },
        { "key": "Team", "value": "payments" }
      ]
    }
  ]
}

A common failure: costs are not allocated to teams. Resources are shared but not attributed, leading to unaccounted expenses.

After the Review

Document the findings in a review report with:

Use the following template for each finding:

### [Pillar]: [Title]

- **Found in**: [Service, Component, Environment]
- **Evidence**:
  - [Log excerpt]
  - [Configuration snippet]
  - [Metrics graph]
- **Impact**: [High/Medium/Low]
- **Priority**: [Critical/High/Medium/Low]
- **Recommendation**: [Actionable step with implementation guidance]

A review is complete when the team has a shared understanding of the system’s strengths and blind spots, and a clear path forward for architectural improvement.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.