Synthetic Monitoring

Synthetic monitoring is the practice of proactively testing system behavior by simulating real user interactions at regular intervals across a distributed infrastructure. It answers: *Is the service available, and is it behaving as expected, even when no real users are present?* This matters most during deployments, during capacity stress, and when diagnosing issues before users report them—especially in systems with high latency, complex dependencies, or global user bases.

Defining a Synthetic Test

A synthetic test is a repeatable, scripted interaction with a system, executed at fixed intervals from one or more locations. It captures the end-to-end behavior of a service, including network latency, API response, rendering time, and error conditions.

Test Definition Format

A synthetic test is described using a declarative, version-controlled configuration that defines:

• The test name and description
• The execution schedule (interval, timezone)
• The entry point (URL, endpoint, command)
• The sequence of steps (HTTP requests, assertions, delays)
• The expected outcomes (status codes, response payloads, timing thresholds)

Example:

name: "Checkout Flow (Web)"
description: "Simulates a full user journey from product page to order confirmation."
schedule:
  interval: "PT5M"     # every 5 minutes
  timezone: "UTC"
entry:
  type: "http"
  url: "https://shop.example.com/checkout"
  method: "GET"
  headers:
    Accept: "text/html"
    User-Agent: "Synthetic Monitor/1.0"
steps:
  - name: "Load product page"
    type: "http"
    url: "/product/123"
    method: "GET"
    assertions:
      - status_code: 200
      - json_path: "$.product.name"
        expected: "Wireless Headphones Pro"
      - duration: "PT1S"
  - name: "Add to cart"
    type: "http"
    url: "/cart/add"
    method: "POST"
    body: '{"product_id": 123, "quantity": 2}'
    headers:
      Content-Type: "application/json"
    assertions:
      - status_code: 201
      - json_path: "$.cart.total"
        expected: 299.98
      - duration: "PT500MS"
  - name: "Submit order"
    type: "http"
    url: "/order/submit"
    method: "POST"
    body: '{"shipping": "express", "payment": "card"}'
    assertions:
      - status_code: 200
      - json_path: "$.confirmation_number"
        expected: "^ORD-[0-9]{6}$"
      - duration: "PT2S"
      - response_size: ">= 1024"

This configuration is stored in tests/ under version control and executed by a synthetic monitor engine.

Running Synthetic Tests

The monitor engine executes the test configuration and records results. It must support:

• Periodic execution (configurable intervals)
• Distributed execution (multiple locations)
• Local state tracking (cookies, session data, variables)
• Failure detection and alerting

Execution Workflow

1. The engine fetches the latest test configuration (e.g., from Git, S3, or a registry).
2. It initializes a test context: session cookies, variables, and a state store.
3. It executes each step in sequence:

• Resolves the URL using DNS and TLS
• Sends the request with headers and body
• Waits for a response (with timeout)
• Parses the response (JSON, XML, HTML)
• Applies assertions (boolean, value-based, pattern-based)

1. If any assertion fails, the step is marked as failed.
2. After all steps, the entire test is marked as passed or failed.
3. Results are written to a time-series database (e.g., InfluxDB, Prometheus, or a custom schema).

Real-Time Execution Example

$ ./synthetic-monitor run --config tests/checkout-flow.yaml --location "us-west-1" --timeout 30s

[2024-04-05T14:32:00Z] Starting test: Checkout Flow (Web)
[2024-04-05T14:32:01Z] Step: Load product page — GET /product/123
  → Status: 200
  → Duration: 412ms
  → Assertion: status_code == 200 — PASSED
  → Assertion: json_path($.product.name) == "Wireless Headphones Pro" — PASSED
  → Assertion: duration < 1s — PASSED
[2024-04-05T14:32:03Z] Step: Add to cart — POST /cart/add
  → Status: 201
  → Duration: 620ms
  → Assertion: status_code == 201 — PASSED
  → Assertion: json_path($.cart.total) == 299.98 — PASSED
  → Assertion: duration < 500ms — FAILED (620ms > 500ms)
[2024-04-05T14:32:06Z] Step: Submit order — POST /order/submit
  → Status: 200
  → Duration: 1980ms
  → Assertion: status_code == 200 — PASSED
  → Assertion: json_path($.confirmation_number) matches "^ORD-[0-9]{6}$" — PASSED
  → Assertion: duration < 2s — FAILED (1980ms)
  → Assertion: response_size >= 1024 — PASSED
[2024-04-05T14:32:07Z] Test result: FAILED (2/3 steps failed)

The engine logs raw HTTP traces, response bodies, and timing breakdowns.

Configuring Execution Locations

Tests run from multiple geographic locations to capture network behavior and detect regional failures.

Location Configuration

Each location is defined by:

• A name (e.g., us-east-1, eu-central-1)
• A geographical coordinate (lat/lon)
• A network profile (latency, packet loss, bandwidth)
• A set of DNS and TLS resolver settings

Example location definition:

locations:
  - name: "us-west-1"
    lat: 37.7749
    lon: -122.4194
    network:
      latency: "PT120MS"
      packet_loss: "0.5%"
      bandwidth: "10Mbps"
    dns:
      servers:
        - "1.1.1.1"
        - "8.8.8.8"
      timeout: "PT10S"
    tls:
      version: "1.3"
      ciphers: ["TLS_AES_256_GCM_SHA512", "TLS_CHACHA20_POLY1305_SHA256"]

The monitor engine uses these to simulate realistic conditions.

Location Selection

When a test runs, it picks a location based on:

• The preferred_location field in the test config
• A round-robin or weighted load-balancing strategy
• A failover policy (e.g., use us-west-1 if available, else eu-central-1)

Common Location Pitfalls

• Latency drift: the configured latency (e.g., 120ms) is applied as a fixed delay, but actual network round-trip varies. A test may pass locally but fail in production due to jitter.
• DNS resolution: the monitor resolves DNS in the location’s context, but fails to account for TTL and cache warming. A test may succeed when DNS is cold, but fail after a cache miss.
• TLS handshakes: TLS is not just a header; it adds 100–400ms of overhead. A test that assumes TLS is "free" may report false pass rates.

Handling State and Dependencies

Synthetic tests often require stateful interactions: authentication, session management, and data setup.

Session Management

A test can maintain state across steps using:

• Cookies (HTTP-only, secure)
• Variables (e.g., {{order_id}}, {{session_token}})
• Persistent storage (e.g., JSON file, in-memory store)

Example:

steps:
  - name: "Login"
    type: "http"
    url: "/auth/login"
    method: "POST"
    body: '{"username": "testuser", "password": "secret"}'
    save:
      cookies: ["session_id", "auth_token"]
      variables:
        user_id: "json_path($.user.id)"
        auth_token: "header('Authorization')"
  - name: "Fetch user profile"
    type: "http"
    url: "/profile/{{user_id}}"
    headers:
      Authorization: "Bearer {{auth_token}}"
    assertions:
      - status_code: 200
      - json_path: "$.name"
        expected: "Test User"

State Initialization

Before a test runs, the monitor may execute a setup phase:

setup:
  - name: "Create test user"
    type: "http"
    url: "/admin/users"
    method: "POST"
    body: '{"name": "Test User", "email": "[email protected]"}'
    save:
      variables:
        user_id: "json_path($.id)"
  - name: "Create test product"
    type: "http"
    url: "/admin/products"
    method: "POST"
    body: '{"name": "Test Product", "price": 99.99}'
    save:
      variables:
        product_id: "json_path($.id)"

Common State Failures

• Stale session: a login token expires, but the test doesn’t refresh it. The test passes on first run, fails on second.
• Missing data: the test assumes a product exists, but the setup step failed silently. No error is logged, but the test fails due to 404 Not Found.
• Variable resolution errors: {{unknown_variable}} resolves to null, but the assertion expects a non-empty string. The error is not surfaced in the log, only in the result payload.

Monitoring and Alerting

Synthetic tests are not just execution engines—they are observability tools.

Result Schema

Each test result is a structured object with:

• test_name, location, start_time, end_time
• duration, status (success, failed, partial)
• steps: list of step results (name, status, duration, assertions)
• metadata: environment, version, run ID, tags

Example:

{
  "test_name": "Checkout Flow (Web)",
  "location": "us-west-1",
  "start_time": "2024-04-05T14:32:00Z",
  "end_time": "2024-04-05T14:32:07Z",
  "duration": 7000,
  "status": "failed",
  "steps": [
    {
      "name": "Load product page",
      "status": "passed",
      "duration": 412,
      "assertions": [
        { "type": "status_code", "expected": 200, "actual": 200, "result": true },
        { "type": "json_path", "path": "$.product.name", "expected": "Wireless Headphones Pro", "actual": "Wireless Headphones Pro", "result": true }
      ]
    },
    {
      "name": "Add to cart",
      "status": "failed",
      "duration": 620,
      "assertions": [
        { "type": "status_code", "expected": 201, "actual": 201, "result": true },
        { "type": "json_path", "path": "$.cart.total", "expected": 299.98, "actual": 299.98, "result": true },
        { "type": "duration", "expected": "500ms", "actual": "620ms", "result": false }
      ]
    }
  ],
  "metadata": {
    "version": "v1.2.3",
    "run_id": "run-7a3f1b2c",
    "tags": ["checkout", "web", "regression"]
  }
}

Alerting on Synthetic Results

Alerts are triggered based on:

• Thresholds: test duration exceeds 1s, or response size < 512 bytes
• Failure rates: more than 10% of runs fail in the last hour
• Trends: 30% increase in latency over 24 hours
• Anomalies: step duration spikes by 2σ from baseline

Alerts are sent via:

• Email (with full result payload)
• Slack (with step-by-step breakdown)
• Webhook (to external systems like PagerDuty, Opsgenie)

Example alert:

{
  "title": "Checkout Flow (Web) Failed in us-west-1",
  "body": "Test failed due to slow 'Add to cart' step (620ms > 500ms).",
  "details": {
    "test": "Checkout Flow (Web)",
    "location": "us-west-1",
    "run_id": "run-7a3f1b2c",
    "timestamp": "2024-04-05T14:32:00Z",
    "steps": [
      { "name": "Add to cart", "status": "failed", "duration": 620 }
    ],
    "raw_response": "..."
  },
  "severity": "critical"
}

Silent Failures

Synthetic monitoring often fails silently:

• A test runs but does not write results to storage.
• A step passes assertion but misses a key part of the response (e.g., json_path("$.total") returns 299.98, but the test expects 299.980).
• A test expects a 200 but receives 201, and the assertion is status_code: 200—but the test does not fail on 201.

Advanced Patterns

Multi-Test Orchestration

Complex scenarios require multiple tests running in sequence:

• Test A: user login
• Test B: order creation
• Test C: payment processing
• Test D: order confirmation

These can be orchestrated using a pipeline:

pipeline:
  - name: "Full User Journey"
    tests:
      - "login.yaml"
      - "create-order.yaml"
      - "process-payment.yaml"
      - "confirm-order.yaml"
    dependencies:
      "create-order.yaml":
        requires: "login.yaml"
      "process-payment.yaml":
        requires: "create-order.yaml"
    schedule: "PT10M"

The pipeline runs tests in order, using results from earlier tests as inputs.

Dynamic Test Parameters

Tests can be parameterized:

parameters:
  - name: "user_type"
    values: ["new", "returning", "vip"]
  - name: "product_category"
    values: ["electronics", "clothing", "books"]

test:
  name: "User Journey by Segment"
  steps:
    - name: "Load category page"
      url: "/category/{{product_category}}"
    - name: "Add to cart"
      body: '{"product_id": 123, "user_type": "{{user_type}}'}"

Each parameter combination runs a separate test instance.

Failure Diagnosis

When a test fails, the monitor should:

• Include raw request and response bodies
• Capture network traces (Wireshark-style packet captures)
• Emit logs at each step (including DNS, TLS, and HTTP layers)

A failure in Add to cart might include:

• Request: POST /cart/add with body and headers
• Response: 201 Created with JSON and Location header
• Error: {"error": "Invalid product ID", "code": "400"}

This allows engineers to reproduce the failure locally with a single command:

$ curl -X POST https://shop.example.com/cart/add \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer xyz123" \
  -d '{"product_id": 123, "quantity": 2}' \
  -v

Synthetic monitoring is not just a test suite—it is a continuous, observability-driven feedback loop that ensures systems behave as expected, even in the absence of real users.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.