Integration Testing: Verifying the Seams Between Systems

Integration testing verifies that independently developed systems work together correctly at their interfaces—what happens when data flows across system boundaries. It focuses on the seams: the APIs, message queues, databases, file formats, and network protocols that connect components. This testing is essential when systems are developed in parallel, deployed independently, or maintained by different teams. It matters when a service fails not because of its own logic, but because another system sent incorrect data, timed out, or changed its schema. It is the safety net between microservices, between a frontend and a backend, between a data pipeline and a reporting engine.

Defining the Integration Scope

Integration testing begins by identifying the systems involved and the boundaries between them. The scope is defined by the data flow and coordination points.

A typical integration test suite validates that a user creates an order in a web application, which triggers a message to a payment service, which in turn updates a customer database and sends a confirmation email. The test verifies that all systems respond correctly and in sequence.

The test configuration specifies the endpoints, credentials, and expected data formats. For example, a test suite using curl to validate API interactions includes:

export ORDER_SERVICE_URL="http://orders:8080"
export PAYMENT_SERVICE_URL="http://payments:8080"
export DATABASE_URL="postgresql://db:5432/orders"
export EMAIL_SERVICE_URL="http://email:8000"

Each system is expected to be available at its endpoint, with credentials provided via environment variables or a shared secrets file.

Common Integration Patterns

The most common integration patterns are synchronous API calls, asynchronous messaging, and shared database state.

Synchronous API Calls

When systems communicate via HTTP APIs, integration tests exercise request-response cycles.

A test for creating a new order validates that a POST /orders request produces a correct 201 Created response with the order ID and timestamp.

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customerId": 123,
    "items": [
      {"productId": 456, "quantity": 2},
      {"productId": 789, "quantity": 1}
    ],
    "total": 249.99
  }' \
  http://orders:8080/orders

The test checks the response body and status code, then verifies that the order was persisted in the database.

SELECT id, customer_id, total, created_at 
FROM orders 
WHERE customer_id = 123 
ORDER BY created_at DESC 
LIMIT 1;

A failure here could mean the API sent invalid JSON, the database failed to insert, or the service did not handle a concurrent request correctly.

Asynchronous Messaging

When systems use message queues (e.g., RabbitMQ, Kafka, Amazon SQS), integration tests must validate that messages are published and consumed correctly.

A test that verifies order creation triggers a payment processing job includes:

Publishing a message to a queue.
Starting a consumer service.
Waiting for the message to be processed.
Validating the side effects.

Using rabbitmqadmin, a command-line tool for RabbitMQ:

rabbitmqadmin publish \
  exchange orders \
  routing_key "order.created" \
  payload '{
    "orderId": 501,
    "customerId": 123,
    "amount": 249.99,
    "currency": "USD"
  }'

The test then waits for the message to appear in the payments.processing queue:

rabbitmqadmin get queue payments.processing \
  --acknowledge \
  --timeout 30

If no message appears, the test fails. The consumer service may have missed the exchange, failed to connect to the broker, or processed the message incorrectly.

Shared Database State

When multiple systems share a database, integration tests validate that state changes are visible across services.

A test for a customer profile service that updates a user’s address triggers a notification service that sends an email when the address changes.

The test starts by inserting a customer record:

INSERT INTO customers (id, name, address, updated_at)
VALUES (123, 'Alice Johnson', '123 Main St', '2024-01-01 10:00:00');

It then runs the customer service, which updates the address:

UPDATE customers
SET address = '456 Oak Ave', updated_at = NOW()
WHERE id = 123;

The integration test checks that the notification service sees the change by querying a notifications table:

SELECT id, customer_id, type, status
FROM notifications
WHERE customer_id = 123
  AND type = 'address_updated'
  AND status = 'sent';

If the notification is missing, the test reveals a gap in the data propagation: either the customer service did not trigger the event, or the notification service did not poll the database.

Setting Up the Test Environment

Integration tests require a consistent, isolated environment that mimics production.

The test environment includes:

A database instance (e.g., PostgreSQL, MySQL).
Message brokers (e.g., RabbitMQ, Kafka).
External services (e.g., email, payment, SMS).
Network configuration (e.g., DNS, service discovery).

A docker-compose.yml file defines the full stack:

version: '3.8'
services:
  db:
    image: postgres:15
    environment:
      POSTGRES_DB: orders
      POSTGRES_USER: testuser
      POSTGRES_PASSWORD: testpass
    volumes:
      - db-data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  rabbitmq:
    image: rabbitmq:3.12-management
    ports:
      - "5672:5672"
      - "15672:15672"
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

  orders:
    build: ./orders-service
    environment:
      DATABASE_URL: postgres://testuser:testpass@db:5432/orders
      RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672
    depends_on:
      - db
      - rabbitmq

  payments:
    build: ./payments-service
    environment:
      DATABASE_URL: postgres://testuser:testpass@db:5432/orders
      RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672
    depends_on:
      - db
      - rabbitmq

volumes:
  db-data:
  rabbitmq-data:

The test suite starts with:

docker-compose up -d

It waits for all services to be ready, using a wait-for-it.sh script:

#!/bin/bash
set -e

HOST=$1
PORT=$2
TIMEOUT=${3:-30}

until nc -z "$HOST" "$PORT"; do
  echo "Waiting for $HOST:$PORT..."
  sleep 1
  TIMEOUT=$((TIMEOUT - 1))
  if [ $TIMEOUT -eq 0 ]; then
    echo "Timeout waiting for $HOST:$PORT"
    exit 1
  fi
done

echo "$HOST:$PORT is available"

This script is called before running tests:

./wait-for-it.sh db 5432 30
./wait-for-it.sh rabbitmq 5672 30

Running the Tests

Integration tests are run as part of a CI pipeline, often in parallel, with a focus on speed and reliability.

A test script might use bash and jq to validate responses:

#!/bin/bash

set -euo pipefail


TEST_RESULTS=()

# Create an order
RESPONSE=$(curl -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customerId": 123,
    "items": [
      {"productId": 456, "quantity": 2}
    ],
    "total": 149.98
  }' \
  http://orders:8080/orders)

STATUS=$(echo "$RESPONSE" | jq -r '.status')
ORDER_ID=$(echo "$RESPONSE" | jq -r '.orderId')

if [ "$STATUS" != "201" ]; then
  echo "Expected 201 Created, got $STATUS"
  TEST_RESULTS+=("Order creation failed: $STATUS")
fi

# Verify in database
DB_RESPONSE=$(psql -t -A -c "
  SELECT id, customer_id, total
  FROM orders
  WHERE id = $ORDER_ID
  AND customer_id = 123
  AND total = 149.98
" postgres://testuser:testpass@db:5432/orders)

if [ -z "$DB_RESPONSE" ]; then
  echo "Order not found in database"
  TEST_RESULTS+=("Database inconsistency")
fi

# Check message queue
MESSAGE=$(rabbitmqadmin get queue payments.processing --timeout 10 --format=json)

if echo "$MESSAGE" | jq -e '.payload | contains("orderId")' > /dev/null; then
  echo "Payment message published correctly"
else
  echo "Payment message missing or malformed"
  TEST_RESULTS+=("Message queue issue")
fi

# Output results
for result in "${TEST_RESULTS[@]}"; do
  echo "FAIL: $result"
done

if [ ${#TEST_RESULTS[@]} -eq 0 ]; then
  echo "PASS"
  exit 0
else
  exit 1
fi

This script checks the HTTP response, validates the database state, and confirms that a message was published to the queue. Each step is a gate: failure at any point halts the test suite.

Failure Modes and Common Pitfalls

Integration tests fail in subtle, often silent ways. The most common failures include:

Timing issues: A message is published but not consumed in time. The test assumes immediate delivery, but the consumer is slow or the broker is under load.

Solution: Add timeouts and polling loops with exponential backoff.

Data format mismatches: A service expects JSON but receives a string. The API returns a timestamp field, but the consumer expects createdAt.

Solution: Validate schema using jq or jsonschema and include examples in test data.

Configuration drift: The test environment uses different credentials than production. The database URL in the test is postgresql://test:pass@db:5432/orders, but the service reads postgresql://db:5432/orders and defaults to testuser and testpass.

Solution: Include a config-check step that logs all configuration values.

Idempotency failures: A service processes the same message twice, but the test expects only one side effect.

Solution: Use unique message IDs and verify deduplication.

Partial failures: A test passes, but only one of two expected events occurs. The database is updated, but the email is not sent.

Solution: Add per-step assertions and detailed logs.

State pollution: Tests run in a shared environment and interfere with each other. The first test creates an order, but the second test expects a clean database.

Solution: Use transactional test setup and rollback on failure.

Maintaining Integration Test Suites

Integration test suites grow complex and fragile over time. They require careful maintenance.

Key practices include:

Test isolation: Each test starts with a clean state: a fresh database, empty queues, and known test data.

Test data management: Use seed data stored in SQL files or JSON files. A seeds/ directory contains:

  -- seeds/customers.sql
  INSERT INTO customers (id, name, address) VALUES
    (123, 'Alice Johnson', '123 Main St'),
    (456, 'Bob Smith', '789 Oak Ave');

Test reporting: Generate structured output, such as JSON or JUnit XML, for CI tools.

  <testsuite name="OrderIntegration" tests="3" failures="1" errors="0" time="2.1">
    <testcase name="CreateOrder" classname="OrderIntegration" time="0.5">
      <failure message="Database inconsistency" type="assertion">
        Expected order to exist with total 149.98, but found 140.00
      </failure>
    </testcase>
    <testcase name="ProcessPayment" classname="OrderIntegration" time="1.2">
      <system-out>
        Message payload: {"orderId": 501, "amount": 149.98}
      </system-out>
    </testcase>
  </testsuite>

Test cleanup: After each test run, roll back the database, clear queues, and shut down services.

  # Cleanup script
  psql -c "TRUNCATE TABLE orders, payments, notifications" postgres://testuser:testpass@db:5432/orders
  rabbitmqadmin purge queue payments.processing
  docker-compose down

Integration testing is not a single activity but a discipline: defining the seam, validating the data flow, handling timing, and maintaining the test environment. It is the bridge between isolated unit tests and full end-to-end validation.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.