Testing Ci Integration | The Garnet Wiki

Testing CI integration ensures that every code change, from a single commit to a full release, is validated across all stages of the continuous integration pipeline. It matters when a team ships code daily and needs confidence that the integration of components, dependencies, and infrastructure is reliable, repeatable, and observable. This page details how to test the end-to-end behavior of CI systems—focusing on the interaction between code changes, build, test, and deployment processes—without relying on any specific vendor’s tooling.

Validating Build and Test Steps

A CI integration fails when the build step produces artifacts that the test suite cannot consume. The validation begins with the build output: the expected artifact format, the presence of metadata, and the consistency of build environment.

To verify a build step, check the output of make build in the project root:

make build
ls -la build/

The build/ directory must contain:

app.tar.gz (the application bundle)
app.tar.gz.sha256 (checksum file)
manifest.json (a JSON file with version, git commit, and build timestamp)

A missing manifest.json or mismatched SHA-256 causes the pipeline to fail silently. The test step expects the manifest.json to contain:

{
  "version": "1.2.3",
  "commit": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2",
  "timestamp": "2024-03-15T10:30:45Z",
  "builder": "github-actions"
}

The test step runs ./test.sh --manifest build/manifest.json. If the manifest is missing or contains invalid JSON, the test script logs:

ERROR: manifest.json is missing or invalid
  - Expected key 'version' at top level
  - Expected key 'commit' to be a 40-character hex string

Failure to validate the manifest format is a common source of flaky CI runs, especially when multiple build systems (GitHub Actions, GitLab CI, Jenkins) write different manifest schemas.

Testing Dependency Resolution

A CI pipeline integrates with external dependencies: container images, package repositories, and configuration sources. The test must verify that the correct versions of dependencies are resolved and used.

A Dockerfile in the project root is built with:

docker build -t myapp:latest -f Dockerfile .

The image must include a VERSION file at the root:

docker run --rm myapp:latest cat VERSION

Expected output:

1.2.3

The build step must also publish the image to a registry. A publish.sh script uses docker push with --all-tags, which pushes myapp:1.2.3, myapp:latest, and myapp:commit-a1b2c3d.

The test step validates the published image via docker inspect:

docker inspect myapp:1.2.3

Expected keys in the output:

{
  "Config": {
    "Env": [
      "NODE_ENV=production",
      "PORT=3000"
    ],
    "Cmd": [
      "/app/start.sh"
    ]
  },
  "Labels": {
    "org.opencontainers.image.source": "https://github.com/org/repo",
    "org.opencontainers.image.version": "1.2.3"
  }
}

A missing Cmd or incorrect Env array causes the test to fail with:

FAIL: Expected 'Cmd' to be ["/app/start.sh"]
  Got: ["/usr/local/bin/start.sh"]

The test also checks the docker-compose.yml file for a deploy section with replicas: 3. If replicas is missing, the deployment step fails to scale correctly.

Validating Test Execution and Reporting

Tests in a CI integration must run in isolation, with consistent outputs and clear pass/fail signals. The test execution is validated by its output format, timing, and metadata.

The test suite runs with npm test -- --reporter=junit --coverage. The output is a junit.xml file in reports/:

<testsuite name="Integration Tests" tests="15" failures="2" errors="1" time="12.34">
  <testcase name="User login with valid credentials" classname="auth" time="2.11">
    <failure message="Expected user to be logged in after 3 seconds" type="AssertionError">
      Expected: true
      Actual:   false
    </failure>
  </testcase>
  <testcase name="API returns 200 on /health" classname="api" time="0.87"/>
</testsuite>

The CI system expects the junit.xml file to be present in reports/junit.xml. If the file is missing or malformed, the pipeline logs:

ERROR: junit.xml not found in reports/
  Expected: reports/junit.xml
  Found:    reports/junit.xml.gz

A common failure is the junit.xml file being generated in a subdirectory (reports/test/junit.xml) but not copied to the expected location. The test step includes a cp command:

cp reports/test/junit.xml reports/junit.xml

The test suite also generates a coverage/lcov.info file. The CI step uses lcov to generate a report and uploads it to a service. The lcov command is:

lcov --capture --directory . --output-file coverage/lcov.info

The test step verifies the coverage with:

lcov --summary coverage/lcov.info

Expected output:

Reading data from coverage/lcov.info
Summary:
  Lines: 87.2% (1523 of 1747)
  Functions: 92.1% (217 of 235)
  Branches: 84.1% (112 of 133)

If the coverage is below 85% for lines, the test fails with:

COVERAGE FAIL: Lines coverage < 85%
  Expected: 85.0%
  Actual:   83.4%

The test step also collects test.log and test-stderr.log files. The CI system expects these to be archived and available in the build artifacts.

Testing Test Environment Isolation

The CI environment must be reproducible and isolated. A test run on Monday must be indistinguishable from one on Friday, even across different machines.

The test suite starts with docker-compose up -d to bring up services: database, cache, message broker, and API gateway. The test script verifies that all services are up:

docker-compose ps

Expected output:

      Name                    Command               State           Ports
--------------------------------------------------------------------------------
db            docker-entrypoint.sh postgres   Up      5432/tcp
cache         docker-entrypoint.sh redis     Up      6379/tcp
broker        docker-entrypoint.sh kafka     Up      9092/tcp
api           npm start                      Up      0.0.0.0:3000->3000/tcp

The test script checks the db container logs for the line:

LOG:  database system is ready to accept connections

If the log is missing this line, the test fails with:

FAIL: Database not ready within 30 seconds
  Expected: "LOG:  database system is ready to accept connections"
  Found:    "LOG:  starting PostgreSQL"

The test also verifies the database schema via psql:

psql -h localhost -p 5432 -U test -d testdb -c "SELECT * FROM users LIMIT 1;"

The expected result is:

 id | name  | email
----+-------+----------------
  1 | Alice | [email protected]

If the table is not created or the data is missing, the test logs:

DATABASE FAIL: Expected row in users table
  Expected: id=1, name='Alice', email='[email protected]'
  Got:      id=2, name='Bob', email='[email protected]'

Validating Deployment and Rollback

Deployment in CI is validated by the presence and correctness of the deployed artifacts, the state of the runtime environment, and the ability to roll back a failed deployment.

The deployment script runs kubectl apply -f k8s/ to update the Kubernetes cluster. The test verifies that the deployment has rolled out successfully:

kubectl rollout status deployment/myapp -n production

Expected output:

Waiting for deployment "myapp" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "myapp" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "myapp" rollout to finish: 2 of 3 updated replicas are available...
deployment "myapp" successfully rolled out

If the rollout takes more than 60 seconds, the test logs:

DEPLOYMENT TIMEOUT: rollout took 72 seconds, expected < 60
  Last 20 lines from pod myapp-789abc:
  ...

The test verifies the deployed version by checking the pod labels:

kubectl get pods -l app=myapp -n production -o wide

Expected output:

NAME                       READY   STATUS    RESTARTS   AGE    IP           NODE
myapp-789abc               1/1     Running   0          45s    10.244.1.5   worker-1
myapp-789abc               1/1     Running   0          45s    10.244.1.5   worker-1
myapp-789abc               1/1     Running   0          45s    10.244.1.5   worker-1

The app=myapp label must be present, and the VERSION label must match the build version.

Validating Rollback Mechanism

A failed deployment must trigger a rollback. The test simulates a deployment with a known version and then introduces a failure in the application.

The deployment script applies a new myapp:1.2.4 image. The test checks the deployment:

kubectl get deployment myapp -n production -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected output:

myapp:1.2.4

The test then updates the deployment with a faulty version: myapp:1.2.4-bad. The start.sh script in the container logs an error:

echo "ERROR: Failed to connect to database" >> /var/log/app.log

The test waits 30 seconds and checks the logs:

kubectl logs deployment/myapp -n production -c app | tail -n 10

Expected output:

ERROR: Failed to connect to database
ERROR: Failed to connect to database
ERROR: Failed to connect to database
...

The CI system detects the failure via a healthcheck endpoint. The test runs:

curl -f http://localhost:3000/health

If the response is not 200 OK, the test logs:

HEALTHCHECK FAIL: /health returned 503
  Body: {"status": "error", "message": "Database not available"}

The rollback step runs:

kubectl set image deployment/myapp myapp=myapp:1.2.3 -n production

The test verifies that the rollback succeeded:

kubectl get deployment myapp -n production -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected output:

myapp:1.2.3

The test also checks that the old pod logs contain:

INFO: Rolling back to version 1.2.3

If the rollback is not triggered, the test logs:

ROLLBACK MISSING: Expected image myapp:1.2.3 but found myapp:1.2.4

Validating Pipeline Artifacts and Metadata

CI integration is not just about execution—it’s about observability. The pipeline must produce consistent, structured artifacts and metadata.

The test validates that the build/manifest.json file is available in the CI artifacts and that the pipeline metadata is recorded in ci-meta.json:

{
  "branch": "main",
  "commit": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2",
  "build_number": 1234,
  "started_at": "2024-03-15T10:30:45Z",
  "finished_at": "2024-03-15T10:35:12Z",
  "duration_seconds": 267,
  "runner": "ubuntu-latest",
  "artifacts": [
    "app.tar.gz",
    "junit.xml",
    "coverage/lcov.info",
    "reports/test.log"
  ]
}

The test script verifies the existence and content of ci-meta.json:

cat ci-meta.json

If any required key is missing, the test fails with:

METADATA FAIL: Missing key 'branch' in ci-meta.json
  Expected: "branch"
  Found:    "branch", "commit", "build_number"

The test also verifies that all artifacts are archived and available at artifacts/ with the correct names.

The pipeline also generates a CHANGELOG.md file. The test checks that it is updated and contains a section for the current version:

## [1.2.4] - 2024-03-15

### Added
- User profile editing
- Export to CSV

### Fixed
- Login timeout on mobile

If the CHANGELOG.md file is missing or does not have a ## [1.2.4] section, the test logs:

CHANGELOG FAIL: Expected section for version 1.2.4
  Expected: ## [1.2.4]
  Found:    ## [1.2.3]

These validations ensure that the CI integration is not just a series of steps, but a coherent, observable, and reliable system that delivers confidence with every code change.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.