Security Testing Automation

Security Testing Automation is the practice of integrating automated security checks into the software development lifecycle, ensuring that every code change is evaluated for vulnerabilities, misconfigurations, and compliance gaps before and after deployment. It matters when teams ship code frequently—daily or multiple times per day—and cannot rely on manual security reviews, which become bottlenecks. It is essential when infrastructure is defined in code, services are distributed across cloud environments, and security policies must be enforced consistently across teams.

Running Security Tests on Code Changes

Automated security testing begins at the moment a developer pushes code. The pipeline triggers a series of checks: static analysis of source code, dynamic analysis of running services, and scanning of container images. These tests run in parallel, report results immediately, and block the merge of pull requests when critical issues are found.

A typical CI job uses git to fetch the latest changes, docker to build images, and curl to invoke APIs. The core of the automation is a test script that calls a security scanner with specific flags. For example, a Python project uses bandit to detect common security issues:

bandit -f json -o results/bandit.json -f html -o results/bandit.html -r src/

This command scans all Python files in src/ and outputs results in both JSON and HTML formats. The JSON output is consumed by the CI system to generate a detailed report, while the HTML is attached to the pull request.

The same pipeline runs snyk on package.json and Dockerfile to detect vulnerabilities in dependencies and container layers:

snyk test --file=package.json --json --output=results/snyk.json
snyk container test --file=Dockerfile --json --output=results/snyk-container.json

Each tool emits a structured output with a consistent schema: a list of issues, each with a id, severity, message, location, and references. The CI system parses this output and marks the pull request with a status: ✅ Pass, ❌ Fail, ⚠️ Warning.

When a pull request contains a security failure, the CI system annotates the code directly: a comment appears with a summary of findings, and inline annotations highlight lines of code where vulnerabilities occur. This is achieved by parsing the JSON output and using curl to post a comment to the pull request API:

curl -X POST \
  -H "Authorization: Bearer $GITHUB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "body": "Found 3 security issues in this PR:\n\n- High: SQL injection in /api/users\n- Medium: Insecure direct object reference in /api/orders\n- Low: Missing input validation on /api/login",
    "issue_number": 123,
    "repo": "myorg/myapp"
  }' \
  https://api.github.com/repos/myorg/myapp/issues/123/comments

The issue_number and repo fields are passed from the CI context. The Authorization header uses a token stored in the CI environment. When a test fails, the CI system sets the pull request status to failure and assigns the security label.

Running Security Tests on Deployed Services

After a service is deployed, automated security testing shifts to runtime. The pipeline now runs dynamic analysis on the live service. The most common approach is to use docker to run the deployed image, kubectl to manage the Kubernetes cluster, and curl to probe endpoints.

A test job in the CD pipeline starts with a kubectl rollout status command to wait for the new deployment to stabilize:

kubectl rollout status deployment/webapp -n production --timeout=10m

Once stable, the pipeline runs owasp-zap as a standalone container, configured to crawl the application and perform active security scans:

docker run --rm \
  -v $(pwd)/zap-reports:/zap/reports \
  -e ZAP_CONFIG="rules=1,2,3" \
  -e ZAP_URL="http://webapp.prod.svc.cluster.local:8080" \
  -e ZAP_CRAWL="true" \
  -e ZAP_ACTIVE_SCANS="true" \
  owasp/zap2docker-stable \
  -i -n -c /zap/config/zap-config.xml

The ZAP_URL environment variable points to the service under test. ZAP_CRAWL enables spidering the entire application, and ZAP_ACTIVE_SCANS triggers a suite of active scans: SQL injection, cross-site scripting, authentication flaws, and more.

The scanner generates an html report in /zap/reports/ and a json report in /zap/reports/scan-results.json. The CI system mounts this directory and uploads the HTML to a web server or S3 bucket:

aws s3 cp zap-reports/ s3://security-reports/zap/ \
  --recursive \
  --acl public-read

The JSON output is parsed to extract critical findings:

{
  "scan": {
    "id": "12345",
    "target": "http://webapp.prod.svc.cluster.local:8080",
    "status": "completed",
    "startTime": "2023-06-12T14:00:00Z",
    "endTime": "2023-06-12T14:15:30Z"
  },
  "alerts": [
    {
      "risk": "High",
      "name": "SQL Injection",
      "description": "The application is vulnerable to SQL injection in the /api/users endpoint.",
      "solution": "Use prepared statements or parameterized queries.",
      "references": [
        "https://owasp.org/www-community/attacks/SQL_Injection",
        "https://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet"
      ],
      "evidence": "GET /api/users?name=John' OR '1'='1",
      "url": "http://webapp.prod.svc.cluster.local:8080/api/users"
    }
  ]
}

The pipeline checks the risk field for High or Critical alerts. If any such alert exists, the pipeline sets the deployment status to failure and sends a notification to Slack:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Security scan failed on production deployment: 2 High-risk issues found.",
    "attachments": [
      {
        "title": "SQL Injection in /api/users",
        "text": "The endpoint is vulnerable to SQL injection. Evidence: GET /api/users?name=John' OR '1'='1",
        "color": "#FF0000"
      }
    ]
  }' \
  https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Integrating Security Tests into Infrastructure as Code

When infrastructure is defined in code—using Terraform, Pulumi, or `CloudFormation—security testing becomes part of the infrastructure lifecycle. The pipeline runs checks on the configuration files before applying changes.

The most common tool is checkov, which validates IaC files against a set of security policies. A Terraform project uses checkov to enforce security best practices:

checkov -f main.tf -f modules/webapp/main.tf \
  --output json \
  --output-dir results/checkov \
  --framework terraform

This command scans two Terraform files and produces a JSON report with a list of checks, each with a check_id, check_name, passed, failed, and file_path. The results are consumed by the CI system to generate a report.

The pipeline uses jq to extract the total number of failed checks:

jq -r '.summary.failed_checks' results/checkov/checkov.json

If the count is greater than zero, the pipeline marks the pull request as failure and adds a comment:

echo "IaC security checks failed: $(jq -r '.summary.failed_checks' results/checkov/checkov.json) issues found." | \
  curl -X POST \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"body": "IaC security checks failed: $(jq -r \".summary.failed_checks\" results/checkov/checkov.json) issues found.", "issue_number": 123, "repo": "myorg/myapp"}' \
    https://api.github.com/repos/myorg/myapp/issues/123/comments

The pipeline also validates the infrastructure against a set of baseline policies. These are defined in a checkov-policy file, which contains a list of rules:

{
  "policies": [
    {
      "id": "AWS-001",
      "name": "Ensure S3 bucket is encrypted",
      "description": "All S3 buckets must have server-side encryption enabled.",
      "severity": "high",
      "condition": {
        "type": "equal",
        "path": "resource.aws_s3_bucket[*].server_side_encryption_configuration",
        "value": "true"
      }
    },
    {
      "id": "AWS-002",
      "name": "Enable CloudTrail logging",
      "description": "CloudTrail must be enabled on the account.",
      "severity": "medium",
      "condition": {
        "type": "exists",
        "path": "resource.aws_cloudtrail_trail"
      }
    }
  ]
}

The pipeline runs checkov with the policy file:

checkov -f main.tf \
  --policy-checks results/checkov-policies.json \
  --output junit \
  --output-dir results/checkov

The junit output is consumed by the CI system to generate a test report. Each policy is a test case, and the results are displayed in the CI dashboard.

Handling Test Failures and Feedback Loops

A key challenge in security testing automation is handling test failures that are not immediately clear. The most common failure is a test that passes locally but fails in CI. This is due to differences in environment, configuration, or data.

The most frequent silent failure is the absence of a ZAP_CONFIG file in the container. The docker run command expects a zap-config.xml file mounted at /zap/config/zap-config.xml, but the file is missing, leading to a default configuration with no active scans. The pipeline logs show no errors, but the scan report is empty.

Another silent failure is the timeout value in kubectl rollout status. If the deployment takes longer than 10 minutes, the command exits with a non-zero status, but the CI system does not mark it as failed. The pipeline continues without waiting, and subsequent steps run on a partially deployed service.

A common confusion is between checkov's --framework terraform and --framework all. The former runs only Terraform-specific checks, while the latter runs all checks across multiple frameworks (Terraform, CloudFormation, Pulumi). The pipeline must explicitly set the framework to ensure consistency.

A sharp edge is the json output format from snyk. The snyk test command emits a vulnerabilities array, but the dependencies array is not used for reporting. When the pipeline expects dependencies to contain the list of vulnerable packages, it misses critical data. The pipeline must use jq to extract the correct fields:

jq -r '.vulnerabilities[] | select(.severity == "high") | .id' results/snyk.json

Another edge is the ZAP_URL environment variable. If the URL is malformed or unreachable, the scanner fails silently. The pipeline must check the ZAP_STARTUP_LOG file, which contains the output of the ZAP startup process:

cat /zap/logs/zap.log | grep -A 5 -B 5 "ERROR"

The grep command extracts all error lines around the ERROR keyword. If the log contains a Connection refused error, the pipeline adds a comment to the pull request:

curl -X POST \
  -H "Authorization: Bearer $GITHUB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "body": "ZAP scan failed: Connection refused to http://webapp.prod.svc.cluster.local:8080.",
    "issue_number": 123,
    "repo": "myorg/myapp"
  }' \
  https://api.github.com/repos/myorg/myapp/issues/123/comments

Finally, the pipeline must handle flaky tests. A security test may pass on first run but fail on subsequent runs due to timing or state. The pipeline uses retry logic with sleep and curl to verify the service before running scans:

until curl -f http://webapp.prod.svc.cluster.local:8080/health; do
  sleep 10
done

This loop waits until the service is healthy before proceeding. The curl -f command returns a non-zero exit code if the HTTP request fails, and the loop continues until the service is up.

Summary

Security testing automation is not a single tool or a single command. It is a pipeline of structured, repeatable checks that run on every code change and deployment. The pipeline uses git, docker, kubectl, curl, jq, aws, and systemctl to orchestrate checks across code, containers, and infrastructure. The output of each tool is a structured JSON or XML report, consumed by the CI system to generate feedback. When a test fails, the pipeline annotates the code, updates the status, and notifies the team. The sharp edges—missing files, silent failures, configuration mismatches, flaky tests—are handled through explicit checks, logging, and retry logic. The result is a reliable, consistent security posture across the software lifecycle.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.