Data Retention Policies

Production engineering guide for data retention policies covering patterns, implementation strategies, and operational best practices.

Data retention policies define how long data must be preserved, when it can be archived or deleted, and under what conditions it is protected from modification or deletion. They are essential when regulatory, legal, or operational requirements demand that data be available for a defined period—such as 7 years for financial records or 5 years for healthcare data—and must be auditable, accessible, and immutable during that window. These policies become critical during compliance audits, system migrations, data breaches, and disaster recovery events.

Defining a Retention Policy

A retention policy is a declarative rule that governs the lifecycle of data. It specifies:

The policy is applied at the storage layer, often through metadata annotations or dedicated configuration. A single policy can be reused across multiple datasets, and policies can be nested or layered.

To define a retention policy, use a standard configuration format such as JSON or YAML. The policy is applied via a policy definition document, which is versioned and managed in source control. The policy is then deployed to the system using a standard deployment mechanism—such as kubectl apply -f policy.yaml or curl -X PUT /policies -d @policy.json.

{
  "name": "financial-records-7-years",
  "description": "All financial transactions must be retained for 7 years.",
  "scope": {
    "data_sources": [
      "finance.transactions",
      "finance.invoices"
    ],
    "filters": {
      "field": "status",
      "value": "finalized",
      "operator": "equals"
    }
  },
  "retention": {
    "duration": "P7Y",
    "mode": "active",
    "immutable": true,
    "grace_period": "P30D"
  },
  "deletion": {
    "trigger": "time-based",
    "schedule": "0 0 1 1 *",  // First day of every year
    "action": "archive-and-delete",
    "archive_location": "s3://archive/finance"
  }
}

The P7Y format follows ISO 8601: P for period, 7Y for 7 years. The schedule field uses cron syntax. active mode means data is immediately accessible; immutable ensures that once a record is marked for retention, it cannot be modified or deleted until the retention period expires.

Applying a Retention Policy

A retention policy is applied to data via policy assignment. This is the process of linking a policy to one or more data entities. The assignment may be explicit or implicit.

Explicit Assignment

The most common method is to attach a policy ID directly to a dataset or file. This is done through a policy assignment table in the metadata store.

INSERT INTO policy_assignments (dataset_id, policy_id, effective_at, assigned_by)
VALUES ('finance.transactions', 'financial-records-7-years', '2023-10-01', '[email protected]');

When a new record is inserted into finance.transactions, the system checks for an active assignment. If one exists, the record inherits the retention rules. The assignment can be validated via a query:

SELECT 
  d.name,
  p.name AS policy,
  p.retention_duration,
  pa.effective_at
FROM datasets d
JOIN policy_assignments pa ON d.id = pa.dataset_id
JOIN policies p ON pa.policy_id = p.id
WHERE d.name = 'finance.transactions';

Implicit Assignment

A policy can also be applied automatically based on naming, schema, or content. For example, a file stored in s3://raw/finance/2023/ is automatically assigned the financial-records-7-years policy if it matches a pattern.

This is done using a policy inference engine that runs on a schedule or on event triggers. The engine scans for new or changed data and applies policies based on rules.


aws s3api list-objects-v2 \
  --bucket raw-data \
  --prefix "finance/2023/" \
  --query 'Contents[].Key' \
  --output text | \
  while read file; do
    policy_id=$(detect-policy-for-file "$file")
    assign-policy "$file" "$policy_id"
  done

The detect-policy-for-file script uses a configuration-driven rule engine:

# detect-policy-for-file
#!/bin/bash
file="$1"

case "$file" in
  *finance*)
    echo "financial-records-7-years"
    ;;
  *customer*|*user*)
    echo "customer-records-5-years"
    ;;
  *support*|*ticket*)
    echo "support-records-3-years"
    ;;
  *)
    echo "default-records-2-years"
    ;;
esac

Enforcing Retention

Once a policy is applied, the system must enforce it. Enforcement occurs through retention enforcement services that monitor data and ensure compliance.

Policy Enforcement Cycle

The enforcement service runs periodically—e.g., daily via systemctl enable retention-enforcer.timer.

It performs the following steps:

  1. Identify all data entities with active policies.
  2. For each, calculate the retention status: is data within its retention window?
  3. Mark data as *retention-eligible*, *retention-locked*, or *retention-exceeded*.
  4. Trigger actions based on state.

The enforcement service logs status changes in a retention_status table:

CREATE TABLE retention_status (
  entity_id TEXT PRIMARY KEY,
  policy_id TEXT,
  status VARCHAR(20) NOT NULL, -- eligible, locked, expired, archived, deleted
  created_at TIMESTAMPTZ,
  updated_at TIMESTAMPTZ,
  metadata JSONB
);

Retention States and Transitions

A policy can define transition rules between states:

"transitions": {
  "eligible": {
    "on": "first-insert",
    "to": "locked",
    "after": "P30D"
  },
  "locked": {
    "on": "expiration",
    "to": "expired",
    "after": "P1D"
  },
  "expired": {
    "on": "deletion-schedule",
    "to": "archived",
    "action": "copy-to-s3",
    "archive_location": "s3://archive/finance"
  }
}

Failure Modes

When enforcement fails, the service logs errors to a standard log format:

{
  "timestamp": "2023-10-01T08:00:00Z",
  "level": "error",
  "service": "retention-enforcer",
  "message": "Failed to lock record 12345: retention period started 2023-09-01, but update received after 2023-09-15",
  "context": {
    "entity_id": "12345",
    "policy_id": "financial-records-7-years",
    "current_status": "eligible",
    "expected_status": "locked",
    "event_timestamp": "2023-09-15T10:30:00Z"
  }
}

Auditing Retention Compliance

Auditing ensures that retention policies are being followed and that data is compliant with their rules. Audits are typically scheduled or triggered by external events.

Audit Query

An audit query checks compliance across all data. It returns a report of all data entities and their retention state.

SELECT
  d.name AS dataset,
  r.status AS retention_status,
  r.created_at AS status_changed,
  p.name AS policy_name,
  p.retention_duration,
  EXTRACT(EPOCH FROM (NOW() - r.created_at)) / 3600 AS hours_since_status_change
FROM retention_status r
JOIN datasets d ON r.entity_id = d.id
JOIN policies p ON r.policy_id = p.id
WHERE r.status = 'expired'
  AND r.updated_at > '2023-09-01'
ORDER BY r.updated_at DESC;

Audit Report Format

The audit result is a table or JSON object with standardized fields:

{
  "audit_id": "retention-audit-2023-10-01",
  "timestamp": "2023-10-01T08:00:00Z",
  "total_records": 472,
  "compliant_records": 413,
  "non_compliant_records": 59,
  "compliance_rate": 0.875,
  "records": [
    {
      "entity_id": "12345",
      "dataset": "finance.transactions",
      "policy": "financial-records-7-years",
      "status": "expired",
      "expected_retention_end": "2023-10-15",
      "actual_retention_end": "2023-10-15",
      "last_update": "2023-10-01T14:00:00Z",
      "notes": "Update after retention start; retention not enforced."
    },
    {
      "entity_id": "67890",
      "dataset": "finance.invoices",
      "policy": "financial-records-7-years",
      "status": "locked",
      "expected_retention_end": "2023-10-15",
      "actual_retention_end": "2023-10-15",
      "last_update": "2023-09-30T09:00:00Z",
      "notes": "Update before retention start; retention enforced."
    }
  ]
}

Audit Automation

The audit is automated via a pipeline that runs daily. It is triggered by a cron job:

# /etc/cron.daily/retention-audit
#!/bin/bash
set -e

AUDIT_ID="retention-audit-$(date +%Y-%m-%d)"
OUTPUT="/var/reports/retention/$AUDIT_ID.json"

echo "Starting retention audit: $AUDIT_ID"

# Run audit query and save result
psql -c "
  SELECT json_build_object(
    'audit_id', '$AUDIT_ID',
    'timestamp', now(),
    'total_records', count(*),
    'compliant_records', sum(case when status = 'expired' then 1 else 0 end),
    'non_compliant_records', sum(case when status != 'expired' then 1 else 0 end),
    'compliance_rate', avg(case when status = 'expired' then 1.0 else 0.0 end),
    'records', json_agg(json_build_object(
      'entity_id', r.entity_id,
      'dataset', d.name,
      'policy', p.name,
      'status', r.status,
      'expected_retention_end', r.created_at + p.retention_duration,
      'last_update', r.updated_at,
      'notes', case when r.status = 'expired' then 'Retention expired' else 'Still within window' end
    ))
  FROM retention_status r
  JOIN datasets d ON r.entity_id = d.id
  JOIN policies p ON r.policy_id = p.id
  WHERE r.updated_at >= now() - interval '1 day'
" > "$OUTPUT"

# Upload to S3
aws s3 cp "$OUTPUT" s3://reports/audits/retention/

# Notify via webhook
curl -X POST https://alerting.service/api/events \
  -H "Content-Type: application/json" \
  -d '{
    "event": "retention-audit-completed",
    "audit_id": "'"$AUDIT_ID"'",
    "report_url": "s3://reports/audits/'"$AUDIT_ID"'.json"
  }'

Managing Policy Lifecycle

Policies are not static. They evolve over time. The policy lifecycle includes:

Policy versions are managed in Git. Each policy is a file in policies/, with a versioned filename:

policies/financial-records-7-years.v1.json
policies/financial-records-7-years.v2.json

A policy is promoted from staging to production via a merge request. The merge request includes:

The policy is rolled out incrementally:

  1. Apply to new data only.
  2. Enable enforcement on existing data.
  3. Monitor compliance for 30 days.
  4. Activate policy on all data.

When a policy is retired, the system:

The retirement process is automated:

# retire-policy
#!/bin/bash
policy_id="$1"
retirement_date=$(date +%Y-%m-%d)

# Update policy status
psql -c "
  UPDATE policies
  SET status = 'deprecated',
      retirement_date = '$retirement_date'
  WHERE id = '$policy_id'
"

# Move data to archive
aws s3 cp \
  "s3://data/$(get-data-location "$policy_id")/" \
  "s3://archive/retired-policies/$policy_id/" \
  --recursive

# Update audit reports
find /var/reports/audits -name "*.json" -exec \
  jq --arg policy "$policy_id" \
     --arg retirement "$retirement_date" \
     'if .records[].policy == $policy
      then .records[].notes += " Retired: $retirement"
      else . end' {} \;

Common Pitfalls

The key to success is a policy validation suite that runs on every deployment and includes:

Failure in any test is a regression in the retention policy system.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.