Data Encryption Comprehensive

Production-grade guide to data encryption comprehensive covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

Data encryption is the cornerstone of data protection across storage and transmission. It matters when data is sensitive, regulated, or exposed to untrusted environments — whether in a cloud database, over a network, or within a mobile app. This page details the full lifecycle of encryption in engineering systems: how to select and apply encryption, how to manage keys, and how to ensure data remains protected from creation to destruction.

Key Management

The foundation of effective encryption is a robust key management system. Keys must be stored securely, rotated regularly, and accessible to services that need them. The primary challenge is deciding how to store and retrieve keys without hardcoding them in code or configuration.

Key Storage and Retrieval

Keys are typically stored in a centralized key management service (KMS) or in a dedicated key vault. The most common access pattern is a simple HTTP API with a GET /keys/{id} endpoint. A service retrieves a key using a key ID and a JWT bearer token:

GET /keys/12345 HTTP/1.1
Host: kms.example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
Accept: application/json

The response is a JSON object:

{
  "id": "12345",
  "name": "database-encryption",
  "algorithm": "AES-256-GCM",
  "key": "aGVsbG8gd29ybGQh",
  "created_at": "2023-01-01T00:00:00Z",
  "updated_at": "2023-10-15T12:34:56Z"
}

The service uses the key field — a base64-encoded raw key — to decrypt data. The key is never persisted in the service’s configuration file, nor is it embedded in the binary.

Key Rotation and Versioning

Keys are rotated periodically, typically every 90 days. A new key is generated and activated, and the system begins using it for new data while continuing to decrypt old data with previous versions. The key versioning is transparent: a service can fetch the current key for a specific ID, and the KMS returns the latest active version.

When a key is rotated, the KMS assigns it a version number. A service may query a key with an optional version query parameter:

GET /keys/12345?version=2 HTTP/1.1
Host: kms.example.com
Authorization: Bearer <token>

If version 2 does not exist, the server returns a 404 with a body:

{
  "error": "KeyVersionNotFound",
  "message": "Key with ID 12345 and version 2 not found",
  "suggested_versions": [1, 3]
}

Failure to handle this error silently causes data loss: a service reads an encrypted record with a key version it does not know, and decryption fails.

Data Encryption at Rest

Data stored on disk, in databases, or in object storage must be protected against physical theft, unauthorized access, and backup compromise.

Database Encryption

A PostgreSQL database can use transparent data encryption (TDE) via the pgcrypto extension. To encrypt a column, the table must be defined with a bytea column and a pgcrypto function call:

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  name TEXT,
  email TEXT,
  encrypted_data BYTEA,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Encrypt data using AES-256-GCM with a key from KMS
UPDATE users
SET encrypted_data = pgp_encrypt(
  (SELECT row_to_json(row) FROM (SELECT * FROM users WHERE id = 1) row),
  (SELECT key FROM keys WHERE id = 12345),
  'cipher-algo=AES256-GCM'
);

The pgp_encrypt function returns a byte array that is stored in the encrypted_data column. Decryption is performed with pgp_decrypt:

SELECT pgp_decrypt(encrypted_data, (SELECT key FROM keys WHERE id = 12345)) AS data
FROM users
WHERE id = 1;

If the key is missing or corrupted, pgp_decrypt returns NULL. A common failure mode is a mismatch between the encryption algorithm and the key: if the key was generated with AES-128-CBC but the decryption uses AES-256-GCM, the output is garbage or a PGP error: invalid header message.

File System and Object Storage Encryption

For file-based systems, the cryptsetup tool manages LUKS (Linux Unified Key Setup) volumes. A volume is created with:

sudo cryptsetup luksFormat /dev/sdb1

The user is prompted to enter a passphrase. The volume is then opened:

sudo cryptsetup open /dev/sdb1 encrypted_volume

This creates /dev/mapper/encrypted_volume. The system mounts it:

sudo mount /dev/mapper/encrypted_volume /mnt/data

The key for the LUKS volume is stored in the volume’s header. When the system boots, it can unlock the volume using a key file:

sudo cryptsetup open /dev/sdb1 encrypted_volume --key-file /etc/keys/volume.key

The key file contains the raw key, and the --key-file flag reads it directly. If the key file is missing, the system fails to mount the volume and logs:

cryptsetup: Failed to open encrypted_volume: No such file or directory

For object storage like AWS S3 or MinIO, encryption is configured at the bucket level. A bucket is encrypted with server-side encryption using a master key from a KMS:

{
  "Bucket": "user-data",
  "ServerSideEncryptionConfiguration": {
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256",
          "KMSKeyId": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
        }
      }
    ]
  }
}

When uploading an object, the client can specify client-side encryption. The object is first encrypted with a data key, which is then encrypted with the master key:

{
  "Key": "a1b2c3d4e5f6",
  "EncryptedKey": "k1k2k3k4k5k6k7k8k9k0",
  "Algorithm": "AES256-GCM",
  "VersionId": "1234567890"
}

The client-side encryption key is stored in the object metadata. If the object is later accessed, the service must fetch the master key from the KMS, decrypt the data key, then decrypt the object body.

Data Encryption in Transit

Data moving between services, across networks, or from client to server must be protected from eavesdropping and tampering.

TLS Configuration

Transport Layer Security (TLS) is the standard for securing network communication. A service configures TLS using a certificate and a private key. A common setup uses Nginx with a server block:

server {
  listen 443 ssl;
  server_name api.example.com;

  ssl_certificate_file /etc/ssl/certs/fullchain.pem;
  ssl_certificate_key_file /etc/ssl/private/key.pem;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512;
  ssl_prefer_server_ciphers on;

  location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

The ssl_ciphers line specifies a list of cipher suites. The server selects the first one from the client’s list that it supports.

A critical failure mode is a mismatch between the server’s cipher suite and the client’s expectations. If the client sends a ClientHello with TLS_AES_128_GCM_SHA256, but the server only supports AES256-GCM-SHA384, the server responds with a TLS handshake failed alert.

API Request and Response Encryption

For API endpoints, services can encrypt the request body and response payload using a shared key. A service receives a request with a Content-Encoding: gzip and X-Encryption-Key: 12345 header:

POST /api/v1/users HTTP/1.1
Host: api.example.com
Content-Encoding: gzip
X-Encryption-Key: 12345
Authorization: Bearer <token>
Content-Type: application/json

{"name": "Alice", "email": "[email protected]"}

The body is a gzipped JSON, but it is also encrypted using the key from the X-Encryption-Key header. The service must:

  1. Fetch the key from the KMS using the ID 12345.
  2. Decompress the request body.
  3. Decrypt the decompressed body using the key.
  4. Parse the JSON.

A common mistake is not verifying the integrity of the encrypted payload. If the encryption uses AEAD (Authenticated Encryption with Associated Data), the service must validate the authentication tag. If the tag is invalid, it returns:

{
  "error": "EncryptionIntegrityCheckFailed",
  "message": "Decrypted payload failed integrity validation"
}

The same process applies to the response. The service encrypts the response body, compresses it, and sets the Content-Encoding and X-Encryption-Key headers:

HTTP/1.1 201 Created
Content-Encoding: gzip
X-Encryption-Key: 12345
Content-Type: application/json

{"id": 123, "name": "Alice", "email": "[email protected]"}

If the client does not expect encrypted responses, it receives a raw, unencrypted JSON and fails to parse it.

End-to-End Encryption

End-to-end encryption ensures that only the sender and recipient can read the data, even if the intermediate systems are compromised.

Message-Level Encryption

A messaging service uses end-to-end encryption to protect messages between users. Each message is encrypted with a session key, which is then encrypted with the recipient’s public key. The message format is a JSON object:

{
  "sender_id": 123,
  "recipient_id": 456,
  "session_key": "a1b2c3d4e5f6g7h8i9j0",
  "encrypted_body": "k1k2k3k4k5k6k7k8k9k0",
  "iv": "0123456789abcdef",
  "tag": "1234567890abcdef",
  "timestamp": "2023-10-15T12:34:56Z"
}

The session_key is encrypted using the recipient’s RSA public key. The encrypted_body is AES-256-GCM encrypted with the session key, using the iv and tag for integrity.

When a user receives a message, the client:

  1. Decrypts the session key using the recipient’s private key.
  2. Uses the session key to decrypt the body, validating the tag.
  3. Parses the JSON.

If the tag does not match, the client logs:

Error: Failed to decrypt message body — invalid authentication tag

The system must handle the case where the session key is corrupted or the public key is outdated. A common failure mode is a mismatch between the encryption algorithm and the decryption logic: if the sender uses AES-128-GCM but the receiver expects AES-256-GCM, the decryption fails silently.

Key Distribution and Forward Secrecy

To enable forward secrecy, the system uses ephemeral key exchanges. Each message is sent with a fresh Diffie-Hellman (DH) key exchange. The sender generates a new DH public key and sends it with the message:

{
  "sender_dh_public": "0123456789abcdef",
  "session_key": "a1b2c3d4e5f6g7h8i9j0",
  "encrypted_body": "k1k2k3k4k5k6k7k8k9k0",
  "iv": "0123456789abcdef",
  "tag": "1234567890abcdef"
}

The recipient computes the shared secret from their private key and the sender’s public key, then derives the session key. This ensures that if the long-term private key is compromised, past messages remain secure.

Failures and Edge Cases

These patterns define the robust practice of data encryption across systems. The core insight is that encryption is not just a configuration step but a continuous, distributed process requiring careful handling of keys, versions, and integrity checks.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.