Schema Evolution

Production engineering guide for schema evolution covering patterns, implementation strategies, and operational best practices.

Schema evolution is the process of modifying data schemas over time while maintaining compatibility with existing data, applications, and downstream consumers. This matters when data pipelines, storage systems, or APIs need to adapt to changing business requirements without breaking existing functionality.

Adding Fields

To add a new field to an existing schema, use the appropriate ALTER statement for your system:

ALTER TABLE users ADD COLUMN phone VARCHAR(20);

When adding fields to a partitioned table, ensure the new column is added at the partition level:

ALTER TABLE sales PARTITION BY RANGE (year);
ALTER TABLE sales ADD COLUMN region_id INT;

Adding nullable fields to existing data is safe but can cause downstream issues if consumers expect all fields to be present. The system will store NULL for existing rows, and any query that references the new field without proper handling may produce unexpected results.

Removing Fields

To remove a field from a schema:

ALTER TABLE users DROP COLUMN phone;

Removing fields is generally safe for storage systems, but consumers may fail if they still reference removed columns. Most systems do not validate column existence during write operations, so removing a field and then writing data with that field will silently succeed.

Changing Field Types

Type changes require careful consideration. Direct type conversion between incompatible types fails:

-- This fails in most systems
ALTER TABLE users MODIFY COLUMN age INT;

For compatible changes, use explicit conversion:

ALTER TABLE users ADD COLUMN age_new INT;
UPDATE users SET age_new = CAST(age AS INT);
ALTER TABLE users DROP COLUMN age;
ALTER TABLE users RENAME COLUMN age_new TO age;

Type widening (e.g., from INT to BIGINT) is generally safe. Type narrowing (e.g., from VARCHAR(100) to VARCHAR(50)) can cause data loss during conversion.

Renaming Fields

Field renaming operations vary by system:

ALTER TABLE users RENAME COLUMN phone TO phone_number;

Some systems support renaming through a direct operation, others require adding new columns and copying data. Rename operations may not be atomic in all systems, leading to temporary inconsistencies.

Handling Schema Changes in Streaming Systems

In streaming systems, schema evolution requires coordination between producers and consumers:

# Example with Kafka schema registry
curl -X POST http://schema-registry:8081/subjects/users-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"phone\",\"type\":\"string\"}]}"
  }'

Streaming systems should support schema evolution with backward compatibility. When a new schema version is registered, existing consumers may continue reading with the old schema while new consumers use the updated schema.

Schema Versioning

Implement versioned schemas by maintaining multiple schema definitions:

-- Create a schema version table
CREATE TABLE schema_versions (
  version_id INT PRIMARY KEY,
  schema_definition TEXT,
  created_at TIMESTAMP
);

INSERT INTO schema_versions VALUES (1, '{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
  ]
}', NOW());

Versioning allows rollback to previous schema versions and maintains a history of changes.

Migration Strategy

Schema migrations should follow a phased approach:

  1. Add new fields in the first phase
  2. Update consumers to handle new fields
  3. Remove old fields in the final phase
-- Phase 1: Add new field
ALTER TABLE users ADD COLUMN created_at TIMESTAMP DEFAULT NOW();

-- Phase 2: Update application code to use new field
-- ... application code changes ...

-- Phase 3: Remove old field if no longer needed
ALTER TABLE users DROP COLUMN legacy_field;

Error Handling

Schema evolution failures can be silent or catastrophic:

# Example error from a schema registry
{
  "error_code": 409,
  "message": "Schema being registered is incompatible with an already registered schema"
}

Incompatible schemas may cause write failures or data corruption. Some systems return errors immediately, others may silently accept incompatible changes.

Compatibility Rules

Backward Compatibility

Newer schema versions must be readable by older consumers:

-- Adding a new optional field is backward compatible
ALTER TABLE users ADD COLUMN middle_name VARCHAR(50);

Forward Compatibility

Older schema versions must be readable by newer consumers:

-- Removing a field requires careful handling
ALTER TABLE users DROP COLUMN old_field;

Data Migration

When changing schema definitions, data migration may be required:

-- Example of data transformation during schema change
UPDATE users SET phone = CONCAT('+1-', phone) WHERE phone IS NOT NULL;

Migration scripts should be idempotent and include error handling:

BEGIN;
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE;
UPDATE users SET email_verified = TRUE WHERE email IS NOT NULL;
COMMIT;

Validation

Validate schema changes after deployment:

# Check if schema change was applied
SELECT column_name, data_type 
FROM information_schema.columns 
WHERE table_name = 'users' AND column_name = 'phone';

Common Pitfalls

Silent Failures

Some systems silently ignore invalid schema changes:

-- This might succeed even though it should fail
ALTER TABLE users ADD COLUMN invalid_field;

Inconsistent Behavior

Different systems handle schema evolution differently. For example, some may allow adding required fields while others require default values.

Consumer Incompatibility

Consumers that don't support schema evolution may break when encountering new schema versions:

# Error from consumer expecting old schema
{
  "error": "Field 'phone' not found in schema",
  "timestamp": "2023-01-01T00:00:00Z"
}

Rollback Procedures

If a schema change breaks consumers, rollback must be possible:

-- Rollback example for field addition
ALTER TABLE users DROP COLUMN phone;

For complex changes, maintain a rollback script that can restore the previous state.

Testing Strategy

Schema evolution should include tests for:

Testing schema changes in staging environments before production deployment is essential.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.