ML Feature Stores

Design and operate feature stores for consistent training-serving feature computation with Feast, Tecton, and custom solutions.

A feature store is a centralized system for managing features used in machine learning models. It provides consistent feature retrieval, versioning, and storage across training and serving environments. This matters when you need to ensure that the same features used to train a model are the same ones used to make predictions, especially at scale.

Managing Feature Definitions

Creating a New Feature Set

Features are defined using a schema that specifies their data type, source, and transformation logic. The system supports multiple data types including numeric, categorical, and temporal values.

INSERT INTO feature_sets (name, description, data_type, source_table, created_at)
VALUES ('user_age_group', 'Age group derived from user birth date', 'categorical', 'users', NOW());

Each feature set must define a primary key column that uniquely identifies records. The system enforces this constraint during insertion.

Defining Feature Transformations

Transformations are applied to raw data before storing features. These transformations are defined in SQL expressions or as Python functions.

INSERT INTO feature_transformations (feature_set_id, transformation_type, expression, description)
VALUES (1, 'sql', 'CASE WHEN age < 18 THEN "minor" ELSE "adult" END', 'Categorize user by age');

Transformation errors cause the entire batch to fail with a detailed error message indicating which feature failed and why.

Versioning Feature Definitions

Feature sets support versioning through a semantic versioning system. Changes to feature definitions create new versions.

UPDATE feature_sets SET version = '1.1' WHERE name = 'user_age_group';

Version changes are tracked in the feature_versions table with timestamps and change descriptions.

Storing and Retrieving Features

Batch Writing Features

Features are written to the store in batches for efficiency. The system requires a consistent primary key across all feature sets.

INSERT INTO features (feature_set_id, entity_id, value, timestamp)
VALUES 
  (1, 'user_123', 'adult', '2023-05-15T10:00:00Z'),
  (1, 'user_456', 'minor', '2023-05-15T10:00:00Z');

Writing features with mismatched primary keys causes a constraint violation error.

Real-time Feature Retrieval

Real-time access uses a REST API endpoint. The system requires an entity ID to retrieve features for a specific record.

curl -X GET "http://feature-store/api/v1/features?entity_id=user_123"

Responses include feature values with timestamps and metadata. Missing features return an empty object rather than an error.

Batch Feature Retrieval

For historical analysis, features can be retrieved in bulk using a query interface.

curl -X POST "http://feature-store/api/v1/features/batch" \
  -H "Content-Type: application/json" \
  -d '{
    "entity_ids": ["user_123", "user_456"],
    "feature_set_names": ["user_age_group"]
  }'

The system returns a structured response with entity IDs as keys and feature values as nested objects.

Managing Feature Data Lifecycle

Feature Expiration and Cleanup

Features can have expiration policies. The system automatically removes expired data based on configured retention periods.

UPDATE feature_sets SET retention_days = 30 WHERE name = 'user_age_group';

Expired features are marked for deletion in the feature_cleanup_queue table and processed by a background job.

Data Validation

All feature values must pass validation rules defined for their feature set.

INSERT INTO feature_validations (feature_set_id, rule_type, rule_value, error_message)
VALUES (1, 'range', '[0, 120]', 'Age must be between 0 and 120');

Validation failures are logged with detailed error information but do not prevent the entire batch from being written.

Feature Schema Evolution

Feature sets can evolve over time. New features are added with explicit versioning.

ALTER TABLE feature_sets ADD COLUMN new_feature_enabled BOOLEAN DEFAULT FALSE;

Backward compatibility is maintained through feature versioning, allowing older models to continue using previous schemas.

Monitoring and Observability

Feature Usage Tracking

The system tracks which features are accessed most frequently and when.

SELECT feature_set_id, COUNT(*) as access_count
FROM feature_access_log
WHERE timestamp > '2023-05-15'
GROUP BY feature_set_id
ORDER BY access_count DESC;

Access logs are stored in a separate table with timestamps and entity IDs for audit purposes.

Feature Quality Metrics

Quality metrics are collected automatically for each feature set including data freshness, completeness, and consistency.

SELECT 
  feature_set_id,
  AVG(timestamp_diff) as avg_freshness,
  COUNT(*) as total_records
FROM feature_quality_metrics
WHERE timestamp > '2023-05-15'
GROUP BY feature_set_id;

Missing data points in quality metrics indicate potential issues with data collection or storage.

Error Monitoring

Feature store errors are logged with detailed information including the specific operation, affected entities, and error types.

SELECT 
  error_type,
  error_message,
  COUNT(*) as frequency,
  MAX(timestamp) as last_occurrence
FROM feature_errors
GROUP BY error_type, error_message
ORDER BY frequency DESC;

Common error types include constraint violations, validation failures, and missing entity references.

Production Considerations

Handling Data Consistency

The system uses transactional writes to maintain consistency across feature sets. Partial failures during batch operations are logged but do not roll back the entire operation.

BEGIN TRANSACTION;
INSERT INTO features (feature_set_id, entity_id, value, timestamp) VALUES (1, 'user_123', 'adult', NOW());
-- If this fails, other inserts continue
INSERT INTO features (feature_set_id, entity_id, value, timestamp) VALUES (2, 'user_123', 150, NOW());
COMMIT;

Performance Optimization

Feature stores support indexing on frequently queried columns to improve retrieval performance.

CREATE INDEX idx_features_entity_timestamp ON features(entity_id, timestamp);

Indexing is managed automatically for common query patterns but can also be customized.

Backup and Recovery

The system maintains backups of feature data with configurable retention policies.

pg_dump -h feature-store-db -U feature_user feature_store_db > backup_20230515.sql

Recovery operations are performed using standard SQL restore procedures.

Common Failure Patterns

Silent Data Loss

Feature store operations can silently fail when data types don't match expected schemas. For example, inserting a string into a numeric field will succeed but store the value as NULL.

-- This succeeds but stores NULL in the numeric column
INSERT INTO features (feature_set_id, entity_id, value, timestamp) 
VALUES (1, 'user_123', 'not_a_number', NOW());

Schema Mismatch Errors

When feature sets are updated with incompatible schemas, existing data may not be accessible until migration is complete.

-- This will fail if any existing records don't meet new constraints
ALTER TABLE features ADD COLUMN required_field VARCHAR(50) NOT NULL;

Race Conditions

Concurrent writes to the same entity can cause inconsistent feature values if transactions aren't properly coordinated.

Time Zone Handling

Feature timestamps are stored in UTC but retrieved in local time zones without explicit conversion, leading to potential confusion.

Missing Feature Errors

Attempting to retrieve features for non-existent entities returns empty responses rather than errors, which can mask data quality issues.

Batch Processing Failures

Large batch operations may partially succeed, leaving some features written while others fail with no clear indication of what was processed.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.