Feature Stores for ML Pipelines

Feature stores are centralized, persistent repositories for machine learning features—precomputed, versioned, and accessible across training and serving. They matter when multiple models share feature logic, when feature engineering is complex or expensive, and when consistency between training and serving is critical. A feature store eliminates the "training-serving skew" that plagues ML pipelines, ensuring that the same feature values used during model training are available at inference time.

Defining and Managing Features

Features are derived values from raw data, computed through transformations, aggregations, or embeddings. A feature definition consists of a name, data type, source, and computation logic. In practice, features are defined once and reused across many models. A feature is not just a column in a table; it is a unit of computation with provenance, lineage, and versioning.

To define a feature, write a specification in a standard format. The most common approach uses a declarative, YAML-based schema:

name: user_avg_order_value
description: "Average order value over last 30 days for each user"
type: float
source:
  table: user_transactions
  columns: [user_id, amount, created_at]
  filter: "created_at >= now() - interval '30 days'"
  join: "user_id"
transform:
  function: "avg(amount)"
  group_by: "user_id"
  window: "30 days"
  materialize: true

This definition is versioned using Git. A new version is created by committing the YAML file to a features/ directory in the project repository. Each feature is stored under a features/<name>/ directory, with a feature.yaml file and an implementation/ directory containing the code to compute it.

The feature store registers the feature using a standard API. A client sends an HTTP POST request to /features:

POST /features HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>
Content-Type: application/yaml

name: user_avg_order_value
description: Average order value over last 30 days for each user
type: float
...

On success, the store returns 201 Created with a Location header pointing to the newly created feature.

Computing and Materializing Features

Features are computed on demand or scheduled. The feature store orchestrates the computation pipeline. A common pattern uses a feature-compute daemon that polls a job queue, such as Kafka or RabbitMQ.

Each feature computation job includes a feature_id, start_time, and end_time. The compute engine reads the feature definition, fetches raw data from a data warehouse (e.g., BigQuery, Redshift), applies the transformation logic, and writes the result to a feature table.

The feature table is a time-series table with a primary key of (feature_id, entity_id, timestamp). It is stored in a columnar format optimized for reads: Parquet, ORC, or Delta Lake. The table is partitioned by timestamp and clustered by entity_id.

A typical job configuration is provided as a JSON file:

{
  "feature_id": "user_avg_order_value",
  "start_time": "2024-01-01T00:00:00Z",
  "end_time": "2024-01-02T00:00:00Z",
  "data_source": "bigquery://project.dataset.transactions",
  "output_path": "s3://feature-store/tables/user_avg_order_value/",
  "execution_mode": "batch",
  "parallelism": 16,
  "cache_ttl": "24h"
}

The compute engine processes this job by:

1. Querying the data source for all rows in the specified time window.
2. Joining the data with any lookup tables (e.g., users).
3. Applying the feature function using a standard SQL or Python pipeline.
4. Writing the result to the output path in Parquet format.
5. Registering the computed version in the feature store metadata.

If the job fails, the engine logs the error to a structured log file. The failure is recorded in the feature store’s jobs table:

SELECT job_id, feature_id, status, error_message, started_at, finished_at
FROM feature_jobs
WHERE feature_id = 'user_avg_order_value'
ORDER BY started_at DESC;

Common failure modes include:

• Missing data: Error: Failed to read from bigquery://project.dataset.transactions: Table not found
• Schema mismatch: Error: Expected column 'amount' but found 'price' in source
• Time zone drift: Warning: Using UTC for feature computation, but data has timestamps in America/New_York

Versioning and Feature Schemas

Each feature version is immutable. When a new version of a feature is computed, the feature store creates a new version and updates the feature’s schema. The schema is stored as a schema.json file alongside the versioned data.

{
  "version": "2.3",
  "created_at": "2024-01-15T10:00:00Z",
  "description": "Updated to include shipping fees",
  "input_schema": {
    "columns": [
      { "name": "user_id", "type": "string" },
      { "name": "amount", "type": "double" },
      { "name": "created_at", "type": "timestamp" }
    ]
  },
  "output_schema": {
    "columns": [
      { "name": "user_id", "type": "string" },
      { "name": "avg_order_value", "type": "double" },
      { "name": "count_orders", "type": "int" }
    ]
  }
}

The feature store maintains a version history. A version is activated by setting it as current in the feature_versions table:

UPDATE feature_versions
SET is_current = true, active_from = '2024-01-15T10:00:00Z'
WHERE feature_id = 'user_avg_order_value' AND version = '2.3';

When a model is trained, it references a specific version of a feature. The model’s metadata.json includes:

{
  "model_name": "user_churn_prediction",
  "feature_version": {
    "user_avg_order_value": "2.3",
    "user_lifetime_value": "1.1",
    "user_last_active_days": "4.0"
  },
  "training_start": "2024-01-10T00:00:00Z",
  "training_end": "2024-01-12T00:00:00Z"
}

Serving Features at Inference Time

At inference, the feature store serves features through a gRPC or HTTP API. The API accepts a batch of entity IDs and returns the latest feature values for each entity, with automatic versioning and fallback.

A request to serve features looks like:

GET /features/batch?entity_type=user HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "entity_ids": ["u123", "u456", "u789"],
  "feature_names": [
    "user_avg_order_value",
    "user_lifetime_value",
    "user_last_active_days"
  ],
  "timestamp": "2024-01-15T11:00:00Z"
}

The response is a JSON object where each entity ID maps to a feature set:

{
  "u123": {
    "user_avg_order_value": 124.5,
    "user_lifetime_value": 1870.2,
    "user_last_active_days": 14
  },
  "u456": {
    "user_avg_order_value": 98.3,
    "user_lifetime_value": 2100.1,
    "user_last_active_days": 21
  },
  "u789": {
    "user_avg_order_value": 150.0,
    "user_lifetime_value": 1400.5,
    "user_last_active_days": 7
  }
}

The feature store performs the following steps:

1. Looks up the latest version of each requested feature.
2. Queries the feature table for the most recent value of each feature for each entity, up to the requested timestamp.
3. If a feature is not available for a given timestamp, it falls back to the latest available version.
4. Applies caching (e.g., Redis or Memcached) to reduce latency for repeated queries.
5. Returns the result with metadata about which versions were used.

A common failure is feature version drift: a model was trained on version 2.3 of user_avg_order_value, but during inference, the feature store returns 2.2. The serving system logs:

WARN: Feature user_avg_order_value version mismatch
  - Expected: 2.3
  - Found: 2.2
  - Fallback to latest version
  - Metadata: {"source": "batch-123", "timestamp": "2024-01-15T11:00:00Z"}

Monitoring and Maintenance

The feature store provides a health check endpoint:

GET /health HTTP/1.1
Host: feature-store.example.com

Returns 200 OK with a JSON body:

{
  "status": "healthy",
  "uptime": 365420,
  "last_computed": "2024-01-15T10:58:00Z",
  "active_jobs": 4,
  "failed_jobs": 2,
  "versioned_features": 127,
  "serving_latency_p95": 18.4,
  "last_cache_flush": "2024-01-15T10:30:00Z"
}

A monitoring dashboard tracks feature freshness. The freshness is computed as the time difference between the latest computed feature and the current system time.

SELECT
  feature_id,
  MAX(computed_at) AS last_computed,
  NOW() - MAX(computed_at) AS freshness
FROM feature_computations
GROUP BY feature_id
HAVING freshness > '24 hours';

The feature store also provides a CLI tool for common operations. The fs command-line tool supports:

$ fs list
user_avg_order_value (float)
user_lifetime_value (double)
user_last_active_days (int)

# Get metadata for a feature
$ fs describe user_avg_order_value
Name: user_avg_order_value
Type: float
Description: Average order value over last 30 days for each user
Current version: 2.3
Last computed: 2024-01-15T10:00:00Z

# Compute a feature for a specific time range
$ fs compute user_avg_order_value --start 2024-01-15 --end 2024-01-16
Job ID: 9842
Status: running
Output: s3://feature-store/tables/user_avg_order_value/2.3/

# Serve features for a list of IDs
$ fs serve --entity-type user --ids u123,u456,u789 --timestamp 2024-01-15T11:00:00Z
{
  "u123": { "user_avg_order_value": 124.5 },
  "u456": { "user_avg_order_value": 98.3 }
}

Common Pitfalls and Sharp Edges

• Feature schema drift: A feature’s output schema changes, but the serving system expects the old schema. The feature store logs Schema mismatch and returns 400 Bad Request with a detailed errors array.

• Timezone confusion: The feature store uses UTC internally, but the input data is in local time zones. Without explicit timezone handling, feature values drift by hours. A common error:

  Error: Feature user_avg_order_value computed in UTC, but inference timestamp is in America/New_York
  Suggestion: Use `timezone: 'America/New_York'` in feature definition

• Versioning conflicts: Two compute jobs for the same feature run concurrently. The store detects a race condition and returns 409 Conflict with a version_conflict field:

  {
    "error": "VersionConflict",
    "current_version": "2.3",
    "new_version": "2.4",
    "conflicting_job_id": "1023",
    "message": "Another job started computation for user_avg_order_value while this job was running."
  }

• Caching inefficiency: The feature store caches feature values, but the cache is invalidated on every new version. A request for a feature version that is not in cache results in a full table scan. The solution is to use a versioned_cache with version as a key.

• Feature dependencies: A feature depends on another feature. The store automatically resolves the dependency chain, but a missing dependency causes a 404 Not Found with a missing_features list in the response.

  {
    "error": "DependencyMissing",
    "missing_features": [
      "user_lifetime_value",
      "user_active_count"
    ],
    "suggestions": [
      "Compute user_lifetime_value for the current timestamp",
      "Update feature dependencies in feature.yaml"
    ]
  }

• Feature deletion: Deleting a feature removes its metadata and versioned data, but the store keeps a deleted_features table for auditing. A soft delete flag is set, and a background job periodically archives old versions to cold storage.

The feature store is not a magic box—it is a distributed, versioned, time-series database with a strong contract between feature definition, computation, and serving. It is the backbone of any ML pipeline that scales beyond a single model.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.