Feature Store Design

Production engineering guide for feature store design covering patterns, implementation strategies, and operational best practices.

Feature stores are centralised repositories that manage the lifecycle of features used in machine learning systems. They ensure consistency between training and serving, support rapid experimentation, and enable teams to share, version, and monitor features across models. Feature store design matters when multiple models depend on overlapping, evolving data, and when training and inference must align precisely—especially in real-time prediction pipelines.

Core Workflow: Feature Definition and Registration

A feature is a named, typed, computed value derived from raw data. It is defined once and consumed by many models. The design begins with defining a feature in a canonical format.

A feature definition is a JSON object with required fields: name, type, description, source, transform, and materialization. The source field specifies the data origin—typically a database table or a stream. The transform field describes how to compute the feature from the source. The materialization field controls how and when the feature values are computed and stored.

{
  "name": "user_avg_order_value",
  "type": "double",
  "description": "Average revenue from user's past orders",
  "source": {
    "table": "orders",
    "key_column": "user_id",
    "timestamp_column": "created_at"
  },
  "transform": {
    "sql": "SELECT user_id, AVG(revenue) as avg_order_value FROM orders GROUP BY user_id"
  },
  "materialization": {
    "schedule": "hourly",
    "target": "s3://feature-store/user_features/"
  }
}

The feature is registered into the store using an HTTP POST to /features.

POST /features HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "name": "user_avg_order_value",
  "type": "double",
  "description": "Average revenue from user's past orders",
  "source": {
    "table": "orders",
    "key_column": "user_id",
    "timestamp_column": "created_at"
  },
  "transform": {
    "sql": "SELECT user_id, AVG(revenue) as avg_order_value FROM orders GROUP BY user_id"
  },
  "materialization": {
    "schedule": "hourly",
    "target": "s3://feature-store/user_features/"
  }
}

On success, the store returns 201 Created with the feature ID in the Location header.

HTTP/1.1 201 Created
Location: /features/12345
Content-Type: application/json

{
  "id": "12345",
  "name": "user_avg_order_value",
  "status": "registered",
  "created_at": "2024-08-10T08:00:00Z"
}

A common failure: missing timestamp_column in the source. If the feature uses time-series data but timestamp_column is absent, the store assumes the table is static and computes features only once—leading to stale values in real-time inference.

Data Flow: Materialization and Update Scheduling

Features are materialized—computed and persisted—on a schedule. The store runs the transform SQL periodically and writes the results to the target location. Materialization is idempotent: repeated runs do not introduce duplicates.

Materialization jobs are triggered by a scheduler that polls the store for features with pending updates. The store exposes a /materialize endpoint that accepts a list of feature IDs.

POST /materialize HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "features": ["user_avg_order_value", "user_total_orders"],
  "start_time": "2024-08-10T08:00:00Z",
  "end_time": "2024-08-10T09:00:00Z"
}

The scheduler runs this for each feature on its schedule. Materialization jobs are persisted in a job table with id, feature_id, status, start_time, end_time, output_path, and error.

A job fails silently if the transform.sql query returns a column not in the feature schema. For example, if avg_order_value is expected but the query returns avg_revenue, the store logs column_mismatch in the job’s error field and marks the job failed.

The materialization.target supports S3-style paths, but only with a strict format: s3://<bucket>/<path>/<feature_name>/. The store expects a directory per feature, with each version as a subdirectory named by YYYY-MM-DD-HH-MM-SS.

s3://feature-store/user_features/
├── user_avg_order_value/
│   ├── 2024-08-10-08-00-00/
│   │   ├── data.parquet
│   │   └── metadata.json
│   └── 2024-08-10-09-00-00/
│       ├── data.parquet
│       └── metadata.json

Each metadata.json contains the feature schema, version, and the transform.sql.

The Scheduling Conundrum

The schedule in materialization can be hourly, daily, or custom. For custom, the store expects a cron-like string.

"materialization": {
  "schedule": "custom",
  "cron": "0 0 * * *"
}

The scheduler runs on a Kubernetes cluster, using kubectl to scale the materialization worker.

kubectl scale deployment feature-materializer --replicas=5 -n feature-store

Workers poll the /jobs/pending endpoint every 30 seconds.

GET /jobs/pending HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>
[
  {
    "job_id": "7890",
    "feature_name": "user_avg_order_value",
    "start_time": "2024-08-10T09:00:00Z",
    "end_time": "2024-08-10T10:00:00Z"
  }
]

The worker pulls the job, fetches the transform.sql, executes it, writes the result, and updates the job status.

A sharp edge: if the start_time is not in UTC, the store assumes it is local to the server and converts it incorrectly. A start_time of "2024-08-10T08:00:00" with no timezone is interpreted as UTC, but if the server is in America/Los_Angeles, the job runs one hour too early.

Another silent failure: a feature with a daily schedule but no cron field. The store defaults to 0 0 * * * (midnight UTC), but if the feature is intended to run at 9 AM local time, the materialization is delayed by 9 hours.

Feature Retrieval: Training and Serving

Features are retrieved via a query interface. The store exposes a GET /features/{name}/values endpoint that accepts a list of keys and a timestamp.

GET /features/user_avg_order_value/values?keys=123,456,789&timestamp=2024-08-10T08:30:00Z HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>

The response is a JSON array of feature values.

[
  {
    "key": "123",
    "value": 54.2,
    "timestamp": "2024-08-10T08:00:00Z"
  },
  {
    "key": "456",
    "value": 78.1,
    "timestamp": "2024-08-10T08:00:00Z"
  },
  {
    "key": "789",
    "value": 32.5,
    "timestamp": "2024-08-10T08:00:00Z"
  }
]

The timestamp parameter controls which version of the feature to use. If no version exists at that timestamp, the store returns the closest earlier version.

The store uses a time-travel mechanism: each materialized version is associated with a timestamp field. During retrieval, the store performs a binary search over the version directory to find the most recent version before the requested timestamp.

A common mistake: using timestamp in the query but not setting the version in the materialization. The store assumes version is the start_time of the materialization job, but if the job runs late, the version is off by minutes.

Real-Time Serving: Feature Lookup and Caching

For real-time inference, the feature store exposes a gRPC service with a GetFeatureValues method.

service FeatureStore {
  rpc GetFeatureValues(GetFeatureValuesRequest) returns (GetFeatureValuesResponse);
}

message GetFeatureValuesRequest {
  repeated string feature_names = 1;
  repeated string keys = 2;
  string timestamp = 3;
}

message GetFeatureValuesResponse {
  repeated FeatureValue values = 1;
}

message FeatureValue {
  string key = 1;
  string feature_name = 2;
  double value = 3;
  string timestamp = 4;
}

The client calls this with a list of feature names and keys.

import grpc
from feature_store_pb2 import GetFeatureValuesRequest
from feature_store_pb2_grpc import FeatureStoreStub

channel = grpc.insecure_channel("feature-store.example.com:50051")
stub = FeatureStoreStub(channel)

request = GetFeatureValuesRequest(
    feature_names=["user_avg_order_value", "user_total_orders"],
    keys=["123", "456"],
    timestamp="2024-08-10T08:30:00Z"
)

response = stub.GetFeatureValues(request)

for value in response.values:
    print(f"{value.key}: {value.feature_name} = {value.value} ({value.timestamp})")

The gRPC server runs in a Kubernetes pod, and the store uses a Redis-backed cache layer to avoid repeated disk reads.

The cache keys are of the form feature:{name}:{key}:{timestamp}.

redis-cli
> GET feature:user_avg_order_value:123:2024-08-10T08:30:00Z
"{\"value\":54.2,\"timestamp\":\"2024-08-10T08:00:00Z\"}"

The store evicts old cache entries using LRU and prewarms the cache for high-traffic keys.

A failure mode: the timestamp in the gRPC request is not in ISO 8601 format. If the client sends "2024-08-10 08:30:00", the store parses it as UTC, but if the server expects Z, it fails to find the versioned data. The error is logged as invalid_timestamp_format with the raw input and expected format.

Feature Versioning and Schema Evolution

Features evolve over time. A new feature may be introduced with a different schema. The store supports versioning of features and schema evolution.

When a feature is registered, it is assigned a version 1.0.0. Each materialization job updates the version and creates a new schema object.

"schema": {
  "version": "1.0.0",
  "fields": [
    {
      "name": "user_id",
      "type": "string",
      "nullable": false
    },
    {
      "name": "avg_order_value",
      "type": "double",
      "nullable": true
    }
  ]
}

The store maintains a schema_versions table with feature_id, version, schema, and created_at.

When a new version is materialized, the store validates the output against the schema and rejects the job if the data does not conform.

A common failure: a feature schema changes, but the transform.sql does not output the new column. The store logs missing_column and unexpected_column in the job error.

The store supports backward compatibility: if a feature is used with a version field in the request, the store returns the appropriate version.

GET /features/user_avg_order_value/values?keys=123&version=1.1.0 HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>

The store selects the versioned directory 1.1.0 and returns values computed from that version.

A sharp edge: the version field in the request is optional. If omitted, the store returns the latest version, but the client must handle version mismatches. If a model was trained on 1.0.0 but the store returns 1.1.0, the model fails to parse the feature values if the new schema has a column not in the old one.

Schema Migration and Feature Registry

A new feature version is registered via a PUT to /features/{name}/versions.

PUT /features/user_avg_order_value/versions/1.1.0 HTTP/1.1
Host: feature-store.example.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "schema": {
    "version": "1.1.0",
    "fields": [
      {
        "name": "user_id",
        "type": "string",
        "nullable": false
      },
      {
        "name": "avg_order_value",
        "type": "double",
        "nullable": true
      },
      {
        "name": "last_order_date",
        "type": "timestamp",
        "nullable": true
      }
    ]
  },
  "description": "Added last order date for user"
}

On success, the store updates the schema_versions table and marks 1.1.0 as the default version for the feature.

The store also maintains a feature_registry table with name, current_version, description, and created_at.

A failure: a client requests version=1.1.0, but the version 1.1.0 is not found in the schema_versions table. The store returns 404 Not Found with the error:

{
  "error": "version_not_found",
  "message": "Feature user_avg_order_value version 1.1.0 not found",
  "available_versions": ["1.0.0", "1.0.1"]
}

Monitoring and Failure Modes

The store exposes a /health endpoint for liveness and readiness.

GET /health HTTP/1.1
Host: feature-store.example.com
{
  "status": "healthy",
  "checks": {
    "database": {
      "status": "ok",
      "latency_ms": 12
    },
    "cache": {
      "status": "ok",
      "latency_ms": 3
    },
    "materializer": {
      "status": "ok",
      "running_jobs": 4,
      "last_heartbeat": "2024-08-10T08:55:00Z"
    }
  }
}

The store logs every feature materialization job to a feature_log table, with job_id, feature_name, status, duration, input_rows, output_rows, and error.

A common alert: materialization jobs that take longer than 5 minutes. The store sends materialization_too_long alerts with the job ID and a query to inspect the job’s logs.

Another failure: a feature is missing in the store. The client receives 404 Not Found with the error:

{
  "error": "feature_not_found",
  "message": "Feature user_avg_order_value not registered",
  "suggested_features": ["user_avg_order_value", "user_average_order_value", "user_avg_order"]
}

The store uses fuzzy matching on the feature name to suggest corrections.

Summary

Feature store design centers on consistency, versioning, and efficient retrieval. It relies on a well-defined schema for feature definitions, a predictable materialization pipeline, and a robust retrieval interface. The key patterns—time-travel, versioning, caching, and schema evolution—are implemented through standard HTTP, gRPC, SQL, and configuration mechanics. The system fails silently but predictably, with well-understood error codes and failure modes. The reader can apply this design to any platform, from cloud-native systems to on-premise data lakes.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.