Recommendation Systems

Recommendation systems are core components of user-facing platforms, delivering personalized content, products, or services based on user behavior, item attributes, and contextual signals. They matter when a platform must surface relevant items from a large pool—e.g., movies in a streaming service, products in an e-commerce site, or posts in a social feed—where relevance significantly impacts engagement, retention, and conversion.

Core Design Goals

Every recommendation system balances accuracy, latency, scalability, and freshness. The primary goal is to compute a ranked list of items for each user, ideally in under 200ms, with minimal computational cost and high throughput. A system that recommends items from a catalog of 10 million items to 1 million active users per day must handle both cold-start problems (new users, new items) and real-time user interactions.

The core pipeline involves:

• Data ingestion: raw interaction logs (clicks, views, purchases) and metadata (item titles, categories, user profiles).
• Feature engineering: transforming raw data into numerical vectors used by models.
• Model training: learning user-item preferences from historical data.
• Online serving: fetching recommendations for active users with low latency.
• Evaluation: measuring relevance and coverage over time.

Feature Engineering

Features are the foundation of a recommendation model. They must capture user behavior, item characteristics, and context. For example, a user’s recent interactions with items in a specific category should be encoded as a weighted vector of item IDs, with decay over time.

A typical feature table is stored as a Parquet file in a distributed filesystem, with the schema:

{
  "user_id": "string",
  "item_id": "string",
  "timestamp": "timestamp",
  "event_type": "string",  // "click", "view", "purchase"
  "category_id": "string",
  "session_id": "string",
  "duration_seconds": "float"
}

Features are computed using a batch pipeline. A standard approach uses dask or Spark to aggregate per-user and per-item statistics:

from pyspark.sql import functions as F
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FeatureEngineering").getOrCreate()

events = spark.read.parquet("/data/events/2024-04-01")


user_clicks = (
    events
    .filter(F.col("event_type") == "click")
    .groupBy("user_id")
    .agg(F.count("item_id").alias("total_clicks"))
)

# Compute per-item view-to-purchase ratio
item_conversion = (
    events
    .filter(F.col("event_type").isin(["view", "purchase"]))
    .groupBy("item_id")
    .agg(
        F.count(F.when(F.col("event_type") == "view", 1)).alias("views"),
        F.count(F.when(F.col("event_type") == "purchase", 1)).alias("purchases")
    )
    .withColumn("conversion_rate", F.col("purchases") / F.col("views"))
)

# Join features into a final dataset
final_features = (
    user_clicks
    .join(item_conversion, on="item_id")
    .repartition(100)
    .write.parquet("/data/features/2024-04-01")
)

The repartition(100) step is critical: too few partitions lead to underutilized compute, while too many cause excessive overhead in serving. A common failure mode is skewed partitions due to high-traffic users, leading to hot shards during online inference.

Model Training

Recommendation models are trained on historical feature data. The most common approach is to use matrix factorization or deep learning models trained on interaction matrices.

For a collaborative filtering model, the training data is a sparse matrix where rows are users, columns are items, and entries are interaction scores (e.g., 1 for click, 2 for view, 3 for purchase). A standard training loop uses PyTorch with torch.utils.data.DataLoader:

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

class InteractionDataset(Dataset):
    def __init__(self, user_ids, item_ids, scores, num_users, num_items):
        self.user_ids = user_ids
        self.item_ids = item_ids
        self.scores = scores
        self.num_users = num_users
        self.num_items = num_items

    def __len__(self):
        return len(self.user_ids)

    def __getitem__(self, idx):
        return {
            "user_id": torch.tensor(self.user_ids[idx], dtype=torch.long),
            "item_id": torch.tensor(self.item_ids[idx], dtype=torch.long),
            "score": torch.tensor(self.scores[idx], dtype=torch.float)
        }

# Training loop
model = nn.Embedding(num_users, embedding_dim=64)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.MSELoss()

dataset = InteractionDataset(user_ids, item_ids, scores, num_users=1000000, num_items=500000)
dataloader = DataLoader(dataset, batch_size=256, shuffle=True)

for epoch in range(10):
    for batch in dataloader:
        user_emb = model(batch["user_id"])
        item_emb = model(batch["item_id"])
        pred = (user_emb * item_emb).sum(dim=1)
        loss = criterion(pred, batch["score"])
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

The choice of embedding dimension (e.g., 64, 128, 256) is often a trade-off between accuracy and memory. A common failure is using the same embedding size for all features, leading to overfitting for low-frequency users and underfitting for high-frequency ones.

Online Serving

Recommendations are served in real time via a REST API. A typical endpoint is:

GET /recommendations?user_id=12345&limit=20&context=homepage

The server returns a list of item IDs with scores:

{
  "user_id": "12345",
  "timestamp": "2024-04-01T12:34:56Z",
  "items": [
    { "item_id": "789", "score": 0.93, "reason": "collaborative_filtering" },
    { "item_id": "456", "score": 0.87, "reason": "content_based" },
    { "item_id": "101", "score": 0.79, "reason": "trending" }
  ]
}

The serving system uses a candidate generation and ranking pipeline:

1. Candidate generation: retrieve a set of candidate items for the user using fast lookups (e.g., user-item co-occurrence, popularity, or a precomputed item-to-item graph).
2. Ranking: apply a learned model to re-rank the candidates.

For candidate generation, a simple implementation uses a precomputed user_item_matrix stored in a Redis cluster:

# Generate candidate items for user 12345
redis-cli HGETALL user:12345:top_items

Response:

{
  "123": 0.87,
  "456": 0.79,
  "789": 0.72,
  "321": 0.65,
  "555": 0.58
}

The ranking model is served as a gRPC service. A client sends a request with user and candidate item features:

message RankingRequest {
  string user_id = 1;
  repeated string item_ids = 2;
  map<string, float> user_features = 3;
  map<string, float> item_features = 4;
}

message RankingResponse {
  repeated RankedItem items = 1;
}

message RankedItem {
  string item_id = 1;
  float score = 2;
  string reason = 3;
}

The model runs on a TensorFlow Serving instance:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "instances": [
      {
        "user_id": "12345",
        "item_ids": ["789", "456", "101"],
        "user_features": { "total_clicks": 24, "avg_duration": 45.3 },
        "item_features": { "category_popularity": 0.81, "views_last_7d": 123 }
      }
    ]
  }' \
  http://tf-serving:8501/v1/models/recommendation:predict

The TensorFlow Serving model must be versioned and served with mlflow or KFServing. A common failure is incorrect model versioning: a new model is deployed but not promoted to latest, leading to inconsistent predictions across the fleet.

Evaluation and Monitoring

Recommendation systems must be evaluated continuously. The primary metrics are:

• Precision@K: proportion of recommended items that are relevant.
• Recall@K: proportion of relevant items that appear in recommendations.
• Mean Average Precision (MAP): average precision across multiple users.
• NDCG@K: normalized discounted cumulative gain.

Evaluation is performed offline on a held-out dataset. A standard command uses scikit-learn and pandas:

from sklearn.metrics import precision_at_k, recall_at_k, ndcg_score
import pandas as pd

# Load test data: user_id, actual_items, predicted_items
test_data = pd.read_parquet("/data/test/2024-04-01")

# Compute metrics
results = []
for _, row in test_data.iterrows():
    user_id = row["user_id"]
    actual = set(row["actual_items"])
    predicted = row["predicted_items"]
    k = len(predicted)

    precision = precision_at_k(actual, predicted, k=k)
    recall = recall_at_k(actual, predicted, k=k)
    ndcg = ndcg_score([list(actual)], [predicted], k=k)

    results.append({
        "user_id": user_id,
        "precision": precision,
        "recall": recall,
        "ndcg": ndcg
    })

# Aggregate metrics
avg_precision = sum(r["precision"] for r in results) / len(results)
avg_recall = sum(r["recall"] for r in results) / len(results)
avg_ndcg = sum(r["ndcg"] for r in results) / len(results)

The k value is often misconfigured: k=10 is common, but user behavior varies—mobile users prefer shorter lists, desktop users longer ones. A silent failure is using the same k across all contexts (homepage, search, email) without tuning.

Failure Modes

• Cold start: new users and new items receive low-quality recommendations. The system may fall back to popularity or content-based models, but the transition is often abrupt. A new user sees only top-10 items, even though they have no interaction history.
• Data drift: features become outdated. A user’s click history from a week ago is not updated in real time, leading to stale recommendations. The feature pipeline may fail silently: logs show data ingestion, but the model sees no updates for 48 hours.
• Model drift: the model's predictions shift over time. A batch feature pipeline runs every 6 hours, but the model is trained on data from 24 hours ago. The serving system uses a model version that is not aligned with the latest feature version.
• Latency spikes: the /recommendations endpoint times out under load. A common cause is a misconfigured gRPC connection: max_message_length is too small for large feature sets, leading to GRPC_ERROR: 8 RESOURCE_EXHAUSTED errors.
• Missing features: a required feature is absent in the user_features map. The model fails to predict, returning null scores. The error is logged, but the cause is not surfaced—feature missing: avg_duration—but the feature pipeline does not validate required fields.

Key Trade-offs

• Batch vs. real-time feature updates: batch pipelines are cheaper but introduce latency. Real-time updates require stream processing with Kafka or Flink, but increase complexity and failure surface.
• Candidate size vs. ranking cost: using 1000 candidates per user requires more storage and bandwidth, but reduces the risk of missing relevant items. The ranking model must be fast enough to process 1000 items in under 100ms.
• Model complexity vs. serving cost: a deep model with attention layers is accurate but requires more memory and compute. A simple linear model is fast but less expressive.

Summary

A recommendation system is not a single model but a coordinated pipeline of data, features, models, and services. The success of the system depends on alignment across training, feature generation, serving, and evaluation. The most common failure is misalignment: a model trained on yesterday’s features, served with today’s data, evaluated on next week’s test set. The system works in isolation, but fails when the pieces are not synchronized.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.