Model Registry Design

Production-grade guide to model registry design covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

Model registry design is the engineering discipline of tracking, versioning, promoting, and auditing machine learning models through their lifecycle. It matters when a team runs multiple model iterations, needs reproducible promotions to production, or must satisfy compliance and audit requirements. Without deliberate registry design, model promotion becomes ad hoc, reproducibility collapses, and incident response requires guessing which version is running where.

Versioning and Lineage

Every model artifact—code, weights, metadata, and data checksums—should be addressed by a content-addressable identifier rather than a mutable name. The most portable pattern stores the model bundle in an object store (S3, GCS, Azure Blob) and records a provenance entry in a relational database. Git can track the container image and pip requirements that produced a given bundle, while the database tracks the mapping from model ID to storage location, framework version, and training dataset hash.

Concrete pattern:

# Store the model bundle; sha256 of the zipped artifacts becomes the model ID
tar czf model-bundle.tgz model.pkl requirements.txt
aws s3 cp model-bundle.tgz s3://model-registry/bundles/sha256-$(sha256sum model-bundle.tgz | cut -d' ' -f1)/model-bundle.tgz

# Record provenance in PostgreSQL; the table is schema-versioned and append-only
psql "dbname=model_registry" -c \
  "INSERT INTO model_versions (model_id, storage_uri, framework, framework_version, data_hash, committed_by, committed_at) \
   VALUES ('sha256-$(sha256sum model-bundle.tgz | cut -d' ' -f1)', \
   's3://model-registry/bundles/sha256-$(sha256sum model-bundle.tgz | cut -d' ' -f1)/model-bundle.tgz', \
   'python', '3.11', 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0', \
   '$(python -c "import hashlib; print(hashlib.sha256(open('training-data.parquet','rb').read()).hexdigest())")', \
   '$USER', now());"

A common silently-failing scenario: the data_hash field is computed from a snapshot that itself changes between runs (e.g., randomly shuffled rows). Two ostensibly identical models receive different hashes, lineage queries return empty sets, and downstream consumers cannot locate the "correct" version. The fix is to lock the data preprocessing pipeline version alongside the model, or hash only the features that are invariant to shuffling.

Stage Management

Models progress through states such as None (unregistered), Staging, Production, and Archived. Transitions should be explicit operations, not implicit side effects of a prediction request. The pattern is a state machine with guarded transitions: Staging → Production typically requires a sign-off or automated validation; Production → Archived occurs when a successor model is promoted.

Concrete pattern using a REST API endpoint that exists in most registries:

# Promote model from Staging to Production; the body specifies the target stage
curl -X POST "https://registry.example.com/api/model-versions/sha256-1a2b3c/transition" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"target_stage": "Production", "promoter": "ci-cd-pipeline", "reason": "auc > 0.95 on holdout set"}'

The request is rejected with 409 Conflict if another transition is in flight for the same model ID, preventing race conditions where two CI runs promote different versions simultaneously. A frequently confused option is the auto_archive flag: setting it to true archives the previous Production version automatically, but if the flag is omitted the old version remains in Production alongside the new one, inflating storage costs and obscuring which version actually serves traffic.

Search and Discovery

Production environments typically need to locate models by criteria such as framework, training date range, metric thresholds, or data provenance. Since the registry database is append-only, queries use standard SQL WHERE clauses on indexed columns. A typical discovery pattern:

# Find all XGBoost models trained after 2024-01-01 with recall > 0.85
psql "dbname=model_registry" -c \
  "SELECT model_id, trained_at, framework, recall_score \
   FROM model_versions \
   WHERE framework = 'xgboost' \
     AND trained_at > '2024-01-01' \
     AND recall_score > 0.85 \
   ORDER BY trained_at DESC \
   LIMIT 20;"

If the registry stores metadata as a JSONB column (common in PostgreSQL deployments), the ->> operator extracts keys for filtering:

# Find models where the "environment" tag equals "production"
psql "dbname=model_registry" -c \
  "SELECT model_id, tags->>'environment' AS env \
   FROM model_versions \
   WHERE tags->>'environment' = 'production';"

A sharp edge: index bloat. When the tags JSONB column grows to thousands of keys per row, ->> queries become sequential scans despite a GIN index. The pattern to learn after it breaks in production is to normalize frequent tag keys into dedicated columns (framework, dataset_id, owner) and keep sparse or ad‑hoc tags in the JSONB column. Another quietly failing case: models registered without any tags cannot be filtered by tag queries, returning zero results even when models exist that match other criteria.

Audit and Compliance

Audit trails require that every state transition, metadata update, and artifact upload be immutable and time-stamped. The design pattern is an append-only log—either a database table with INSERT ONLY privileges, or an external append‑only store such as Apache Kafka or a log‑structured merge‑tree. Each event carries: actor, timestamp, target model ID, action, and a checksum of the preceding event to detect tampering.

Concrete pattern using PostgreSQL row security:

-- Table is created with a trigger that records the actor from JWT claims
CREATE TABLE model_audit (
  id BIGSERIAL PRIMARY KEY,
  model_id UUID NOT NULL,
  action TEXT NOT NULL CHECK (action IN ('CREATE', 'TRANSITION', 'UPDATE', 'DELETE')),
  actor TEXT NOT NULL,
  actor_claims JSONB NOT NULL,
  previous_state TEXT,
  new_state TEXT,
  timestamp TIMESTAMPTZ NOT NULL DEFAULT now(),
  event_hash UUID NOT NULL GENERATED ALWAYS AS (
    md5(concat_ws('|', model_id, action, actor, to_char(timestamp, 'YYYY-MM-DD\"T\"HH24:MI:SS'), previous_state, new_state))
  ) STORED
);

-- Row‑level security restricts readers to compliance team and model owners
ALTER TABLE model_audit ENABLE ROW LEVEL SECURITY;
CREATE POLICY compliance_view ON model_audit
  USING (has_role('compliance_team') OR actor = current_user);

A silently failing mechanism: if the JWT middleware drops the actor claim or maps it to a generic service account, every audit row records the same actor, destroying forensic value. The pattern to learn after it breaks in production is to validate the JWT signature and extract the sub claim at the registry API gateway before any database write, and to reject writes where the actor is a service account unless the action is CREATE of a new model version.

Promotion Workflows

Promotion from Staging to Production is often gated by automated validation (metrics on a held‑out set, schema compatibility checks, container image digest verification) and optionally by human sign‑off. The workflow pattern separates concern: a CI pipeline runs validation jobs and emits a pass/fail status; a separate approval service holds the promotion in a Pending sub‑state until an authorized user transitions it to Production.

Concrete pattern using webhook‑driven CI:

# CI pipeline step: evaluate model on holdout set, exit 0 if auc >= 0.92
python evaluate.py \
  --model s3://model-registry/bundles/sha256-1a2b3c/model.pkl \
  --data s3://holdout-set/features.parquet \
  --metric auc \
  --threshold 0.92

# If the step passes, post a status update that the promotion service watches
curl -X POST "https://ci.example.com/api/jobs/model-eval/sha256-1a2b3c/status" \
  -H "Authorization: Bearer $CI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "pass", "metric": "auc", "value": 0.942, "commit": "a1b2c3d4"}'

The promotion service, receiving the webhook, calls the transition endpoint:

curl -X POST "https://registry.example.com/api/model-versions/sha256-1a2b3c/transition" \
  -H "Authorization: Bearer $REGISTRY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"target_stage": "Production", "promoter": "ci-cd-pipeline", "reason": "auc >= 0.92 on holdout set"}'

A confused‑for‑other‑option pattern: the wait_for_approval flag defaults to false. When true, the transition API returns 202 Accepted and enqueues the request; a separate GET /model-versions/sha256-1a2b3c/status poll reveals PENDING_APPROVAL. Omitting the flag causes the transition to block the calling CLI until the state changes, which can hang if no one approves the promotion. The sharp edge learned after production incidents: always set wait_for_approval explicitly in automation, and add a timeout webhook that escalates unapproved promotions after 48 hours.

Common Failure Modes Learned in Production

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.