ML Experiment Tracking

Manage reproducible experiments with MLflow, Weights and Biases, and DVC for versioning data, code, and model artifacts.

ML Experiment Tracking *Reference Page*

This is the process of recording, organizing, and comparing machine learning training runs to reproduce results, compare hyperparameters, and track model performance over time. It matters when you need to debug failed models, understand what configurations led to success, or audit how experiments evolved.

Recording Experiments

To record an experiment, log metrics, parameters, and artifacts during training. The standard format is a structured key-value store with timestamps. Use the following SQL schema for a basic tracking system:

CREATE TABLE experiment_runs (
    id UUID PRIMARY KEY,
    name TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    status TEXT,
    parameters JSONB,
    metrics JSONB,
    artifacts JSONB
);

Each run must have a unique identifier. Use UUIDs for consistency across systems.

Parameters are logged as key-value pairs:

{
  "learning_rate": 0.001,
  "batch_size": 32,
  "epochs": 100
}

Metrics include loss, accuracy, or any scalar values from training:

{
  "train_loss": 0.45,
  "val_accuracy": 0.87,
  "train_time_seconds": 1200
}

Artifacts are files or binary objects that can be stored in a blob store (e.g., S3, GCS). They may include model checkpoints, plots, or logs.

Starting an Experiment

Use a command-line interface to begin tracking. Example:

ml-experiment start --name my-model-training \
  --param learning_rate=0.001 \
  --param batch_size=32 \
  --param epochs=50

The system returns a UUID for the run.

If no --name is provided, use a generated name like experiment-<timestamp>.

If parameters are not passed, they default to empty JSON ({}).

Logging Metrics

After starting a run, log metrics periodically during training. Use this command:

ml-experiment log-metric --run-id <uuid> \
  --metric train_loss=0.35 \
  --metric val_accuracy=0.89

Metrics are stored as key-value pairs and can overwrite previous values with the same key.

If --run-id is missing, log to the last started run. If no run has been started, return an error:

Error: No active experiment. Start one first with 'ml-experiment start'.

Logging Artifacts

Artifacts are logged using a dedicated command:

ml-experiment log-artifact --run-id <uuid> \
  --path model-checkpoint.pth \
  --name checkpoint

If the artifact is large, upload it to a storage backend and store a reference in the database.

If --name is not provided, use the basename of the file. If the path does not exist, return an error:

Error: File 'model-checkpoint.pth' not found.

Listing Experiments

List all experiments with:

ml-experiment list

Output includes UUID, name, start time, and status:

ID                                 NAME                    CREATED_AT           STATUS
a1b2c3d4-e5f6-7890-abcd-ef1234567890 model-train-1           2025-04-01T10:00:00Z running
e0f1a2b3-c4d5-6789-0123-456789012345 model-train-2           2025-04-01T10:05:00Z completed

Comparing Experiments

Compare runs by parameters or metrics:

ml-experiment compare --run-id <uuid1> --run-id <uuid2>

Output shows differences in parameters and metrics side-by-side. If a parameter is missing from one run, it shows as null.

Retrieving Run Details

Use this to fetch full metadata for a run:

ml-experiment get --run-id <uuid>

Response includes all stored data:

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "model-train-1",
  "created_at": "2025-04-01T10:00:00Z",
  "status": "running",
  "parameters": {
    "learning_rate": 0.001,
    "batch_size": 32
  },
  "metrics": {
    "train_loss": 0.45,
    "val_accuracy": 0.87
  },
  "artifacts": [
    {
      "name": "checkpoint",
      "path": "/artifacts/a1b2c3d4-e5f6-7890-abcd-ef1234567890/checkpoint.pth"
    }
  ]
}

Stopping Experiments

Mark a run as complete:

ml-experiment finish --run-id <uuid>

If --run-id is not provided, finish the last started run.

If the run is already finished or does not exist, return an error:

Error: Run 'a1b2c3d4-e5f6-7890-abcd-ef1234567890' is not active.

Error Handling and Failure Modes

If ml-experiment is called without a subcommand, it returns:

Error: No command provided. Use 'ml-experiment start', 'log-metric', etc.

If a run ID is invalid (e.g., malformed UUID), return:

Error: Invalid run ID 'abc'. Must be a valid UUID.

If --metric or --param flags are passed without values, return:

Error: Metric or parameter must have a value (e.g., --metric loss=0.5).

If an artifact path is invalid or inaccessible, the system may silently skip it, or raise an error depending on configuration.

Rollback and Reproduction

To reproduce an experiment, re-use its parameters:

ml-experiment start --name reproduction-run \
  --param learning_rate=0.001 \
  --param batch_size=32 \
  --param epochs=50

Use the same run ID to ensure the system can compare results.

If an artifact is missing or corrupted, the tracking system does not stop logging. It logs a warning and continues:

Warning: Artifact 'checkpoint' not accessible. Continuing.

Data Consistency

All writes are atomic per run. If logging fails mid-run, partial data may be stored. To avoid inconsistencies, always call finish explicitly.

If the system crashes mid-logging, a run can be left in an inconsistent state (e.g., with no final metrics). Use a periodic cleanup job or manual check to identify such runs.

Versioning and Metadata

Run metadata includes a version field:

{
  "version": "1.0",
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "model-train-1",
  "created_at": "2025-04-01T10:00:00Z"
}

Version can be incremented manually or via a schema migration. The system does not enforce versioning automatically, so it must be handled by the client.

Configuration and Backends

The tracking system uses SQL as its default backend. If using a different storage engine (e.g., PostgreSQL, SQLite), the schema remains the same, but queries may vary.

For example, in PostgreSQL:

CREATE TABLE experiment_runs (
    id UUID PRIMARY KEY,
    name TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    status TEXT,
    parameters JSONB,
    metrics JSONB,
    artifacts JSONB
);

In SQLite, JSONB can be replaced with TEXT, and JSON parsing happens in the application layer.

If an artifact is stored in a blob store, log its URI:

{
  "name": "model",
  "path": "s3://bucket/models/model-123.pth"
}

Limitations

The system does not track code versions or environment details (e.g., Python packages). That must be done manually.

It does not support nested metrics or arrays of metrics. If you need to log multiple losses, use separate keys:

ml-experiment log-metric --run-id <uuid> \
  --metric train_loss=0.35 \
  --metric val_loss=0.41

If a run is not finished explicitly, it may be marked as running indefinitely.

Example Full Workflow

Start an experiment:

ml-experiment start --name lr-001-batch-32 --param learning_rate=0.001 --param batch_size=32

Log metrics during training:

ml-experiment log-metric --run-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 --metric train_loss=0.35 --metric val_accuracy=0.89

Log an artifact:

ml-experiment log-artifact --run-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 --path model-checkpoint.pth --name checkpoint

Finish the run:

ml-experiment finish --run-id a1b2c3d4-e5f6-7890-abcd-ef1234567890

Compare with another run:

ml-experiment compare --run-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 --run-id e0f1a2b3-c4d5-6789-0123-456789012345

Tools and Commands

The system is built on standard Unix conventions. The CLI is minimal, with no optional flags or complex subcommands beyond what's needed for tracking.

Commands include:

No configuration file is required. All settings are passed via command-line flags.

Common Pitfalls

  1. Not calling finish leads to runs that appear "running" forever.
  2. Logging metrics without a run ID causes errors.
  3. Artifacts not found during logging do not halt execution but may leave warnings.
  4. Reusing a run ID for multiple experiments overwrites previous data.
  5. Metrics are overwritten by new values with the same key.

Future Considerations

The system does not support experiment branching, parameter sweeps, or automated hyperparameter tuning out of the box. These features must be implemented at the client layer using the core tracking API.

Platforms in this space include tools that follow this pattern, but no specific product is defined by its configuration file or UI. The approach described here is generic and portable across implementations.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.