Ai Experiment Tracking | The Garnet Wiki

Ai Experiment Tracking is the practice of systematically logging, comparing, and managing the parameters, code, data, metrics, and artifacts of machine learning experiments. It matters when training models across multiple configurations, evaluating new features, or iterating on model architectures—especially when teams run dozens of experiments in parallel, each with distinct hyperparameters, datasets, and evaluation scores.

Defining and Running an Experiment

An experiment is a single training run of a model with a specific set of inputs: code version, hyperparameters, data splits, and environment. To define an experiment, a system must capture both *what* was run and *how* it was run. The core artifact is a experiment.json file, stored at the root of the experiment’s output directory, with fields:

{
  "experiment_id": "exp-2024-05-17-123456",
  "name": "ResNet50 on CIFAR-10 with dropout",
  "project": "image-classification",
  "code_version": "a3b5c7d",
  "start_time": "2024-05-17T12:34:56Z",
  "end_time": "2024-05-17T13:45:22Z",
  "parameters": {
    "batch_size": 64,
    "learning_rate": 0.001,
    "dropout_rate": 0.3,
    "optimizer": "Adam",
    "epochs": 100
  },
  "data_version": "data-v1.2.0",
  "environment": {
    "python_version": "3.11.7",
    "package_versions": {
      "torch": "2.2.0",
      "scikit-learn": "1.5.0",
      "numpy": "1.26.0"
    }
  }
}

The experiment is initiated by a command that registers the run in a central registry. This command is typically invoked by a train.py script or a batch job:

python train.py \
  --data-root /data/cifar10 \
  --epochs 100 \
  --batch-size 64 \
  --learning-rate 0.001 \
  --dropout-rate 0.3 \
  --output-dir /experiments/exp-2024-05-17-123456 \
  --log-to-tracker

The --log-to-tracker flag triggers the upload of the experiment.json and all outputs to a tracking service. The tracking service responds with an experiment_id, which is used to later query or update the experiment.

Logging Metrics and Artifacts

Metrics are logged during training, typically in a metrics.csv file, but also via API calls. Each metric is a key-value pair logged at a timestamp, with the possibility of multiple metrics per step.

step,loss,accuracy,val_loss,val_accuracy
100,0.912,0.721,0.884,0.743
200,0.834,0.773,0.812,0.789
300,0.765,0.812,0.751,0.820

A service can ingest these metrics via HTTP POST to /api/v1/experiments/{id}/metrics, with a JSON body:

[
  {
    "step": 400,
    "timestamp": "2024-05-17T13:44:10Z",
    "metrics": {
      "loss": 0.712,
      "accuracy": 0.835,
      "val_loss": 0.701,
      "val_accuracy": 0.842
    }
  },
  {
    "step": 500,
    "timestamp": "2024-05-17T13:44:25Z",
    "metrics": {
      "loss": 0.687,
      "accuracy": 0.851,
      "val_loss": 0.683,
      "val_accuracy": 0.859
    }
  }
]

Artifacts—model weights, evaluation reports, confusion matrices, plots—are stored as files and linked to the experiment. The system uses a artifacts/ directory within the experiment output, with subdirectories for each type:

/experiments/exp-2024-05-17-123456/
├── artifacts/
│   ├── models/
│   │   └── best_model.pth
│   ├── plots/
│   │   ├── confusion_matrix.png
│   │   └── learning_curve.png
│   └── reports/
│       └── classification_report.json
├── experiment.json
└── metrics.csv

A registry service maintains a mapping from experiment ID to artifact paths. When a new artifact is written, a POST /api/v1/experiments/{id}/artifacts call is made with a JSON body:

{
  "type": "model",
  "path": "artifacts/models/best_model.pth",
  "metadata": {
    "checkpoint_step": 500,
    "val_accuracy": 0.859,
    "file_size_bytes": 107374182
  }
}

Querying and Comparing Experiments

Experiments are compared by querying a central database. The query interface is a REST API with common endpoints:

GET /api/v1/experiments?project=image-classification&status=completed
GET /api/v1/experiments/{id}/metrics?start_step=100&end_step=500
GET /api/v1/experiments/{id}/artifacts?type=model

Results are returned as JSON arrays with consistent structure:

[
  {
    "experiment_id": "exp-2024-05-17-123456",
    "name": "ResNet50 on CIFAR-10 with dropout",
    "project": "image-classification",
    "start_time": "2024-05-17T12:34:56Z",
    "end_time": "2024-05-17T13:45:22Z",
    "status": "completed",
    "parameters": {
      "batch_size": 64,
      "learning_rate": 0.001,
      "dropout_rate": 0.3,
      "optimizer": "Adam"
    },
    "metrics": {
      "final_accuracy": 0.859,
      "final_val_accuracy": 0.842,
      "training_time_seconds": 4226
    },
    "artifacts": [
      {
        "type": "model",
        "path": "artifacts/models/best_model.pth",
        "metadata": {
          "checkpoint_step": 500,
          "val_accuracy": 0.859
        }
      },
      {
        "type": "plot",
        "path": "artifacts/plots/learning_curve.png"
      }
    ]
  }
]

A common pattern for comparing experiments is to fetch all experiments in a project with their final accuracy and training time, then sort by accuracy and overlay plots.

Sub-Section: Visual Comparison

A user might query all experiments in a project and generate a comparison table:

curl -X GET 'http://tracker.example.com/api/v1/experiments?project=image-classification&status=completed' \
  -H 'Accept: application/json' \
  -o experiments.json

Then use jq to extract key fields:

jq -r '.[] | [.name, .parameters.dropout_rate, .metrics.final_accuracy, .metrics.training_time_seconds] | @tsv' experiments.json

This produces tabular output:

ResNet50 on CIFAR-10 with dropout	0.3	0.859	4226
ResNet50 on CIFAR-10 with dropout	0.5	0.841	4210
ResNet50 on CIFAR-10 with dropout	0.2	0.837	4180

This table reveals that a dropout rate of 0.3 yields the best accuracy, even though the training time is similar across runs.

Tracking Configuration and Pipeline State

Experiments are often part of a larger pipeline. The configuration of the pipeline itself—data preprocessing, feature extraction, model training—must be tracked as part of the experiment. A pipeline.yaml file defines the steps:

name: "Image Classification Pipeline"
stages:
  - name: "data-prep"
    image: "data-prep:v1.2"
    command: ["python", "preprocess.py", "--input", "/data/raw", "--output", "/data/processed"]
    inputs:
      - data_version: "data-v1.2.0"
    outputs:
      - artifact: "processed_data.tar.gz"
  - name: "model-training"
    image: "train-model:v2.1"
    command: ["python", "train.py", "--epochs", "100"]
    inputs:
      - data_version: "data-v1.2.0"
      - model: "baseline-model.pth"
    outputs:
      - artifact: "best_model.pth"
    parameters:
      batch_size: 64
      dropout_rate: 0.3

When an experiment is created, the pipeline configuration is attached to it. The tracking system stores the pipeline.yaml and links it to the experiment via a pipeline_version field in experiment.json.

The system can validate that a given experiment matches a known pipeline version by comparing the pipeline.yaml hash to a stored pipeline_version in the database. If the pipeline configuration drifts—e.g., a new preprocess.py script is added—the experiment’s pipeline_version may not match the expected one, leading to a pipeline_mismatch warning.

Failure Modes and Common Pitfalls

Silent Failure: Missing Metrics

If a training script logs metrics to a metrics.json file but forgets to upload it, the tracking system only sees the experiment.json and no metrics. This results in experiments with final_accuracy: null and no learning curves. The root cause is a missing POST /api/v1/experiments/{id}/metrics call.

Confusion: `step` vs `epoch`

Many scripts log metrics per epoch, but the tracking system expects step as the primary dimension. If a training loop logs:

[
  { "epoch": 1, "accuracy": 0.721 },
  { "epoch": 2, "accuracy": 0.773 }
]

but the tracking service expects step, the system assumes step 1 is epoch 1, step 2 is epoch 2, and so on. When the batch size is 64 and there are 50,000 training examples, one epoch is 781 steps. A model that trained for 10 epochs logs 10 entries, but the tracking system interprets this as 10 steps, not 10 epochs.

To resolve this, the tracking system must be told the batch_size and train_size at experiment setup. The metrics.json file should include step or epoch with a step field for every metric entry.

Confusion: Artifact Path vs Artifact ID

Artifacts are stored as files, but the system uses a logical artifact_id to reference them. If an experiment uploads artifacts/models/best_model.pth, the system may store it as model-abc123 and register the path as artifacts/models/best_model.pth. But when a new experiment uses the same model file, the tracking system does not recognize it as the same artifact unless the artifact_id matches.

This leads to duplicate models in the registry. The solution is to use a content-based hash (e.g., SHA-256) of the file as the artifact_id, and to store the path as metadata.

Silent Failure: Out-of-Order Metrics

When a training process runs on multiple GPUs or in a distributed setup, metrics are logged from different processes. If the timestamp field is not set or is inaccurate, the tracking system can receive metrics in an out-of-order sequence.

For example:

[
  { "step": 100, "timestamp": "2024-05-17T13:44:10Z", "accuracy": 0.721 },
  { "step": 200, "timestamp": "2024-05-17T13:44:08Z", "accuracy": 0.773 }
]

The second metric arrives before the first, but step 200 is logged earlier than step 100. The tracking system may then display a learning curve that drops between steps 100 and 200, even though accuracy improved.

The fix is to require that timestamp be accurate to within 100ms and that all metrics be sorted by timestamp before ingestion.

Managing Experiment Lifecycles

Experiments are not static. They evolve through states: created, running, completed, failed, archived. A PUT /api/v1/experiments/{id}/status call updates the state:

{
  "status": "running",
  "message": "Training step 350 of 500"
}

A monitoring process checks for experiments that have been running for more than 24 hours without a status update. If no status update is received in the last 24 hours, the experiment is marked as stalled.

When a model is selected as the best, its is_best flag is set:

PUT /api/v1/experiments/exp-2024-05-17-123456
{
  "is_best": true,
  "selected_by": "accuracy",
  "selected_at": "2024-05-17T13:45:22Z"
}

A separate service can then promote the best model to production by copying the model artifact to a models/ directory and updating a current_model.json file.

Versioning and Reproducibility

Reproducibility is ensured by linking every experiment to its code version, data version, and environment. The tracking system maintains a registry of:

code_versions (Git commits, SHA-1s)
data_versions (dataset hashes, version strings)
environment_versions (package manifests, lock files)

When an experiment is created, it is linked to one entry in each registry. The system can then:

Reconstruct the full environment from code_version and environment_version.
Recreate the data pipeline using data_version.
Replay the experiment by checking out the code_version, installing environment_version, and running the pipeline.yaml.

A common error is a data_version mismatch: the experiment was trained on data version data-v1.2.0, but the evaluation script uses data-v1.1.0. The tracking system detects this via a data_version field in experiment.json and issues a data_version_mismatch alert.

Summary

Ai Experiment Tracking is the foundation of iterative machine learning. It captures the full context of a run: code, data, parameters, metrics, and artifacts. It enables comparison, reproducibility, and continuous improvement. The key is consistency: every experiment must be registered with the same structure, and every metric and artifact must be linked to a canonical experiment_id. When the system breaks, the failure is not in the model— it is in the tracking.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.