Fine-Tuning Pipelines | The Garnet Wiki

Fine-tuning pipelines are iterative workflows that adapt pre-trained models to domain-specific data through repeated training, validation, and deployment cycles. They matter when a model’s performance degrades in production due to data drift, distribution shift, or evolving user needs—when the gap between a model’s initial accuracy and its real-world effectiveness demands continuous adaptation. This page details the practical mechanics of building, operating, and refining pipelines that produce robust, up-to-date models, from initial configuration to failure diagnosis.

Defining the Pipeline

A fine-tuning pipeline orchestrates the end-to-end process of training a model on a dataset, validating it against a held-out set, and deploying the resulting artifact into a serving environment. The pipeline is typically triggered by a new dataset version, a model checkpoint, or a scheduled cadence. It consists of a sequence of stages: data preparation, training, evaluation, artifact packaging, and model deployment.

Each stage is a discrete unit of work, often implemented as a job or task in a workflow engine. The pipeline’s state is tracked via a central registry—such as a database, a file system, or a version control system—where each run is uniquely identified by a run ID, timestamp, and input metadata.

The pipeline’s core inputs are:

A base model (e.g., facebook/opt-1.3b, google/flan-t5-small)
A dataset (e.g., s3://datasets/finance/2024-06-15/)
A configuration file (config.yaml) defining training hyperparameters
A set of evaluation metrics (e.g., accuracy, f1, perplexity)

Configuration Structure

The pipeline expects a standard config.yaml format, with the following keys:

pipeline:
  name: "finance-text-classification"
  version: "v1.2"
  schedule: "0 3 * * *"  # Cron expression: daily at 3 AM
  dataset:
    source: "s3://datasets/finance/2024-06-15/"
    format: "parquet"
    split: "train"
  model:
    base: "google/flan-t5-small"
    output_dir: "models/finance/2024-06-15/"
    strategy: "deepspeed"
  training:
    epochs: 10
    batch_size: 16
    learning_rate: 1e-4
    gradient_accumulation_steps: 4
    weight_decay: 0.01
    warmup_steps: 500
    save_steps: 1000
    evaluation_strategy: "steps"
    eval_steps: 500
  evaluation:
    metric: "f1"
    threshold: 0.7
    test_dataset: "s3://datasets/finance/2024-06-15/test.parquet"
  deployment:
    target: "kubernetes"
    service_name: "text-classifier"
    replicas: 4
    rolling_update: true

This configuration is consumed by the pipeline orchestrator, which may be a cron job, a CI/CD system, or a dedicated workflow engine like Airflow or Prefect.

Running a Pipeline

The pipeline is executed via a command-line script that processes the configuration and drives the workflow:

./run_pipeline.sh --config config.yaml --run-id 20240615-123456

This command invokes a series of stages, each of which may fail independently and produce logs, artifacts, and status updates.

Stage 1: Data Preparation

The pipeline begins by loading the dataset from the configured source. It expects the data to be partitioned in a consistent format—typically Parquet or JSONL—and validates the schema against a known specification.

If the data is missing or malformed, the pipeline fails with a structured error:

{
  "error": "DataValidationFailed",
  "run_id": "20240615-123456",
  "stage": "data_preparation",
  "details": {
    "expected_columns": ["text", "label"],
    "missing_columns": ["text"],
    "sample_rows": [
      { "text": "The stock market opened higher today", "label": "equities" },
      { "text": "Interest rates are rising", "label": "macroeconomics" },
      { "text": "AI models are getting cheaper", "label": "technology" }
    ],
    "schema_mismatch": [
      { "column": "label", "expected_type": "string", "actual_type": "int32" }
    ]
  }
}

The pipeline can be configured to auto-correct common issues, such as:

Renaming columns via a rename_map in config.yaml
Converting column types (e.g., int32 → string)
Filtering out rows with missing values in critical columns

Stage 2: Training

The training stage uses Hugging Face Transformers with PyTorch. The pipeline invokes transformers.Trainer with the following command:

python train.py \
  --model_name_or_path google/flan-t5-small \
  --train_file s3://datasets/finance/2024-06-15/train.parquet \
  --validation_file s3://datasets/finance/2024-06-15/val.parquet \
  --output_dir models/finance/2024-06-15/ \
  --num_train_epochs 10 \
  --per_device_train_batch_size 16 \
  --gradient_accumulation_steps 4 \
  --learning_rate 1e-4 \
  --warmup_steps 500 \
  --weight_decay 0.01 \
  --evaluation_strategy steps \
  --eval_steps 500 \
  --save_steps 1000 \
  --load_best_model_at_end True \
  --metric_for_best_model f1 \
  --greater_is_better True

This command is executed in a Docker container that includes:

PyTorch 2.1
Transformers 4.40
Accelerate 0.24
Deepspeed 0.10 (for distributed training)

The training process logs metrics to a JSON file at logs/train_metrics.json:

[
  { "step": 1000, "loss": 1.847, "f1": 0.721, "eval_loss": 1.792 },
  { "step": 2000, "loss": 1.712, "f1": 0.745, "eval_loss": 1.723 },
  { "step": 3000, "loss": 1.623, "f1": 0.768, "eval_loss": 1.681 }
]

Silent failure: if eval_loss increases while f1 stabilizes, the pipeline may miss early overfitting. The --load_best_model_at_end flag ensures the final model is the one that achieved the best validation f1, but only if the evaluation is done at the same frequency as eval_steps.

Stage 3: Evaluation

After training completes, the pipeline runs a validation step using the test_dataset defined in config.yaml. It evaluates the final model on the test set and computes the metric specified in the configuration.

A failure occurs when the test set is missing or when the evaluation script returns an unexpected output format:

{
  "error": "EvaluationFailed",
  "run_id": "20240615-123456",
  "stage": "evaluation",
  "details": {
    "expected_metrics": ["f1", "accuracy", "precision", "recall"],
    "actual_metrics": {
      "f1": 0.768,
      "accuracy": 0.812,
      "precision": 0.743
    },
    "missing_metrics": ["recall"],
    "error_log": "Failed to compute recall: label mismatch in test dataset"
  }
}

The evaluation script must produce a results.json file in the model output directory:

{
  "run_id": "20240615-123456",
  "model": "google/flan-t5-small",
  "test_set": "s3://datasets/finance/2024-06-15/test.parquet",
  "metrics": {
    "f1": 0.768,
    "accuracy": 0.812,
    "precision": 0.743,
    "recall": 0.721
  },
  "confusion_matrix": [
    [123, 21],
    [18, 145]
  ]
}

If threshold is specified in the config.yaml, the evaluation script applies a threshold to the model’s output probabilities and recalculates metrics accordingly.

Stage 4: Artifact Packaging

Once evaluation passes, the pipeline packages the trained model into a standard artifact format. The output directory is archived into a .tar.gz file with a predictable name:

models/finance/2024-06-15/model-20240615-123456.tar.gz

The archive contains:

pytorch_model.bin — the trained weights
config.json — model configuration
tokenizer_config.json — tokenizer settings
vocab.txt — vocabulary file
model_card.md — metadata and training details
results.json — evaluation results

This artifact is uploaded to a model registry (e.g., a S3 bucket or a database table) with metadata:

INSERT INTO model_artifacts (
  run_id,
  model_name,
  version,
  model_path,
  created_at,
  metrics,
  dataset_version,
  training_config
) VALUES (
  '20240615-123456',
  'finance-text-classification',
  'v1.2',
  's3://models/finance/2024-06-15/model-20240615-123456.tar.gz',
  '2024-06-15T12:34:56Z',
  '{"f1": 0.768, "accuracy": 0.812}',
  '2024-06-15',
  '{"epochs": 10, "batch_size": 16, "learning_rate": 1e-4}'
);

Stage 5: Deployment

The final stage deploys the model to a serving environment. The pipeline updates a Kubernetes manifest or a Docker Compose file with the new model version and triggers a rolling update.

The deployment process uses kubectl apply with a template that replaces the image field in a Deployment resource:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: text-classifier
  labels:
    app: text-classifier
spec:
  replicas: 4
  selector:
    matchLabels:
      app: text-classifier
  template:
    metadata:
      labels:
        app: text-classifier
    spec:
      containers:
        - name: model-server
          image: ghcr.io/ai-models/text-classifier:20240615-123456
          ports:
            - containerPort: 8000
          env:
            - name: MODEL_PATH
              value: "/models/finance/2024-06-15/"

The pipeline verifies deployment success by polling the model’s health endpoint:

curl -v http://text-classifier:8000/health

A failure occurs if the response does not include "status": "healthy" or if the response time exceeds 500ms.

Common Pitfalls and Silent Failures

Missing test_dataset in config: The pipeline runs evaluation on the validation set but fails to report test set performance. The evaluation section may be ignored if the test_dataset key is misspelled as test_data or test_set.

Mismatched evaluation_strategy and eval_steps: If evaluation_strategy: "epoch" but eval_steps: 500, the pipeline evaluates every 500 steps but only saves the best model at the end of each epoch, leading to suboptimal model selection.

Incorrect save_steps: A save_steps value that is not divisible by eval_steps causes the model to be saved at different intervals than evaluation, leading to confusion when comparing model versions.

Missing model_card.md: The artifact is uploaded, but the model registry fails to display metadata because the model_card.md file is missing or contains invalid Markdown.

Rolling update without readiness check: The pipeline triggers a kubectl rollout restart but does not wait for all pods to become ready. A new model version is active while some pods are still booting, leading to inconsistent predictions.

Threshold application in evaluation but not in inference: The evaluation uses a threshold of 0.7, but the inference service uses 0.5. This leads to a mismatch between expected and actual model behavior.

Monitoring and Diagnosing Pipeline Failures

Pipeline runs are tracked in a central table. A failed run is identified by:

Exit code ≠ 0
Presence of error.json in the run directory
Missing model-*.tar.gz file
Incomplete or missing results.json

The pipeline logs all stage durations, resource usage (CPU, GPU, memory), and checkpoint intervals. A run that takes more than 1.5× the median duration is flagged as potentially slow.

Failure diagnosis begins with the run_id and proceeds through:

Check logs/run-20240615-123456.log for stage-level errors
Inspect artifacts/run-20240615-123456.tar.gz for model and data consistency
Query the model registry for metadata and metric trends
Compare the test_dataset against the train_dataset using statistical summaries

A common failure is a model that performs well on training data but poorly on test data—indicating overfitting, data leakage, or a mismatch in domain. The pipeline should include a data_analysis stage that computes and logs feature statistics (mean, variance, skew) for both datasets.

Iterating the Pipeline

The pipeline is not a one-off process. Each run produces a new model version that can be used as the base for the next run. The pipeline can be configured to:

Automatically retrain when a new dataset version is available
Trigger a new run on model performance drop (e.g., f1 < threshold)
Compare two runs side-by-side using a compare_runs.sh script that computes delta metrics and visualizes confusion matrices

A new run may inherit the configuration from a previous run, with overrides in a overrides.yaml file:

training:
  epochs: 12
  learning_rate: 5e-5
  warmup_steps: 1000
evaluation:
  threshold: 0.75

This enables rapid experimentation and continuous improvement without manual reconfiguration.

Fine-tuning pipelines are the engine of model evolution. Their success depends on precise configuration, consistent data, and careful monitoring. When they break, they do so in ways that reveal deeper issues in data, model, or deployment—each failure a clue to a larger system.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.