Self Supervised Learning | The Garnet Wiki

Self Supervised Learning

Self supervised learning trains models on data without human-labeled examples by creating pretext tasks from the data itself. It matters when labeled data is expensive or scarce, especially in domains like natural language processing, computer vision, and speech recognition where unlabeled data is abundant.

Pretext Tasks and Data Preparation

Self supervised learning relies on designing pretext tasks that extract meaningful representations from raw data. The most common approach uses contrastive learning where the model learns to distinguish between similar and dissimilar data points.

For text data, a typical pretext task involves predicting which words are missing from a sentence or reconstructing corrupted text. For images, models might predict rotation angles, patch order, or colorization.

A minimal example of this in PyTorch looks like:

import torch
import torch.nn as nn

class ContrastiveModel(nn.Module):
    def __init__(self, encoder):
        super().__init__()
        self.encoder = encoder

    def forward(self, x1, x2):
        z1 = self.encoder(x1)
        z2 = self.encoder(x2)
        return z1, z2


model = ContrastiveModel(encoder=MyVisionTransformer())

The key configuration parameter is the loss function used for training. Common choices include InfoNCE loss and contrastive loss.

Training Configuration

Training typically requires careful balance of hyperparameters including learning rate, batch size, and temperature scaling.

Example Training Loop

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

for epoch in range(100):
    for batch in dataloader:
        optimizer.zero_grad()
        z1, z2 = model(batch['x1'], batch['x2'])
        loss = criterion(z1, z2)
        loss.backward()
        optimizer.step()

The batch size must be large enough to support contrastive learning but small enough to fit in GPU memory. Typical values range from 64 to 512 depending on the model and hardware.

A common failure mode occurs when the temperature parameter is set too high, leading to vanishing gradients. The default value of 0.1 usually works well for most pretext tasks.

Data Augmentation Techniques

Data augmentation creates multiple views of the same sample for contrastive learning. For images, this may include random cropping, color jittering, or Gaussian noise addition.

For text, augmentation might involve synonym replacement, back translation, or sentence shuffling.

Example Augmentation Pipeline

import torchvision.transforms as transforms

transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),
    transforms.ToTensor()
])

The augmentation strategy should preserve semantic meaning while introducing enough variation for the model to learn meaningful representations.

Model Architectures

Self supervised learning works with various architectures including transformers, CNNs, and hybrid models. The encoder must be designed to output embeddings that capture semantic features.

In vision tasks, models often use ResNet or Vision Transformer backbones. For language, BERT or similar transformer-based encoders are common.

The encoder's output dimension should match the pretext task requirements. For instance, if using InfoNCE loss, the embedding size must be compatible with the contrastive batch dimensions.

Evaluation and Transfer Learning

Once trained, self supervised models can be fine-tuned on downstream tasks. The evaluation metric depends on the original task type.

For image classification, accuracy on a held-out dataset is standard. For NLP, metrics like perplexity or F1 score may apply.

Example Fine-Tuning

# Freeze pre-trained encoder weights
for param in model.encoder.parameters():
    param.requires_grad = False

# Add new classification head
classifier = nn.Linear(embedding_dim, num_classes)
model.classifier = classifier

A critical failure point is forgetting to freeze the encoder weights during fine-tuning. This leads to overfitting and poor generalization.

Common Pitfalls and Failure Modes

Incorrect Loss Scaling

The InfoNCE loss function requires proper normalization. Without correct scaling, gradients become unstable or vanish.

# Incorrect
loss = torch.nn.functional.cross_entropy(z1, z2)

# Correct
logits = torch.matmul(z1, z2.T) / temperature
loss = torch.nn.functional.cross_entropy(logits, labels)

Insufficient Data Diversity

Training with homogeneous data limits the model's ability to generalize. This manifests as poor performance on diverse datasets.

Overfitting to Pretext Task

Models may memorize the pretext task rather than learning useful representations. Validation on downstream tasks reveals this failure.

Configuration Parameters

Key configuration parameters include:

temperature: Controls softmax sharpness, typically 0.1
batch_size: Influences gradient stability, usually 64–512
learning_rate: Often 0.001 for Adam optimizer
num_epochs: Typically 50–200 depending on dataset size

Hardware and Memory Considerations

Self supervised training is memory-intensive due to the need to store multiple views of data. GPU memory usage scales with batch size and model complexity.

For large datasets, distributed training across multiple GPUs or nodes may be necessary. The key challenge is maintaining consistent contrastive pairs across workers.

Monitoring Training Progress

Loss values should decrease steadily during training. Sudden spikes or plateaus indicate problems. Common metrics include:

Contrastive loss
Accuracy on pretext task
Embedding similarity metrics

A failing model often shows loss plateaus before a sharp drop, indicating learning has stagnated. This usually requires adjusting the learning rate or batch size.

Data Requirements

Self supervised learning benefits from large volumes of unlabeled data. The quality and diversity of this data directly impacts final performance.

Training datasets should cover the full range of expected input variations. For images, this means diverse lighting, angles, and backgrounds. For text, it means varied writing styles and domains.

Implementation Patterns

Standard Training Loop Structure

Define encoder architecture
Create pretext task loss function
Set up optimizer and learning rate schedule
Process data through augmentation pipeline
Train with batched examples
Validate on pretext task accuracy

Model Export for Inference

After training, models must be exported to support downstream inference. Common formats include ONNX or native PyTorch state dictionaries.

torch.save(model.state_dict(), 'pretrained_model.pth')

Batch Processing for Evaluation

During evaluation, batch processing ensures efficient use of compute resources.

with torch.no_grad():
    for batch in dataloader:
        embeddings = model.encoder(batch['data'])

Troubleshooting Common Issues

Gradient Vanishing

When gradients become too small, training stalls. Solutions include:

Reducing temperature parameter
Increasing learning rate
Adjusting batch size
Using gradient clipping

Memory Limitations

GPU memory overflow occurs when batch sizes are too large or models are too complex. Solutions include:

Reducing batch size
Using mixed precision training
Implementing gradient checkpointing
Distributing training across multiple GPUs

Poor Representation Quality

If final representations are uninformative, check for:

Insufficient data diversity
Incorrect loss function parameters
Inappropriate pretext task design
Overfitting to the pretext task

Conclusion

Self supervised learning enables effective model training when labeled data is scarce. Success depends on choosing appropriate pretext tasks, configuring training properly, and maintaining sufficient data diversity. The method works across modalities and architectures, making it a versatile tool for representation learning.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.