Differential Privacy | The Garnet Wiki

Differential privacy is a mathematical framework for protecting individual data in statistical analyses, especially when aggregating sensitive information from many users. It guarantees that the output of a query—such as a model training run or a report—reveals little about any single individual’s data, even to an adversary with full knowledge of all other data. It matters when collecting user behavior, medical records, or financial activity, and when models are trained on data that must remain private: privacy is not just a property but a contract.

Core Concepts and Workflow

Differential privacy introduces noise to data or query outputs so that adding or removing one individual’s record changes the output distribution only slightly. The core unit is the privacy budget, denoted by ε (epsilon), which controls the trade-off between privacy and utility. Smaller ε means stronger privacy but more noise, larger ε allows more accurate results with less privacy.

The standard workflow begins with a privacy mechanism that adds noise to raw or aggregated data. Common mechanisms include the Laplace mechanism for real-valued outputs and the Gaussian mechanism for higher-dimensional or vector outputs. The mechanism is parameterized by sensitivity, which measures how much a query’s output changes when one record is added or removed.

For example, to compute the average income in a dataset with differential privacy, the sensitivity is the range of possible values (max − min), and Laplace noise is added with scale = sensitivity / ε.

Setting the Privacy Budget

To configure a differential privacy process, engineers must define:

ε (epsilon): target privacy budget
δ (delta): probability of privacy failure (used in (ε, δ)-differential privacy)
Δf: global sensitivity of the query function
σ: noise scale (for Gaussian mechanism)

In practice, a common pattern is to define these in a configuration block or script:

differential_privacy:
  epsilon: 0.5
  delta: 1e-5
  mechanism: "gaussian"
  global_sensitivity: 10000.0
  noise_scale: 20000.0
  privacy_units: "user-level"

The privacy_units field specifies whether privacy is at the user level (one user contributes one record) or record level (each data row is a user’s contribution). A user-level privacy mechanism can be composed from record-level ones using composition.

Composing Mechanisms

When multiple queries or steps are applied to a dataset, privacy budgets are consumed across the pipeline. The basic composition theorem states that applying k mechanisms each with ε_i privacy budget results in total privacy budget ∑ε_i. For tighter bounds, the advanced composition theorem uses a quadratic term:

> Total ε = √(2k ln(1/δ)) · ε + k · ε²

This is critical when training a machine learning model with multiple steps: data preprocessing, feature extraction, model fitting, and evaluation—each potentially adding noise.

In code, this composition is applied by accumulating ε and δ across steps:

from dataclasses import dataclass
from typing import List

@dataclass
class PrivacyBudget:
    epsilon: float
    delta: float

def compose_privacy(budgets: List[PrivacyBudget]) -> PrivacyBudget:
    total_epsilon = sum(b.epsilon for b in budgets)
    total_delta = sum(b.delta for b in budgets)
    return PrivacyBudget(total_epsilon, total_delta)

A common failure mode: engineers forget to account for privacy accounting when applying multiple mechanisms. The output of a model trained with two separate differentially private queries is often less private than expected because each query’s ε was not adjusted for composition.

Implementing Differential Privacy in Practice

Data Collection Pipeline

In a typical data collection pipeline, user data flows from edge devices (mobile apps, IoT sensors) to a central analytics system. To preserve privacy, the pipeline applies differentially private aggregation at each stage.

A common pattern uses local differential privacy (LDP), where each user adds noise to their own data before sending it to the server. This reduces the server’s need to trust the data pipeline.

Example: collecting user preferences in a mobile app.

Each user computes a histogram of their preferred categories (e.g., movies, music, books), then adds Laplace noise to each bin. The noise scale is set to:

> scale = sensitivity / ε = 1 / ε

for a binary input (user checked a category or not), since sensitivity is 1.

The server collects these noisy histograms and aggregates them without further noise. The result is a differentially private estimate of the overall distribution.

A real configuration for this step:

{
  "pipeline": "user_preferences",
  "source": "mobile_app",
  "aggregation": {
    "method": "local_dp",
    "noise_distribution": "laplace",
    "epsilon": 0.8,
    "sensitivity": 1,
    "output_format": "json"
  }
}

A silent failure: the client uses Gaussian noise instead of Laplace, but the server assumes Laplace. The resulting aggregation is less accurate and the privacy budget is underutilized.

Model Training with DP-SGD

For training machine learning models with differential privacy, Differentially Private Stochastic Gradient Descent (DP-SGD) is the standard. It adds noise to gradients during training and clips them to limit sensitivity.

The algorithm steps:

Clip each gradient vector to a fixed norm (e.g., L2 norm ≤ 1)
Add Gaussian noise scaled to the clipping norm and privacy budget
Average gradients over a mini-batch
Update model parameters

A typical configuration in a training job:

training:
  algorithm: "dp_sgd"
  batch_size: 64
  clipping_norm: 1.0
  noise_multiplier: 1.5
  epochs: 100
  learning_rate: 0.01
  privacy_budget:
    epsilon: 3.0
    delta: 1e-6

The noise multiplier is the ratio of the standard deviation of the noise to the clipping norm. It is closely related to ε via:

> ε = ∑_{t=1}^T (1 / σ_t) · (1 / √(2 log(1/δ)))

where σ_t is the noise standard deviation at step t.

A failure mode: engineers set the clipping_norm too high, resulting in excess noise and poor model accuracy. Conversely, setting it too low causes gradient clipping to lose precision.

Another subtle issue: privacy accounting is not done per epoch but across all steps. The default privacy accounting is advanced composition, but many pipelines use basic composition, underestimating the total ε.

To compute privacy budget during training, use:

def compute_dp_sgd_privacy(epochs, batch_size, noise_multiplier, delta=1e-6):
    # Assume constant step size and noise
    total_steps = epochs * (num_records // batch_size)
    sigma = noise_multiplier
    # Use advanced composition: ε = √(2T ln(1/δ)) / σ + T / (2 * σ²)
    epsilon = (2 * total_steps * sigma**2)**-0.5 * (2 * total_steps * sigma**2) + total_steps / (2 * sigma**2)
    return {"epsilon": epsilon, "delta": delta}

Querying Differentially Private Datasets

When querying a dataset protected via differential privacy, engineers must know whether the query is interactive or non-interactive.

In non-interactive queries, all queries are known in advance (e.g., a dashboard with fixed metrics). The privacy budget can be partitioned across queries, and each query can be designed to consume a portion of the budget.

In interactive queries, users explore the data, and each query can be adapted based on previous results. Here, composition theorems are essential.

A typical query pattern using curl:

curl -X GET \
  "https://analytics.example.com/api/dp/query" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT AVG(age) FROM users WHERE country = 'US'",
    "privacy_budget": 0.5,
    "mechanism": "gaussian",
    "sensitivity": 10.0
  }'

The server returns the result with added noise and metadata:

{
  "result": 35.2,
  "noise_scale": 20.0,
  "privacy_budget_used": 0.48,
  "mechanism": "gaussian",
  "sensitivity": 10.0,
  "timestamp": "2024-06-10T12:00:00Z",
  "query_id": "q-abc123"
}

A common error: the client expects the result to be in Laplace noise, but the server returns Gaussian-noised output. The client must adjust the scale factor, and the privacy budget is misallocated.

Another edge: post-processing. Once a differentially private output is produced, it can be transformed without losing privacy. For example, applying a threshold filter or a machine learning model to the output does not consume additional privacy budget.

But if the post-processing is not explicitly noted, the client assumes the output is final. The server must document that post-processing is allowed and that the output is post-processed.

Common Failure Modes and Pitfalls

Misunderstanding ε and δ: engineers often set δ too high (e.g., 0.01) but expect δ to be small. The privacy guarantee fails when a single user’s data causes a significant deviation in output.

Underestimating sensitivity: the sensitivity is computed from the data distribution, but the real-world data deviates from the assumed distribution. A query on income data assumes max-min = 100k, but actual data spans 200k. The noise scale is too small, and the result is less private.

Failing to account for composition: multiple queries are applied, but the total privacy budget is not computed. The result is a model or report that appears private but is actually less private than advertised.

Noise distribution mismatch: a client expects Laplace noise but receives Gaussian. The client must convert scale: Laplace scale = σ, Gaussian scale = σ × √(2), so the client must multiply Gaussian scale by √2 to use it as Laplace.

Incorrect privacy accounting in DP-SGD: the number of training steps is computed incorrectly due to variable batch size or missing steps. The privacy budget is either too small or too large.

Ignoring post-processing: a differentially private result is transformed into a histogram, but the transformation is not documented as post-processing. Users assume the histogram is the final output, but the original noise was not accounted for.

Integrating DP into Engineering Workflows

Engineers should integrate differential privacy into CI/CD pipelines, model serving, and data validation.

In a CI job:

jobs:
  - name: train-dp-model
    runs-on: gpu-medium
    steps:
      - uses: actions/checkout@v3
      - run: pip install torch differential-privacy
      - run: python train_dp_model.py --epsilon 2.0 --delta 1e-6 --epochs 50
      - run: |
          python validate_dp.py --model ./outputs/dp_model.pt
          # Output: "Privacy budget used: ε=1.95, δ=9.8e-7"
      - name: Upload DP Report
        run: |
          echo "ε: 2.0" >> report.md
          echo "δ: 1e-6" >> report.md
          echo "Achieved ε: 1.95" >> report.md
          echo "Achieved δ: 9.8e-7" >> report.md

In a model serving pipeline, the model endpoint returns both the prediction and the privacy metadata:

{
  "prediction": 0.72,
  "confidence_interval": [0.68, 0.76],
  "privacy_budget": {
    "epsilon": 0.5,
    "delta": 1e-5,
    "used": 0.48,
    "mechanism": "gaussian"
  },
  "timestamp": "2024-06-10T13:30:00Z"
}

A service consuming this endpoint must validate that the privacy budget is sufficient for its use case.

Summary

Differential privacy is not a one-off configuration but a continuous engineering discipline. It requires engineers to think about data as a privacy-protected resource, and to track the consumption of privacy budgets across the entire pipeline—from data collection to model serving. The key to success is rigorous privacy accounting, consistent noise distributions, and explicit documentation of the privacy assumptions at each stage.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.