AI Infrastructure GPU Cluster Management

Managing GPU clusters for AI workloads involves orchestrating compute resources across nodes with shared storage, scheduling jobs, and maintaining cluster health. This process matters when deploying distributed training or inference systems that require consistent access to GPU hardware and networked storage.

Cluster Provisioning and Node Registration

Use standard Kubernetes manifests to define GPU-enabled nodes:

apiVersion: v1
kind: Node
metadata:
  name: gpu-node-01
  labels:
    node-role.kubernetes.io/gpu: "true"
    accelerator: nvidia-tesla-v100

Nodes must be registered with appropriate taints to prevent non-GPU workloads from running on them:

kubectl taint nodes gpu-node-01 accelerator=nvidia-tesla-v100:NoSchedule

NVIDIA's device plugin must run on each node to expose GPU resources. The plugin registers with the kubelet and makes GPUs available as a resource type:

kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com\/gpu}'

Failure to register the device plugin results in jobs requesting GPUs but never getting them, producing errors like:

0/3 nodes are available: 3 node(s) didn't have enough resource: nvidia.com/gpu.

Resource Allocation and Scheduling

GPU resources are tracked as custom resources in Kubernetes. A pod requesting a GPU must specify it explicitly:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: training-container
    resources:
      limits:
        nvidia.com/gpu: 1
      requests:
        nvidia.com/gpu: 1

Pods without explicit GPU requests will not be scheduled on GPU nodes, even if they are available. The scheduler will show:

0/3 nodes are available: 3 node(s) didn't have enough resource: nvidia.com/gpu.

If a pod is scheduled but cannot access the GPU, check the device plugin logs for errors such as:

E0123 12:34:56 device_plugin.go:123] failed to register device plugin: rpc error: code = Unknown desc = context deadline exceeded

Storage and Data Access

Shared storage is essential for distributed training jobs. Use NFS or cloud object storage mounted as volumes:

apiVersion: v1
kind: PersistentVolume
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: nfs-server.example.com
    path: /shared-data

Mount this volume in the pod:

spec:
  containers:
  - name: training-container
    volumeMounts:
    - name: shared-data
      mountPath: /data
  volumes:
  - name: shared-data
    persistentVolumeClaim:
      claimName: shared-data-claim

Jobs fail silently if the storage is unreachable. The pod will start but training processes will hang or exit with no clear error message.

Job Scheduling and Lifecycle Management

Submit jobs using Kubernetes job objects:

apiVersion: batch/v1
kind: Job
metadata:
  name: training-job
spec:
  template:
    spec:
      containers:
      - name: train
        image: my-trainer:latest
        resources:
          limits:
            nvidia.com/gpu: 2
      restartPolicy: Never

Jobs can be configured to retry on failure:

spec:
  backoffLimit: 3

The job controller does not automatically clean up completed pods unless explicitly configured. Use a finalizer or cleanup script to handle this.

Monitoring and Health Checks

Monitor GPU utilization with standard tools:

nvidia-smi -q -d UTILIZATION -l 1

Use Kubernetes metrics-server for cluster-wide resource usage:

kubectl top nodes
kubectl top pods --all-namespaces

Health checks should include:

• GPU memory availability
• Node CPU and network utilization
• Pod restart counts
• Job completion status

A node can appear healthy in the cluster but have a broken GPU driver. The kubelet will mark it ready, but jobs will fail with:

Error: failed to allocate GPU resources for container

Troubleshooting Common Failures

Driver Mismatch Issues

Mismatched NVIDIA drivers between nodes and containers cause silent failures. The node's driver version must match the container's expected driver version.

Check node driver version:

nvidia-smi -q -d DRIVER_VERSION

Check container driver compatibility:

docker run --gpus all nvidia/cuda:11.0-base-ubuntu20.04 nvidia-smi -q -d DRIVER_VERSION

Resource Exhaustion

Jobs fail with "OOMKilled" when requesting more GPU memory than available. The system does not prevent over-allocation.

Check memory usage:

nvidia-smi pmon -c 1

If a pod requests 4 GPUs but only 2 are available, it will remain in pending state until resources are freed.

Network Partitioning

Jobs fail with timeouts when nodes lose network connectivity. This is particularly problematic for distributed training jobs that require communication between nodes.

Monitor network health:

kubectl get nodes -o jsonpath='{.items[*].status.conditions[?(@.type=="NetworkUnavailable")].status}'

If a node reports "NetworkUnavailable: true", jobs scheduled on it will hang until connectivity is restored.

Storage Access Errors

Jobs fail with permission denied or "no such file" errors when shared storage becomes unavailable.

Check volume status:

kubectl get persistentvolumeclaim -A

Cluster Maintenance and Updates

Update the device plugin by restarting the daemonset:

kubectl rollout restart ds/nvidia-device-plugin-daemonset -n kube-system

When upgrading Kubernetes, ensure the node's GPU driver version is compatible with the new control plane version. The device plugin must be updated before upgrading the cluster.

Use rolling updates for node maintenance to avoid downtime:

kubectl drain gpu-node-01 --ignore-daemonsets
kubectl uncordon gpu-node-01

Resource Quotas and Limits

Define resource quotas at the namespace level to prevent one team from consuming all available GPUs:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
spec:
  hard:
    nvidia.com/gpu: 8

Quotas do not apply to individual pods unless they are explicitly limited. A pod can request unlimited GPU resources and still be scheduled successfully, even if it exceeds the quota.

Node Labels and Taints

Use labels to group nodes by GPU type:

kubectl label nodes gpu-node-01 accelerator=nvidia-tesla-v100
kubectl label nodes gpu-node-02 accelerator=nvidia-a100

Taint nodes to prevent non-GPU workloads from consuming resources:

kubectl taint nodes gpu-node-01 accelerator=nvidia-tesla-v100:NoSchedule

Misconfigured taints can cause pods to remain in pending state indefinitely.

Logging and Diagnostics

Capture GPU-specific logs from the device plugin:

kubectl logs -n kube-system -l name=nvidia-device-plugin

Use standard Kubernetes events to track scheduling issues:

kubectl get events --sort-by=.metadata.creationTimestamp

GPU-related errors are often logged in the kubelet's logs:

journalctl -u kubelet -f

Performance Optimization

Set resource requests and limits appropriately to avoid overcommitment:

resources:
  requests:
    memory: "2Gi"
    cpu: "1"
    nvidia.com/gpu: 1
  limits:
    memory: "4Gi"
    cpu: "2"
    nvidia.com/gpu: 1

Use pod affinity to keep related workloads on the same node:

affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          app: training
      topologyKey: kubernetes.io/hostname

Cluster Scaling

Scale GPU nodes using standard Kubernetes commands:

kubectl scale deployment gpu-node-deployment --replicas=5

Monitor cluster status during scaling:

kubectl get nodes -o wide
kubectl get pods -A

Scaling failures occur when new nodes cannot register with the control plane or when device plugins fail to start.

Job Retry and Failure Recovery

Configure jobs with appropriate restart policies and backoff limits. Jobs that fail due to GPU issues should be retried automatically:

spec:
  backoffLimit: 5
  template:
    spec:
      restartPolicy: Never

Implement custom failure detection logic to handle GPU-specific errors, as these often appear as generic container failures.

Resource Monitoring and Alerting

Set up monitoring for GPU memory usage, node health, and job status. Alerts should trigger when:

• GPU memory usage exceeds 90%
• Node becomes unreachable
• Jobs fail with GPU-related errors
• Pod restart counts increase rapidly

Use standard Prometheus metrics exposed by the device plugin and Kubernetes metrics server to build dashboards and alerts.

Backup and Recovery Procedures

Maintain backups of cluster state, including:

• Node configurations
• Job definitions
• Resource quota settings
• Storage volume configurations

Recovery procedures should include:

• Restoring node labels and taints
• Recreating failed jobs
• Reconfiguring resource quotas
• Verifying storage access

Cluster state can be exported using:

kubectl get nodes -o yaml > nodes-backup.yaml
kubectl get jobs --all-namespaces -o yaml > jobs-backup.yaml

These procedures ensure that GPU clusters remain functional and recoverable in case of failures or upgrades.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.