Kubernetes Operators: Custom Resource Management

Kubernetes Operators: Custom Resource Management

Kubernetes Operators extend the native Kubernetes API by registering Custom Resources and embedding operational logic as a control loop that continuously reconciles desired state against actual cluster state. They are the standard pattern for managing complex, stateful workloads—databases, message queues, distributed storage—where lifecycle events such as initial provisioning, rolling upgrades, node failures, and backup restoration require more than what built-in controllers and Services provide. This reference covers how to define and register Custom Resource Definitions, create and manage Custom Resources, write a minimal operator loop, evolve schemas across versions, and troubleshoot the failure modes that typically surface in production.

Define a Custom Resource Definition

A Custom Resource Definition (CRD) registers a new top-level resource with the Kubernetes API server. Once installed, the server exposes the resource at a URL derived from the names.plural field, and kubectl commands become available without additional tooling.

The canonical CRD manifest uses a small, well‑defined set of fields. Here is a minimal example that registers a PostgreSQL resource:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqls.example.com
spec:
  group: example.com
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                storageGB:
                  type: integer
                username:
                  type: string
            status:
              type: object
              properties:
                phase:
                  type: string
      served: true
  names:
    plural: postgresqls
    singular: postgresql
    kind: PostgreSQL
    shortNames: [pg]

After saving this as postgres-crd.yaml, install it with the standard kubectl primitive:

kubectl apply -f postgres-crd.yaml

Verification:

kubectl get crd postgresqls.example.com

If the CRD has not yet propagated to the API server, kubectl returns:

Error: the server could not find the requested resource

If the manifest contains an invalid schema field, the server replies in its customary format:

Error: custom resource definition "postgresqls.example.com" is invalid: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.storageGB: Invalid type: expected "integer" got "number"

The kubectl explain command can be used to inspect the schema from the terminal:

kubectl explain customresourcedefinitions.spec.versions[0].schema.openAPIV3Schema

CRD validation is strict about the openAPIV3Schema shape; any deviation from the JSON Schema draft-07 grammar will be rejected at install time. There is no implicit fallback—if the schema is malformed, the CRD stays in Failed condition and no instances of the kind can be created until the definition is corrected.

Create and Manage Custom Resources

With the CRD installed, the Kubernetes API now serves the new resource type. Creating an instance is a matter of applying a standard Kubernetes object that uses the newly registered apiVersion and kind.

Example instance manifest:

apiVersion: example.com/v1alpha1
kind: PostgreSQL
metadata:
  name: my-postgres
spec:
  storageGB: 10
  username: admin

Apply it:

kubectl apply -f postgres-instance.yaml

Listing instances uses the plural form from the CRD definition:

kubectl get postgresqls.example.com

Output will include the NAME, AGE, and any observed STATUS fields if the CRD includes a status subresource. Describing a specific instance:

kubectl describe postgresql my-postgres

If the CRD was never installed, or the API group is not registered, kubectl returns:

Error: the server could not find the requested resource "postgresqls.example.com"

Editing an instance is done with the familiar kubectl edit:

kubectl edit postgresql my-postgres

This opens the default editor with the resource YAML; saving commits a kubectl apply under the hood. Patching is also supported for partial updates:

kubectl patch postgresql my-postgres -p '{"spec":{"storageGB":20}}'

Status updates are managed by the operator control loop (see the next section). If the CRD does not expose the subresources.status field, kubectl get will omit the status block entirely, and attempting kubectl get postgresql my-postgres -o yaml will return a resource with an empty status: section rather than an error.

Write a Minimal Operator Control Loop

Operators replace hand‑crafted imperative scripts with a declarative control loop that the Kubernetes ecosystem can manage as a native Deployment. The Operator Framework (formerly operator-sdk) provides the scaffolding to generate this loop in the language of your choice; the most common path is Go via sigs.k8s.io/controller-runtime.

Scaffold a new operator project:

operator-sdk init --domain example.com --repo github.com/example/pg-operator

Create the API definition for the kind you have already registered as a CRD:

operator-sdk create api --group example.com --version v1alpha1 --kind PostgreSQL

The command generates the following structure (excerpt):

apis/v1alpha1/postgres_types.go
controllers/postgres_controller.go
config/manager/manager.yaml

The generated controller file contains a Reconcile method whose signature mirrors the controller‑runtime API:

func (r *PostgreSQLReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
    // Fetch the PostgreSQL Custom Resource
    postgres := &examplev1alpha1.PostgreSQL{}
    if err := r.Get(ctx, req.NamespacedName, postgres); err != nil {
        return reconcile.Result{}, client.IgnoreNotFound(err)
    }

    // Desired‑state logic: ensure the underlying PostgreSQL pod matches spec
    // Example: check if a StatefulSet exists, create or update it as needed
    // ...

    // Update the resource status so kubectl get reflects the current state
    postgres.Status.Phase = "Ready"
    if err := r.Status().Update(ctx, postgres); err != nil {
        return reconcile.Result{}, err
    }

    // Re‑queue after a fixed interval, or when an observed condition changes
    return reconcile.Result{RequeueAfter: 30 * time.Second}, nil
}

The manager.SetupWithManager function binds the reconciler to the manager’s event sources:

func (r *PostgreSQLReconciler) SetupWithManager(mgr manager.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1alpha1.PostgreSQL{}).
        Complete(r)
}

Deploy the controller:

operator-sdk run --manager-image=quay.io/operator-framework/operator-sdk-builder:latest

If the operator pod starts before the CRD is installed, the manager will be unable to list or watch the new resource, and the controller logs will emit:

failed to watch postgresqls.example.com: the server could not find the requested resource

If RBAC is insufficient, the operator will fail to perform get, list, or update actions on the custom resource, producing logs such as:

error: failed to run "": exit code 1

Inspect the service account and role bindings with kubectl auth can-i to confirm permissions.

Handle Schema Evolution & Version Migration

Custom Resources evolve as the application matures. A CRD can serve multiple versions simultaneously; each version may be marked served: true (exposed via the API) and referenceable: true (used as a default for new resources). When a CR is created with an older API version, a conversion webhook may transform the object into the newest served version before it is persisted.

Example: you add a backupIntervalHrs field to the spec. The updated CRD YAML is reapplied:

kubectl apply -f postgres-crd.yaml

If a conversion webhook is configured, the API server automatically translates existing instances. Without one, older instances remain in their original shape, and new instances adopt the extended schema. Attempting to upgrade a cluster without a conversion webhook may result in the API server returning:

Error: failed to convert "postgresql.example.com/v1alpha1": unknown field "backupIntervalHrs" on the server: cannot convert

The Operator Framework provides tooling to generate a conversion webhook skeleton:

operator-sdk generate webhook --defaulting --conversion

Validate a generated bundle (which includes CRDs, manifests, and optional webhooks) with:

operator-sdk bundle validate ./bundle

A mis‑configured conversion webhook that returns the same version it received can cause the API server to loop, consuming CPU and eventually triggering a pod restart. Logs typically show:

reconciler went into a tight loop converting v1alpha1 to v1alpha1

Upgrade paths for the operator itself are typically handled by deploying a new Deployment image and allowing the reconciliation loop to pick up the updated controller code. CRD changes require a separate kubectl apply step; the operator will not automatically reconfigure existing resources to match a new schema—manual intervention or a migration script is needed.

Debugging and Sharp Edges

Several failure modes surface only after an operator has been running in production for some time. Being aware of them reduces mean‑time‑to‑recovery.

RBAC drift – If the operator’s service account loses the watch verb on the custom resource, the controller cache stalls and no events are processed. The symptom is a silent stop in reconciliation; kubectl logs <pod> shows no new entries after the initial sync. Fix with kubectl create role / kubectl create rolebinding or inspect existing bindings with kubectl get role <name> -o yaml.

Conversion webhook mis‑configuration – A webhook that returns a 5xx or times out causes the API server to reject the admission request, and the kubectl apply command fails with:

  Error: admission webhook "validation.example.com" failed: ...

The request never reaches the object store, so the resource appears absent even though the manifest is valid.

Status subresource absence – Many operators forget to enable subresources: {status: true} in the CRD. The result is that kubectl get my-postgres -o yaml omits the status: block entirely, and downstream tooling that expects a phase field will either error out or fall back to defaults. The operator itself may still reconcile successfully, but observability pipelines lose visibility.

Defaulting conflicts – CRDs support server‑side defaulting via the default keyword in openAPIV3Schema, and admission webhooks can also inject defaults. If both are active, the one that runs last wins. In practice, this means a webhook default of storageGB: 20 can silently override a CRD default of storageGB: 10, and the operator will reconcile against the webhook‑applied value without any error message. The only clue is a diff between the declared spec and the observed runtime state.

Watch cache staleness – controller‑runtime maintains an internal cache of watched resources. After a CRD update (e.g., adding a new field), the cache may continue serving the old schema for several seconds, causing the reconciler to reject the new field with a validation error that looks like a client‑side bug. Flushing the cache requires a manager restart or manager-cache invalidation calls in custom code.

Upgrade loop without CRD update – Deploying a new operator binary while leaving the CRD unchanged often results in the operator logging “reconciling existing resources” but making no substantive changes. The operator may report Phase: Progressing indefinitely because the controller logic expects a field that the old CRD does not provide. The resolution is to kubectl apply the updated CRD first, then roll the operator.

These edges are not exhaustively documented in official Kubernetes guides because they arise from the interaction of three moving parts: the API server, the CRD definition, and the operator’s reconcile loop. Tracking them in a production environment typically involves correlating kubectl get events, operator pod logs, and API server audit records.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.