Data Governance Frameworks | The Garnet Wiki

Data governance frameworks in a compliance context are the set of policies, processes, and technical controls that determine how data is collected, classified, stored, accessed, and retired across an organisation’s infrastructure. They matter when regulatory regimes (GDPR, CCPA, HIPAA, etc.) require demonstrable control over the data lifecycle, or when internal risk management demands that data assets be discoverable, lineage-traced, and protected from unauthorized movement. This page focuses on the engineering side: how to instrument, enforce, and audit those controls using real tooling and generic patterns that map to any platform.

Defining Scope and Inventorying Assets

The first operational step in any data governance framework is knowing what data exists, where it lives, and how it is classified. Without a reliable inventory, policy enforcement is guesswork and audit evidence is incomplete.

Listing and Classifying Data Sources

A common pattern across platforms is to expose an API or database view that returns every managed data store along with its metadata tags. The following curl command fetches a JSON catalogue from a typical metadata service endpoint; replace $METADATA_HOST and $TOKEN with your environment’s values:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://$METADATA_HOST/api/v1/datastores?include_tags=true" | jq '.[] | {id, type, tags}'

If your metadata lives in a PostgreSQL-backed catalogue, the equivalent query is:

SELECT id, source_type AS type,
       jsonb_object_agg(key, value) AS tags
FROM   metadata_tags
GROUP  BY id, source_type;

Tag schemas vary, but a frequent convention is classification: public|internal|restricted and retention: years. Missing or misspelled tags are the single most common root cause of governance gaps; a tag that is restricted in one system may be internal in another, and no standard reconciliation runs silently allow the drift.

Enforcing Policies and Rules

Once the inventory is in place, the next goal is to attach rules that determine what actions are permitted on each data asset. Enforcement is typically expressed as policy-as-code, validated on commit or on every resource change.

Policy-as-Code Patterns

A policy language evaluates a JSON or YAML document against declared constraints. The following Rego snippet—runnable with the open-source opa binary—denies any write to a table tagged classification: restricted unless the request includes a justified ticket reference:

package data.gov

deny[reason] {
  input.method == "POST" or input.method == "PUT"
  not input.tags.classification == "restricted"
  reason := sprintf("write to restricted data requires ticket reference; path=%v", [input.path])
}

To test this policy locally against a sample request file request.json:

opa test data.gov policy.rego request.json

If the policy is violated, opa returns a structured violation entry; if the request complies, the test suite passes. The same pattern translates to JSON Schema validation, OIDC claim checks, or Kubernetes admission webhook rules—mechanics that are product-agnostic.

Validating Against Schemas

Data contracts are often expressed as JSON Schema documents. A typical curl invocation against a schema-validation endpoint:

curl -s -X POST "$POLICY_ENDPOINT/validate" \
  -H "Content-Type: application/json" \
  -d @data_contract.json | jq '.valid, .errors[]?.message'

A frequent sharp edge: a schema may declare type: string and format: uuid, but many validators ignore the format keyword by default. A request with a malformed UUID passes validation, only to fail later when the downstream consumer invokes a UUID parser and raises invalid UUID format. The lesson: always pair schema validation with a runtime type check, or explicitly enable format checking in your validator configuration.

Monitoring for Drift and Violations

Policies are useless if drift goes undetected. Monitoring focuses on three vectors: tag consistency, policy evaluation outcomes, and access‑log anomalies.

Watching for Changes

Real-time awareness is typically achieved by streaming events from the metadata or identity store. The following kubectl command watches for changes to DataStore custom resources in a Kubernetes namespace:

kubectl get datastores -n compliance -w -o json | jq '.object.metadata.resourceVersion'

For non-Kubernetes stores, a long‑polling curl pattern works:

while true; do
  curl -s "$METADATA_HOST/api/v1/datastores?since=$LAST_RV" | jq -r '.[].tags.classification'
  sleep 30
done

Dropped connections, TLS handshake failures, or missing since parameters are common failure modes; each produces a non‑200 HTTP status or an empty payload that, if unhandled, causes the loop to re‑query at full rate and throttle the metadata service.

Interpreting Alerts

When a policy evaluation fails, the emitted error typically contains the request path, the violated rule identifier, and the current tag state. A representative output fragment:

{
  "violation": "restricted_write",
  "path": "/datastores/db-prod/users",
  "tags": {"classification": "restricted"},
  "requested_by": "svc-ci-cd"
}

Silently, some systems only log a numeric exit code without a message, making post‑mortem debugging harder. Adding a --verbose flag or configuring the enforcer to emit structured JSON logs mitigates this.

Remediation and Evidence Export

When a violation is confirmed, the workflow shifts to remediation and audit‑ready evidence collection.

Applying Fixes

Remediation scripts often follow a pattern of updating tags, rotating credentials, or moving data to a compliant store. A psql update that re‑classifies a database after a policy review:

UPDATE datastores
SET    tags = jsonb_set(coalesce(tags, '{}'), '{classification}', '"internal"'::jsonb)
WHERE  id = 'db-prod-02'
  AND (tags->>'classification') IS DISTINCT FROM 'restricted';

If the update affects zero rows, the script should log no matching records rather than failing; the “zero‑row” case commonly indicates the tag was already corrected manually or the asset was decommissioned.

Exporting Audit Records

Regulatory audits require a tamper‑evident export of all policy decisions, tag changes, and remediation actions. The following curl POST to a generic audit‑ingestion endpoint streams a month’s worth of events in JSON Lines format:

curl -s -X POST "$AUDIT_ENDPOINT/export" \
  -H "Authorization: Bearer $AUDIT_TOKEN" \
  -d '{"start": "2025-07-01T00:00:00Z", "end": "2025-07-31T23:59:59Z", "format": "jsonl"}' \
  -o audit_export.jl

The resulting file can be piped into jq for aggregation or loaded into a SIEM. A frequent pitfall: the export endpoint may truncate records older than 90 days unless an explicit retention parameter is supplied, producing incomplete audit packages. Always verify the endpoint’s default window and adjust the request bounds accordingly.

Sharp Edges

What silently fails, what confuses, what you only learn after it breaks in production:

Tag propagation delays. Metadata systems often cache tag assignments for several minutes. A policy that runs immediately after a tag update may evaluate against the stale value, producing a false‑negative. Adding a wait period or querying the source of truth directly before enforcement eliminates this class of false confidence.
Overlapping policy languages. Many organisations run both Rego/OPA rules and JSON Schema validation in the same pipeline. If the schema allows a field and the Rego rule denies it, the first evaluator wins depending on ordering. Document the evaluation sequence explicitly; otherwise, a change in one layer can appear as a regression in the other.
Schema format quirks. As noted, format keywords in JSON Schema are advisory. Validators that silently ignore them let malformed data through, causing downstream consumers to crash with unexpected token or invalid base64. Pin a specific validator version and enable format checking in CI to catch these before promotion.
Export window defaults. Audit exporters frequently default to a 30‑ or 90‑day lookback. If a compliance audit covers a longer period, the exported package will be incomplete unless the request explicitly overrides the window. Treat the default as a configuration trap, not a feature.
Credential leakage in logs. Policy‑as‑code runners often print the full request object on failure, which may include bearer tokens or data identifiers. Route stderr to a protected sink and scrub sensitive fields before logging; a single PRIVATE_KEY= line in a CI log can compromise an entire environment.

These patterns—inventory via API or SQL, enforcement through policy‑as‑code or schema validation, real‑time monitoring through streaming or long‑polling, and remediation/export with explicit bounds—form a practical, tool‑agnostic backbone for data governance frameworks in any compliance‑driven engineering organization. The concrete commands and error fragments above map directly to the mechanics you’ll encounter; the sharp‑edges notes are the details that separate a working pipeline from one that appears functional until the first audit or breach lays bare the gaps.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.