Data Governance & Data Catalog

Data governance defines how data is managed, used, and controlled within an organisation. It establishes policies, roles, and standards for data quality, security, and compliance. Data catalog is the mechanism through which these governed assets are documented, discovered, and accessed. This page covers practical aspects of setting up and maintaining a data governance system with a catalog, focusing on what you need to do rather than abstract principles.

Setting Up Metadata Storage

Metadata about data assets must be stored in a structured way. A minimal implementation uses a relational database table:

CREATE TABLE data_asset (
    id UUID PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    description TEXT,
    owner VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    schema JSONB,
    tags TEXT[],
    lineage JSONB
);

Use psql to insert initial data:

psql -d your_database -c "
INSERT INTO data_asset (id, name, description, owner, schema)
VALUES (
  '123e4567-e89b-12d3-a456-426614174000',
  'customer_orders',
  'Orders from customers with timestamps',
  'data-team',
  '{\"fields\": [{\"name\": \"order_id\", \"type\": \"string\"}, {\"name\": \"customer_id\", \"type\": \"string\"}]}'
);
"

Registering Data Assets

Each data asset must be registered in the catalog. The registration process involves:

1. Creating an entry in the metadata store
2. Defining ownership and access controls
3. Specifying schema information
4. Associating tags and documentation

Use a simple HTTP endpoint to register assets:

curl -X POST http://catalog.example.com/api/v1/assets \
  -H "Content-Type: application/json" \
  -d '{
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "name": "customer_orders",
    "description": "Orders from customers with timestamps",
    "owner": "data-team",
    "schema": {
      "fields": [
        {"name": "order_id", "type": "string"},
        {"name": "customer_id", "type": "string"}
      ]
    }
  }'

The response should include a status code and metadata:

{
  "status": "success",
  "asset_id": "123e4567-e89b-12d3-a456-426614174000",
  "version": "1.0"
}

Managing Data Ownership

Ownership defines who is responsible for a data asset. It's critical for access control and accountability. The ownership model includes:

• Primary owner
• Secondary owners
• Access groups

Use a database table to track ownership:

CREATE TABLE asset_ownership (
    asset_id UUID REFERENCES data_asset(id),
    role VARCHAR(50),
    user_or_group VARCHAR(255),
    PRIMARY KEY (asset_id, role, user_or_group)
);

To update ownership:

curl -X PUT http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000/ownership \
  -H "Content-Type: application/json" \
  -d '{
    "primary_owner": "data-team",
    "secondary_owners": ["analyst-team"],
    "access_groups": ["business-users"]
  }'

Defining Data Quality Rules

Data quality rules specify what constitutes valid data for a particular asset. These rules are enforced during ingestion and reported in the catalog.

Example rule definition:

curl -X POST http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000/rules \
  -H "Content-Type: application/json" \
  -d '{
    "rule_name": "customer_id_not_null",
    "description": "Customer ID must not be null",
    "type": "null_check",
    "field": "customer_id"
  }'

Tagging and Classification

Tags provide a flexible way to categorize assets. They are stored as an array in the metadata table.

curl -X PUT http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000 \
  -H "Content-Type: application/json" \
  -d '{
    "tags": ["finance", "customer", "daily"]
  }'

Tags are case-sensitive and should follow a consistent naming convention. Common patterns include:

• department (e.g., marketing, sales)
• data_type (e.g., transactional, analytical)
• sensitivity (e.g., public, confidential, restricted)

Schema Management

Schema definitions are stored in JSON format and should include field names, types, and constraints. The schema is versioned.

curl -X PUT http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000/schema \
  -H "Content-Type: application/json" \
  -d '{
    "version": "1.0",
    "fields": [
      {"name": "order_id", "type": "string", "nullable": false},
      {"name": "customer_id", "type": "string", "nullable": false},
      {"name": "order_date", "type": "timestamp", "nullable": true}
    ]
  }'

Access Control

Access control is managed through roles and permissions. A minimal implementation uses a table for mapping users to asset access:

CREATE TABLE asset_access (
    asset_id UUID REFERENCES data_asset(id),
    user_or_group VARCHAR(255),
    permission VARCHAR(20),
    PRIMARY KEY (asset_id, user_or_group, permission)
);

Permissions are read, write, and admin. Example access update:

curl -X PUT http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000/access \
  -H "Content-Type: application/json" \
  -d '{
    "access": [
      {"user_or_group": "data-team", "permission": "read"},
      {"user_or_group": "analyst-team", "permission": "read"},
      {"user_or_group": "admin-team", "permission": "admin"}
    ]
  }'

Versioning Assets

Each change to an asset should create a new version. This is managed through a versioning system that tracks schema changes, metadata updates, and ownership modifications.

curl -X POST http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000/versions \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Added order_date field",
    "changes": [
      {"field": "order_date", "type": "timestamp"}
    ]
  }'

Data Lineage Tracking

Lineage shows how data flows from source to destination. It's stored as a JSON object with source and target information.

curl -X PUT http://catalog.example.com/api/v1/assets/123e4567-e89b-12d3-a456-426614174000/lineage \
  -H "Content-Type: application/json" \
  -d '{
    "source": "orders_raw",
    "destination": "orders_enriched",
    "transformation": "join_customer_data"
  }'

Querying the Catalog

Search and filter functionality allows users to find assets by name, tags, owner, or schema.

Example search query:

curl -X GET 'http://catalog.example.com/api/v1/assets?tag=customer&owner=data-team'

Response includes a list of matching assets with basic metadata:

{
  "assets": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "name": "customer_orders",
      "description": "Orders from customers with timestamps",
      "owner": "data-team",
      "tags": ["customer", "finance"]
    }
  ]
}

Error Handling

Catalog operations return specific HTTP status codes:

• 200 OK for successful operations
• 400 Bad Request for malformed input or validation errors
• 401 Unauthorized for access denied
• 404 Not Found for missing assets
• 409 Conflict when trying to create a duplicate asset

Example error response:

{
  "error": "Asset already exists",
  "code": "ASSET_EXISTS",
  "status": 409
}

Common Pitfalls and Gotchas

Silent Failures

1. Missing schema field: If a field is added to the schema but not in the actual data, it silently fails during validation. Always validate that schema fields exist in data.

1. Case sensitivity in tags: Tags are case-sensitive. A tag Customer is different from customer.

1. JSON parsing errors: Malformed JSON in schema or metadata causes silent failures in catalog operations.

Configuration Issues

1. Ownership conflicts: Adding an owner who already has access can create redundant entries without clear precedence.

1. Versioning confusion: Updating a schema without creating a new version can lead to inconsistent data expectations.

1. Tag normalization: Without enforcement, tags like customer, Customer, and CUSTOMER can exist simultaneously, making search unreliable.

Operational Risks

1. Catalog downtime: Catalog unavailability prevents asset registration and access checks. Use a simple heartbeat endpoint to detect issues.

1. Data inconsistency: If data is updated in the catalog but not in the actual source, users see outdated information.

1. Access control drift: Changes to user roles or groups can leave asset access definitions stale.

Monitoring and Maintenance

Set up monitoring for:

1. Catalog health: Regularly check that API endpoints respond correctly.
2. Data freshness: Ensure metadata reflects actual data state.
3. Access logs: Track who accesses what assets and when.
4. Error rates: Monitor for repeated failures in catalog operations.

Example health check endpoint:

curl -X GET http://catalog.example.com/health

Response should be:

{
  "status": "healthy",
  "timestamp": "2023-10-01T10:00:00Z"
}

Migration Considerations

When migrating from an older catalog system:

1. Data validation: Ensure all existing assets are correctly represented in the new schema.
2. Ownership mapping: Confirm that ownership relationships are preserved during migration.
3. Tag consistency: Standardize tag formats to prevent duplicates.
4. Version history: Preserve versioning information where possible.

Example migration command:

curl -X POST http://catalog.example.com/api/v1/migrate \
  -H "Content-Type: application/json" \
  -d '{
    "source_system": "old-catalog",
    "assets_file": "/path/to/assets.json"
  }'

The process should return a summary of imported assets and any errors encountered.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.