Internal Developer Portals | The Garnet Wiki

Internal Developer Portals (IDPs) are centralized, self-service platforms that empower developers to discover, adopt, and operate software components and infrastructure services within an organization. They matter when teams grow beyond a handful of engineers and multiple teams begin building, deploying, and maintaining services with divergent practices, tooling, and documentation. An IDP becomes essential when onboarding a new developer takes more than a week, when service ownership is unclear, and when teams struggle to find or reuse existing components.

Discovering Services and Components

Developers need to locate existing services, APIs, libraries, and infrastructure templates without relying on tribal knowledge or ad hoc Slack threads. An IDP enables discovery through a searchable catalog of reusable assets, each with metadata, documentation, and usage examples.

A service is registered in the portal via a manifest.yaml file placed in the root of a Git repository:

name: user-profile-service
version: 1.2.0
description: Manages user profiles and preferences
maintainer: [email protected]
tags:
  - api
  - user
  - auth
  - microservice
  - golang
  owners:
    - [email protected]
    - [email protected]
  - [email protected]
  dependencies:
    - auth-service
    - notification-service
  links:
    docs: https://docs.acme.com/user-profile-service
    source: https://github.com/acme/user-profile-service
    deploy: https://dashboard.acme.com/applications/user-profile-service

This manifest is consumed by a catalog service via a GET /catalog endpoint that returns a JSON array. Each entry includes a type, name, version, and description, with optional icon and tags fields.

To search for components, a developer makes a GET /catalog request with query parameters:

GET /catalog?query=auth&tags=api&sort=stars&limit=10

The response includes pagination and filtering metadata:

{
  "items": [
    {
      "type": "service",
      "name": "auth-service",
      "version": "2.1.0",
      "description": "Centralized authentication and session management",
      "icon": "/icons/auth.svg",
      "stars": 42,
      "tags": ["api", "auth", "golang", "jwt"],
      "links": {
        "docs": "https://docs.acme.com/auth-service",
        "source": "https://github.com/acme/auth-service",
        "deploy": "https://dashboard.acme.com/applications/auth-service"
      }
    }
  ],
  "pagination": {
    "next": "/catalog?query=auth&tags=api&sort=stars&limit=10&offset=10",
    "total": 147
  }
}

A common failure mode is stale or missing manifests. When a new service is deployed but not registered, developers may use a service without knowing it exists. The IDP detects this by polling Git repositories every 15 minutes, using git fetch and git ls-remote to detect new branches and tags. If a repository contains a manifest.yaml but no catalog entry, the portal suggests a POST /catalog/scan to trigger a manual scan.

Consuming a Service or Component

Once discovered, a developer selects a service to consume. The IDP presents a detailed view with a quick-start guide, configuration schema, and deployment steps.

To integrate auth-service into a new application, a developer follows the guide, which includes a curl command to register the service in the application’s configuration:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "service": "auth-service",
    "config": {
      "base_url": "https://auth.acme.com",
      "timeout_ms": 3000,
      "cache": {
        "enabled": true,
        "ttl_seconds": 300
      }
    }
  }' \
  https://api.acme.com/v1/applications/my-app/services

The API returns a 201 Created response with a Location header pointing to the new service instance:

HTTP/1.1 201 Created
Location: https://api.acme.com/v1/applications/my-app/services/auth-service
Content-Type: application/json

{
  "id": "auth-service-8c7b",
  "service": "auth-service",
  "config": {
    "base_url": "https://auth.acme.com",
    "timeout_ms": 3000,
    "cache": {
      "enabled": true,
      "ttl_seconds": 300
    }
  },
  "status": "active",
  "created_at": "2024-07-15T10:30:00Z"
}

The service’s configuration schema is defined in a schema.json file in the service’s repository, which is loaded dynamically from https://github.com/acme/auth-service/schema.json:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Auth Service Configuration",
  "type": "object",
  "properties": {
    "base_url": {
      "type": "string",
      "format": "uri",
      "description": "Base URL of the auth service endpoint"
    },
    "timeout_ms": {
      "type": "integer",
      "minimum": 100,
      "default": 2000
    },
    "cache": {
      "type": "object",
      "properties": {
        "enabled": { "type": "boolean", "default": false },
        "ttl_seconds": { "type": "integer", "minimum": 10, "default": 60 }
      }
    }
  },
  "required": ["base_url"]
}

The IDP validates the input against this schema before saving. If the timeout_ms value is not an integer, the API returns a 400 Bad Request with a detailed error:

{
  "error": "Invalid configuration",
  "details": [
    {
      "field": "config.timeout_ms",
      "message": "Expected integer, got string",
      "value": "3000"
    }
  ]
}

A subtle failure occurs when a service schema changes but the IDP caches the schema for 10 minutes. If a new field jwt_issuer is added, but the cache is not invalidated, developers may miss the new option until they manually refresh or until the cache expires.

Deploying and Managing Environments

Developers use the IDP to create and manage environments—staging, production, canary, etc.—for their applications. Each environment is a named instance of a service with its own configuration and deployment pipeline.

To create a new staging environment, a developer selects my-app in the portal and clicks "Deploy New Environment". The IDP presents a form based on the application’s environment.yaml:

name: my-app
environments:
  staging:
    description: "Staging for feature branch deployments"
    base_url: "https://staging.my-app.acme.com"
    replica_count: 2
    image: "my-app:latest"
    config:
      logging: "debug"
      feature_flags:
        user_profile_v2: true
        dark_mode: false
    deployment:
      strategy: "rolling"
      rollout_steps:
        - "build image"
        - "run tests"
        - "update load balancer"
        - "enable canary"

After filling in the form, the IDP triggers a deployment via a POST /deployments request:

POST /deployments
Content-Type: application/json

{
  "application": "my-app",
  "environment": "staging",
  "version": "v2.0.1",
  "config": {
    "logging": "info",
    "feature_flags": {
      "user_profile_v2": true,
      "dark_mode": true
    }
  },
  "trigger": "manual",
  "user": "[email protected]"
}

The deployment service responds with a 202 Accepted and a Location header to a deployment status endpoint:

HTTP/1.1 202 Accepted
Location: /deployments/7b3a2c1
Content-Type: application/json

{
  "id": "7b3a2c1",
  "application": "my-app",
  "environment": "staging",
  "status": "pending",
  "started_at": "2024-07-15T11:05:45Z",
  "steps": [
    { "name": "build image", "status": "running", "started_at": "2024-07-15T11:06:00Z" },
    { "name": "run tests", "status": "pending", "started_at": null }
  ]
}

The deployment pipeline uses kubectl to apply a Helm chart to a Kubernetes cluster, with the environment-specific configuration:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-staging-config
  namespace: staging
data:
  log_level: info
  feature_flags: |
    {
      "user_profile_v2": true,
      "dark_mode": true
    }
EOF

A failure mode is when the deployment step run tests fails but the deployment status does not reflect this. The IDP polls the deployment service every 30 seconds, but if the test runner fails to write a status.json file to a shared S3 bucket, the IDP sees the deployment as running indefinitely. The solution is to require each pipeline step to emit a status.json file with status, message, and duration_ms fields.

Integrating Feature Flags and Observability

An IDP integrates feature flags and observability to help developers understand and control their services in real time.

A feature flag is declared in a flags.yaml file in the application’s repository:

name: my-app
flags:
  user_profile_v2:
    description: "New user profile UI with dark mode support"
    default_value: false
    rollout:
      strategy: "percentage",
      percentage: 10
    segments:
      - name: "internal-users",
        filter: "user.role == 'employee'"
        value: true
      - name: "beta-users",
        filter: "user.tier == 'beta'"
        value: true
    metadata:
      owner: "[email protected]"
      documentation: "https://docs.acme.com/feature/user-profile-v2"

The IDP exposes a /features API to list all flags and their current state. A developer can toggle a flag via a PATCH /features/user_profile_v2 request:

PATCH /features/user_profile_v2
Content-Type: application/json

{
  "value": true,
  "context": {
    "user": {
      "id": "u123",
      "role": "employee"
    }
  }
}

The response includes the updated flag state and a Cache-Control: public, max-age=60 header to avoid redundant requests.

The IDP also integrates with observability tools. It fetches metrics from a Prometheus instance via a GET /metrics endpoint:

GET /metrics?service=user-profile-service&interval=1h&query=rate(http_requests_total[5m])

The response is a list of time-series data points in a standard format:

[
  {
    "timestamp": "2024-07-15T11:00:00Z",
    "value": 127.3
  },
  {
    "timestamp": "2024-07-15T11:05:00Z",
    "value": 132.1
  }
]

A common issue is missing or misaligned timestamps. If the IDP expects ISO 8601 timestamps but the Prometheus endpoint returns YYYY-MM-DD HH:MM:SS, the frontend fails to render the graph. The fix is to normalize all timestamps to ISO 8601 before rendering.

Extending the Portal with Custom Workflows

Developers extend the IDP with custom workflows using a simple event-driven system. Events are published to a Kafka topic named portal.events in JSON format:

{
  "event": "service.registered",
  "timestamp": "2024-07-15T10:30:00Z",
  "payload": {
    "service": "auth-service",
    "version": "2.1.0",
    "repository": "https://github.com/acme/auth-service"
  }
}

The portal listens for events and triggers actions. For example, when a service.registered event arrives, it creates a catalog.entry in a PostgreSQL database:

INSERT INTO catalog (name, version, description, tags, metadata)
VALUES (
  'auth-service',
  '2.1.0',
  'Centralized authentication and session management',
  '{"api", "auth", "golang", "jwt"}',
  '{"source": "https://github.com/acme/auth-service", "docs": "https://docs.acme.com/auth-service"}'
);

A failure mode occurs when the event consumer fails to process a message. The IDP uses kafka-consumer-group with auto.commit.interval.ms=5000, but if the database connection is slow, messages accumulate in the consumer group, and a new service registration may be delayed by up to 30 seconds. The solution is to use transactional producers and consumers, ensuring that each event is processed exactly once.

Troubleshooting and Diagnostics

When a developer encounters an issue, the IDP provides diagnostic tools. A GET /diagnostics endpoint returns a JSON report of the portal’s health:

{
  "status": "degraded",
  "services": {
    "catalog": {
      "status": "up",
      "latency_ms": 42,
      "last_error": null
    },
    "deployments": {
      "status": "down",
      "latency_ms": 1200,
      "last_error": {
        "code": "500",
        "message": "Database connection failed",
        "timestamp": "2024-07-15T10:45:30Z"
      }
    }
  },
  "dependencies": [
    {
      "name": "Kafka",
      "status": "healthy",
      "latency_ms": 15,
      "connected": true
    },
    {
      "name": "Prometheus",
      "status": "warning",
      "latency_ms": 210,
      "connected": true,
      "last_query": "rate(http_requests_total[5m])"
    }
  ]
}

A common failure is when the deployment service is down but the diagnostics report it as up due to a stale cache. The solution is to use a heartbeat mechanism: the deployment service sends a PUT /heartbeat every 30 seconds, and the diagnostics service marks it down after 90 seconds of no heartbeat.

Key Failure Modes

A manifest.yaml file is missing or misnamed (MANIFEST.yaml), causing the catalog service to skip the repository.
The schema validation fails silently when a schema.json file is not available at the expected path.
The deployment status is not updated when a pipeline step fails to emit status.json.
Feature flag updates are not reflected in the application until the application reloads the flag configuration every 5 minutes, causing delays in feature rollouts.
The IDP caches environment configurations but does not invalidate the cache on environment.yaml changes, leading to stale configurations.

Internal Developer Portals are not just a collection of links and documentation—they are systems that coordinate discovery, deployment, configuration, and observability. Their value lies not in what they do, but in how they fail, and in how developers recover from those failures.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.