Platform Observability Integration

Production-grade guide to platform observability integration covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

This reference page covers the integration of observability primitives—metrics, logs, and distributed traces—into an internal developer platform. It matters when your platform team needs to expose health data to operators, when application teams need to emit instrumentation that the platform collects and aggregates, or when the pipeline between application and observability backend breaks. You are likely reading this if you are instrumenting a new service, debugging missing metrics or traces, or configuring the platform-wide observability stack.

Exposing Metrics from Platform Services

Platform services make health data available through a standard HTTP /metrics endpoint that returns Prometheus-formatted text. The endpoint typically listens on port 9090 or 9100, bound to 0.0.0.0 or the service cluster IP. A basic health check with curl reveals whether the endpoint is reachable and returning the expected format:

curl -s http://platform-service:9090/metrics | head -n 20

A healthy response begins with # HELP and # TYPE comments followed by metric families in the form metric_name{label_pair} value. If the response returns HTML, a 404, or a connection refused error, the metric export is misconfigured or the service is not running.

Common misconfiguration patterns include:

A sharp edge that catches teams off guard: metric label cardinality. If a service exports a metric with a label whose cardinality grows unbounded—such as user_id or request_path—Prometheus may emit warnings about sample limit exceedance or, in severe cases, trigger an OOM kill on the scraper. The platform team should enforce metric naming conventions and review label sets during code review.

Collecting Component Logs

Component logs are the primary debug source when metrics or traces stop flowing. In a Kubernetes-based platform, pod logs are accessed through kubectl:

kubectl logs -n platform <pod-name> -c <container-name>

The -c flag selects a specific container within a pod that may run multiple components. If the pod has just restarted, --previous fetches logs from the terminated instance:

kubectl logs -n platform <pod-name> --previous

For node-level diagnostics—such as kubelet events, container runtime logs, or operating system messages—systemd’s journal is the standard source:

journalctl -u kubelet --since "2 hours ago" | grep -i "metric\|runtime"

Log format matters. Most platform-deployed agents expect JSON-structured logs. A plain‑text log line like 2024-03-15T12:34:56 ERROR connection timeout may be parsed incorrectly by log aggregators that expect structured fields. Conversely, a JSON log missing a required field such as level or message may be dropped during ingestion. When migrating log sources, verify that the expected key names match the downstream parser’s schema.

A silently failing edge case: the kubelet log truncation limit. By default, kubelet retains the last 4 MiB of container logs per pod. If a verbose service writes aggressively, older entries are evicted without warning. Operators may assume a log never existed, leading to wasted debug time. Periodic rotation or sidecar log forwarders that ship to an external backend mitigate this.

Propagating Trace Context Across Platform Boundaries

Distributed traces require consistent context propagation across service boundaries. The W3C Tracecontext standard defines two headers that should appear on every outbound HTTP request:

A curl command that injects a trace context into a request looks like this:

curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" http://downstream-service/api/endpoint

If the downstream service does not forward the traceparent header, or if a middleware strip it, the trace span becomes a root span in the new service, breaking the parent‑child relationship. Platform teams should enforce header propagation at the reverse‑proxy or service mesh layer.

A sharp edge that appears in production: trace context loss through HTTP redirects. Some HTTP clients, particularly older curl versions or SDKs without explicit redirect handling, drop custom headers during 301/302 responses. The result is a trace that starts at the final destination but has no parent, creating orphan spans. Modern service meshes (e.g., those implementing the Envoy or Linkerd data plane) typically preserve headers across redirects, but custom HTTP clients must be coded explicitly to forward traceparent.

Another silent failure: missing X-Request-ID injection. Many platforms use a request‑ID header for correlation across logs and metrics. If the header is not set at the edge and not propagated downstream, log lines and metric samples from different services cannot be reliably joined in a debugging session. The platform should provide a middleware or proxy that generates and injects X-Request-ID: <uuid> on every inbound request.

Instrumenting Custom Services

Application teams add observability to their services through one of three patterns: manual instrumentation, auto‑instrumentation agents, or OpenTelemetry SDKs. The most portable approach across platforms is environment‑variable‑driven auto‑instrumentation.

For OpenTelemetry‑based agents, the OTEL_RESOURCE_ATTRIBUTES environment variable sets service identity that appears on every exported span and metric:

export OTEL_RESOURCE_ATTRIBUTES=service.name=my-api,deployment.environment=production

The agent, usually started as a sidecar or system‑level process, reads this variable at startup and embeds the attributes in the resource detection step. If the environment variable is set after the agent has already started, it has no effect—the agent must be restarted for new attributes to take effect. This is a common pitfall during rolling deployments: a config change that adds OTEL_RESOURCE_ATTRIBUTES appears in the pod spec, but the sidecar container still uses the old values because it was not restarted.

Auto‑instrumentation agents for language runtimes (e.g., oteljavaagent, otelpythonagent) are invoked via Java -javaagent or Python OTEL_PYTHON_INSTRUMENTATION flags. These agents attach to the running runtime and begin emitting spans without code changes. A frequent error is specifying an incorrect agent path or version mismatch between the agent and the runtime’s OpenTelemetry protocol version. The agent may start successfully but emit spans with an unsupported protocol version, causing the collector to reject them with a 400 Bad Request response.

Debugging Missing Telemetry

When metrics, logs, or traces stop appearing in the observability backend, a systematic check list narrows the fault domain. The following commands and verification steps apply across most platforms:

1. Verify the endpoint is reachable

curl -v http://collector:4317/v1/metrics

A 200 OK or 202 Accepted response indicates the collector is listening and accepting data. A 401 Unauthorized means authentication is misconfigured. A Connection refused means the collector pod is not running or the service mesh routing is broken.

2. Check authentication and TLS

If the collector requires mTLS, verify that the service account or client certificate is mounted correctly:

kubectl exec -n platform <collector-pod> -- ls /etc/certs/

Missing or expired certificates cause silent drops—the client sends data, the collector receives it but drops it because TLS verification fails. No error is returned to the sender, making this particularly insidious.

3. Confirm the correct protocol endpoint

The OpenTelemetry collector receiver endpoint for metrics is typically /v1/metrics. For logs, it is /v1/logs. For traces, /v1/traces. Sending metrics data to the traces endpoint, or vice versa, results in a 400 Bad Request with a message like invalid protocol buffer format. The curl command must match the data type to the endpoint:

# Metrics
curl -X POST http://collector:4317/v1/metrics -d @metrics.pb

# Traces  
curl -X POST http://collector:4317/v1/traces -d @trace.json

4. Inspect collector back‑pressure

When the collector’s internal queue fills, it returns 429 Too Many Requests. The client should implement exponential back‑off. Without it, bursts of traffic cause repeated 429 responses, and the client may interpret them as transient failures and retry, amplifying the load. Platform teams should configure the collector’s batch_timeout and max_export_batch_size to match expected traffic patterns.

5. Check for schema version mismatches

If a service upgrades its instrumentation library but the collector expects an older protocol version, spans may be dropped without error logging. The collector’s debug endpoint often provides version information:

curl http://collector:13133/debug/vars | jq '.otelcol_build'

Comparing the collector build version with the agent version helps determine whether a version skew is the root cause.

Maintaining the Pipeline Across Platform Iterations

Observability integration is not a one‑time setup. As the platform evolves—new service mesh versions, updated base images, changed Kubernetes distributions—the integration points shift.

Agent version alignment is the most common maintenance task. When the platform upgrades the OpenTelemetry collector from version 0.80.0 to 0.85.0, the sidecar agent version should ideally be updated in lockstep. Mismatched versions may result in new fields being emitted that the older collector does not recognize, or deprecated fields being removed, causing unexpected data loss. The platform changelog should always list observability component upgrades alongside application runtime upgrades.

Schema compatibility requires foresight. If the platform adds a new required attribute to the resource detection step (e.g., deployment.region), all emitting services must be updated to include it. Without a coordinated rollout, some services will emit spans missing the new attribute, which downstream dashboards may render as null or omit entirely. The platform team should maintain a versioned instrumentation guide that lists required attributes per platform version.

A sharp edge learned after breakage: collector config reloads. Reloading the collector’s configuration file (often signaled via otelcol --config-reload) does not in‑flight‑reset active spans. Spans that have already been started under the old configuration continue under the old rules. If the reload changes a processor’s behavior—such as adding a batch processor with a new timeout setting—in‑flight spans may be emitted with the old timeout, or dropped if the new configuration rejects them. The safe practice is to restart the collector pod after config changes, accepting a brief period of no data ingestion.

Platforms in this space include OutSystems, Mendix and Retool.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.