Sre Dashboards | The Garnet Wiki

Orientation

SRE dashboards serve as the primary visual interface for real‑time system state. They matter whenever an incident demands quick situational awareness, a deployment needs health verification, or a team reviews capacity trends between meetings. This reference page tells you how to construct, maintain, and troubleshoot dashboards using concrete, real‑world tools—without tying any mechanism to a specific vendor’s product internals. You’ll find exact commands, real flags, and the failure modes that surface when things go quiet in production.

1. Locating or restoring a dashboard definition

When a dashboard disappears after a config change or you inherit an unfamiliar cluster, the first step is finding the manifest that describes it. Most SRE stacks store dashboard definitions in Kubernetes ConfigMaps or plain files under version control.

To search across all namespaces for a dashboard named cluster‑load:

kubectl get configmaps --all-namespaces -o json | jq -r '.items[] | select(.metadata.labels.dashboard // "" == "cluster-load") | .metadata.name'

If the definition lives in a git repository, a targeted grep will surface the file quickly:

grep -r "cluster-load" ~/infra/ --include="*.yaml" --include="*.json"

Once located, you can extract the raw manifest for inspection or editing:

kubectl get configmap cluster-load -n monitoring -o yaml > /tmp/cluster-load.yaml

Exact error text you may encounter

If the ConfigMap name is miss‑typed or the object has been deleted:

Error from server (NotFound): configmaps "cluster-load" not found

If the manifest references a metric query that the API server cannot parse:

Error: parse error at line 12: unknown function 'rate'

2. Crafting a dashboard manifest from scratch

A dashboard definition is typically a JSON or YAML document that declares panels, each of which references a metric expression and specifies visual settings. The structure is deliberately generic so it maps onto most metric‑visualisation systems.

Here is a minimal YAML fragment that could be saved as request-rate.yaml:

title: "Request Rate (5m avg)"
panels:
  - title: "incoming requests"
    height: 8
    targets:
      - expr: rate(http_requests_total[5m])
        legendFormat: "$host"
        refId: A
    type: "graph"
    gridPos: {x: 0, y: 0, w: 24, h: 8}

To apply this manifest to a Kubernetes‑based monitoring stack:

kubectl apply -f request-rate.yaml

If the stack watches ConfigMaps and reloads automatically, the panel appears immediately. If a restart is required, use the service‑specific reload mechanism (see the next section).

Common field patterns (vendor‑agnostic)

Field	Typical purpose
`title`	Human‑readable panel label
`expr` / `query`	Metric expression or SQL statement
`legendFormat`	Series legend template (e.g., `$host`, `$instance`)
`refId`	Internal identifier used when a panel has multiple targets
`type`	Rendering style (`graph`, `table`, `stat`, etc.)
`gridPos`	Position and size in the dashboard grid

3. Fetching metrics to populate a panel

Before wiring an expression into a dashboard, it is prudent to verify that the metric system returns the expected series. Most SRE metric endpoints expose an HTTP API that accepts a query parameter and returns JSON‑encoded time series.

Exact command

curl -sG 'http://localhost:9090/api/v1/query' --data-urlencode 'query=rate(http_requests_total[5m])'

Expected JSON shape (truncated)

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {"job": "nginx", "handler": "request"},
        "value": [1704067200, "125.3"]
      }
    ]
  }
}

If the query references a metric that does not exist in the current window:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": []
  }
}

If the query syntax is malformed, the API returns:

{
  "error": "parse error: unexpected function name",
  "status": "error"
}

Using the result in a dashboard

Copy the expr value verbatim into the panel’s targets[].expr field. If the query returns multiple series, the dashboard renderer will plot each as a separate line or bar, depending on the panel type.

Verifying with `psql` when metrics are persisted

When metrics are stored in a relational store for historical trending, a direct SQL check can confirm data availability:

psql -h localhost -U sre -d metrics -c \
  "SELECT metric_name, avg(value) FROM dashboard_samples \
   WHERE ts > now() - interval '5 minutes' \
   GROUP BY metric_name LIMIT 5;"

Typical output:

 metric_name  |          avg
--------------+------------------------
 http_req_total | 124.73829787234043
 request_latency | 0.04215384615384615

If the table is empty or the column names differ, the dashboard will render blank— a common silent failure (see the sharp‑edges section).

4. Reloading and refreshing without a full restart

Dashboard definitions and metric configurations are often hot‑reloaded, but the exact mechanism depends on the service. Knowing the right flag saves time and avoids unnecessary pod restarts.

Exact command for a Prometheus‑style API server

Many metric endpoints expose a /reload endpoint that triggers a configuration re‑parse without stopping the process:

curl -X POST http://localhost:9090/-/reload

A successful response:

Reload succeeded.

If the request is made while the server is performing a snapshot or the config file has a syntax error:

Error: configuration reload failed: parse error at line 3

Reloading systemd‑managed SRE services

When the metric collector or dashboard renderer runs as a systemd service, a reload signal can be sent directly:

systemctl reload myservice

Equivalent to sending SIGHUP to the main process, this re‑reads config files and updates internal state. If the service does not support SIGHUP, the command exits with:

Job for myservice failed because the control process exited with error code.

Checking the journal reveals the underlying cause:

journalctl -u myservice --no-pager -n 30

What “refresh” means for the consumer

Even after the server reloads, browser‑cached dashboard pages may still show stale panels. Most visualisation front‑ends provide a “re‑load” button that forces a fresh HTTP fetch of the definition and metrics. If your stack uses a front‑end proxy, ensure Cache‑Control headers are set appropriately; otherwise, a hard browser refresh (Ctrl+F5) may be required.

5. Sharp edges: what silently fails, confused options, production breakage

Stale data because the query range is too wide

A common quiet failure occurs when a panel’s refreshInterval is set to 15s but the underlying query spans 5m. The renderer may clip or down‑sample aggressively, making the graph appear flat or empty despite active traffic. The fix is to align the panel’s interval with the query’s window, or to adjust the query’s lookback (rate(http_requests_total[1m]) instead of [5m]).

Confused flags: `refreshInterval` vs `evaluationInterval`

In many dashboard systems, refreshInterval controls how often the UI requests new data from the backend, while evaluationInterval (in alerting rules) determines how frequently alert conditions are checked. Setting refreshInterval: 1s on a dashboard that queries a expensive PromQL expression can throttle the metric system; conversely, a high refreshInterval makes incident response feel sluggish. The two are independent—changing one does not affect the other.

Dashboard definition that references a deleted metric

If a panel’s expr references a metric that was renamed or removed, the panel renders blank and the browser console may log:

Error: query returned no series for panel "incoming requests"

The dashboard does not raise an alert because the visualisation layer treats an empty result set as “no data” rather than “error”. The remedy is to grep the manifest for the old metric name before deploying:

grep -r "old_metric_name" ~/dashboards/ || echo "no references found"

Reload that appears to succeed but does not pick up changes

When a systemctl reload command returns “Job for myservice finished, failed, etc.” but the dashboard still shows old panels, the service may use a separate config‑watch mechanism (e.g., inotify) that requires a full restart. In such cases, systemctl restart myservice is the reliable path, albeit with a brief outage window.

Production breakage story (generic pattern)

During a recent incident, a team merged a dashboard YAML that added a new panel with expr: sum by (instance) (rate(http_requests_total[1m])). The query ran fine in isolation via curl, but after kubectl apply, the entire dashboard froze. Investigation revealed that the sum by aggregation conflicted with an existing legendFormat that already incorporated $instance, causing the renderer to emit duplicate series and eventually hit the per‑panel series limit. The team resolved it by removing the redundant sum by and simplifying the legendFormat. The lesson: always validate a new panel’s query both in the API sandbox and in a staging dashboard before merging to production.

Checklist before pushing a dashboard change

Run the metric query via curl; confirm non‑empty result set.
Apply the manifest to a staging cluster; verify the panel renders as expected.
Check that refreshInterval and any alert evaluationInterval are appropriate for the query cost.
Ensure no other panel in the same dashboard references the same metric with conflicting aggregations.
After merge, trigger a reload (curl -X POST /-/reload or systemctl reload) and confirm the update propagates without a full restart.

Following the patterns and commands above will let you construct dashboards that stay in sync with your metrics, reload cleanly after configuration changes, and surface failures early—before they manifest as opaque blank panels during an incident.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.