01 · Notice the gap
A service with a metrics port, and nothing scraping it.
Every day the monitoring scan lists the workloads in the cluster and
joins them to their Services and to every existing
ServiceMonitor and PodMonitor. A ServiceMonitor selects
Services, not pods, so the join follows the real chain: monitor →
Service → workload, honouring namespace selectors.
orders-api has a port called metrics on
9102. No monitor reaches it. Workloads scaled to zero on purpose are
skipped, so a parked relic never gets a monitor.
02 · Check the endpoint really serves
It asks the endpoint before it writes a line of YAML.
A port name is a hint, not proof. Before proposing anything, the
assistant fetches /metrics through a read-only probe and
checks that it answers with Prometheus exposition format.
That same response tells it which metric families exist, which is
what the dashboard will be built from later.
03 · Propose the ServiceMonitor
Labels copied from the live Service, not guessed.
The generated ServiceMonitor selects the Service by its actual
labels, names the actual port, and carries the release label your
Prometheus operator or Alloy instance uses to pick monitors up.
Where the Helm chart already ships a monitor behind a values flag,
you get the values flip instead of a hand-written manifest,
so the chart stays the owner.
04 · Merged. Data starts flowing.
Your collector does the scraping. The bot reads the result.
After you merge, Flux applies the monitor, your collector
(Prometheus, or Grafana Alloy) starts scraping, and the series land
in your metrics backend.
The assistant never writes to that pipeline. It only reads from the
backend, per tenant, when it builds and checks the dashboard.
05 · Find or write the dashboard
Community dashboards first. Generated only when none fit.
For well-known exporters (node, Redis, PostgreSQL, Traefik…) it
prefers the maintained grafana.com dashboard and adapts it to
your datasource and namespace variables.
orders-api is your own code, so there is nothing to
import. The assistant drafts a dashboard from the metric families it
actually saw: request rate, latency histogram, errors, business
counters, runtime.
06 · Prove every panel
Each query runs against your data before you see it.
Every panel expression is executed against the metrics backend.
Panels that return nothing are dropped and counted. A dashboard with
no grounded panel is never shipped quietly: it is held for review
with the reason named.
rate() over a gauge is caught and escalated.- Every
$variable must be declared.
- Datasource and namespace are variables, never hard-coded.
07 · Ship it the GitOps way
A ConfigMap in git. Grafana's sidecar does the rest.
The dashboard is wrapped in a ConfigMap with the
grafana_dashboard label, placed in the tenant's folder,
and stamped with how many queries were grounded and when.
Merge it and Flux applies it. The Grafana sidecar picks it up. No
click-ops, no dashboard that only exists in one Grafana's database.
08 · The result
A dashboard that maps one-to-one onto what the service exposes.
Every panel is one of the metric families the probe saw
in step 2. Nothing invented, nothing empty.
Committed dashboards are audited for known defects and repaired by
MR. Re-checking them on a schedule, so a panel that goes dark after
a service change gets noticed too, is in development.