Somebody set the Mimir querier to 5.6 GiB of memory a long time ago, "to be safe". Eight replicas later, the cluster reserves 45 GiB for a component that has never used more than 80 MiB. This is how the assistant notices, decides what the right number is, and refuses to let a model's guess go below what was measured.
The metrics scan runs daily. For each metrics tenant it queries CPU and memory working set, the configured requests, OOM kills and restarts for every container in every namespace, over a 30-day window.
Nothing is collected by the assistant itself. It reads the Prometheus or Mimir data you already have, read-only.
A pre-filter picks the interesting ones, always measured against the request:
mimir-querier requests 5.6 GiB and uses about 64 MiB:
roughly 1% of what it reserves.
An average hides bursts, so the sizing never rests on it alone. For each container the assistant keeps the average, the 95th percentile and the peak working set, taking the highest value across all 8 replicas that ran in the window.
Here the three numbers sit close together: 64 MiB average, 76 MiB p95, 80 MiB peak.
Volatility is peak divided by p95. At 1.25 or below the workload is flat: there are no real spikes, so a big gap between request and limit is just reserved memory nobody uses.
Above 1.25 it is spiky, and a second number decides how much headroom to reserve: namespace concurrency, whether the pods around it tend to peak at the same time or take turns.
The querier: 80 / 76 = 1.06, flat.
The measured numbers, the classification and the current manifest go to the language model together with the sizing rules for each case. It returns a small structured proposal: which values to change, and why.
It is allowed to say "keep as proposed". Tuning for the sake of tuning is not the goal.
Models sometimes ignore their own rules, so the hard limits are enforced again in code after the model answers:
Here the model suggested a 64 MiB request. That is below the 80 MiB peak, so it was raised.
The change is written back into the file in place, so comments, key order and every other value stay exactly as they were. Values are rounded up to standard steps (64, 96, 128 MiB…), never down.
For a HelmRelease the edit goes into spec.values, where
the chart takes it from, not into rendered output.
Every resource-touching MR carries the measurements it was sized from: classification, volatility, namespace concurrency, average, p95 and peak, how many replicas were observed, and which safety clamps fired. You can check the arithmetic without opening Grafana.
# working set, per container avg by (namespace, pod, container) (container_memory_working_set_bytes{container!=""}) # CPU, per container rate(container_cpu_usage_seconds_total{container!=""}[5m]) # what each container asked for kube_pod_container_resource_requests{resource="memory"} # OOM kills that survive the restart kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} → 1,284 containers · 61 namespaces · 30 days
volatility = peak / p95 = 80 / 76 = 1.06 # ≤ 1.25 → flat concurrency = 14.2% # namespace pods rarely peak together
{
"requests": { "memory": "64Mi" },
"limits": { "memory": "128Mi", "cpu": "2800m" },
"rationale": "Flat workload (1.06). Request to typical use,
limit about 1.5x peak. Existing CPU limit kept."
}
values: querier: replicas: 8 # sized for query fan-out, see runbook resources: requests: cpu: 500m - memory: 5.6Gi + memory: 96Mi limits: - cpu: 2800m - memory: 5.6Gi + memory: 128Mi
Illustrative numbers, real shape: the evidence block and the clamp notes are what every resource-touching MR carries today.
gitops-ai/rightsize-mimir-querier-94a2 into main · 1 file · +2 −3The querier component of the mimir HelmRelease in monitoring requests 5.6 GiB of memory per replica and has used at most 80 MiB in the last 30 days, across all 8 replicas. This MR brings the request to 96 MiB and the limit to 128 MiB, and removes the CPU limit.
The workload is flat (peak / p95 = 1.06), so request and limit are kept close: the request covers the observed peak, the limit leaves about 1.5× peak of headroom. Values are rounded up to the nearest standard step.
About 44 GiB of reserved memory across 8 replicas, which the scheduler can now give to other pods.
| Signal | Value | Source |
|---|---|---|
| Classification | flat | peak / p95 ≤ 1.25 |
| Volatility (peak / p95) | 1.06 | working set, 30d |
| Namespace concurrency | 14.2% | joint vs summed peaks in monitoring |
| Memory avg / p95 / peak | 64Mi / 76Mi / 80Mi | max across replicas |
| Replicas observed | 8 | 30-day window |
| OOM kills / restarts (24h) | 0 / 0 | kube-state-metrics |
| Current request / limit | 5.6Gi / 5.6Gi, CPU limit 2800m | HelmRelease values |
| Applied request / limit | 96Mi / 128Mi, no CPU limit | after clamps |
values:
querier:
replicas: 8 # sized for query fan-out, see runbook
resources:
requests:
cpu: 500m
- memory: 5.6Gi
+ memory: 96Mi
limits:
- cpu: 2800m
- memory: 5.6Gi
+ memory: 128Mi
mimir HelmRelease; querier pods roll one at a time.container_memory_working_set_bytes, container querier).OOMKilled terminations on the querier over the next week.Revert this MR. The next nightly scan re-measures either way, and a workload that starts hitting its limit is picked up by the "over 85%" rule.
Checked the cited peak against the tenant's 30-day data and the chart's values path for querier.resources. Request covers peak, limit is within bounds. LGTM.
The model proposes a number. Measured data and fixed rules decide whether it ships.