Silencing an alert is easy. Finding out why it fired, and changing the configuration so it does not fire again, is the part that gets postponed. The assistant picks up each new alert, investigates it with read-only tools, and asks one question before it writes anything: could a change in git have prevented this?
Every 15 minutes the assistant collects alerts from PagerDuty, from Alertmanager (what is firing right now, with silences), and from Prometheus alert history (which catches alerts that flap faster than a page).
Each alert is handled once. If the same rule fires on twenty filesystems, those collapse into one "this is widespread" case instead of twenty investigations.
The alert's target (kind, namespace, name) is resolved from its labels, and a snapshot of the live object and its git manifest is fetched before the investigation starts. If the alert body is thin, the rule definition itself is read.
If no tracked repository declares the workload, the outcome is already known: an issue, never an edit to an unrelated file.
The investigation is a loop of read-only tool calls against the cluster. The pod's last state says OOMKilled, exit code 137, seven restarts. Events confirm the pattern: back-off, restart, back-off.
The cluster identity behind these tools can read. It cannot change anything.
Memory working set over the last two days: it climbs every night during the catalogue reload, touches the 512 MiB limit, and the container is killed. Daytime use is well below.
That is not a leak (it drops back each time) and not random. It is a known peak that no longer fits.
Where you have enabled log access for a cluster, the assistant reads a short tail of the previous container, the one that died. Here the last line before the kill is the catalogue reload with a much larger item count than a year ago.
Without log access it does not guess. It says what it could not see and hands you the command to look.
Root cause: the catalogue grew, the reload's peak outgrew a memory limit set long ago. A limit is a manifest value, so the answer is yes, and the output is a draft MR.
Had the cause been outside git (a node problem, a workload in a repository you do not track, a bug that needs a code change), the answer would be no, and speculative "bump it just in case" fixes are explicitly not allowed.
Deterministic checks run on the drafted fix before anyone sees it:
Anything that fails is held for human review with the reason named.
The MR carries the alert, the evidence trail and the reasoning. The issue carries what was observed, the copy-paste commands to finish the diagnosis, and the likely fix for each outcome.
The loop does not end at the proposal. An issue whose alert stays quiet for long enough is closed with a note. A merged fix whose alert keeps firing is escalated with a link back to the MR.
→ prometheus_alert_rule("KubePodCrashLooping") expr: max_over_time(kube_pod_container_status_waiting_reason{ reason="CrashLoopBackOff"}[5m]) >= 1 → read_workload_manifest_from_git(Deployment, shop, orders-api) found: shop-deploy · apps/shop/orders-api/deployment.yaml resources.limits.memory: 512Mi requests.memory: 384Mi last change to resources: 14 months ago
→ kubernetes_describe_object(Pod, shop, orders-api-7d9f8c-x2kqp) lastState.terminated: reason: OOMKilled exitCode: 137 restartCount: 7 → workload_restart_history(shop, orders-api) 7 restarts / 48h · all OOMKilled · all between 02:00 and 02:20 → kubernetes_list_events(shop, orders-api) BackOff Back-off restarting failed container api ×23 Killing Container api exceeded its memory limit ×7
02:10:03 INFO scheduler: starting nightly catalogue reload 02:10:04 INFO catalogue: fetching snapshot from products-db 02:10:41 INFO catalogue: decoded 1,940,212 items (prev 1,211,480) 02:10:58 INFO catalogue: building search index (container terminated: OOMKilled)
orders-api in deployment.yaml, sized from the measured peak.resources: requests: - memory: 384Mi + memory: 512Mi limits: - memory: 512Mi + memory: 768Mi
Illustrative, shaped like the real output: the alert, the investigation trail and the sizing reasoning travel with the one-file change.
gitops-ai/alert-orders-api-oomkilled-5c81e0a2 into main · 1 file · +2 −2KubePodCrashLooping (critical) on shop/orders-api, container api, firing since 02:14. Third firing in 7 days.
The container is OOMKilled during the nightly catalogue reload (02:10). The catalogue grew from about 1.2M to 1.9M items; the reload briefly holds the old and new copies, and the peak now exceeds the 512 Mi limit set 14 months ago. Daytime usage (≈300 Mi) is unaffected. Memory drops back after each restart, so this is not a leak.
Request 384 Mi → 512 Mi, limit 512 Mi → 768 Mi. The observed peak is capped by the old limit, so the true need is not known exactly: the new limit leaves 1.5× the last observed peak, and the post-merge checks below confirm it.
Streaming the reload instead of double-buffering it would remove the peak altogether. That is an application change and is noted here for the service owners, not proposed.
| # | Tool (read-only) | Finding |
|---|---|---|
| 1 | prometheus_alert_rule | Crash-loop rule, 5 min window |
| 2 | read_workload_manifest_from_git | apps/shop/orders-api/deployment.yaml, limit 512Mi |
| 3 | kubernetes_describe_object | lastState OOMKilled, exit 137, 7 restarts |
| 4 | workload_restart_history | All 7 restarts between 02:00 and 02:20 |
| 5 | kubernetes_list_events | BackOff ×23, memory-limit kills ×7 |
| 6 | prometheus_query_range | Nightly climb to the limit, daytime ≈300Mi |
| 7 | kubernetes_pod_logs (previous) | Killed while building the index after reload |
Second pass (OOM class): confirmed under-resourced rather than leaking; memory returns to baseline after every restart.
containers:
- name: api
resources:
requests:
cpu: 250m
- memory: 384Mi
+ memory: 512Mi
limits:
- memory: 512Mi
+ memory: 768Mi
orders-api; pods restart once with the new limits.OOMKilled in kubectl get pod -n shop -l app.kubernetes.io/name=orders-api after 02:30.KubePodCrashLooping for shop/orders-api stays resolved.The assistant watches for it: a merged fix whose alert keeps firing gets an escalation issue and a comment on this MR.
Restart times match the reload window in the cited logs; the change touches only the alerted container's memory. Request stays below limit. LGTM.
An alert is a symptom. Most of the rules below exist to stop a confident-sounding fix for the wrong thing.