01 · Read everything
Flux status and an hour of Warning events.
Every two hours the scan lists all HelmReleases and Kustomizations
and picks out the ones that are Ready=False for a real
failure reason: upgrade failed, build failed, health check failed,
dependency not ready and so on.
It also collects every Warning event from the last hour and groups
them by the object they are about. An object needs at least five
warnings before it is worth a look.
02 · Spot churn is not an incident
A node going away on schedule is filtered, carefully.
On a spot cluster, nodes are reclaimed and replaced all the time,
and each replacement produces a handful of node warnings. Those are
dropped only when both hold: the node carries the provider's
spot label, and every reason in the group is a known churn reason.
Anything else on the same node, such as memory pressure or a
read-only filesystem, is still reported, spot or not.
03 · A pod starting up is not broken
Probe bursts on healthy workloads fold into one digest.
Rescheduled pods fail a few readiness checks while they boot. If a
group of warnings is only probe failures, and the workload is
fully available right now, every pod Ready with zero restarts and
no OOM kills, it does not get its own issue.
Those groups fold into one low-risk digest per cluster, so the
pattern stays visible without paging anyone.
04 · Drift, once, not per release
Twelve drifting releases, one digest.
With Helm drift detection in warn mode, healthy releases emit a
drift warning on every reconcile. Rather than twelve
"investigate this HelmRelease" issues, drift goes to a single
per-cluster drift digest.
The digest reports. It never proposes switching drift correction on
or ignoring fields: that decision stays yours.
05 · What is left is real
One release that stopped trying.
payments-api failed its upgrade and exhausted its
retries. That matters: a stalled HelmRelease will not retry
on its own, and its parent Kustomization shows as failing too.
The parent Kustomization's issue does not stop at "dependency not
ready": it names the child release that is holding it up, with that
release's recent events and, where it can find the pods, a log tail.
06 · Which change did this?
A join with the MRs merged just before it broke.
The failure's first-seen time is joined with MRs merged in the 30
minutes before it. Candidates are scored on the files they touched,
the names they share with the failing object, and how close in time
they landed. Weak matches are discarded rather than guessed.
Here one MR stands out: the chart upgrade for
payments-api, merged seven minutes earlier.
07 · Grouped by cause
From 63 events to a short list.
Scheduling failures on a cluster that is simply out of memory
become one capacity issue, not one per workload. An ExternalSecret
that cannot sync is linked to the workloads that consume its
Secret, so the chain reads as one story.
Grouping findings that share an identical error message (one
missing CRD version behind eleven failing releases) is
In development.
08 · Closed when it is fixed
Quiet is not the test. Healthy is.
A Flux failure issue is closed when the release is back to
Ready=True at or above the version that failed, and
stays that way. A lull in events is not enough on its own.
Warning-event findings are short-lived by design: they re-alert
daily while the problem continues, and stop when it does.