Home / Stories / Cluster state
Story · Operations Live

Sixty-three Warning events. One issue worth reading.

A busy cluster is never quiet. Spot nodes come and go, pods fail a few probes while they start, Flux reports harmless drift on every reconcile. Somewhere in that noise is the one release that is actually broken. The assistant reads all of it every two hours, throws away what it can prove is routine, and turns the rest into a short list of issues, grouped by cause.

Starts fromEvery Flux HelmRelease and Kustomization, and the last hour of Warning events, every two hours.
Looks atFlux conditions, events, pod state and previous-container logs, node spot labels, recently merged MRs, the manifests in git.
You getA short list of issues, grouped by cause, with the evidence, a recovery walk and the likely culprit MR, closed again once the cluster is healthy.
01 · Read everything

Flux status and an hour of Warning events.

Every two hours the scan lists all HelmReleases and Kustomizations and picks out the ones that are Ready=False for a real failure reason: upgrade failed, build failed, health check failed, dependency not ready and so on.

It also collects every Warning event from the last hour and groups them by the object they are about. An object needs at least five warnings before it is worth a look.

02 · Spot churn is not an incident

A node going away on schedule is filtered, carefully.

On a spot cluster, nodes are reclaimed and replaced all the time, and each replacement produces a handful of node warnings. Those are dropped only when both hold: the node carries the provider's spot label, and every reason in the group is a known churn reason.

Anything else on the same node, such as memory pressure or a read-only filesystem, is still reported, spot or not.

03 · A pod starting up is not broken

Probe bursts on healthy workloads fold into one digest.

Rescheduled pods fail a few readiness checks while they boot. If a group of warnings is only probe failures, and the workload is fully available right now, every pod Ready with zero restarts and no OOM kills, it does not get its own issue.

Those groups fold into one low-risk digest per cluster, so the pattern stays visible without paging anyone.

04 · Drift, once, not per release

Twelve drifting releases, one digest.

With Helm drift detection in warn mode, healthy releases emit a drift warning on every reconcile. Rather than twelve "investigate this HelmRelease" issues, drift goes to a single per-cluster drift digest.

The digest reports. It never proposes switching drift correction on or ignoring fields: that decision stays yours.

05 · What is left is real

One release that stopped trying.

payments-api failed its upgrade and exhausted its retries. That matters: a stalled HelmRelease will not retry on its own, and its parent Kustomization shows as failing too.

The parent Kustomization's issue does not stop at "dependency not ready": it names the child release that is holding it up, with that release's recent events and, where it can find the pods, a log tail.

06 · Which change did this?

A join with the MRs merged just before it broke.

The failure's first-seen time is joined with MRs merged in the 30 minutes before it. Candidates are scored on the files they touched, the names they share with the failing object, and how close in time they landed. Weak matches are discarded rather than guessed.

Here one MR stands out: the chart upgrade for payments-api, merged seven minutes earlier.

07 · Grouped by cause

From 63 events to a short list.

Scheduling failures on a cluster that is simply out of memory become one capacity issue, not one per workload. An ExternalSecret that cannot sync is linked to the workloads that consume its Secret, so the chain reads as one story.

Grouping findings that share an identical error message (one missing CRD version behind eleven failing releases) is In development.

08 · Closed when it is fixed

Quiet is not the test. Healthy is.

A Flux failure issue is closed when the release is back to Ready=True at or above the version that failed, and stays that way. A lull in events is not enough on its own.

Warning-event findings are short-lived by design: they re-alert daily while the problem continues, and stop when it does.

cluster prod-eu · Warning events · last 60 min63 events
Node/spot-b-7q2x     PreemptScheduled        ×3
Node/spot-b-7q2x     NodeNotReady            ×4
Pod/search-api-6c…   Unhealthy readiness probe ×9
Pod/web-5f8d…        Unhealthy readiness probe ×6
HelmRelease/grafana  DriftDetected           ×6
HelmRelease/loki     DriftDetected           ×6
…10 more releases    DriftDetected
HelmRelease/payments-api UpgradeFailed       ×5
Kustomization/apps   DependencyNotReady      ×7
63 events · 16 objects · 2 Flux objects Ready=False
spot-churn filter · Node groups
  • spot-b-7q2x · gke-spot=true · PreemptScheduled, NodeNotReady dropped
  • spot-c-k81m · gke-spot=true · TerminationNotice, NodeShutdown dropped
  • spot-a-2m9d · gke-spot=true · NodeNotReady, MemoryPressure kept
  • stable-b-01 · no spot label · NodeNotReady kept
unknown label, mixed reasons or pressure → always reported
probe-burst check · live workload state
  • search-api · readiness ×9 · 3/3 ready · 0 restarts → digest
  • web · readiness ×6 · 4/4 ready · 0 restarts → digest
  • ledger-sync · liveness ×8 · 1/2 ready · 4 restarts own issue
2 bursts folded into one low-risk probe digest · 1 kept
12 HelmReleasesDriftDetected · all Ready=True
Drift digest · prod-euone issue, one row per release
digest excerpt
release            ready   drift warnings
grafana            True    6
loki               True    6
… 10 more, all Ready=True
report only: no drift-correction or ignore-rule change proposed
flux get · prod-eu
KUSTOMIZATION  READY  REASON
apps           False  DependencyNotReady
  └ HelmRelease payments/payments-api not ready

HELMRELEASE                 READY  REASON
payments/payments-api       False  UpgradeFailed
  Stalled=True  RetriesExceeded
  values don't meet the specifications of the schema:
  additional property "replicaCount" is not allowed
stalled: helm-controller has stopped re-attempting · parent Kustomization blocked
deploy-failure correlator · MRs merged 30 min before first failure
  • !311 Upgrade payments-api chart 2.4.1 → 3.0.0 path ✓ name ✓ 7 min · 0.95
  • !309 Add dashboard for orders-api path ✕ name ✕ 22 min · 0.04
  • !310 Bump search-api image path ✕ name ✕ 12 min · 0.11
score = 0.50 · path + 0.30 · name + 0.20 · time proximity
below 0.30 → discarded · clear winner needs a 0.15 margin
63warning events
16objects
6after filters
5issues
issues filed this pass
  • Stalled Flux HelmRelease (retries exhausted): payments/payments-api risk:medium
  • Investigate Warning events: Deployment ledger-sync risk:medium
  • Kustomization apps: blocked by payments/payments-api risk:medium
  • Investigate Warning events: Node spot-a-2m9d risk:medium
  • Probe-burst digest · drift digest (existing, updated) risk:low
closure check · payments/payments-api
  • values fix merged replicaCount → replicas
  • Ready=True UpgradeSucceeded
  • deployed chart 3.0.0 ≥ flagged 3.0.0 ok
  • still healthy on the following sweeps sustained
Issue closed by the assistant
payments-api Ready at 3.0.0 across consecutive checks · closing

The issue you get.

A live failure has no diff to review, so this one arrives as an issue, not an MR. Illustrative, with the sections the assistant writes today. The parent Kustomization gets its own shorter issue pointing here.

Open

Stalled Flux HelmRelease (retries exhausted): payments/payments-api

issue in platform/prod-eu-deploy · opened by gitops-assistant
category:reliabilitysource:cluster_state_prod-eurisk:medium
OverviewLikely causeConditionsLive vs git
What is failing

HelmRelease payments/payments-api is Ready=False (UpgradeFailed) and Stalled with retries exhausted. The chart's values schema rejects replicaCount. Kustomization apps is blocked on it (DependencyNotReady). The previous release is still serving.

Recovery walk
  1. Stalled: the helm-controller has stopped re-attempting and will not retry on its own, but a spec or values push, or a chart-version change, resets the failure counters and re-attempts.
  2. Diagnose and fix the cause first: kubectl describe helmrelease.helm.toolkit.fluxcd.io payments-api -n payments, helm history payments-api -n payments.
  3. If the fix does not change the spec or values (for example a repaired registry or Secret), reset with flux reconcile helmrelease payments-api -n payments --reset. A plain reconcile does not clear the failure count.
Merged just before the first failure
MRPathNameBefore failureScore
!311 Upgrade payments-api chart 2.4.1 → 3.0.0apps/payments/payments-api/helmrelease.yaml7 min0.95

Chart 3.0.0 renames replicaCount to replicas and enforces a values schema. The HelmRelease still sets the old key, which the schema now rejects.

Suggested fix: rename the key in spec.values. Pushing the change also resets the stalled release.
Conditions on the HelmRelease
TypeStatusReasonMessage
ReadyFalseUpgradeFailedHelm upgrade failed for release payments/payments-api with chart payments-api@3.0.0: values don't meet the specifications of the schema(s): additional property "replicaCount" is not allowed
StalledTrueRetriesExceededFailed to upgrade after 4 attempt(s)
ReleasedFalseUpgradeFailedprevious revision (chart 2.4.1) still deployed
First seen

09:19, recorded once, so retries do not move the clock the correlator anchors on.

ObjectIn gitVerdict
HelmRelease payments/payments-apiapps/payments/payments-api/helmrelease.yamlin_git: a real failure, not uninstall residue
Kustomization appsclusters/prod-eu/apps.yamlin_git: blocked by the release above

The cluster access behind this issue is read-only. The fix is a change in git; this issue proposes it, it does not apply it.

G
gitops-assistant 2 sweeps later

Values fix merged; payments-api is Ready at chart 3.0.0 and has stayed healthy across consecutive checks. Closing.

What keeps it honest.

Filtering noise is only safe if the filter can never swallow a real problem. Every filter below fails towards reporting.

Positive evidence to dropNode churn is dropped only for nodes that carry the spot label. An unknown or unlabelled node is always reported.
Hard reasons always emitMemory pressure, read-only filesystems and OOM kills are reported even on a spot node, even if someone configured them as churn.
Healthy means proven healthyA probe burst is only folded when the workload is fully available with zero restarts. A pod with no status to read counts as not proven.
Folding keeps the evidenceDigests reproduce each member's warnings. Consolidation shortens the list; it never hides what was seen.
Weak blame is no blameA merged MR is only named as the likely cause when it scores clearly above the rest. An old failure is never pinned on a recent merge.
Closing needs health, not silenceIssues close on a sustained Ready state at the right version. Any unreadable state keeps the issue open.

Where this stands.

Live
  • Flux HelmRelease and Kustomization failure issues with recovery walks
  • Warning events with event tables and previous-container log tails
  • Spot-churn, probe-burst, drift and memory-pressure digests
  • Deploy-failure correlation with merged MRs
  • Live-vs-git evidence on failures
  • Closing Flux failure issues once healthy
In development
  • One issue per shared error message across many findings
  • Folding workload issues into the ExternalSecret that caused them (linking is live)
Roadmap
  • Recognising a spot replacement wave across many workloads and namespaces as one event
← Previous storyAvailability that survives spot preemption