Spot capacity is cheap because the provider can take a node back at any moment. Some teams even treat that as a free chaos monkey. It only works if every workload is laid out to survive losing a node. This is how the assistant finds the ones that are not, and what it proposes so the next preemption is a non-event.
The cluster-state scan records the shape of each cluster: how many availability zones it spans, whether it has spot nodes, and whether it runs only on spot. It reads node labels and nothing else.
Here: three zones, four spot nodes. checkout-api runs
two replicas, but both landed on the same node.
payments-operator and orders-worker run a
single replica each, on that same node.
The provider reclaims node a-1 with a few seconds of
notice. Everything on it stops at once.
checkout-api had two replicas and still went fully
dark: replicas that share a node share its fate. Until the scheduler
finds room and the new pods pass their readiness checks, nothing
serves checkout.
The same gaps are visible in git long before any node dies. The availability rules walk every workload daily and check it against the cluster it runs on:
This agent does not call a language model at all. Each rule is a plain check with a fixed fix, so a finding can be reproduced and explained line by line, and it costs nothing to run every day.
Rules that need topology skip themselves when the zone count is unknown, rather than guessing. When no clean diff can be written, the finding becomes an issue instead of a risky MR.
On a multi-zone cluster, a multi-replica workload without
topologySpreadConstraints gets two: one keyed on the node,
one on the zone, each with maxSkew: 1 and the
workload's own selector labels.
Both are ScheduleAnyway: a best-effort spread that can
never leave a pod Pending when capacity is short. You can tighten it
to DoNotSchedule if you have guaranteed room.
A multi-replica workload with no PodDisruptionBudget gets one, in a new file next to the workload, wired into the directory's kustomization.
To be precise about what that buys: a PDB governs voluntary evictions, such as node drains, pool upgrades and autoscaler scale-downs. A spot preemption does not ask permission. That is what the spread in step 05 is for. The two cover different failures, so you get both.
A controller-manager running alone gets a proposal for 3 replicas: leader election means only one works at a time, the others take over when its node dies. It ships as a "judgment call" draft, and it is skipped for controllers whose standbys cannot pass readiness.
On a spot-only cluster, a terminationGracePeriodSeconds
above 30 is trimmed to 30: spot never waits longer, and a long grace
also slows every voluntary drain. Trimming further, to 15 s when
every live pod is provably on spot, is
In development.
With the replicas spread, losing a-1 costs one
checkout-api replica, not the service. The operator's
standby in another zone takes the lease. The lost pods are
rescheduled in the background.
orders-worker still has one replica, so it still
blips. Proposing extra replicas for ordinary single-replica apps,
with evidence that the app tolerates it, is
Roadmap: some singletons are singletons
on purpose.
if kind in (Deployment, StatefulSet, HelmRelease) and replicas ≥ 2 and not has_topology_spread(workload) and cluster.availability_zones ≥ 2: # unknown → skip propose(spread_by_host_and_zone(workload)) # no clean diff → issue
template: spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: checkout-api + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: checkout-api containers: - name: api
+apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: checkout-api-pdb + namespace: shop +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: checkout-api
resources: - deployment.yaml - service.yaml + - poddisruptionbudget_checkout-api.yaml
spec: - replicas: 1 + replicas: 3 template: spec: containers: - args: ["--leader-elect"] # standbys wait for the lease
spec: - terminationGracePeriodSeconds: 120 + terminationGracePeriodSeconds: 30
One rule, one small MR. This is the spread proposal for checkout-api; the PodDisruptionBudget arrives as its own MR so each can be reviewed and reverted on its own. Illustrative, shaped like the real output.
gitops-ai/topology-spread-checkout-api-7b2e into main · 1 file · +14Deployment shop/checkout-api runs 2 replicas on a 3-zone cluster but declares no topologySpreadConstraints. Multiple replicas landing on the same node defeats the HA intent: a single-node failure takes all replicas down at once.
Two soft constraints on the pod template: spread by kubernetes.io/hostname and by topology.kubernetes.io/zone, maxSkew: 1, selecting on the workload's own labels.
A hard DoNotSchedule can leave a replica Pending when the cluster cannot satisfy the skew, for example while a spot node is being replaced. Tighten it only if you have guaranteed node and zone capacity.
A separate draft adds a PodDisruptionBudget (minAvailable: 1) for the same workload.
To opt out for this resource, add the skip annotation with the value availability.
| Signal | Value | Source |
|---|---|---|
| Availability zones | 3 | node zone labels, last cluster scan |
| Spot nodes present | yes | node spot / preemptible labels |
| Spot only | yes | no on-demand pool |
| Effective replicas | 2 | spec.replicas in git |
| Existing spread / PDB | none / none | workload and sibling files |
No language model was used for this finding or this diff.
template:
spec:
+ topologySpreadConstraints:
+ - maxSkew: 1
+ topologyKey: kubernetes.io/hostname
+ whenUnsatisfiable: ScheduleAnyway
+ labelSelector:
+ matchLabels:
+ app.kubernetes.io/name: checkout-api
+ - maxSkew: 1
+ topologyKey: topology.kubernetes.io/zone
+ whenUnsatisfiable: ScheduleAnyway
+ labelSelector:
+ matchLabels:
+ app.kubernetes.io/name: checkout-api
containers:
- name: api
kubectl get pods -n shop -l app.kubernetes.io/name=checkout-api -o wide shows the two replicas on different nodes, in different zones.Revert this MR. The constraints are soft, so they never block scheduling in the meantime.
Selector labels match the Deployment's pod template; constraints are soft; the cluster has three zones. LGTM.
A rule that fires on the wrong workload is worse than no rule. Most of the work is in the cases where it stays quiet.