Incidents do not respect team boundaries. The cause of a Kubernetes alert can be a network uplink; the cause of a failed Ansible run can be a Vault pod three layers down that nobody noticed was sealed. This story shows where the assistant is heading: joining what it knows across inventory, network, CI, cluster and cloud, so you get one root-cause finding instead of a pile of symptoms. Most of it is designed, not yet built, and each step says which.
orders-api is timing out, payments reports
its database unreachable, a node exporter goes quiet, a queue backs
up. Each alert, read on its own, points at a different service and
a different owner.
Today each one gets its own investigation and, if it warrants one, its own issue. Sixteen honest findings about the wrong thing.
A deterministic co-firing correlator already knows that
KubeletDown explains the pod alerts on that node, and
that a scraper going dark makes its targets look down. It elects a
root and annotates the others as dependents, confined to one tenant
so an outage elsewhere is never mis-labelled as "fix the root".
It ships switched off and only annotates; it does not yet collapse N issues into one. And it cannot see what is outside the cluster: a switch, an uplink, a VLAN.
Each alert's instance or node label is
joined, in-process, to a NetBox record: address to host, host to
interface, interface to switch port and VLAN. Fourteen of the
sixteen alerts land on hosts behind the same access switch uplink.
The join is strict. A key that matches two hosts abstains rather than guessing, and "not in NetBox" is only said when the read was complete.
For VM and bare-metal estates monitored by Zabbix, its problems become one more alert source and its host inventory (OS, hardware, resources) sits beside NetBox as observed context.
Here Zabbix raised an interface-down problem on the switch uplink about ninety seconds before the first Kubernetes alert. Two independent sources, one cause.
Inventory is core-infrastructure detail. Addresses, MAC addresses, serials, VLAN ids, racks and site names are used for the join and then dropped. What reaches an LLM is a closed-schema card: role, platform family, status, update cadence, whether an owner is declared and an opaque team handle.
Topology is expressed as relations between opaque handles, not as invented addresses. Any field not on the allowlist is dropped by default, and a canary test proves no raw value survives.
The dependents are folded into a single issue that names the shared uplink, lists every alert it explains, and states what could not be joined. The two alerts that did not fit the pattern keep their own investigation.
The scheduled Ansible apply mints its SSH certificate from Vault
before any play runs. Since last night it fails at that step:
HTTP 503, Vault is sealed. The playbook repo is fine.
The problem is somewhere else.
The failure signature (vault … 503 … sealed) is
classified deterministically. The CI script itself names the Vault
address and auth mount, so the dependency is declared, not guessed.
The cluster map resolves that Vault to its HelmRelease in the
cluster repo.
In the cluster, the Vault pods are Running, 0/1 ready,
zero restarts. The one auto-unseal attempt at startup failed
against Google Cloud KMS, and nothing ever tried again.
A rule in the availability pack looks for exactly this shape: a
Vault HelmRelease with KMS auto-unseal and no liveness probe.
Readiness only takes a sealed pod out of the Service; a liveness
probe on /v1/sys/health restarts it, and the restart
retries the unseal.
The rule fires on its own, outage or not. The attribution in step 8 is what lets the MR cite the playbook failures as evidence.
name: db-07.dc1.example primary_ip: 192.0.2.47 interface: eth0 · aa:bb:cc:… switch_port: sw-dc1-r4 Gi1/0/17 vlan: 112 serial: XK4… owner: (a person) comments: free text
host: h-5c1e role: database platform: vm status: active update_cadence: monthly owner_declared: true owner_team: team-91a0 address_scope: private behind: sw-3f02 (uplink u-1)
$ ci/mint-ssh-cert requesting fleet SSH certificate from Vault vault PUT auth/fleet/login: HTTP 503 ['Vault is sealed'] ✕ job failed before ansible-playbook started scheduled applies, newest first ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ … 12 in a row
spec: chart: { spec: { chart: vault } } values: server: ha: enabled: true config: | seal "gcpckms" { key_ring = "…" crypto_key = "vault-unseal" } # no livenessProbe
Illustrative, and on the roadmap as a whole: the probe values and the reasoning are the ones the rule is being built with; the cross-repo evidence section is the planned attribution step.
gitops-ai/vault-autounseal-liveness into main · 1 file · +7 −0This Vault unseals itself through Google Cloud KMS. Vault attempts auto-unseal once, at startup. If that attempt fails, the pod stays sealed and Running indefinitely: readiness only removes it from the Service, and nothing restarts it.
This MR enables the chart's liveness probe on /v1/sys/health?standbyok=true. A sealed pod fails the probe, is restarted, and the restart retries the unseal.
initialDelaySeconds: 60, then failureThreshold: 6 × periodSeconds: 10 with a 3 s timeout: a KMS blip of up to a minute does not cause a restart loop.standbyok=true keeps healthy standbys at 200. It does not mask 474 (a standby that lost its active node), so that case restarts too.If this Vault is uninitialised (501) or runs as an Enterprise DR / performance standby (472 / 473), this probe would restart-loop it. Check that neither applies before merging.
The scheduled Ansible apply in the playbook repository has failed 12 times in a row at the certificate step with HTTP 503 ['Vault is sealed']. The CI script declares this Vault's address and auth mount, so the failures are attributed here, to the repository that owns the Vault.
| Evidence | Source |
|---|---|
| 12 consecutive failed scheduled applies, same signature | playbook repo · CI job logs |
| Vault address + auth mount named in the CI script | playbook repo · declared edge |
| 3 pods Running, 0/1 ready, 0 restarts, sealed | cluster · read-only |
| KMS auto-unseal, no liveness probe | cluster repo · this HelmRelease |
A comment on the latest failed run links this MR. No change is proposed there: the playbook is correct, it was the first thing to notice.
valuesFrom or post-renderer could override the probe.vault status shows Sealed: false on all replicas.Revert this MR. The probe is the only change.
server:
ha:
enabled: true
config: |
…
+ livenessProbe:
+ enabled: true
+ path: "/v1/sys/health?standbyok=true"
+ initialDelaySeconds: 60
+ periodSeconds: 10
+ failureThreshold: 6
+ timeoutSeconds: 3
Confirmed the seal stanza is live config and KMS-backed, and that the probe path tolerates standbys. The restart-on-sealed behaviour is the intended fix. LGTM.
Joining domains multiplies what can go wrong. These are the design rules, not afterthoughts.