A webhook signing key was created with kubectl create secret
during an incident. Another one sits base64-encoded in a YAML file in
the repo. Both work, until the namespace is rebuilt or someone rotates
the key. The assistant turns each into an ExternalSecret backed by your
Vault, seeds Vault from the running value, and checks every diff it
writes for credentials that should not be there.
The assistant lists Secrets in application namespaces and looks for an ExternalSecret that produces each one. It reads names, types and key names only: its cluster access is read-only and it never sees a secret value.
Here: payments/stripe-webhook, two keys, created by hand,
and a committed smtp-relay Secret in the repo.
Secrets written by a chart's own bootstrap Job, or named after a Helm release in the same namespace, are left alone: re-shipping them would fight the chart.
TLS and CA material goes to review, because a key-value store path that turns out empty would overwrite a working certificate. A committed Secret whose values are plainly not credentials is proposed as a ConfigMap instead.
The ExternalSecret mirrors the Secret's keys, one
remoteRef per key. When the cluster has exactly one
ClusterSecretStore it is used; with several, your configured default
wins, otherwise the MR asks you to choose.
For a committed Secret, the same MR deletes the static manifest, so git and the ExternalSecret never both own it.
vault kv put.An ExternalSecret pointed at an empty path would replace a working Secret with an empty one. So the MR pairs it with a PushSecret that copies the live value into Vault once, on merge, and never overwrites or deletes it afterwards.
The value travels from the cluster to Vault inside your cluster. It never passes through the assistant, the model, or the MR.
When a HelmRelease is migrated or adopted, its values are scanned for
credential-shaped leaves: a password key, a token pasted into a
startup script. Each is rewritten to a reference, and one
ExternalSecret supplies them all through valuesFrom.
Anything ambiguous is declined, not guessed: the MR then carries a scaffold and a note instead.
Every diff from every agent is checked for literal-looking credentials: API keys, tokens, passwords, client secrets. A hit withholds the MR.
ExternalSecret edits get their own checks: a renamed
remoteRef key, a template reverted to the old engine,
a rewrite that matches everything, deletionPolicy
flipped to Delete.
A diff that makes a workload reference a Secret that exists nowhere,
in git or in the cluster, is held: the next pod start would fail
with CreateContainerConfigError.
When an ExternalSecret stops syncing, the issue names the workloads that consume its Secret, so the stale credential is fixed before the next rollout picks it up.
When the External Secrets Operator stops serving an old API version, every Helm release that still renders it fails at its next upgrade. Those failures are reported today, one issue per release; grouping them into one issue per cause is built and not yet switched on.
Roadmap Before an operator upgrade, sweep live ExternalSecrets for defaults the new version removes (such as a stored template engine version) and propose declaring them.
apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: { name: stripe-webhook, namespace: payments } spec: refreshInterval: 1h secretStoreRef: { kind: ClusterSecretStore, name: vault } target: { name: stripe-webhook, creationPolicy: Owner } data: - secretKey: signing-secret remoteRef: { key: payments/stripe-webhook, property: signing-secret } - secretKey: api-key remoteRef: { key: payments/stripe-webhook, property: api-key }
kind: PushSecret spec: updatePolicy: IfNotExists # never overwrite what Vault holds deletionPolicy: None # removing this never deletes the Vault data selector: { secret: { name: stripe-webhook } }
values: db: host: orders-db - password: s3cr3t-from-2021 migrate: command: - - sh -c "curl -H 'Token: 9f2c…' http://…" + - sh -c "curl -H \"Token: ${MIGRATE_TOKEN}\" http://…" + env: [ { name: MIGRATE_TOKEN, valueFrom: { secretKeyRef: … } } ] + valuesFrom: + - kind: Secret + name: orders-api-externalized
✗ shop/orders-api no matches for kind "ExternalSecret" in version "external-secrets.io/v1alpha1" ✗ shop/cart-api no matches for kind "ExternalSecret" in version "external-secrets.io/v1alpha1" ✗ payments/billing-api no matches for kind "ExternalSecret" in version "external-secrets.io/v1alpha1" … 8 more, same message
Illustrative, shaped like the real conversion MR: same title format, same PushSecret pairing, same pre-merge conditions.
gitops-ai/convert-secret-payments-stripe-webhook into main · 3 files · +34 −0Secret payments/stripe-webhook exists in the cluster with no ExternalSecret and no manifest in git. It cannot be recreated if the namespace is rebuilt, and rotating it means editing the cluster by hand. This MR makes Vault its source of truth.
vault (the only store in this cluster).updatePolicy: IfNotExists, deletionPolicy: None).apps/payments/kustomization.yaml.The assistant reads Secret metadata only and cannot see the PushSecret's sync status. Check it after merge (below).
payments/stripe-webhook (the PushSecret needs write once).kubectl -n payments get pushsecret stripe-webhook: Synced.kubectl -n payments get externalsecret stripe-webhook: SecretSynced, same two keys.With creationPolicy: Owner, deleting this ExternalSecret later also deletes the Secret. deletionPolicy: Retain does not prevent that.
+ apps/payments/externalsecret_stripe-webhook.yaml (new, 17 lines) + apps/payments/pushsecret_stripe-webhook.yaml (new, 15 lines) apps/payments/kustomization.yaml + - externalsecret_stripe-webhook.yaml + - pushsecret_stripe-webhook.yaml
Key names match the live Secret, store name matches the cluster's only ClusterSecretStore, no value appears in the diff. LGTM.
Secret changes fail quietly and late. Most of the rules below exist because of a failure with exactly that shape.