Home / Stories / Orphans & relics
Story · Hygiene & cost In development

The things nobody owns any more, found and written up with evidence.

Every long-lived estate collects leftovers: a directory of YAML that no Kustomization points at, a ServiceMonitor from a chart that was uninstalled a year ago, a 50 GiB volume nothing mounts, a database that was hibernated "for now". The assistant looks at git and the cluster side by side, proves what is unused, and hands you an issue. It never deletes anything itself.

Starts fromYour GitOps repositories and the live objects in each cluster.
Looks atFlux Kustomizations and their inventories, HelmReleases, monitors and policies, PVCs, pods and workload templates, CloudNativePG clusters.
You getIssues that name each leftover, why it is believed unused, and what was checked. Cleanup stays your decision.
01 · Two directions

Git without a cluster, and cluster without a git.

An orphan is a break in the chain between what is written down and what runs. The assistant walks it from both ends: YAML in git that nothing deploys, and live objects that nothing in git declares.

Each direction is a separate, deterministic scan. No model decides whether something is an orphan.

02 · YAML nothing deploys Live

A directory that no Kustomization reaches.

The daily wiring audit computes the closure of every Flux Kustomization over the repository tree. Files outside that closure are never applied, however correct they look.

Where the fix is a one-line stitch into a parent kustomization.yaml, you get a draft MR. A whole unreferenced directory becomes an issue, because only you know whether it is future work or a forgotten copy.

03 · Objects nothing declares Live

Uninstall residue: still running, owned by nobody.

Every four hours the cluster side is walked the other way. ServiceMonitors, PodMonitors, PrometheusRules and NetworkPolicies that are neither declared in a scanned repository nor managed by a live HelmRelease are reported as an issue.

A leftover NetworkPolicy still shapes traffic, and a stale PrometheusRule still pages. That is why these are worth finding.

04 · What prune: false keeps alive Live

Removed from git, never removed from the cluster.

Flux only garbage-collects when a Kustomization has prune: true. With it off, deleting a file in git leaves the object running. The assistant compares each Kustomization's own applied inventory with the live objects that carry its labels.

An object must show up as orphaned on three consecutive runs before it is reported, one digest per cluster. The assistant never proposes flipping prune: that is your call.

05 · Idle volumes In development

A volume nothing has mounted for months.

The relic census lists PersistentVolumeClaims older than a threshold (90 days by default) and checks them against every live pod and every workload template, including CronJobs and Deployments scaled to zero, so a paused consumer still counts as a consumer.

StatefulSet claims are matched to their ordinals: a volume reserved by a StatefulSet parked at zero replicas is left alone. Built and tested, opt-in, not yet switched on in production.

06 · Hibernated databases In development

A database put to sleep, that nothing will ever wake.

A CloudNativePG cluster with hibernation on keeps its storage and costs money. The census looks for any reference to its -rw, -r or -ro services in pod and workload env, args, commands and volumes.

No reference anywhere and it is reported as a relic. Secret and ConfigMap contents are not read, and the issue says so.

07 · Absence must be proven

"I could not see it" is never "it is not there".

An orphan claim is a claim of absence, so every read behind it must be complete. If listing HelmReleases, pods or workloads was denied, truncated or malformed, the scan withholds the verdict for that cluster and logs why, rather than report everything as unused.

Anything you want left alone can carry a skip annotation, and the scan will not mention it again.

08 · Where it is going Roadmap

Owners who left, siblings that drifted, and a price on each.

Planned: resources whose declared owner or team no longer exists, and environments that drifted apart from their siblings (a file that is identical in staging and production today, changed in only one).

Planned: an estimated monthly cost next to each relic, so the list can be sorted by what cleanup is worth. Today the issue carries the size and storage class, not a price.

gitwhat is written down
Fluxclosure + inventory
clusterwhat is running
four orphan shapes
  • in git, never applied wiring audit
  • in cluster, declared nowhere cluster-side orphan scan
  • deleted from git, still live prune posture digest
  • declared and live, but unused relic census
wiring audit · Kustomization closure over the repo tree
clusters/production/apps.yaml  → path: ./apps/production
apps/production/kustomization.yaml
   shop/orders-api/
   shop/checkout-web/
   payments/
 apps/production/shop/reports-worker/   # 3 files, in no resources: list
 apps/production/legacy-search/         # 7 files, whole directory unreferenced
reports-worker → draft MR (one-line stitch) · legacy-search → issue
cluster-side orphan scan · cluster prod-eu
  • ServiceMonitor shop/orders-api declared in git
  • PrometheusRule monitoring/kube-prometheus-stack managed by HelmRelease
  • PrometheusRule monitoring/elasticsearch-alerts chart uninstalled · declared nowhere
  • NetworkPolicy shop/allow-legacy-search declared nowhere
2 objects neither in git nor owned by a live release → issue
Kustomization flux-system/apps · prune: false
status.inventory # what Flux applied last reconcile
  shop_orders-api_apps_Deployment
  shop_orders-api__Service
  shop_checkout-web_apps_Deployment
live, labelled kustomize.toolkit.fluxcd.io/name=apps
  shop/orders-api        Deployment  in inventory
  shop/checkout-web      Deployment  in inventory
  shop/promo-banner      Ingress     not in inventory
  shop/promo-banner      Service     not in inventory
seen orphaned on run 1 · 2 · 3 → included in this cluster's digest
relic census · PersistentVolumeClaims older than 90 days
  • shop/data-orders-db-0 mounted by pod orders-db-0
  • shop/exports referenced by CronJob nightly-export (suspended)
  • search/data-opensearch-2 StatefulSet at 0 replicas · reserved
  • platform/data-auth-postgresql-0 1,280 days · 50Gi · no mount anywhere
  • shop/import-scratch 412 days · 200Gi · no mount anywhere
2 unattached volumes · 250Gi retained · opt-in lane
Cluster.postgresql.cnpg.io platform/auth-db
metadata:
  annotations:
    cnpg.io/hibernation: "on"
# consumer search in pods and workload templates
auth-db-rw   0 references
auth-db-r    0 references
auth-db-ro   0 references
# the live auth service connects to auth-db-v2-rw
hibernated, no consumer → relic (Secret/ConfigMap payloads not read)
read completeness · cluster staging-eu
  • list PersistentVolumeClaims complete
  • list pods complete
  • list workloads (CronJobs) 403 · incomplete
planned signalsroadmap
  • owner team no longer exists planned
  • staging and production twins diverged planned
  • estimated monthly cost per relic planned
# planned: sibling drift
apps/staging/shop/orders-api/configmap.yaml     sha 3f1c…
apps/production/shop/orders-api/configmap.yaml  sha 9a07…  # identical until last month
Illustrative. Not built yet.

The issue you get.

Orphans and relics arrive as issues, not merge requests: deleting data is a decision, not a diff. Illustrative example of the relic digest, shaped like the one the census writes.

Issue

Relic PersistentVolumeClaim digest (unattached-pvc-relic, 2 resources) on cluster prod-eu

opened by gitops-assistant · issue only, no branch, no diff
category:costsource:orphan_cluster_resourcerisk:low
OverviewEvidenceHow it decidedNext steps
Summary

Two PersistentVolumeClaims on prod-eu are older than 90 days and are not mounted by any live pod or referenced by any workload template, including suspended CronJobs and StatefulSets scaled to zero.

Is this retained data still needed, or should it be considered for garbage collection after verifying consumers and recovery requirements? This is an issue-only advisory; no destructive action is proposed.

PVCAgePhaseStorage classSize
platform/data-auth-postgresql-01,280 daysBoundssd50Gi
shop/import-scratch412 daysBoundstandard200Gi

Both rows: no PVC mount reference found in live pods or workload pod templates.

Checked before this issue was opened
  • Every live pod's volumes: no reference to either claim.
  • Every Deployment, StatefulSet, DaemonSet, Job and CronJob template, including paused and scaled-to-zero ones.
  • StatefulSet claim templates matched by ordinal, so a volume reserved by a parked StatefulSet is not reported.
  • All listings complete. A denied or partial read would have withheld this issue.
  • Neither volume carries the skip annotation.
  • Confirm nobody outside the cluster (a laptop kubectl cp, a migration plan) still expects the data.
  • Confirm a backup exists if the data may be needed later.
  • Remove the claim through git if it is declared there, so Flux does not recreate it.
  • Or keep it: add the skip annotation and this issue will not come back.
G
gitops-assistant deduplicated per volume set

The same set of volumes produces one issue, not one per run. A new unattached volume starts a new digest.

What keeps it honest.

An orphan report is a claim that something is unused. Every guard below exists because a wrong one costs data.

Issues, not deletionsRelics and cluster-side orphans are reported as issues. The assistant's cluster access is read-only, and it never proposes a delete for retained data.
Absence needs a complete readA denied, truncated or malformed listing withholds the verdict for that cluster instead of turning a failed read into "everything is orphaned".
Paused is not abandonedSuspended CronJobs, Deployments at zero and StatefulSets parked at zero replicas still count as consumers of their volumes.
Persistence before reportingprune: false leftovers must be seen on three consecutive runs, so a mid-rollout object is not reported.
Never flips your policyThe prune digest names the leftovers; it never proposes turning prune on. That stays an operator decision.
Opt out per objectA skip annotation on the resource silences it for good, for relics you keep on purpose.

Where this stands.

Live
  • Wiring audit: YAML no Kustomization reaches (stitch MRs, directory issues)
  • Cluster-side orphans: monitors, rules and NetworkPolicies declared nowhere
  • prune: false leftover digest from Flux inventories
  • Orphaned PodDisruptionBudgets in cluster-state checks
In development
  • Relic census: unattached PVCs past an age threshold (built, opt-in, not yet on in production)
  • Relic census: hibernated CloudNativePG clusters with no consumer (same)
Roadmap
  • Resources whose owner or team has left
  • Drift between sibling environments
  • Estimated monthly cost on each relic
← Previous storyNaming conventions