Backups fail quietly. The nightly Velero run reports
PartiallyFailed because one volume could not be
snapshotted; the database backup has been failing since a credential
rotated. Nobody looks until the day a restore is needed. The assistant
checks every few hours and opens one issue per broken schedule, with
the evidence and the likely cause.
Backup objects and CloudNativePG Backup objects in the cluster.
The backup scan lists Velero Backup objects and
CloudNativePG Backup objects created in the last 48
hours (both configurable), straight from the cluster API, read-only.
Here: 14 Velero runs and 8 database backups in the window.
Velero runs in Failed or
PartiallyFailed count. A partial failure often means
the Kubernetes objects were saved but a volume was not: exactly the
part you need for a restore.
Runs are grouped by their Schedule, so seven failed nights become one finding, not seven.
For each failing schedule the scan collects the error messages of the most recent runs, the Schedule's spec and status from the cluster, and the Schedule and BackupStorageLocation manifests from your repo.
The error usually names the layer: storage credentials, bucket, snapshot permissions, a node agent.
Failed CNPG backups are grouped by the Cluster that owns
them. The Cluster's status and its spec.backup block
(object store, credentials reference, retention) are attached as
context.
The model reads the evidence and writes a short diagnosis: the most likely cause, what to check first, and how to confirm the fix. If the analysis is unavailable, the issue is still opened with the raw phase and error, never dropped.
Backup findings are opened as issues, labelled reliability, high risk. The fix is often outside git (a cloud permission, a bucket policy, a rotated key), and a wrong storage change can make things worse.
That caution is earned: a storage-location prefix removed to fix one error once made the whole location invalid, because the bucket was shared. The full issue is below.
If the BackupStorageLocation is Unavailable, Velero
attempts nothing, so there is nothing failed to find. Planned: check
the storage location itself, and the age of the newest successful
backup per schedule, so silence is not read as health.
Also planned: flag runs whose volume snapshot list is empty when the schedule includes volumes.
Planned: draft MRs for schedule and retention changes, and for a missing BackupStorageLocation, grounded in the bucket's actual layout before touching a prefix.
And when an upgrade bumps a backup plugin, compare the cloud permissions each version documents, so a new required permission is in the upgrade MR instead of in next month's failed restore.
phase: PartiallyFailed errors: 3 error: snapshot shop/data-orders-db-0: 403 permission denied (create snapshot not allowed for the backup identity) volumeSnapshotsAttempted: 3 completed: 0 # Schedule (cluster) spec.template: { includedNamespaces: [shop, payments], snapshotVolumes: true, ttl: 720h } status.lastBackup: 2026-09-23T04:00:12Z # git: infrastructure/velero/schedule_daily-full.yaml · backupstoragelocation_default.yaml
spec.backup: barmanObjectStore: destinationPath: s3://acme-backups/billing-pg s3Credentials: { accessKeyId: { name: billing-pg-s3, key: ACCESS_KEY_ID } } retentionPolicy: 30d status.lastFailedBackup: 2026-09-23T00:00:04Z
403. The backup identity can read disks but may not create snapshots.shop and check that volumeSnapshotsCompleted equals volumeSnapshotsAttempted.spec: schedule: "0 4 * * *" template: - ttl: 72h0m0s + ttl: 720h0m0s # match the 30-day retention the runbook promises snapshotVolumes: true
backup plugin v1.7 → v1.9, documented cloud permissions + one permission added in v1.9, missing from the current role
Illustrative, shaped like the real backup issue: same title format, same labels, same evidence sections. Backup findings are issues today, not merge requests.
prod · namespace veleroThe Velero schedule daily-full has produced PartiallyFailed backups on the last 2 runs in the 48-hour window. Kubernetes objects are saved; no volume snapshot completed (0 of 3), so persistent data in shop and payments is not restorable from these backups.
Every snapshot error is a 403 on snapshot creation. The backup identity can list disks but lacks permission to snapshot them. The Schedule and BackupStorageLocation in git are unchanged in this period, which points at the cloud role rather than the manifests.
Velero schedule daily-full: 2 backup runs failed. Restores from these runs would bring back objects without their data.
| Run | Phase | Error |
|---|---|---|
| daily-full-20260923040012 | PartiallyFailed | snapshot shop/data-orders-db-0: 403 permission denied |
| daily-full-20260922040009 | PartiallyFailed | snapshot shop/data-orders-db-0: 403 permission denied |
infrastructure/velero/schedule_daily-full.yamlinfrastructure/velero/backupstoragelocation_default.yamlvelero backup create verify-snapshots --include-namespaces shop.volumeSnapshotsCompleted equals volumeSnapshotsAttempted in velero backup describe.A backup alert that cries wolf gets muted, and a muted backup alert is how data gets lost.
PartiallyFailed is treated as a failure, because the missing part is usually the data.