A database migration is booked for Saturday morning. Without a silence, it pages the on-call engineer four times. With a silence clicked in the UI, it might still be muting critical alerts in three years. Here the window is a small YAML file in git with a hard end time, and everything downstream follows from it.
A real audit of one estate found silences created years earlier with an end date more than a century away, a two-word comment, and matchers for alert rules that no longer existed anywhere. They muted nothing useful and hid whatever came next under the same name.
The fix is not a better UI. It is making every silence a reviewed change with a reason and an end.
Silences live in your alerting-config repo under
silences/<tenant>/<id>.yaml: matchers, a
start, an end at most 30 days later, a comment and a link to
the change that justifies it.
The rare genuinely long-lived silence has a loud escape hatch,
permanent:, which requires a reason, an approver and a
review_by date.
The merge request pipeline runs a small, self-contained validator: unknown keys, duplicate keys and multi-document files are refused, the tenant must match its directory, the id must match the file name, the window must be bounded, every regex must compile.
It runs with no network and no secrets, because in many Alertmanager setups a "read-only" certificate can also write. The same validator is the loader the assistant uses, so review and runtime never disagree about the schema.
After merge, a job on the default branch prints a plan, then creates the silence. Each silence carries a sentinel comment with its id and generation, so a past window is never re-created and an edited end time is reconciled on the next pass.
The lint ships with the assistant; the applier is a small, tested script that runs in your pipeline. If a tenant cannot be read, the job fails instead of guessing.
An Alertmanager silence does nothing for a PagerDuty incident that was already open: the escalation policy keeps re-notifying. So every 15 minutes the same pipeline snoozes open incidents that a live declared silence covers, until the silence ends.
It also resolves incidents whose alert group has stopped firing, which a silenced alert would otherwise never do on its own.
The migration trips PostgresReplicationLag at 06:40.
Alertmanager matches it to the silence and sends nothing.
The assistant's alert triage still sees it, with the silence attached as context: silenced alerts are annotated, not dropped, so if something outside the planned change breaks, the triage knows which alerts were expected.
Two hours and again thirty minutes before the end, if alerts covered
by the silence are still firing, a Slack message says so.
That is the moment to extend it deliberately (edit
ends_at in a new MR) or accept the page.
At ends_at the silence expires. Nothing has to be
remembered or cleaned up.
A read-only audit walks every tenant's live silences and the loaded rule set. Deterministic rules flag an unbounded or over-30-day expiry, a silence that matches no loaded rule, an empty or broken matcher, and a permanent grant whose review date has passed.
A second report inventories the other ways an alert can be muted (inhibit rules, mute intervals, severity demotions) and their expiry debt. If a source cannot be read, the answer is "unknown", never "clean".
Planned: when the assistant proposes a risky upgrade hop, it also drafts the matching silence file as a companion MR, sized to the window the hop needs. Recurring maintenance windows (a weekly patch slot, as a route-level mute interval) are also planned.
Today your team writes the silence file. The rest of this page runs in production.
+tenant: acme # must match the directory +doc_id: orders-db-pg16-2026-09-26 # must match the file name +generation: 1 # bump only to re-issue +matchers: + - name: namespace + value: orders-db + - name: alertname + value: Postgres(ReplicationLag|Down|TooManyConnections) + isRegex: true +starts_at: 2026-09-26T06:00:00Z +ends_at: 2026-09-26T10:00:00Z # ≤ 30 days after start +comment: "orders-db Postgres 15 → 16 migration, primary failover expected" +link: https://git.example.com/acme/platform/-/merge_requests/318
$ python3 ci/silences_lint.py --self-check silences/ ✓ self-check: validator matches the vendored checksum ✓ acme/orders-db-pg16-2026-09-26.yaml schema · keys · types ✓ tenant "acme" matches directory ✓ window 4h (cap 30d) ✓ 2 matchers · regex compiles note: regex checked with Python re; Alertmanager uses RE2 0 failing · 0 skipped
tenant acme live sentinels: 3 declared: 4 CREATE orders-db-pg16-2026-09-26@g1 06:00Z → 10:00Z skip batch-reindex-2026-09-12@g1 past window, never re-created ok search-rebuild-2026-09-24@g2 content matches comment: "gitops-silences:orders-db-pg16-2026-09-26@g1: orders-db …" applied 1 · 0 errors
SNOOZE incident PostgresDown orders-db until 10:00Z covered by orders-db-pg16…@g1 RESOLVE incident CronJobFailed batch alert group no longer firing in any tenant note added to both incidents with the silence id
orders-db is inside a declared silence for the Postgres 16 migration.orders-db-pg16-2026-09-26 ends in 2h. 1 covered alert still firing: PostgresReplicationLag.Today your team opens this MR; drafting it alongside an upgrade is on the roadmap. Everything after "merge" runs in production now. Illustrative example.
silence/orders-db-pg16 into main · 1 file · +13 · pipeline passedThe orders-db cluster moves from Postgres 15 to 16 on Saturday. The operator performs a switchover, so PostgresDown and PostgresReplicationLag will fire for a few minutes each. This silences that alert family for orders-db only, for four hours.
Any other namespace; any other Postgres alert name; the application's own SLO alerts. If orders-api starts failing checkout, you will still be paged.
+tenant: acme +doc_id: orders-db-pg16-2026-09-26 +generation: 1 +matchers: + - name: namespace + value: orders-db + - name: alertname + value: Postgres(ReplicationLag|Down|TooManyConnections) + isRegex: true +starts_at: 2026-09-26T06:00:00Z +ends_at: 2026-09-26T10:00:00Z +comment: "orders-db Postgres 15 → 16 migration, primary failover expected" +link: https://git.example.com/acme/platform/-/merge_requests/318
CREATE orders-db-pg16-2026-09-26@g1 and applies it.ends_at in a new MR; the reconcile picks it up.Silence is bounded (4h) and matches 3 loaded rules in tenant acme: no findings in the next audit.
A silence is a way to stop people being told something. It gets stricter handling than any other config.