Home / Stories / Vault audit logs
Story · Secrets & access In development

A service keeps getting "permission denied". Is it a missing grant, or someone probing?

Vault writes every request to its audit log, including the ones it refuses. Most teams only look at that log after an incident. The assistant reads it every day, groups the denials, decides which ones are a policy gap and which ones look like probing, and only ever proposes a policy change for the first kind.

Starts fromVault's audit log, streamed to Loki by your existing log collector.
Looks atDenied and successful requests per entity, the entity's bound policies, and your Vault policy repo in git.
You getOne rolling digest issue for everything suspicious, and a small draft MR for a clear, low-risk policy gap.
01 · The log you already have

Vault's audit device, tailed into Loki.

A stdout audit device makes Vault print one JSON record per request and one per response. Your log collector already ships pod logs to Loki, so the audit trail lands there with no extra agent.

The assistant runs a read-only LogQL query against that stream. It holds no Vault token and cannot change Vault.

02 · Find the refusals

Parse every record. Keep the denials.

Each audit record is parsed as JSON. A denial is a response that carries permission denied, together with who asked (the entity and its bound policies), what they asked for (path and operation) and when.

Records that do not parse are skipped and counted. A failed read never turns into "no denials".

03 · Group them

One row per (entity, path), not one alert per line.

246 denials in two days is usually one misconfigured client, not 246 problems. Denials are clustered by entity and path, with a count, first and last seen, and how many distinct path prefixes the entity was refused on.

A single occurrence is noise. A cluster needs to repeat before it is considered at all.

04 · Widen, probe or ambiguous

A fixed rule decides, not a model.

The classifier is deterministic code, with no LLM involved:

  • Probe: the path is sensitive (sys/, auth/, the audit device, key-management paths), or the entity is unknown, or it was refused on three or more different prefixes.
  • Widen: a known entity, refused again and again on one ordinary path prefix, with plain operations.
  • Ambiguous: everything in between.
05 · Probes and grey areas stay issues

One digest issue, refreshed every run.

Probe and ambiguous clusters are never turned into a policy change. They roll into a single audit-denial digest issue per Vault repo, with a cause table that is recomputed on every run, so you get one place to look instead of a new ticket per entity.

Each row says how the denials arrived: a steady rhythm reads as a misconfigured client, a burst reads differently. When there are too few timestamps to support that label, it is withheld and the shortfall is stated.

06 · A clear gap becomes a draft MR In development

Append one stanza to one policy. Nothing else.

For a widen cluster with exactly one candidate policy and one operation, the assistant renders the change against the policy file in git: it adds the missing capability to an existing path block, or appends a new block for the denied path. The rest of the file is left byte for byte as it was.

This path is built and switched on, but has not yet fired on a real production denial, which is why it is marked in development.

07 · What it will never widen

A safety layer that fails closed.

Every finding passes one guard before it is emitted. A widen can never target an escalation surface (sys/, auth/, identity/, CA and key management), never grant sudo, never touch an audit device, and never edit the protected root policy.

If the renderer is unsure about anything, the finding stays an issue.

08 · Did the grant land?

The next run checks for successes, not just silence.

For each entity the assistant also looks for its successful responses in the same window. "The denials stopped and the service now reads the path" is visible from the issue itself.

That lookup only runs when the entity is a real, unique Vault entity id. A shared login name or IP address would mix up clients, so the lookup is skipped and the issue says so.

Loki · {namespace="vault"} · audit streamlive tail
09:41:02 request  entity=8f2c… orders-api  read   secret/data/shop/orders-api/db
09:41:02 response entity=8f2c… orders-api  read   secret/data/shop/orders-api/db     200
09:41:07 request  entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key
09:41:07 response entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key  permission denied
09:43:15 response entity=1a90… ci-deployer update secret/data/platform/ci/token    200
09:51:07 response entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key  permission denied
09:52:40 response entity=—      approle     list   sys/policies/acl                  permission denied
10:01:07 response entity=8f2c… orders-api  read   secret/data/shop/payments/psp-key  permission denied
Vaultstdout audit device
Log collectorpod logs
Lokiaudit stream
Assistantread-only LogQL
one audit record, parsed
{
  "type": "response",
  "time": "2026-09-18T10:01:07Z",
  "auth": {
    "entity_id": "8f2c…",
    "display_name": "kubernetes-shop-orders-api",
    "policies": ["default", "shop-orders-api"]
  },
  "request": { "operation": "read", "path": "secret/data/shop/payments/psp-key" },
  "error": "1 error occurred:\n\t* permission denied\n\n"
}
  • entity real entity id · bound policy shop-orders-api
  • operation read → capability "read"
  • timestamp from the record itself, sanity-checked
denial clusters · last 48h
entitypathcountprefixes
orders-apisecret/data/shop/payments/psp-key2461
approle (no entity)sys/policies/acl311
report-jobsecret/data/* (5 prefixes)585
billing-workersecret/data/shop/billing/…122
ci-deployersecret/data/platform/tmp1below threshold
353 denials → 4 clusters worth a look · 1 single occurrence ignored
classify_cluster · deterministic
  • orders-api · psp-key known entity · 1 prefix · read → WIDEN
  • approle · sys/policies/acl sensitive path · no entity → PROBE
  • report-job · 5 prefixes breadth ≥ 3 → PROBE
  • billing-worker · 2 prefixes neither → AMBIGUOUS
widen → draft MR candidateprobe → issueambiguous → issue
Issue · Vault audit-denial digest · 3 causesupdated this run
category:securitysource:vaultrisk:high
causeclasstimeline
sys/policies/acl · approleprobe31 over 4h, single burst
report-job · 5 prefixesprobe58 over 2d, ~1 per 50m (steady)
billing-worker · 2 prefixesambiguousrate withheld (3 of 12 timestamped)
policies/shop-orders-api.hcl+4
path "secret/data/shop/orders-api/*" {
  capabilities = ["read"]
}

path "secret/metadata/shop/orders-api/*" {
  capabilities = ["list"]
}
+
+path "secret/data/shop/payments/psp-key" {
+  capabilities = ["read"]
+}
category:securitysource:vaultrisk:medium
safety.passes_all_guards · fail-closed
  • widen sys/policies/acl escalation surface · refused
  • widen pki/root/sign-intermediate CA management · refused
  • add sudo never renderable
  • touch sys/audit/* audit device is untouchable
  • edit the root policy protected · operator-only
  • read on secret/data/shop/payments/psp-key ordinary path · one policy · allowed
anything the guard cannot prove safe stays an issue
orders-api · psp-key · denials vs successes
policy merged denied succeeded
Interleaved successes: 212 since 14:05 · last denial 14:01 · the grant landed
Illustrative. Absence is stated too: "Interleaved successes: NONE found".

The merge request you get.

Illustrative. The widen path is built and enabled but has not yet fired on a real production denial, so this is the shape it takes, not a copy of one it sent.

Draft

Vault: grant shop-orders-api read on secret/data/shop/payments/psp-key

from gitops-ai/vault-widen-shop-orders-api-5b21 into main · 1 file · +4
category:securitysource:vaultrisk:mediumxrev::LGTM
OverviewEvidenceChangesBefore you merge
Summary

The Vault entity for orders-api (Kubernetes auth, namespace shop) was refused read on secret/data/shop/payments/psp-key 246 times over two days. Its only non-default policy is shop-orders-api, which grants nothing under shop/payments/.

Classification: widen

Known entity, one path prefix, one plain operation, no wildcard, not an escalation surface. That is the pattern of a missing grant, not of probing.

Is this access legitimate?

The assistant cannot know whether orders-api should read the payment provider key. That is a product decision. Confirm it before merging, or close this MR and fix the client instead.

Timeline

246 denials over 2d13h, ~1 per 15m (steady). First seen 2026-09-16 21:04Z, last seen 2026-09-19 10:01Z. A steady rhythm like this usually means a client retrying on a schedule.

Interleaved successes

1,380 successful responses for the same entity in the window, all on secret/data/shop/orders-api/*. None on the denied path.

Source query
{namespace="vault"} |= "permission denied" | json
  | auth_entity_id="8f2c…" | request_path="secret/data/shop/payments/psp-key"
 path "secret/metadata/shop/orders-api/*" {
   capabilities = ["list"]
 }
+
+path "secret/data/shop/payments/psp-key" {
+  capabilities = ["read"]
+}
Checked before this MR was opened
  • Target path is not under sys/, auth/, identity/, an audit device, or CA / key management.
  • Capability set is read only. No sudo, no new write verb.
  • The policy is not the protected root policy. The rest of the file is unchanged.
  • Exactly one candidate policy is bound to the entity.
After merge, expect
  • Your Vault repo's CI applies the policy on the default branch.
  • Denials for this cluster stop within the client's retry interval.
  • The next run reports successes on the path and closes the loop in the issue history.
R
independent reviewer second opinion, no pipeline context

Diff is a single read grant on one exact path, no wildcard. Grant scope matches the evidence. Merge is a product call on whether the service should hold this key. LGTM on shape.

What keeps it honest.

Vault is a high-blast-radius system. Everything here is deterministic code, with no model in the loop.

No Vault credentialsThe assistant reads the audit trail from Loki and the policies from git. It cannot read secrets or change Vault; your own CI applies merged policy.
Escalation surfaces are off limitsA widen on sys/, auth/, identity/, audit devices or CA and key management is refused, whatever the evidence says.
Probes are never "fixed"A denial that looks like probing becomes an investigation item, never a policy change that would make the probe succeed.
No invented factsRates and "steady" labels are withheld on thin samples; success lookups are skipped when the requester is not uniquely identifiable, and the issue says why.
Fails open for the run, closed for the findingA Loki or parse error never blocks the daily run, and never produces a finding. An unsure guard suppresses.
One digest, not a ticket stormIssue-only causes share one rolling issue per repo; partial Loki evidence is marked so fewer rows cannot imply the problem went away.

Where this stands.

Live
  • Audit-log denial triage from Loki, clustered by entity and path
  • Deterministic widen / probe / ambiguous classifier
  • Rolling audit-denial digest issue with timeline and success telemetry
  • Fail-closed Vault safety layer
In development
  • Draft MRs for unambiguous widen findings (enabled, awaiting its first real production witness)
Roadmap
  • A separate paging alert for probe patterns on sensitive paths
  • Comparing declared policy with what is live in Vault (needs a read-only Vault token)
← Previous storyReviewing your team's merge requests