Home / Stories / Ansible CI/CD
Story · Beyond Kubernetes Roadmap

Your Ansible CI is green. Is your fleet?

A well-run Ansible repository already lints every change and shows a --check --diff plan on each merge request. What nobody automates is the layer above: noticing that the hourly apply has been failing since Tuesday, that a green run was actually a no-op, or that a merge request is showing another merge request's plan. That layer is where the assistant is going. Parts are built; the MR-producing lanes are on the roadmap.

Starts fromAn Ansible repository (roles, inventories, playbooks) whose CI runs the playbooks: check on merge requests, apply on a schedule.
Looks atCI run history and job logs, the workflow files, role defaults and version pins, the repos those playbooks depend on.
You getRun-health alerts on the scheduled apply, draft MRs for CI hygiene and role pin upgrades, and failures traced to their cause.
01 · A different kind of repo

No HelmReleases, no Kustomizations. Just roles.

An Ansible repository looks nothing like a Flux repository. The Kubernetes-shaped agents parse nothing in it, so the failure mode is not a wrong proposal. It is silent zero coverage: the repo is scanned, nothing is found, and nothing is said.

Treating Ansible repositories as first-class targets starts with admitting that, and building lanes that understand them.

02 · What CI already does

Lint and plan on every MR. Apply every hour.

The typical setup: a merge request runs syntax-check, ansible-lint and a --check --diff plan; main applies on push and on a schedule. The scheduled apply is this repo's reconciler, the Ansible equivalent of Flux.

The MR gate is good at what it checks. It structurally cannot report its own reconciler dying.

03 · Watch the reconciler In development

OK, stale, red, or unknown. Never "probably fine".

A deterministic run-health check reads the scheduled runs and gives one of four verdicts. Stale: no scheduled run in two cadences. Red: the last K scheduled applies all failed. Unknown: no token, an API error, or incomplete pages, and unknown never reads as OK.

Thresholds are tuned by replaying the full run history, not by waiting a week. Today it runs report-only; turning it into an issue is on the roadmap.

04 · Read the run log In development

A PLAY RECAP is structured data.

A parser turns an ansible-playbook job log into facts: plays, tasks, per-host results and the recap. In --check mode, changed>0 is a genuine drift signal against the live hosts.

It is honest about its ceiling: a run log cannot prove a named resource exists, so it answers "can't tell", never "absent". The parser is built; no production lane consumes it yet.

05 · CI hygiene Roadmap

The plan on this MR belongs to a different MR.

The plan step writes to a fixed /tmp/plan.txt on a self-hosted runner whose /tmp survives between jobs. The publish step runs if: always(). When this MR's plan never ran, reviewers see yesterday's plan from someone else's change.

That misleads reviewers and discloses diffs across merge requests. It is also detectable from the workflow file alone.

06 · CI hygiene Roadmap

Green every hour, doing nothing every hour.

A guard step that finds a required secret missing sets ready=false and lets the job finish green, "rather than failing hourly". Every scheduled apply becomes a successful no-op, and a run-health check that only reads the status says OK.

The rule flags the pattern; the fix is to fail loudly on scheduled runs, or to read step outcomes and demote a skipped run to unknown.

07 · Role pins Roadmap

Versions live in role defaults, not in requirements.yml.

In practice the pins that matter sit in roles/*/defaults/main.yml (*_version, *_ref, image digests) and in the CI image's build arguments. A deterministic lane reads them, looks up upstream releases, and proposes the same evidence-based target the Kubernetes upgrade lane uses.

08 · Failure triage Roadmap

When the playbook is the victim, not the cause.

A failed job's log tail is classified against known signatures: a sealed Vault, a registry pull error, DNS. When the repo declares the dependency, the failure is attributed to the repo that owns it, and the proposal lands there. The cross-domain story follows one such failure all the way to a missing probe.

09 · The proposal

A small workflow fix, with the reason spelled out.

The fix is two lines: a per-run path and a guard on the publish step. The MR explains the mechanism, shows the evidence from the repo's own history, and warns that the MR itself triggers the check job on the fleet. The full MR is below.

fleet-ansible · repository tree
.ci/workflows/ansible.yaml
inventories/prod/hosts.yml
inventories/prod/group_vars/all.yml
roles/node_exporter/defaults/main.yml
roles/inference/defaults/main.yml
roles/hardening/tasks/main.yml
site.yml
Dockerfile
  • HelmRelease / Kustomization lanes 0 files understood
  • naming, resources, availability nothing to parse
silent zero coverage is the default, not a misfire
MRsyntax-check
ansible-lintproduction profile
--check --diffplan in job summary
mergemain
scheduled applyevery hour
fleethosts converge
the MR gate never sees the scheduled apply fail
scheduled applies · last 36 hours · oldest first
OKsuccess in window
STALE> 2 cadences
REDK in a row
UNKNOWNnever OK
backtest over full history: the single blip at run 4 does not fire; the streak does
job log · ansible-playbook site.yml --check --diff
TASK [node_exporter : install binary] ****************
ok: [host-01]
changed: [host-02]
PLAY RECAP *******************************************
host-01 : ok=41 changed=0 unreachable=0 failed=0
host-02 : ok=41 changed=1 unreachable=0 failed=0
  • mode: check dry-run drift probe
  • host-02 · node_exporter : install binary drift: live differs from repo
  • does resource X exist? can't tell, never "absent"
.ci/workflows/ansible.yaml · check job
runs-on: self-hosted   # host mode: /tmp persists
steps:
  - id: plan
    run: ansible-playbook site.yml --check --diff | tee /tmp/plan.txt
  - name: Publish the plan
    if: always() && steps.guard.outputs.ready
    run: cat /tmp/plan.txt >> $STEP_SUMMARY
plan step failed · summary shows a plan stamped the previous day
.ci/workflows/ansible.yaml · apply job · guard
  - id: guard
    run: |
      if [ -z "$KNOWN_HOSTS" ]; then
        echo "runner busy, skipping this run"
        echo ready=false >> $OUTPUT
      fi
12 green runs · 0 tasks executed · status-only health says OK
roles/node_exporter/defaults/main.yml
node_exporter_version: 1.8.1
node_exporter_checksum: sha256:…
node_exporter_web_listen: ":9100"
1.8.1pinned
1.8.2patch
1.9.1target
1.10.0-rcskipped
upstream releases read · no breaking change applies · checksum updated with the version
Scheduled apply · REDsame failing step, 12 runs
Log-tail signaturevault … 503 … sealed
Declared dependencyCI script names the Vault
Owning repoproposal lands where the cause is
Draft · Publish the check-mode plan from a per-run path
-    run: ansible-playbook site.yml --check --diff | tee /tmp/plan.txt
+    run: ansible-playbook site.yml --check --diff | tee "$RUNNER_TEMP/plan.txt"
-    if: always() && steps.guard.outputs.ready
+    if: always() && steps.plan.outcome == 'success'
category:securitysource:ansible_cirisk:low
↓ the full merge request is below

The merge request you would get.

Illustrative and on the roadmap: the rule, its evidence and its fix come from a real finding in an Ansible repository we run, but this MR is not yet produced automatically.

Draft

Publish the check-mode plan from a per-run path, only when the plan ran

from gitops-ai/ansible-ci-plan-per-run-path into main · 1 file · +3 −3
category:securitysource:ansible_cirisk:low
OverviewEvidenceValidationChanges
Summary

The check job writes its plan to the fixed path /tmp/plan.txt. The runner is self-hosted in host mode, so /tmp persists between jobs. The publish step runs under always() and does not check whether the plan step succeeded.

Result: when the plan step fails or is skipped, the MR's job summary shows the previous job's plan, which may belong to a different merge request.

Why it matters
  • Reviewers approve a change based on a diff that is not its own.
  • Plan output from one merge request is disclosed on another.
Fix

Write the plan under $RUNNER_TEMP, which is per job, and publish it only when the plan step succeeded.

CheckResult
Job runs on a persistent (host-mode) runnerruns-on: self-hosted
Producer writes a fixed shared pathtee /tmp/plan.txt
Publisher runs regardless of producer outcomeif: always() && …, no steps.plan.outcome
Observed in historyA failed check job published a plan stamped the previous day.
Checked before this MR was opened
  • The workflow still parses, and every step id it references exists.
  • No other step reads /tmp/plan.txt.
  • The open-bot-MR limit for this repo (1, to protect a runner shared with the hourly apply) is not exceeded.
After merge, expect
  • A merge request whose plan step fails shows no plan, instead of a stale one.
  • A merge request whose plan succeeds shows exactly its own plan.
       - id: plan
-        run: ansible-playbook site.yml --check --diff | tee /tmp/plan.txt
+        run: ansible-playbook site.yml --check --diff | tee "$RUNNER_TEMP/plan.txt"
       - name: Publish the plan
-        if: always() && steps.guard.outputs.ready
+        if: always() && steps.plan.outcome == 'success'
-        run: cat /tmp/plan.txt >> $STEP_SUMMARY
+        run: cat "$RUNNER_TEMP/plan.txt" >> $STEP_SUMMARY

What keeps it honest.

An Ansible repo's CI touches real hosts. The guards are about not making that worse.

Unknown is not OKA missing token, an API error or an incomplete page of runs gives UNKNOWN. A health check that cannot see is never reported as healthy.
Tuned on history, not on hopeRun-health thresholds are chosen by replaying every past scheduled run and counting would-fire and false-fire cases per threshold.
Explicit opt-in for CI side effectsBecause a bot MR triggers a check job on the fleet, writing to an Ansible repo needs a dated, per-repo acknowledgement in config, plus a cap on open bot MRs.
Drafts only, verifiedProposals are draft MRs. Where the forge marks drafts by title prefix, the draft state is read back after creation; if it did not stick, the MR is closed and the proposal fails.
Negative controls per ruleEach hygiene rule ships with a workflow that does it right (per-run path, outcome gating, failing on a missing secret) and must stay quiet on it.
No plan hunks in fixturesTest fixtures made from real job logs have diff hunks stripped and hosts masked, and are reviewed by a person before they are committed.

Where this stands.

Live
  • Ansible repositories hosted on GitLab get the same CI/CD, access and runner review as every other repo
  • Evidence-based upgrade targeting (the Kubernetes lane the role-pin lane will reuse)
In development
  • Scheduled-apply run-health (OK / STALE / RED / UNKNOWN), report-only today
  • Run-log parser for ansible-playbook job logs, not yet wired into a lane
  • Read access to Ansible repos on Forgejo
Roadmap
  • Run-health as an issue on the repo
  • CI hygiene rule pack: stale artefacts on persistent runners, guard-skips that hide a dead reconciler
  • Role and collection pin upgrades as draft MRs
  • Failure triage from CI job logs, attributed across repos
  • Inventory and group_vars understanding
← Previous storyDNS as code