Skip to content

Agents in CI: fix the build, not the pipeline

AI may change what CI checks. Never how.

Ask an agent to make CI green and it has two ways to get there. It can fix the code. Or it can change how CI checks the code: add continue-on-error: true to the failing test step, lower a coverage threshold, skip a test. Both end in a green check. Only one of them means anything.

Terminal: referee audit --base main reports 2 findings that weaken checks: .github/workflows/ci.yml line 10 adds continue-on-error: true (ci_continue_on_error), and cart_test.go line 6 adds t.Skip("flaky on CI") (go_skip); exit=20 Terminal: referee audit --base main reports 2 findings that weaken checks: .github/workflows/ci.yml line 10 adds continue-on-error: true (ci_continue_on_error), and cart_test.go line 6 adds t.Skip("flaky on CI") (go_skip); exit=20
An agent asked to make a red build green, in a small demo repository. referee audit finds both shortcuts it took: continue-on-error on the test step and a skipped test.

To git, both are an ordinary commit. So the question is not “did CI pass?” but “did anything change what passing means?”. Four ideas answer it; the linked guides show how Itervox and the standalone referee apply them.

Idea 1: sort every change by what it touches

Section titled “Idea 1: sort every change by what it touches”

A fix or a new test is what CI checks. A workflow file, the test runner’s config or a coverage threshold is how CI checks. Sort every file a change touches into a lane; the highest lane decides:

LaneTypical filesWho decides
autocode and testsCI, as usual
second_opinionlockfiles, tool versions, sensitive codea reviewer other than the writer
humanCI config, deploy tooling, test and coverage config, the lanes policy itselfa person

The default human lane covers CI config, test-runner and coverage config, CODEOWNERS and Makefile. The second-opinion lane is empty until you fill it. referee always adds its policy file to the human lane, so a change cannot loosen the rules it is judged by.

Watch the files that change how without looking like CI. A version bump in a lockfile or go.mod can swap the linter or the test runner, and no CI file changes. Build manifests are not in the default human lane (by path, their tool sections cannot be told apart from their dependencies), so put them in the second-opinion lane yourself:

# .referee.yml on your default branch
second_opinion:
- "go.mod"
- "go.sum"
- "package.json"
- "pnpm-lock.yaml"

The same failing test, two branches:

One bug, two fixes, two lanes

Terminal: on the agent's branch, git diff --stat main shows .github/workflows/ci.yml and cart_test.go changed; referee lanes --base main prints lane: human, policy .referee.yml read at the merge base, human (1): .github/workflows/ci.yml, auto (1): cart_test.go; exit=20 Terminal: on the agent's branch, git diff --stat main shows .github/workflows/ci.yml and cart_test.go changed; referee lanes --base main prints lane: human, policy .referee.yml read at the merge base, human (1): .github/workflows/ci.yml, auto (1): cart_test.go; exit=20
The agent's branch touches the CI workflow, so the whole change takes the human lane: exit 20, a person decides.
Terminal: on the fix branch, git diff --stat main shows only cart.go changed; referee lanes --base main prints lane: auto, auto (1): cart.go; exit=0 Terminal: on the fix branch, git diff --stat main shows only cart.go changed; referee lanes --base main prints lane: auto, auto (1): cart.go; exit=0
The real fix changes cart.go only: the auto lane, exit 0, and it ships on green.

See Lanes for how Itervox holds a human-lane push, and referee for the policy format.

Idea 2: put the permissions outside the model

Section titled “Idea 2: put the permissions outside the model”

A prompt that says “don’t touch the pipeline” is a request, not a rule. The rule has to live where the agent cannot edit it:

  • The policy comes from the base branch. referee reads it with git show <base>:.referee.yml, never from the branch under review. A pull request that edits the policy is judged by the old one.
  • The referee comes from outside the branch. The Action builds it from its own pinned ref.
  • Pushes go through a hook. For the runs it starts, Itervox puts a pre-push hook and shims for git, gh and glab in front of the agent. See Guardrails.

In Itervox: the agent's pipeline change waits for you

Needs attention panel with one row: DEMO-1, Pipeline change: the agent wants to push changes to .github/workflows/ci.yml, awaiting reply, with Reply and Open buttons Needs attention panel with one row: DEMO-1, Pipeline change: the agent wants to push changes to .github/workflows/ci.yml, awaiting reply, with Reply and Open buttons
The push that edits the CI workflow is held. It shows up in Needs attention with the files and the git command.
The reply box with the text: Rejected: fix the code, not the pipeline. The CI config stays as it is. The reply box with the text: Rejected: fix the code, not the pipeline. The CI config stays as it is.
Reject it with a reason. The agent gets your reply and fixes the code instead.

The agent that wrote a change does not decide that it passed. Its “tests pass” is a claim:

  • The judge runs where the writer cannot reach. Run the agent in one job and the referee in a later job the agent’s job cannot change.
  • The second-opinion lane needs another reviewer. In Itervox, merge_pr refuses such a pull request until a reviewer on a different backend than the writer approves it. See Cross-review preset.
  • “Done” comes with a signed record. referee run (and a profile’s verify list in Itervox) runs each check itself and signs the real exit code and the tree it ran on, with a key the agent cannot read. referee verify --tree . then proves the record is about this code, not an earlier version. See Signed checks.

A signed run, and two ways to fake one

Terminal: referee run --name test on main records exit=1; on the fix branch it records exit=0 with tree c307fc7a9af5; referee verify --tree . prints ok for the check and the seal, all 2 records verify, exit=0 Terminal: referee run --name test on main records exit=1; on the fix branch it records exit=0 with tree c307fc7a9af5; referee verify --tree . prints ok for the check and the seal, all 2 records verify, exit=0
referee run executes the tests itself and signs the real exit code and tree. On the fix branch, verify passes.
Terminal: after rewriting main's record from exit 1 to 0, referee verify fails with attest: signature does not verify, exit=1; after appending a line to cart.go, verify --tree fails with attest: record is for a different tree, exit=1 Terminal: after rewriting main's record from exit 1 to 0, referee verify fails with attest: signature does not verify, exit=1; after appending a line to cart.go, verify --tree fails with attest: record is for a different tree, exit=1
Edit the record's exit code and the signature fails. Change the code after the run and the tree no longer matches.

Idea 4: let AI reviewers find, let tests decide

Section titled “Idea 4: let AI reviewers find, let tests decide”

An AI review is a second opinion, good at finding things. It is never what turns CI green. A finding is a lead; a test that fails before the fix and passes after it is the verdict.

Two examples from Itervox’s own history:

  • A check that passed by running nothing. The first version of the chaos test script (pull request #140) read its package list from go list. When go list failed, the script got an empty list, ran zero packages, and printed chaos: all packages passed. A review flagged it. A run of the script with a fake go that exits 42 proved it. The fix fails the run on a failed or empty listing.
  • Guard bypasses that passed the existing tests. Fuzzing the guardrails in 0.3.3 found holes the existing tests did not cover: git push --no-verif (git accepts abbreviations of --no-verify) skipped the hook; git -c include.path=… could load a file that replaces the hooks; gh pr -R owner/repo merge slipped past the self-merge check; and the lane matcher was exponential in ** segments on deep paths. Each one now has a test.

In both cases the existing tests were green.

Add one workflow that sorts each pull request into lanes, and make it a required check in branch protection:

name: referee
on:
pull_request:
permissions:
contents: read
jobs:
lanes:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0 # the merge base and the base branch's policy
persist-credentials: false
- uses: vnovick/itervox/referee@<sha> # pin to a commit SHA
with:
mode: lanes

referee lanes exits 0 for the auto lane, 10 for second opinion, 20 for the human lane and 2 on an error; it never returns 0 for an error. With the default fail-on: human, exit 20 fails the job: a person must look before the pull request merges. To let a maintainer approve with a label, and to add mode: audit for weakened checks, use the full example in the referee guide.

Elsewhere, brew install vnovick/tap/referee (or use itervox referee) and run referee lanes --base origin/main in any CI step.