Agents in CI: fix the build, not the pipeline
AI may change what CI checks. Never how.
Ask an agent to make CI green and it has two ways to get there. It can fix
the code. Or it can change how CI checks the code: add
continue-on-error: true to the failing test step, lower a coverage
threshold, skip a test. Both end in a green check. Only one of them means
anything.
To git, both are an ordinary commit. So the question is not “did CI pass?”
but “did anything change what passing means?”. Four ideas answer it; the
linked guides show how Itervox and the standalone referee apply them.
Idea 1: sort every change by what it touches
Section titled “Idea 1: sort every change by what it touches”A fix or a new test is what CI checks. A workflow file, the test runner’s config or a coverage threshold is how CI checks. Sort every file a change touches into a lane; the highest lane decides:
| Lane | Typical files | Who decides |
|---|---|---|
auto | code and tests | CI, as usual |
second_opinion | lockfiles, tool versions, sensitive code | a reviewer other than the writer |
human | CI config, deploy tooling, test and coverage config, the lanes policy itself | a person |
The default human lane covers CI config, test-runner and coverage config,
CODEOWNERS and Makefile. The second-opinion lane is empty until you fill
it. referee always adds its policy file to the human lane, so a change
cannot loosen the rules it is judged by.
Watch the files that change how without looking like CI. A version bump in
a lockfile or go.mod can swap the linter or the test runner, and no CI file
changes. Build manifests are not in the default human lane (by path, their
tool sections cannot be told apart from their dependencies), so put them in
the second-opinion lane yourself:
# .referee.yml on your default branchsecond_opinion: - "go.mod" - "go.sum" - "package.json" - "pnpm-lock.yaml"The same failing test, two branches:
One bug, two fixes, two lanes
See Lanes for how Itervox holds a human-lane push, and referee for the policy format.
Idea 2: put the permissions outside the model
Section titled “Idea 2: put the permissions outside the model”A prompt that says “don’t touch the pipeline” is a request, not a rule. The rule has to live where the agent cannot edit it:
- The policy comes from the base branch.
refereereads it withgit show <base>:.referee.yml, never from the branch under review. A pull request that edits the policy is judged by the old one. - The referee comes from outside the branch. The Action builds it from its own pinned ref.
- Pushes go through a hook. For the runs it starts, Itervox puts a
pre-push hook and shims for
git,ghandglabin front of the agent. See Guardrails.
In Itervox: the agent's pipeline change waits for you
Idea 3: the writer never judges
Section titled “Idea 3: the writer never judges”The agent that wrote a change does not decide that it passed. Its “tests pass” is a claim:
- The judge runs where the writer cannot reach. Run the agent in one job and the referee in a later job the agent’s job cannot change.
- The second-opinion lane needs another reviewer. In Itervox,
merge_prrefuses such a pull request until a reviewer on a different backend than the writer approves it. See Cross-review preset. - “Done” comes with a signed record.
referee run(and a profile’sverifylist in Itervox) runs each check itself and signs the real exit code and the tree it ran on, with a key the agent cannot read.referee verify --tree .then proves the record is about this code, not an earlier version. See Signed checks.
A signed run, and two ways to fake one
Idea 4: let AI reviewers find, let tests decide
Section titled “Idea 4: let AI reviewers find, let tests decide”An AI review is a second opinion, good at finding things. It is never what turns CI green. A finding is a lead; a test that fails before the fix and passes after it is the verdict.
Two examples from Itervox’s own history:
- A check that passed by running nothing. The first version of the chaos
test script (pull request #140) read its package list from
go list. Whengo listfailed, the script got an empty list, ran zero packages, and printedchaos: all packages passed. A review flagged it. A run of the script with a fakegothat exits 42 proved it. The fix fails the run on a failed or empty listing. - Guard bypasses that passed the existing tests. Fuzzing the guardrails
in 0.3.3 found holes the existing tests did not cover:
git push --no-verif(git accepts abbreviations of--no-verify) skipped the hook;git -c include.path=…could load a file that replaces the hooks;gh pr -R owner/repo mergeslipped past the self-merge check; and the lane matcher was exponential in**segments on deep paths. Each one now has a test.
In both cases the existing tests were green.
Start with one job
Section titled “Start with one job”Add one workflow that sorts each pull request into lanes, and make it a required check in branch protection:
name: referee
on: pull_request:
permissions: contents: read
jobs: lanes: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 with: fetch-depth: 0 # the merge base and the base branch's policy persist-credentials: false - uses: vnovick/itervox/referee@<sha> # pin to a commit SHA with: mode: lanesreferee lanes exits 0 for the auto lane, 10 for second opinion, 20 for
the human lane and 2 on an error; it never returns 0 for an error. With
the default fail-on: human, exit 20 fails the job: a person must look
before the pull request merges. To let a maintainer approve with a label, and
to add mode: audit for weakened checks, use the full example in the
referee guide.
Elsewhere, brew install vnovick/tap/referee (or use itervox referee)
and run referee lanes --base origin/main in any CI step.