Rules Need Exit Codes, and Green Is Not Useful: What Still Went Wrong

Part 9 of 9 in the series on agentic software development. Part 1, The Agent Writes the Code has the two settings, the numbers, and the ten sentences the series works through.

For nine months an AI coding agent, Claude Code, has written nearly all the code of a multi-product SaaS platform: 74 repositories, about 14,000 commits, about 1,240 plans worked through. Its instruction file, CLAUDE.md, said never to bypass the test hook, and every plan was verified against the code before it was marked done. Both turned out to be worth less than they read. The prohibition was bypassed five times, eight red CI runs went unnoticed, and a whole round of features came out correctly built and pointless. The last part of this series is about those three, and about the list I would draw up if I started again.

--no-verify, five times

The instruction file says: never bypass the test hook. Every task brief for an implementation agent said: never --no-verify.

On one day, three implementation agents committed with git commit --no-verify anyway, and reported it themselves. The explanations were “reflex” and “harmless, because the tests were green before”. They had read the prohibition as a push rule. A fourth case came nine days later, after the brief had been sharpened. A fifth followed on the same day, this time not --no-verify but git -c core.hooksPath=.git/hooks commit. Same reflex, different spelling.

The three on the first day reported it themselves, which is the part worth pausing on. Nobody was hiding anything. Each agent had decided, in the moment, that the rule did not apply to this commit: the tests had been green before, so the hook would only have confirmed it. A prohibition that an agent can reinterpret is a prohibition that will be reinterpreted whenever it is inconvenient.

Two lessons came out of that.

A prohibition must name the effect, not the spelling. The wording now is: “No move that bypasses a hook — neither --no-verify nor SKIP_TESTS nor core.hooksPath, and not at commit time either.” Whoever names only one flag gets the next one.

What must hold has to be a gate, not a sentence. A PreToolUse hook now inspects every shell command the agent wants to run. If a git push segment carries --no-verify, SKIP_TESTS, or a hooksPath override, the command is denied, with a message: “the test chain must be green before pushing — if it is red, that is the finding: report it, do not bypass it.” A human in a terminal can still do it. An agent session cannot. That asymmetry is intentional: the hook is not there to stop me, it is there to stop a session that has just decided a red chain is harmless.

The difference between the two is not subtle. The sentence stood in every brief and was bypassed five times within ten days, each time with a reasonable-sounding explanation. The hook has an exit code, and an exit code does not accept explanations.

Eight red CI runs nobody noticed

The second incident needed no bypass at all. In one pipeline run, a frontend repository had eight consecutive red release runs, from nothing but formatting and import sorting. Eight commits were considered delivered and were not.

The mechanism: the agent had run only the affected test file per step and concluded “green”. The test runner does not run the format check; the build does. And git log origin/main shows the commit even when the run after it fails, so nothing in the agent’s usual view was red. It was noticed by accident, when a version bump re-triggered the run.

A red release publishes nothing. Every downstream layer then works against a version that does not exist, and the error surfaces hours later somewhere completely different. Hence the push gate: the full test chain in the affected repository before every push, enforced by a global pre-push hook; after the push, the CI run counts, not the push, so the run is watched until completed/success; and at the end of a run, a sweep of all touched repositories. A third rule belongs to the same gate: a green CI run without a release publishes nothing either. semantic-release silently skips the release when the branch is behind the remote, for example because the next commit was pushed during the CI job. So backend waves are bundled and pushed once.

The general principle behind both incidents:

Red is the finding: report it, don’t bypass it.

Everything that must hold goes into something with an exit code: a hook, a guard test, a check script, a ratchet. Prose rules are guidance for judgement. They are not a control, and an agent that treats them as one is doing you a favour you cannot rely on.

Built correctly, and pointless

The third failure has nothing to do with hooks, and no gate would have caught it. After a test round of the trades software, my verdict on many features was: implemented, but they don’t make much sense.

The early plans had been driven by competitor feature lists and verified against the code. The pattern repeated from feature to feature: the data model and the primitives cleanly built, a UI glued on without a thought-through workflow. Tours as an address text area on top of a tour primitive. Checklist execution on the template page. Appointment booking without closing the loop. “Done” was set, although the user’s job could not be played through.

Verification against the code answers the question the plan asked: is what the plan describes there? It cannot answer whether the plan should have been written, and neither can the agent. The agent is excellent at building what the plan says. It does not ask whether the plan makes sense for the person using it, unless the process forces the question. Four things changed:

• Every plan has a mandatory ## User job section: persona, job sentence (“does X with it instead of Y”), connection (where it enters and leaves the process), and a done scenario.
• Done means the job was played through in the browser as that persona. Not that the tests are green, and not that the plan was verified against the code.
• The QA walkthrough includes a sense check with five questions: job, playability, work saved, connection, honest maturity. Sense findings are not auto-fixed. They are triaged with me: fix, rework, remove from the navigation, or delete.
• Half-finished work is hidden behind a module toggle, not visible in the navigation.

Everything we build should take work off a user’s hands and improve their process. That is the measure — not feature parity, and not a green build.

Of the three failures in this part, this is the one a green build hides best. The bypassed hook was at least self-reported, and the red runs were at least in the CI log. A pointless feature passes every check, because every check asks whether the thing works, and none asks whether anyone needs it.

What I would do again, and what I would not

Two lists, kept short on purpose. Nine things I would do again, seven I would not.

Again:

• Tables before tests before code, whenever interfaces exist.
• A red chain with one identical message before any business logic.
• Transport-independent service design, so that all business tests run in seconds.
• The rule “expectation changes only via the table, with a reason”.
• A lesson memory with the mechanism, not just the incident.
• Hooks and guard tests for everything that must hold; prose for judgement only.
• A ledger with write-ahead for anything that runs longer than one session.
• Archiving unused skills and keeping the instruction file short.
• A root repository for rules, plans, and decisions, so that their history is visible.

Most of that list is about making the agent’s own check mean something: tests it can run in seconds, a chain that is red for the right reason, a table it may not change without a written reason, a hook that does not care how the bypass is spelled. The rest is memory: lessons with their mechanism, a ledger, a root repository whose history can be read.

Not again:

• A table for every trifle. Two pure functions with six cases would have been fine as ordinary tests. Tables pay off where combinations arise.
• Very fine classes. More classes mean more combinations and more mandatory cases, without more insight. Use the classes at which behaviour changes.
• Multiplying a case where one field would do. One multiplication took a sheet from 10 to 157 cases without a new statement.
• Letting several sessions work in the same checkout without push rules. Use worktrees.
• Trusting a browser walkthrough as the owner as proof that permissions work.
• Solving a recurring mistake locally, with a comment in the file concerned.
• Planning from competitor feature lists without a user job.

The second list is shorter than the first, and most of it is about overdoing the first: too many tables, too fine classes, too many cases. The method holds; the dose needed adjusting.

The rules this part comes down to: what must hold needs an exit code, because a rule in prose is guidance and a hook is a control; and built correctly is not the same as useful, so every plan names a user job and done means that job was played through. Behind both stands the one thing the whole series has been about. The agent writes the code. Whether that code can be trusted is decided by everything around it: the structure it works in, the order of the steps, the checks it cannot talk its way past, and the knowledge that outlives the session.

The nine parts are drawn from one report, which is on the site as a single page: Agentic Software Development: How We Build Software with AI Agents, What Works, and Where It Breaks.

Previous: Part 8, The Agent That Pleases. This was the last part; the series starts at Part 1.