Writing Tests Got Cheap. Specifying Them Didn't.

An AI writes a Playwright spec faster than any of us. Ask for “tests for the login” and something plausible appears within seconds. That is precisely why the interesting question has moved. Whether those tests cover the cases that matter, whether they will still cover them next month, and whether anyone but their author can read them — all three are unanswered. Generating the test was never the hard part. Deciding what to test, and holding that decision still, is.

This is a report from a day and a night in August, spent on one small feature area of our own SaaS — registration and login — with a model doing the typing and a decision table doing the deciding. The numbers are the ones we measured. The day is written up at length in the working notes in the repository; this is the short version, and the one argument it comes down to.

The drift

Working through a table of validation rules, the model adjusted seven expectations to match what the application actually did. The application accepted a company name consisting of three spaces; the table said it should be rejected; the model “fixed” the table.

The correction came back immediately: “If the Excel says we expect a format check, then we expect one. That it isn’t implemented is a different matter.”

That sentence is the argument in one line; the field report makes it at length. The short form: a model leans towards what it can observe. Faced with a failing test and a passing application, it resolves the contradiction in favour of the thing it can measure, and left alone it arrives at tests that describe what the software does. From then on a bug and a feature look the same. The table is the fixed point. It states what should happen; the distance between that and reality is the bug list, not an inconsistency to be smoothed away — and a person can see that distance, because a table is a grid of fields, classes and crosses, not three hundred lines of generated code that nobody diffs.

One grid

A decision table lists the fields of a form, the equivalence classes each field can fall into — valid, empty, whitespace only, too long, malformed — and one column per test case, marking which class applies. The registration table from that day has three rows and four cases and covers 100 % of the combinations it declares. That is the whole specification, and a domain expert can read it.

Two things made it scale beyond a toy. First, there are two kinds of table. Data tables (User, CompanyDE) define fields and classes and produce nothing on their own; test-case tables (Registration, Login) define cases and pull the classes in by reference. The registration table contains no field definitions at all. It says: session logged out, a user exists or does not, the input is valid or invalid — with the sixteen invalid users arriving as a single range reference, ref::User::[E_1-16]. Second, the secondary-data section of a table is the description of the base state: what must be true before the test types anything. The generator reads it off instead of inferring it. A new entity means one more row, not a change to the generator code.

Where Playwright enters: the Nanook table produces data, the specs are generic functions per page. There is no generated .spec.ts to maintain. The suite this happened in — 117 tables, 1,981 cases — is described in the field report.

What the table asked

Filling in a cell labelled “too long” forces a question: how long is too long? Nobody knew. Measuring the schema answered it: in the packages we sampled, 894 text columns had no upper bound and 192 had one. The registration form had no limit at all, neither in the form schema nor in the database. A whole document could be stored as a user name. The table found this, not a test run. A prompt for “some tests for registration” would have produced a test for a limit that does not exist, and it would have passed.

The coverage arithmetic asked a second question. The first complete table reported 1,168 % coverage — a structural error, not a rounding one: marking a field as “any valid value” in every column multiplies combinations that were already counted. The fix is a cascade — a field is open only in the error columns of the fields before it and closed from its own block onward — and after restructuring the table sat at exactly 100 %, 8 of 8 combinations, no duplicates. The practitioner’s caveat that came with it deserves quoting: “The cascade pattern is only an aid, and it doesn’t always work.” The arithmetic tells you when a table is inconsistent; it does not design the table. Later, another table sat at 99.2 % with six classes that had no test case at all, because “any of these” counts as covered while the generator only ever picks the preferred one. The percentage is the wrong question. “Does every class have its own test case?” produces an action, and a twenty-line checker answers it — on its first run it found a gap in a table written ten minutes earlier.

Failing for the wrong reasons

The first browser run came back 18 of 18 red, including the happy path. That last detail is the lesson. Eighteen red tests look like a verdict on the application, but if the valid case fails too, the verdict is about the measuring instrument. The cause was a cookie banner, a dialog layered over the page that intercepted every click. Twenty minutes of runtime, zero statements about the software. An all-red result is not a finding. It is a suspicion, about yourself — and it applies to AI-written tests specifically, because a model will happily generate a hundred tests that all fail for one shared, invisible reason, and the failure list reads like diligence.

After the fix, the run — by then twenty cases — came back 10 green, 10 red, 43 seconds. Now the red ones meant something, and they sorted into four causes. Five cases where the form accepts what it should reject. Three where the server accepts what it should reject — including the same email address twice: 200 twice, two different user ids, while the database, checked before reporting, holds a unique constraint on the address and no duplicate among its 60 users. So the row is not written twice, but the client is told it succeeded; for a user that is indistinguishable from a bug. One case where a rule exists but lives one layer further back than assumed. And one nobody was looking for: the backend can register an account through Apple, and the form offers no button. That is not a bug report a human would have filed, because nobody looks for a feature they never saw. It fell out of a table that enumerated the possibilities.

What the model is genuinely good at

Generators, scaffolding, and chasing a defect through an object graph. The defect that proved it: generation produced 3 of 7 login cases, silently. The instruction was to write a unit test that reproduces it before changing anything. The first fixture was red for the wrong reason — a missing header row made the parser discard the sheet — and was rebuilt. The real failure was two defects, not one: a referenced table’s existence was checked but the instance was indexed unchecked, and directives inherited from a referenced table were re-pointed at a node that never receives data. Measured with two instance ids side by side, fixed with a diff limited to the two corrections, 197 tests green, and the symptom measured again end to end: 7 of 7. The reason both had survived so long: the library logged the error to a logger the test harness never read. The instrument was blind.

And scope. A form with type="email" and no noValidate lets the browser block submission before the application validates, in the browser’s language, while the translated message sits unrendered in the bundle. The customer form in the same application had this right, with a fifteen-line comment dated three days earlier. The finding needed a human-shaped idea: check whether the message the user sees is the message we wrote. The blast radius needed a machine: sixteen more forms with the same defect across five products, listed in about four seconds. Neither is impressive alone. The pair took under a minute, and the follow-up belongs in a lint rule, so it cannot be relearned again.

What the file remembered

Late in the day the spreadsheet a human opens still stopped at two countries, while sixteen had been generated into a different workbook. The tests were green on sixteen; the artefact a person works with was already behind by lunchtime. Worse, the generated sheets had none of the formulas that make the hand-built CompanyDE readable — 278 of them, counting the marks per field group so that a zero in a case column shows an incomplete case without running anything. CompanyDE had been built by the model too, in the same session, one context compaction earlier. By the afternoon the model was generating a poorer version of a thing it had designed that morning, and the project’s own skill file had said how to do it all along, unread.

Three consequences, and they are the practical core of this piece. Write the format down where the work happens — a README next to the spreadsheets, not in a chat log. When generating an artefact that already exists, read the existing one first. And a skill file is worth nothing unread. The model’s memory of its own decisions is the least durable artefact in the room, less durable than the spreadsheet, the skill file or the commit message.

That is the through-line. A test suite is a measuring instrument and has to be calibrated before its readings mean anything, and the instrument’s own specification is an artefact too — one that decays faster than anything else. The table outlived the model’s memory of how to build it. It is the shared language between the domain expert, the developer and the model; everything downstream is derived, and the table is the thing you maintain. Writing tests got cheap. Specifying them didn’t.

Start with the Nanook Quickstart; the Claude Code skill drafts the first table, and the login example shows one end to end.