How We Test a SaaS Application with Nanook

We build SaaS products — an invoicing API and dasHandwerk, a field-service application for the trades. The end-to-end suite grew the way these suites do: one spec file per screen, then one per feature, then one per bug. Nobody planned it. Nobody had to. Every file was reasonable on the day it was written.

This is a field report on what we did when that suite stopped paying for itself: what it looked like before, what we changed, what it cost, what it caught — and the two assertions we got wrong ourselves along the way.

The Suite That Stopped Paying for Itself

Before we touched anything, we measured. The page-object layer of one product had 55 page classes and 703 element getters — and exactly 2 functions with a data contract, meaning a function that takes a typed record and fills a form from it. Everything else took no arguments and returned a locator.

The consequence, in one sentence: how you operate a Radix select was written down up to 55 times. Each page class knew how to open a dropdown, wait for the listbox, and click the option. When the dropdown component changed, the suite did not break in one place. It broke in as many places as had copied the pattern, and it broke on different days, because each copy had drifted a little.

That is the volume problem, and it is the one everybody sees. It was not the real problem.

The real problem was that the specification lived in TypeScript. Every spec file had, somewhere near the top, a const CASES = [...] — an array of inputs and expected outcomes. That array is the specification: it says which inputs are valid, which are rejected, and what the system must do in each case. And it belonged to whoever writes TypeScript, not to whoever knows the domain — all the test case design there ever was, locked in a file only one side could edit. The person who could tell you that a proposal over an existing booking must not raise a conflict could not read the file that should have said so. When the two disagreed, the file won by default, because nobody who knew better ever opened it.

What We Changed

We had been using Nanook equivalence class tables for some time, but in the way the tool originally suggested: write a table, generate a spec file from it, commit the spec file. That turned out to be the expensive half.

The table is the source, not a generator input

After generation, a file lay around per surface and wanted maintaining. It was still a spec file, still TypeScript, still the place where the specification actually lived — the table had merely been its first draft. Within weeks the generated files had been hand-edited, and the tables were stale.

We inverted it. The table stays the source, and one runner reads suite descriptions as data. A suite is a JSON file that names a table, a page, and how the table’s fields map to that page. The runner loads the suite, generates the cases from the table at run time, and drives each one. Nothing is generated into the repository.

The result is <app>/tests/suiten.spec.ts — one file, not one per surface. It is short, it is generic, and it has not been edited for a new screen since.

Three commitments that carry the approach

The runner is not the interesting part. Three decisions about what goes into it are.

1. The suite JSON knows no Playwright vocabulary. It describes cases, not clicks. There is no click, no fill, no selector in it. It maps table fields to a page and leaves the operating of the form to the page module. Someone who knows the domain and not the tooling can read it, and can read the table it points at.
2. The expectation belongs at the boundary where it holds — ui or api. When we converted the company form, it had 41 cases. 26 of them were pure statements about a validation schema: too long, empty, wrong format. Against the procedure, those 26 run in seconds. Through the browser, the same case costs 20–60 s, and what it measures is the same schema plus a form that forwards to it. A test that pushes a schema statement through a browser does not measure more, it measures slower. So a suite declares its boundary, and the same table runs at whichever one holds the statement.
3. A report that names the table column. When a case fails, the report says the test case name, the column, the page and the field. Without that, a red run leads straight back into the handwork the runner just abolished: open the table, find the column, guess which field the message is about. With it, the failure is a cell reference.

The Shape of a Table

A Nanook decision table has one column per test case and one row per equivalence class. Ours settled into a fixed skeleton: a section Secondary data for what must already exist before the test starts, above a section Primary data for what the test itself enters, then Expected reaction (what the user sees) and Expected effect (what the system does). Here is the skeleton for an appointment form on a dispatch board, reduced to the rows that matter:

The markers are Nanook’s: x chooses exactly this class for the case. Where several classes of one field are marked, a is the preferred one and is what actually gets generated, while e marks the other valid classes so that they count for coverage without being generated — that is the CASCADE pattern, and it is why a handful of columns can cover every class. What a marker never does is state a negative. “Must not happen” is written as its own row with an x: no conflict on the board. A table states expectations positively, one row per statement. Read the OK_proposalOver column for the effect of the open markers: slot state and time both carry a/e, because a proposal must not raise a conflict over anything, however it lies — that one column claims four of the eight combinations, and the Summary row shows the arithmetic.

The fifth case is the counter-probe, not decoration. Read the last two rows. Four cases demand “conflict findable”, one demands “no conflict”. A probe that always reports a conflict fails the fifth case. A probe that never reports one fails the four. Only a probe that actually looks at the board passes all five. We learned to distrust any effect that had only one row in the table — it can be satisfied by a probe that returns true. The contradiction between two cases is what turns a value into an assertion.

Preconditions are an axis, not a path

“First create an appointment, then book over it” reads like a scenario. Written as a script, it is one: setup, act, assert, in a straight line. We had dozens of those.

In a table, a pre-existing state is not a step. It is an axis — a row in Secondary data — and as an axis it crosses with every other class in the table. The moment “slot taken by a booking” is a row rather than a setup call, it combines with “kind: proposal” whether or not anyone thought of it.

That is the payoff case in the skeleton above: OK_proposalOver, a proposal over an existing booking, which must be saved and must not report a conflict — while a real booking over the same slot must. Nobody on the team had written that as a script. Not because it was hard, but because it only occurs to you when you see the grid: two kinds of appointment, two states of the slot, four cells, and one of them empty.

Where the Values Come From

The skeleton above hid one column of the workbook: the generator column. It is the reason a table needs no fixture files — the data is produced from the table, at run time. Reduced to a single test case so the column has room to breathe:

Three kinds of entry, and a prefix tells them apart. No prefix is a static value, copied as-is. gen: calls a registered generator — Faker ships with Nanook, and a custom one is ten lines plus a one-line registration. The number after gen: is the instance id: the name and the phone above both say 1, so they come from the same generated person — different ids, or none, would produce two unrelated people.

The quiet star is ref:. The appointment’s customer is not described a second time — it references the customer table’s own OK case, and the processor generates that case and takes its id. That is what “secondary data” looks like in practice: the precondition of one table is the output of another, with no setup script in between. Change what a valid customer is, and every table that builds on one follows.

A Flow Table Ties the Tables Together

Decision tables answer “which combinations”. They deliberately do not answer “in which order” — and for the cases where the order is the statement, we keep flow sheets. This is a real one from our suite, with the identifiers translated to English:

One row is one journey; the columns are its stations, read left to right. <fn:…> calls a registered function — here the base state, a company with a signed-in user. A column headed by a table name plays a case from that decision table: DE_OK_1 is filled into the form exactly as the data table defines it. <pc:customer> clicks an action on the page class. And <mode:check> is the payoff: the same case, read back instead of entered — open the record and assert that every field holds what DE_OK_1 says it should. The expectation is not written a second time; the data table is both the input and the oracle.

Read TC2 for the counter-move: the rejected case DE_E_1-1 runs through the same stations, and the check column is empty — a customer that was refused has nothing to read back. The flow sheet adds only what the decision table cannot say: the order, and what must still be true at the end of it. Everything else stays in the data tables, which is why our 24 flow sheets stay at two rows instead of becoming a second test suite.

Where It Stands Now

That is across two products, an invoicing API and a field-service SaaS. Alongside the decision tables sit 24 flow sheets for the cases where order genuinely is the statement — the section above shows one.

The number that convinced us was not in the table. One module — a dispatch board with 43 procedures and 84 call sites in the UI — had no tests at all. It now has 92 cases in six tables, built in a single session. The runner, the page contract and the report already existed; the session only had to say what the board must do.

What It Caught

To be clear about the order of things: the point of the switch was maintenance. The defects were a side effect. That is exactly why they are worth reporting — nobody set out to find them, and the tables found them anyway.

The one that had been green for two days

A sort assertion on a list view. A field had been renamed in the row reader; the assertion had not been told. It compared the sorted column with the expected order and passed:

const got = rows.map((r) => r.dueDate)   // reader renamed: every entry is undefined
expect(got).toEqual([...got].sort())      // [undefined, …] equals [undefined, …]

A dead reader does not make an assertion fail. It makes it empty — and empty looks exactly like correct. Behind the green: sorting ascending returned descending, and had done so for two days.

The table-driven runner surfaced it within minutes of the switch, for a reason that generalises. The same row reader serves twenty other cases that do have values in that column. The moment one of them asked for a real date and got undefined, the report named the column, and the rename was visible in the diff.

The class of defect tables find on their own

The rest were less dramatic and more instructive, because they are all of one shape.

• A missing .trim() before .min() — three spaces pass a minimum-length check — in five separate locations. Not because anyone hunted for it, but because every table asks the same question of every text field: what happens with whitespace only?
• A whole module answering rule violations with 500 instead of 4xx. One wrapper, 41 procedures, every one of them turning a validation error into a server error. Each table has a column that expects a rejection; every one of those columns went red on the status code at once.

The pattern: a table asks the boring questions everywhere, and boring questions are the ones humans skip. Nobody writes the whitespace-only test for the fifth text field on a form. The table does not know it is the fifth.

The findings that arrived before the first run

One more class of catch, and we can only half measure it: filling a table is itself an audit. A column forces a decision into every cell — you cannot leave a field vaguely “handled” the way a scenario script can simply not mention it. Writing the country classes for the customer sheet meant asking the application which countries it actually serves — and the answer was five different lists in five places, among them sixteen in one form, fourteen in another, thirteen in the backend. You could create a Romanian customer but not a Romanian supplier, and a Canadian customer silently fell back to US dollars. None of that came from running a test. It came from having to write a row.

The same forcing works on the AI that helps build the tables, and this one we can date: working through a validation sheet, the model adjusted seven expectations to match what the application actually did — the app accepted a company name of three spaces, the table said reject, so the model “fixed” the table. The correction became a house rule:

If the table says we expect a format check, then we expect one. That it is not implemented is a different matter.

A model has a strong pull towards the observable: it sees a failing expectation and a passing application and resolves the contradiction in the direction of what it can measure. Left alone, it converges on the tests describe what the software does — at which point a bug and a feature are indistinguishable. The table is the fixed point that stops that drift, and because it is a grid of fields and crosses rather than three hundred lines of generated code, a human can see the drift happening. Whether the authoring effect alone justifies the switch we cannot prove — but a noticeable share of our findings arrived while writing columns, before anything ran.

Two Mistakes We Made

This section is deliberate. A field report without it is a brochure.

An equation whose two sides come from the same number

The stock module has a balance: what is on hand equals the opening quantity plus movements. Our first assertion for it derived the opening quantity from the same record it was checking:

const base = stockAt - sum(movements)   // then: expect(base).toBeGreaterThanOrEqual(0)

It was green. It was also unable to find the defect its own case was named after. If the service had double-counted a movement, stockAt would carry the error and base would absorb it. The assertion checked that a number minus part of itself is a non-negative integer — which it always is when the data is integers.

The fix was to read the opening quantity independently, from the receipt that created it. Only then was a mutation probe possible at all: with two independent sources, breaking the service makes them disagree. With one, there is nothing to disagree with.

An optional field switched off the compiler

The page modules share a contract for what a list view reports back to the runner. One line of it read:

ariaSort?: string | null   // optional — and three of four page modules never set it

The probe for sort order read that field. Three of the four modules never wrote it. So the probe compared undefined === 'ascending', which is always false, and the check passed empty — the same failure shape as the dead reader, produced by a question mark.

We made the field mandatory. The type checker named a third module immediately, one we had not suspected. And the same run went from 15 to 20 red — the five new ones being exactly the cases that had measured nothing before. That is the number to watch after a fix like this: if the red count does not rise, the field was not being read.

Verify the Verifier

After the second mistake we made it a rule: every new assertion gets a mutation probe. Break the thing it guards on purpose, and check which cases fall — not merely that some do. A probe that turns everything red is as uninformative as one that turns nothing red.

Each exact. Where prediction and measurement disagree, the assertion is wrong, not the application — that is the point of predicting first.

One practical rule: always over a file copy, never git checkout. A checkout after the probe discards every uncommitted change in that file, and the probe usually runs in the middle of other work. Copy the file aside, mutate, measure, copy it back.

What Tables Do Not Solve

Three things, and we hit all three.

Order-as-the-statement stays a flow. “Finalize the invoice, then confirm it can no longer be edited” is a chain; the sequence is the assertion. A decision table has no notion of “then”. We keep 24 flow sheets for exactly this, and we stopped trying to bend tables around it.

A table freezes what it describes. Pouring a half-built module into 92 cases pins down a state that is still meant to change. Every subsequent change then produces red that is not a defect, and the team learns to ignore red. Tables belong on surfaces whose behaviour is decided.

The runner still needs a domain probe. The table says what must happen; something has to know how to look. “Conflict findable on the board” needs code that opens the board and looks. That code must observe and report — never assert. A probe with its own expect pulls the expectation out of the table and into the place where nobody reads it, and we are back to the specification living in TypeScript. We wrote that rule down and still broke it twice.

Would We Do It Again

Yes, and earlier. The win was not fewer tests; there are more now than before. The win was that the specification moved out of TypeScript and into a form the people who know the domain can read, correct and extend — that, not the tooling, is what data-driven testing bought us. The maintenance curve flattened as a direct result: a new screen is a table and a suite file, and the runner has not changed for it.

If you want to start, the Quickstart covers installation, the equivalence class guide explains the markers, and the guides on matrix tables, cross-table references and custom generators cover what we leaned on most. If you would rather have a model draft the first table, AI-Assisted Equivalence Class Tables shows how; and Manual vs. Automated is the argument for generated data in the first place.

Your suite is probably already a table — it is just written in TypeScript. Start with the Quickstart and turn one screen into an equivalence class table; the tutorial walks through the markers.