Testing E-Invoices for 40 Country/Format Combinations

EN 16931 is the European standard for electronic invoices. It defines the business terms an invoice carries — BT-1 is the invoice number, BT-5 the currency, BT-31 the seller’s VAT identifier — and it defines them once, for everyone. Then every country narrows it. A national narrowing is called a CIUS, and there is one almost everywhere the standard applies: XRechnung in Germany, FatturaPA in Italy, CIUS-RO in Romania, Peppol BIS across the Peppol network, PINT in Japan, Oman and the United Arab Emirates. Each one says which optional terms become mandatory, which values are allowed, and which combinations are forbidden.

For us that meant 40 country/format combinations, each with its own rule book, each rule book with hundreds of rules. The rules are not prose. They are Schematron — executable XPath assertions with an identifier, a severity and a test expression, versioned and published as files. BR-DE-17 is a rule. PEPPOL-EN16931-R120 is a rule. You can pin the file, hash it, and run it against an invoice, and it will tell you which rule objects.

That sounds like the easy kind of specification-based testing problem: the oracle is written down. It is not. Generating a hundred thousand cases across forty dialects is a morning’s work. Knowing that those cases test the standard, and not our implementation of it, is the hard part — and we got it wrong before we got it right.

The Oracle Problem, in One Measurement

Before any of the tables existed, we had 726 green assertions on the German invoice type code. Every one of them checked that InvoiceTypeCode came out as 380. They passed on every input, every run, for months.

380 is a commercial invoice. It is also our default. What those 726 assertions proved was that our default arrives in the output — whatever you send in. Then a customer asked for partial invoices. XRechnung codes a partial invoice as 326, and BR-DE-17 names the code in its text. 726 green tests, and not one of them had ever seen the case, because the case had come out of our own catalogue, and our catalogue did not know it.

It got worse when we looked at where the allowed codes lived. The allow-list in our code had been transcribed by hand from the Schematron. It contained 386 — a code BR-DE-17 does not know. No test had ever compared the transcript with the source, because both were ours.

The expected value comes from the pinned specification artefact — Schematron, XSD, code list — never from a constant in src/. A test built from our own code confirms our code. A test built from the specification finds the bug in it.

What a Table Looks Like When the Rules Are the Input

One sheet per rule area, one column per case

We build the tables with Nanook, whose decision tables read left to right: rows are the things that can vary, columns are the cases, and a marker in a cell says which class the case takes. For a rule book the rows are the business terms the rules touch, grouped by rule area, above a base variant that says which invoice we start from. The expected reaction sits at the bottom: which rule fires, or that none does.

A class in these sheets is not “valid” or “invalid”. It is a grip — a concrete way of taking hold of a field so that exactly one rule objects: omit, too-long:100, not-in-list, wrong-value, value-limit:-1, group-removed:BG-8. And the grip is not written by hand. It is derived from the rule’s own test expression:

string-length(cbc:Name) <= 100   →   too-long:100
count(cac:PaymentMeans) <= 1      →   too-many:1

The class comes from the file with the hash, so when the rule changes the grip changes with it.

The a and e markers are the CASCADE pattern: the products of the classes sum to the whole, and the coverage formula in the sheet comes out at 100 %. That formula is the reason the next section exists.

The CASCADE triangle, and why 0.03 % was the most useful number of the project

Here is the wrong turn, told plainly, because it is the lesson. A script generated the sheets — 819 of them. From that moment on we worked on the script. Every run rewrote the whole workbook, and the table became a printout. Nobody opened it any more.

When we finally did open one in Excel, its summary row said 0.03 % coverage. The CASCADE triangle was missing entirely: the script had written x markers and never emitted the a/e cascade. Every case tested one grip and left the rest of the row undefined. The generator log had said nothing, because from the generator’s point of view every sheet was exactly as specified.

A table a human reads and edits is a conversation about test cases. A script that emits tables is a monologue, and a bug in the script always looks correct in the script’s own output. So we wrote the rule down: the table is the source, not the generator’s output. The test for whether you actually live by it is short: change a cell, re-run the generator — is your change still there? If the answer is no, the table is a printout, whatever the file name says.

Multiplicity, or: one code is not a code list

ISO 4217 has 178 currency codes. A rule that checks BT-5 against the list is 178 rules in a trench coat. Writing 178 columns for it is absurd. Writing one column with EUR in it is a lie: it tests one code and reports the list as covered.

Nanook’s multiplicity section is the way out. A number in the multiplicity row clones the case definition n times (tc.1 … tc.n), and a generator directive walks the referenced code list, one entry per clone. One class, every code, and the sheet stays readable.

This is where the numbers move. The same tables went from a few thousand cases to 105,000 without a single new column. That was the figure we quoted for weeks. It was also wrong, and the way it was wrong is the best story in this piece.

The 105,000 that never ran

The workbooks said 105,000 cases. The suite ran 13,042. Nobody had compared the two, because they lived in different places — one in Excel, one in a test reporter.

The multiplicity had been written onto the referenced data sheets: 52 document types, 179 currencies, 200 country codes, each with the correct count in its multiplicity row. The executing sheet — the one whose columns actually become cases — carried 1 everywhere. Nanook clones where it generates. A referenced sheet is consulted for values, not executed, so its multiplicity row is read by nobody. Result: zero clones, and from every code list exactly one code was ever tested.

The rule that came out of it: what a referenced sheet says about values holds; what it says about the number of cases has to reach the generator. We lifted the multiplicity onto the executing sheets and the count went from 13,042 to 88,609 — 76,253 of them clones.

How it was found is the transferable part. We had raised the CI sample from 400 to 2,500 cases and the executed test count came back bit-identical. Not close — identical. That was the same signature as a slice(0, n) bug we had hit earlier in the project, and it is a signature worth memorising: when a number does not move although it should, suspect the instrument, not the thing.

Clones that repeat themselves would be idle work, so the counter-check went in the same day: 32 clone groups, none of them with two identical documents. EN_Seller_E_14 has 200 clones and produces 200 distinct invoices. A guard holds that, and we verified the guard the only way a guard can be verified — by switching the instance number off. It went red and named the group.

Three Run Levels, Because 105,000 Is Not a CI Job

The full level is simply everything the suite builds — the 88,609 from the multiplicity fix above (the section title keeps the number the workbooks used to claim). The decision behind the split was made in one sentence, and we kept it verbatim:

“CI should have a sample, not the 6,500. Fifty is enough. Locally, everything must run.”

The mistake we made here is worth more than the table. The sample took the first 80 cases per kind. Then we added two new grips, which produced 72 new cases — and the number of executed tests stayed bit-identical. The first 80 were the same first 80 they had always been. The sample was blind to every addition, and the measurement confirmed the error instead of showing it: a stable count looked like a stable suite.

The fix is a few lines. Take every k-th case, deterministically — a red sample has to be reproducible, so never random:

const k = Math.max(1, Math.floor(cases.length / SAMPLE_SIZE))
const sample = cases.filter((_, i) => i % k === 0)
// deterministic: same input, same sample — and a new case shifts the stride

Immediate yield after the change: two failures the first-80 sample would never have reached.

What It Caught

The tables were built to prove conformance. What they actually found was that fields we accept never reach the document. Eight of them, each confirmed the same way: a request carrying the field, a 200 in response, and an invoice without it. The six with the widest blast radius:

These are our own generators, and that is why the story lands: the transfer succeeds, the validator is happy, and the invoice is wrong. BG-11 is mandatory in several EU states for a non-resident seller; without it the document is factually incorrect and nobody notices. BG-10 says where to pay. Drop it and the customer pays the seller instead of the factor, and the receivable stays open.

Two further findings we filed rather than fixed, because each is a product decision: normalizeUnitCode maps any unknown unit to C62 (“piece”), and an unfitting tax category is silently replaced by a default. Garbage in, valid invoice out.

Four Derivations We Built, Measured, and Deleted

This is the section that makes the post worth reading. Each of these was plausible, each was wrong, and in each case it was the run that said so — not a review, not an argument.

1. The context bridge. Many rules name no business term in their text, but every rule names an XPath in its context. So: map context → term → input field, and derive the grip from there. Measured: 14 of 56 contexts mapped, several of them wrongly. Deleted. Then, a day later, we fixed the rule source we had been reading — and re-measured out of habit: 491 of 764. The reason for the rejection had never been the bridge; it had been a broken input. A discarded derivation deserves re-measuring the moment its input changes, and this one now carries most of our mappings.
2. The group grip on categories. Remove a whole group to break a rule. Correct for BG-8, the buyer address: no group, rule fires. Wrong for 231 rules where the group is merely the scene the rule plays in — removing it satisfies the rule’s condition trivially, the rule never evaluates, and the case goes green having tested nothing.
3. Reason codes by index. BT-98 is the allowance reason code, BT-105 the charge reason code. Their paths assumed order — .0 is the allowance, .1 the charge — rather than identity. When a document carried a charge and no allowance, the grip landed on the wrong entry. A grip that hits the wrong field one time in two is not coverage; it is noise with a rule id attached.
4. The note subject code. BR-CL-08 wants a code from UNCL 4451, carried as #AAI# at the start of the note text. Faking that prefix looked airtight. It failed in 102 of 117 cases: our notes field is free text, the generator passes it through verbatim, and the convention simply does not exist in our model. The grip was testing a format we never write.

The pattern across all four: the mapping was right and the effect was not. On paper each derivation reached the rule it was aimed at. Only the run tells whether the rule actually evaluated. The cheapest instrument for that turned out to be raising the sample size once — 80 to 400 — and reading what fell out.

The Ratchet That Read Progress as Regression

We guard the numbers. “Unreachable classes” — grips the base data cannot produce — is a count that may only fall, and a test holds it as an upper bound per rule family. Then it rose by exactly 1 in all 36 families at once.

Nothing had regressed. Two new grips reached more rules, and each touched one more field our base data does not carry. A single upper bound cannot tell progress from regression — and worse, it can be satisfied by leaving cases out. Delete the new grips and the guard is green again, having protected nothing.

The fix is two numbers instead of one:

expect(unreachable).toBeLessThanOrEqual(CEILING)   // may only fall
expect(executableViolations).toBeGreaterThanOrEqual(FLOOR)   // may only rise

Progress that raises both is acceptable; a change that lowers the ceiling by dropping cases now trips the floor. Two numbers that pull in opposite directions cannot be gamed by removing work.

What the Numbers Say Now

The last row deserves honesty, and it makes the strongest point in the post: most of it is not work. The three largest blocks alone cover half the gap: 2,447 of those rules are German rules sitting in a Dutch rule family — a case for them would assert XRechnung behaviour on a Dutch invoice. 706 are Spanish tax categories in a German family. 379 are structurally unbreakable, because our generator rounds every amount to two decimals and the rules test for a third. A case for any of these would be wrong, not missing.

And most of the remainder is not a test problem either. 91 of the 201 business terms have no input field at all. Line-level allowances and charges (BG-27, BG-28) do not exist in our model — there is a single discount: {type, value, reason} per line. No grip can reach a rule about a field the API cannot receive. Those rules need a feature, not a grip.

That distinction was the most valuable output of the exercise. Not the coverage number — learning to say, rule by rule, which part of the gap is testing work and which part is product work.

Then We Added the Missing Fields — and Four More Rules Bit

Once the tables showed which terms the input never carried, the fix was a feature. Three plans, three releases, and the input model gained the fields the rules had been pointing at. The moment those fields were rendered into a document for the first time, four national rules objected — to values that had been perfectly fine for as long as they were being thrown away.

The lesson generalises past invoicing: a field the generator used to discard becomes a new assertion the first time it is written — and a national profile may forbid it. Filling a gap is not neutral. It changes what the document says, and every dialect gets a vote on what it now says.

The sharpest find came free. lib-invoice-outbound-fr/ubl.ts called the same builder twice — copy-paste, identical comment, two lines apart. UBL-SR-04 says the element may appear once, and it is fatal. The duplicate had sat there for months without a single failure, because without a value the block writes nothing, and twice nothing does not show up. Only when the fixture finally carried BT-18 did dead code turn into a red test.

What We Would Tell Someone Starting This

1. Pin the specification, then read it. Every expected value traces to a file with a SHA. If it traces to a constant in src/, you are testing a transcript — and ours contained a code BR-DE-17 does not allow.
2. Open the workbook. Read coverage in the file, not in the generator log. Our 0.03 % was in the file; the log said nothing.
3. Let the sample scatter. slice(0, n) goes blind the moment you add anything, and a blind sample reports stability.
4. Give every guard a counter-metric. One number can always be gamed by removing cases. A ceiling needs a floor.
5. Re-measure what you discarded. Four derivations died here; one came back when its input was fixed, and now carries 491 mappings.
6. Fill the fixture before you trust the coverage. Half our findings were fields the input never carried — and an empty field cannot fail a test, however many rules point at it.

The smaller version of this story — one SaaS, one runner, 1,981 cases — is in How We Test a SaaS Application with Nanook. Matrix tables, custom generators and the table tutorial are in the guide.

If your domain has a specification, your test oracle is already written — in someone else’s repository. Start with the Quickstart, then read the multiplicity section: it is the difference between testing one currency code and testing all 178.

Building against EN 16931, Peppol or a national CIUS? We would like to compare notes — the hard part is never the XML.