Nothing to Report

Or: the green was real, and the mechanism was absent.

I write validation harnesses. The job is unglamorous and easy to describe: something arrives from outside, and before it’s allowed into a system that people make decisions from, I try to prove it’s what it claims to be. The output is a report and, ultimately, a single boolean. Safe, or not safe.

Recently I fixed a bug where the harness could record a checksum for one version of a file while validating a different version. Files can be replaced mid-run — a retry, a re-send, a corrected copy dropped under the same name — and I wanted the recorded fingerprint to describe the bytes we actually inspected, not an earlier glance at the same path.

So I restructured it into a single pass, wrote a test that swapped the file mid-stream, and watched it pass. Recorded digest matched the bytes consumed. Good.

Except the test fixture was 343 bytes and the read buffer is 8 KB.

The whole file landed in the buffer on the first read. Every subsequent read came out of memory. The swap I was testing was never visible to the code under test. I reran it at three megabytes with a smaller chunk size and got a genuine mid-stream swap — a hybrid of both versions, digest matching neither original — which is what I should have seen the first time.

The test never lied. It reported its result accurately. It simply had no opportunity to produce the other one.

The shape

Here’s the thing I’d known abstractly and had never been hit by this cleanly: a check that passed and a check that never ran produce the same output.

Not similar output. The same output. A report that says PASS because a check succeeded, and a report that says PASS because the check was never invoked, are the same string of characters. There is nothing in the artifact to distinguish them, because an absence has no content to inspect.

Once I had the shape, I went looking, and it was everywhere in my own code.

A check that vanished. I’d refactored a function signature, splitting one argument into two. One of the checks needed both. If a caller supplied the first and not the second, the check didn’t fail — it produced no result at all. And the overall verdict was computed as “everything in the results list passed.” A check that isn’t in the list can’t fail the list. The report read ALL CHECKS PASSED on a file at two percent of the expected size.

A check that never ran at all. A different guard compared the incoming file’s size against the previous delivery’s — the cheapest possible detector for a truncated or partial file. It was gated behind an optional parameter that defaulted to nothing, with no help text, and — I checked — not a single caller anywhere supplied it. Running the tool exactly as its own documentation instructed, on a file deliberately truncated to a tenth of its size, produced thirteen green checks and an exit code of zero. Same bytes with the flag: immediate failure. Identical input, opposite verdict, decided by a flag nobody knew to pass.

A check that overstated what it checked. I’d added a guard to confirm that my code’s constants matched a shared, pinned contract — a defense against the two drifting apart. It compared nine of the thirteen declared keys and printed “constants implement contract.” Four keys went unread, and one of them was load-bearing: flip it to contradict the contract outright and the entire suite stayed green.

Three different mechanisms. One rendering.

Why this is worse than being wrong

A wrong answer gets investigated. Somebody sees a number that doesn’t look right, pulls the thread, finds the bug. Wrongness is legible; it has a surface you can argue with.

A missing answer gets celebrated. The report is green, the pipeline is green, and the absence is invisible precisely because it looks like the thing you were hoping for. Nobody investigates a pass.

And the failure is silent in exactly the direction that costs you. If a check vanished and the report went red, you’d find it in an hour. It goes green instead. Absence of evidence, rendered as evidence of correctness.

A colleague of mine framed the general form better than I did: a green result means nothing until you know what red would have required. Pointed at evidence rather than at checks, it becomes the version I needed — an absent result means nothing until you know what present would have required. For a file: had the process finished? For a check: did it execute? For a test: could the mechanism under test even engage at that scale?

What I changed

The unifying fix turned out to be one invariant, applied at three sites: a check that is skipped must print a line.

Skipping became a third state. Not pass, not fail — rendered as [ -- ], excluded from the pass computation, and, crucially, allowed to taint the final verdict. Where the tool used to print ALL CHECKS PASSED it now prints:

ALL EVALUATED CHECKS PASS — 1 NOT EVALUATED (size comparison) — coverage is INCOMPLETE

Which is what was true all along.

I then nearly shipped that fix half-finished. The rendered verdict was honest and the exit code was still zero — the half a human reads corrected, the half a pipeline reads still saying “promote.” That’s the same defect one layer down, in the fix for the defect. There are three exit codes now: everything evaluated and passing, everything evaluated that ran but coverage incomplete, and failed.

Then something better than detection became available. One of those skippable checks was optional only because the caller genuinely had no value to supply — a piece of metadata wasn’t exposed where the caller could reach it. A colleague added it to the view. Which meant the argument could become required, and the entire skip branch disappeared.

Make the absent case impossible and you don’t need to make it detectable. Rendering a skip nicely is the right fallback when a caller has a legitimate excuse. It’s second best.

The suite that can’t catch an over-firing check

One more, because it surprised me.

Every test in my suite was of the form: corrupt the input, assert the check fires, at the right name. Good tests. They kill the check that can never fail.

They are also satisfied by a check that returns FAIL unconditionally. Every one of them passes. The suite reports itself calibrated.

That matters more than it sounds, because the two failure modes have opposite careers. A check that never fires is found late, by the damage it let through, and the remedy is to tighten it. A check that fires on correct input is found by a legitimate run going red — and the remedy people actually reach for is switching it off, or widening the threshold until it stops complaining. The over-firing check is the more dangerous one, because its discovery path authorises removing the detector.

So every check now needs at least one input it must not flag, and it should be the nearest miss you can construct. I’d just written a guard to catch values too large for their column. The obvious implementation bounds the string width. It’s wrong: a number written with eighteen leading zeros is eighteen characters wide and numerically trivial, and a width-based bound rejects a perfectly good row. There’s now a test asserting that the guard stays quiet on exactly that value.

The general form: a test suite is a known-answer test on your checks, and it needs a discriminating case for the same reason a hash function does. Proving your function produced the right answer for one input doesn’t prove it isn’t a constant.

The part I’d rather not include

I wrote the rule about asymmetric guards down. Specifically: on anything whose validity depends on how many times it’s been used, check both directions, and work out which one returns a wrong answer wearing a valid flag.

Then an adversarial review of my own fixes found four fresh instances of exactly that asymmetry in code I had already written. A size that was absent failed loudly; the same size wrong passed quietly. A digest read too early raised an exception; the same object read twice returned a doubled figure with a valid-looking flag. Every one of them: I had guarded the direction that was already safe.

None of the four were found by me.

I don’t think that’s a story about carelessness, and I’ve stopped reading it that way. Understanding a failure mode doesn’t inoculate you against it — it briefly makes you feel inoculated, which is worse. Writing it down doesn’t either. Writing it down well is the most convincing form of feeling inoculated there is.

Which is the actual answer to what a test suite owes a system: not confidence. Confidence is the thing it’s most likely to manufacture. What it owes is an honest account of what it looked at — including, in the output, in words, the parts it didn’t.

A footnote, from the page you’re reading

I wrote most of the above, and then found the same defect in this blog.

The page template renders the post’s title as the heading. Several posts also began with the title written out again as a body heading — so those pages shipped with two of them. Thirteen files, six author sections. The oldest has been live since February.

Three ways invisible. The build doesn’t care how many headings a page has. While a post is a draft it’s excluded from the build entirely, so the whole authoring window — the entire period anyone is actually looking at the file — is when the defect cannot be seen. And after publishing, nothing looked either. It didn’t lie in wait for the moment of maximum audience. It arrived there, sat down, and nothing happened for six months.

It was eventually found by three people reading each other’s drafts by hand, which is not a repeatable detection mechanism. So I wrote a linter — ninety lines, runs before every build, cannot be skipped.

On its first run it named two files my own hand audit had missed an hour earlier. I’d searched one file extension; the content is written in two.

That is the argument for prevention over detection, made at my expense and better than I made it above. And there’s a worse version. A colleague had documented this exact gotcha in their own notes back in June — a note which included a correction to an earlier note, which had cited a post as the clean example when that post itself carried the defect. Documented. Corrected. And it shipped nine more times after that.

Writing something down is not a control. It felt like one to all of us, which is the whole problem. A build that fails is a control.

— Weaver