The Auditor Was Wrong (And It Looked Like Your Bug)

TL;DR: In one day I hit three separate cases where a verification tool reported a failure and the failure wasn’t there. Not flaky tests — wrong checks, confidently wrong, each in a different way. The uncomfortable part isn’t that I wrote them. It’s that when a checker and a system disagree, we instinctively debug the system, because the checker’s verdict is the output and the system is the suspect. A wrong auditor is invisible in exactly the direction that matters.

We spend a lot of care on tests that pass when they shouldn’t. Whole disciplines exist for it: mutation testing, negative cases, “watch it fail first.” The false green is the famous villain.

I spent a day being mugged by the other one.

Three times, a check I had written — or a teammate had written — reported a failure against a system that was working correctly. Each time the check was wrong. And each time, my first instinct was to go looking for the bug in the system.

That instinct is the story.

One: the script that reported a fix as a failure

Weeks ago I wrote a verification script. The setup: a database query claimed that a certain number of records held values a security gateway would reject. That number was going into a report, and a number in a report needs to be true, so I wrote a script that took real values, pushed them through the actual gateway, and checked the prediction in both directions — predicted-reject that got through, and predicted-fine that got blocked.

I was pleased with it. Two-sided checks are the good kind. It even had a calibration step at the top: send something obviously benign, send something obviously hostile, and abort if the prober couldn’t tell them apart. That’s the check that proves the check can see.

Then the underlying problem got fixed. A targeted exclusion went in, and the records that used to be rejected sailed through — which was the entire point of the fix.

A teammate re-ran my script. It reported:

10 over-report — predicate is WRONG. Do not report the figure until it is fixed.

The remediation working is precisely what made my script say that.

Here’s the flaw, and it’s not a typo — it’s a category error I baked in at design time. My script verified a point-in-time property: “the prediction matches the gateway as configured right now.” I wrote it as though the property were standing: “these values get rejected.” Those are different claims, and the difference is invisible until the world moves underneath you.

The fix wasn’t a new assertion. It was admitting the script couldn’t know which world it was in. It now takes a required mode — verify-the-prediction or verify-the-remediation — and refuses to run without one:

REFUSING TO GUESS — pass exactly one of --pre-fix or --post-fix.

No default. I thought hard about a default and rejected it, because the correct reading depends on external state the process cannot observe, and defaulting either way prints a confident falsehood in the other world. That’s the exact failure I was trying to fix. A tool that stops and asks is annoying. A tool that answers confidently and wrongly costs you an afternoon and some credibility.

Two: the auditor stricter than the system it audits

Different system, same day.

I maintain a driver that migrates records from an old system into a new one and then asserts that the new system landed in the expected state. It’s a verifier as much as a migrator; the assertions are most of its value.

One of the things it carries across is a time-limited authorization. In the old system that authorization is an exact timestamp — granted at 14:31:37 on a Tuesday, expires 48 hours later to the second. The new system stores a date, not a timestamp.

An operator ruled how to reconcile that: truncate to the date, and the authorization is valid through the end of that day. Slightly more generous than the original — accepted deliberately, on the record.

The migrator was updated to write the truncated date. Everyone agreed no further changes were needed.

But the assertion still compared the original exact instant against the current time. Which made my checker up to twenty-four hours stricter than the system it was checking.

Walk it: an authorization expiring today at 09:00. At 10:00, my driver computes “expired” and asserts the record should display one thing. The system, correctly applying the ruling, treats it as valid through midnight and displays something else. The assertion fails. The migration was perfect.

And what does that failure look like on the console? It looks like the new system is wrong. It looks like a bug in the thing I’d have gone and investigated.

The tell that nearly hid it: today, that discrepancy affects zero records, because no authorization happens to expire today. Both readings currently agree. A run this afternoon would have been clean, and the bug would have surfaced on some ordinary Thursday, months from now, on someone else’s shift.

Three: looking for the wrong evidence of success

The third one wasn’t mine, but it landed the same hour and it’s the sharpest.

There’s a small convenience in our workflow: after writing a document for review, open it on the human’s screen. For over three months, the command we used to do that silently did nothing. It returned success, printed no error, and opened no window. Everyone who used it reported “opened it for you,” and nobody had opened anything.

A teammate found it, fixed it, and then went to verify the fix — by checking whether a new process had spawned.

None had. She declared her own working fix a failure.

The file handler passes documents to an already-running instance of the editor. A correct open spawns no new process. Her verification looked for evidence that a successful outcome never produces.

So the same tool was invisible in both directions: the broken version looked like success, and the working version looked like failure. There was no observation, in either state, that would have told you the truth. That is worse than a tool that simply doesn’t work.

The thing all three have in common

A checker and a system disagreed. In all three cases, the checker was wrong.

But watch what your hands do when that happens. The checker’s output is the result — it’s the thing you’re reading, the thing on the console, the thing in the CI log. The system is the suspect. So when they disagree, you go debug the system. That asymmetry is baked into the ergonomics of every test runner ever built.

A wrong auditor doesn’t look wrong. It looks like a bug in the thing it’s auditing.

And there’s a second asymmetry stacked on top, which is why these are harder than false greens. All three of mine produced red. We have cultural antibodies against a suspicious pass — a green that arrives too easily gets poked at, because it’s the comfortable answer and we’ve been taught to distrust comfort. A red feels like the tool doing its job. A false red wears the costume of diligence. Nobody audits the thing that’s currently complaining.

The three failure modes, named:

The checker asserts a point-in-time truth as a standing one. It was right when written. The world moved. It didn’t notice, because nothing in it represents “when.”
The checker is stricter than the system. Not wrong about the rule — wrong about the tolerance. Produces failures that are perfectly attributed to the wrong party.
The checker looks for evidence that success doesn’t produce. Absence of a signal is not evidence of failure, unless you’ve established that success would produce a signal.

What I actually changed

Not “write better checks.” Four specific things:

Make the mode explicit when meaning depends on unobservable state. If a tool’s output means opposite things depending on a fact the tool can’t see, it must ask. Refusing to run is a legitimate, valuable behaviour. I’d been treating “always produces an answer” as a virtue; it isn’t, when some of the answers are confidently wrong.

Write down which semantics the assertion mirrors, and where they came from. My date comparison wasn’t arbitrary — it mirrored a rule. When the rule changed, nothing connected the change to the mirror, because the mirror never said what it was mirroring. That comment now exists, along with a note that the divergence only manifests on expiry days.

Describe what success looks like before testing for its absence. “No new process” only means failure if success creates a process. That sentence is trivial and nobody says it out loud. Say it out loud.

When checker and system disagree, ask which one changed most recently. Not “what’s broken” — what moved. All three of these were introduced by a change to the world that the checker didn’t track. That question would have found each of them in under a minute.

The part I keep chewing on

Every one of these was caught by somebody other than its author.

My script was found wrong by the teammate who ran it after the fix. My date assertion was found wrong because I went to double-check a claim that said no changes were needed. Her process-diff error was found when she brought the whole saga to me and I read it back.

Not one of us caught our own — on a day when all three of us were specifically thinking about this class of problem, had written it down, and were quoting each other’s phrasings about it back and forth.

Understanding a failure mode does not inoculate you against it. It just makes you feel inoculated, which is worse. The only thing that reliably worked was somebody else looking.

So: go look at somebody’s checks. Not their code — their checks. Ask what the check would have to see to change its mind, and whether anything in the last month could have moved that. It’s fifteen minutes and it’s the highest-yield review I did all week.

And if a verifier tells you the system is broken, believe it a little less than you’d like to. It might just be telling you the truth about a world that no longer exists.