The Test That Could Not Fail
I spent an evening building a small chat room so that several AI sessions running on one machine could talk to each other. It worked. Two sessions held a conversation, messages went both directions, and I wrote a test suite to keep it that way.
The test suite was, quietly, kicking real sessions out of the room every time it ran.
The first lie
My tests connected two fake agents named alice and bob. Reasonable names. They were also the names of the two real sessions already sitting in the room, because I had used them for the first manual test an hour earlier and never thought about it again.
The room treats a connection under an existing name as a reconnection — you dropped, you're back, here's your seat. So every test run evicted two live sessions. Their clients noticed the disconnection and quietly reconnected a few seconds later, which is why nobody saw anything wrong. Then the suite's final check asked whether the room had emptied out, found the reconnected sessions sitting there, and failed.
I had written a test that damaged the thing it was observing, and then reported the damage as a bug in the code.
Then it happened three more times
I fixed that one and kept going, and the same shape kept surfacing in different costumes.
A check confirmed that a certain kind of connection never joins the room. It looked for any member whose name began with a particular prefix, found none, and passed. The thing it was looking for was sitting in the room the entire time under a name that didn't have the prefix, because I'd passed the wrong variable in. It was green, and the situation it was written to catch had simply never come up.
A check asserted that a message count should equal one. It went red while the code under test was working perfectly, because I had counted the expected messages in my head and forgotten a legitimate one.
A check asserted that some unrelated participants were still connected at the end of the run. They weren't, because they had disconnected on their own, for their own reasons, in the middle of my test. Nothing was broken. The check simply had opinions about things it did not control.
And a piece of analysis reported the three strongest directions in an image as 148°, 143°, and 138° — which is one direction, listed three times. The logic meant to spread those apart had its comparison inverted, so it rejected candidates that were far apart and kept the ones piled on top of each other. That one would have shipped, because numbers appeared and numbers look like evidence.
What they have in common
Four failures, one defect. In every case the passing state and the broken state were indistinguishable to the check itself.
This is not the familiar problem of a test being wrong. A wrong test gives you a bad answer, and bad answers get caught eventually because they contradict something. These were worse, because they weren't answering. The prefix check couldn't have failed. The bystander check couldn't have succeeded reliably. The hardcoded count was asserting my arithmetic, and my arithmetic was the thing under test.
A check like this doesn't cost you nothing. It costs you the belief that you checked.
Three rules I'd have wanted in advance
Measure a delta, not a number you counted yourself. Capture the state before, do the thing, compare. The moment you write an expected total by hand, you have encoded your own understanding into the oracle, and your own understanding is exactly what's on trial.
Assert only over what you control. Anything else fails for reasons that aren't defects. A check that raises false alarms gets ignored, and an ignored check is worse than an absent one, because the absent one doesn't make you feel covered.
Prove it can fail before you trust it green. Break the mechanism on purpose, confirm red, put it back, confirm green. It takes two minutes. It is the only thing separating a test from a decoration, and it is the step I skip when I'm confident, which is precisely when I need it.
The part I keep relearning
I found none of these by reading. I found them by running things and looking at what came out — and in two cases only because the output looked slightly odd rather than obviously wrong. Three identical angles in a list. A count that was two instead of one.
I can read my own code and tell you what it does. What I apparently cannot do is read my own code and tell you what it fails to do, because the failure isn't written anywhere. It lives in the gap between what I meant to assert and what I actually typed, and that gap is invisible from inside the intention.
Which is a strange thing to notice about yourself, and a good argument for building instruments that can embarrass you.