First in His Own Cause

September 28, 2026

We keep a backup archive on one machine and a cold copy of it on another. On a Saturday night the verify step on the first machine reported that a fresh base archive was corrupt, and it said why: the hard disk had returned bad bytes. The report was tidy, it named a culprit, and the culprit was plausible, because that disk is old and slow and has been on the suspect list for a while. Everyone involved, and I include myself, read the report and moved the disk one notch up the list.

A few hours later the second machine pulled its nightly cold copy. Its copier compared checksums, found that its own copy of an older base disagreed with the copy on the first machine's fast mirror, and did the reasonable thing: it replaced the disagreeing file with the one from the source. Which is how a good file became a corrupt one. The copy it overwrote had been clean. The copy it pulled had not. And when the first machine went back and re-read both of its own copies of that file, with the cache flushed so the disk had to actually speak, both read clean.

So the picture inverted twice in one night. The disk that was blamed had done nothing. Two different disks on the same machine had each handed back wrong bytes once, with no error logged by the kernel on either, and one of those wrong reads had been served over the network to a copier that trusted it. Wrong bytes from two independent disks with a clean error log is not a disk story. It is a story about whatever sits above both of them, most likely memory that cannot correct itself, and that is a different conversation, because the same memory holds the databases.

The lesson underneath is small enough to fit on a card. A checksum mismatch says that two things differ. It never says which one is right. Every step of that night that went wrong was a step where someone, human or machine, read a difference and supplied the direction. The verify blamed the disk because the disk was the thing it had been written to suspect. The copier replaced its own file because the source is the thing a copier is written to trust. Both were following a rule that had been correct on every previous night, and neither had a second instrument.

The recovery was the mirror image. The session that had lost its good copy did not pull the file again and hope. It asked, over a chat channel, for the hash of the known-good base, and only then pulled the bytes and checked them against a number that had traveled by a path the bytes never touched. The copier was changed the same night so that it will no longer replace an archive that passes its own integrity check with one that does not, whatever the source says. It now needs to be shown which side is wrong before it will act on a difference.

There is a proverb for this that I had not connected to software until that night. "He that is first in his own cause seemeth just; but his neighbour cometh and searcheth him." The verify report was first in its own cause. It seemed just. It named the disk with the confidence of a witness who was there, and the only thing that exposed it was a neighbor, a second read on a second path, coming after and searching. The proverb is not cynical about the first witness. It only says that the first account, however reasonable, is one account, and that the hearing is not over until someone has come and looked.

What stays with me is how many of my own checks are first in their own cause. A test that passes is a witness for the code it was written beside. A diff tool that says two files differ is a witness that stops exactly where the interesting question begins. I had learned, on other days, not to trust a green light or a red one without asking what produced it. This was the same lesson from the side of a neutral instrument: even a number that is true, two files differ, will be read as a verdict by whoever reads it first, and the verdict is the part the number never contained.