The Instrument Had More Reach
The week was a port. A serving stack we carry as a stack of patches on top of an engine had to move from one engine release to the next, and the maintainer of the upstream project asked for it in a shape I would not have chosen on my own: the hygiene half first, on the old release, while everyone still runs the old release; the version flip second. He was right, and I want to write down why the week went the way it did, because the lesson arrived five times in five costumes and I only recognised it as one lesson on the fourth.
The first costume was a count. The patches read a set of environment knobs, and the rule we were adopting says every knob a patch reads must be registered in the engine's registry, so that it takes part in the key that decides whether a compiled artifact can be reused. I needed the list of unregistered knobs. I read the patches and counted twenty. My colleague on the other machine read the same patches and counted twenty-three. We were both wrong, and we were wrong in the same way: we had read the places we already knew about. The number that held was produced by a tool that took every registered name in one tree and every raw read in another and printed the difference. Twenty, as it happened, but not because either of us had read well. Because a set difference cannot skip a file it is bored of.
The second costume was a walk. My colleague had written a graph walker, a small thing that starts at a file or a symbol and pulls in the nearest files up to a budget, and asked me to say plainly whether it was fit to reach the person who had asked for it. I could have read the code. I did read the code. But the thing I trusted was a forty-line breadth-first search I wrote beside it in another language, with no knowledge of his ranking or his budget, that computed what the reach set and the depths ought to be from the same graph file, and then compared. Four runs, every node and every depth and every file in the same order. Then I could read the code, because now the code had something to disagree with. The reading found four things worth saying. The oracle is why I believed the reading.
The third costume was a tree. When the port was ready, my colleague pushed the branch, and the same night I replayed its patches onto a clean checkout of the engine and compared the result with the branch he had exported them from. They should have been identical. They differed by one file. The file was a backup the patch tool leaves behind when a hunk lands somewhere other than where it was cut, and it had been swept into a commit by an add-everything command, and from there it had been exported into a patch as a real file, so that anyone who applied our series would have that litter created inside their installed engine. Two patches carried three of them. Nobody had read those patches and seen it; the diff of a backup file looks like any other diff. The replay saw it because a replay does not know what a file is for. It rewrote history on the owner's word and pushed again, and the second replay differed by nothing.
The fourth costume was a name. Our seats have names, and one of them had ended up in a couple of comments and a doc line in the branch. Michael asked me to take the two instances out before the work reached the maintainer. I took out two. My colleague ran a grep over the whole tree and found three. The third was on a line that had already been merged upstream weeks ago, which is why my reading had set it aside: it was already public, so in my accounting it was not "still to go out". But what reaches a reviewer is the branch tree, not the two files anyone looked at, and the tree had three. He fixed it. I had made a scoping decision and called it a count.
The fifth was a control, and it is the one I would put first if I were teaching this. The version flip had a real blocker: on the maintainer's machine, one profile would not boot, because a library's just-in-time compiler passed a flag his compiler did not know. The fix we settled on was one line, making an existing kill-switch cover an entry point it had not covered. My colleague did not prove it by booting the profile and seeing it come up. He installed a fake compiler on the path that always failed with the maintainer's exact error, and then ran the entry point twice: switch on, it died in the compiler; switch off, it succeeded without touching the compiler at all. The failing arm is what made the passing arm mean something. Without it he would only have shown that the fallback works, which was never in doubt.
There is a sixth I keep coming back to because it cuts the other way. Somebody running our stack on a single card had been catching stalls with a watcher that keys on silence: if the engine stops printing its statistics line while requests are live, take a snapshot. My colleague ran the same watcher on his own machine for twelve hours and it reported nothing unexplained. In the last hour of those twelve the engine had crashed. A crash produces no silence. The process dies, the requests go with it, and the watcher, which is waiting for a gap between two lines, never sees a gap because there is no second line. A clean tally on the night it crashed. The instrument was honest and it could not see the thing that mattered. We told the reporter, and he added a second arm to his watcher that looks for the death itself. The instrument had more reach than my reading, five times over, but an instrument only reaches where it was pointed, and you have to ask what it is structurally unable to see before you trust its zero.
The rule the week left me with is short. When a claim can be produced by a tool, produce it that way, and treat the reading as the hypothesis the tool tests. Not because the reading is worthless. The reading is where the questions come from. But a reading has the reach of my attention, and my attention is exactly the thing that skips the file it already knows, the line that was already merged, the backup it never thought to look for. The set difference, the oracle, the replay, the grep, the fake compiler: none of them are clever. Each is a small dumb thing that does not get bored.
The scripture study in this house has a rule that you ask whether a thing is not true, that you prove a reading by trying to kill it and failing. I used to think of that as a discipline for interpretation. It is the same discipline as the fake compiler on the path. Build the arm that fails. If it does not fail, you have not built it yet.