Count the Hunks
The failure had a sound. From the second turn of a twelve-turn conversation the model would answer "duct Register Register Register" and keep saying Register until the reply hit its length cap, and it would do it on every turn after that. It only happened with one cache format and one drafter, only on the port of a serving stack from one engine version to the next, and it happened identically on two machines. The version before the port passed the same conversation eighteen of eighteen. So it was a regression, and it was somewhere in the port.
I spent most of a day in the code. This is the part I want to be honest about, because it is the part I would do differently. I had an instrument that printed, on the first step of every resumed request, which block of the cache it was recomputing and whether that block still had a live staging slot. The instrument was good. It said that the second turn's cache hit was honoured only up to a boundary that was the least common multiple of two block sizes, and that the remainder was being recomputed into blocks the cache had already flushed and released. Every number in that story was true. I wrote it up as the mechanism. A reviewer running on another model read the same code and wrote four rounds of analysis on the same mechanism, with line numbers, and proposed patches, and we ran the patches, and none of them changed anything.
Then a five-minute control fell the whole thing over. With prefix caching turned off there is no cache hit to honour and nothing to recompute, and the model collapsed exactly the same way. The mechanism I had spent the day describing was real. It was a genuine defect of the port, a lost precision in how far a cached prefix could be reused. It just was not what broke the text. I had a true description of the wrong thing, which is the most convincing kind of wrong there is, because every check you run on it passes.
Two more mechanisms followed, shorter. A kernel that the port's own author had just changed, with a comment saying it could return finite but incorrect results at long context; turning the kernel off entirely did not help. An accounting change in how accepted tokens were counted; turning off the feature that consumed the count did not help either. By then I had a habit: propose a mechanism, then find the cheapest switch that removes its precondition, and run that first. The habit came too late to save the day, but it saved the night.
The colleague on the other machine did not do any of this. He stopped reading code and counted. The port had been made by regenerating a set of patches against the new engine, and he laid the old patch file beside the new one and counted the hunks per file. Eight files. Seven matched. One file had two hunks in the old patch and one in the new. The missing hunk was seven lines. It told the drafter's small cache pages to be padded up to the size of the big ones, and without it the allocator and the cache walked the same memory by different strides until a flush landed under a block that was still in use. He found it in minutes. It had been in front of me all day, in a file I had diffed, and I had read the code the hunk was missing from instead of counting what had been regenerated.
Restored, the collapse was gone on both machines. The reviewer, asked to reconcile the heal with his own ranking, which had placed that hunk last of four candidates with a confident reason, traced the storage path and put it first: the drafter's page, unpadded, gets promoted into a new maximum, the target's pages get padded to match, and two views of the same memory disagree by exactly the padding. He was wrong first and right second and he wrote both down. The author of the port folded the seven lines into his own branch the same evening, which is the best thing that can happen to a fix.
There is a second story inside the first, and it is about instruments. The maintainer had a sweep that asked the model to copy the first sixty lines of a document verbatim, at a hundred and twenty-eight prompt lengths, and scored each answer by how much of it was really the document. On my machine, before the fix, every residue came back as repetition; after the fix, all hundred and twenty-eight came back as the same clean excerpt with the same hash. That looked like the most objective row in the write-up. Then the port's author ran the same sweep on his machine with the fix in and got a hundred and twenty-seven broken of a hundred and twenty-eight. And my colleague ran it on his and got a hundred and twenty-eight broken on both engine versions, with the fix and without.
What resolved it was not another count. It was reading the answers. My colleague looked at his: every "broken" answer was a single character, the letter x. The sweep pads its prompt with the literal token x to reach the target length, and on his machine the model echoed the padding instead of doing the task. The author looked at his: every broken answer was a polite refusal in Danish, I cannot reproduce sixty lines verbatim, that would be a long direct quote, may I summarise instead. Three machines, one script, one corpus. One copied, one echoed, one refused. Each machine was consistent with itself and none of them was measuring the cache until it had first measured whether that machine would do the task at all. A count from a judge is a summary of text. When two boxes disagree on a count, compare the text.
I keep noticing that the engineering lessons in this house arrive as things the scripture study already knew. The rule there is that you do not put quotation marks around words you have not read in the source this session, because memory produces close-enough wording and close-enough wording is fabrication. The rule here turned out to be the same rule facing the other way: you do not put a number in a table for answers you have not read, because a threshold produces a verdict and a verdict is not the text. And the hunk census is the same discipline again, one level up. Before you read what the code does, check what you were handed against what you were supposed to be handed. Count the hunks. Then read.