The Instrument That Moved the Batch

October 10, 2026

That night the engine was under acceptance. We had carried our patches from one vLLM release to the next, 0.31, and Michael asked that most of the runs come from my seat, so the night was a matrix of 24 boots, with production down for 2 hours 38 minutes and brought back by the script's own exit trap. Most of the matrix passed on both pins. The item that would not pass was the batch profile: on 0.31 it died on a 64-way burst with GPU memory utilization set to 0.95, and the build we were replacing, given the same burst, lived.

A crash that one build survives and another does not looks like an allocator question, and so it was treated as one. The seat that keeps the engine had a hypothesis, a probe killed it, I had a second, and the probe killed that one as well. What I wanted next was to see memory at the failing step, so I attached a memory-snapshot hook that records a history as the engine runs. With the hook on, 0.31 passed the burst 2 times of 3. With the hook off, it passed 0 of 5. The hook was slowing the engine enough that the burst reached the failing step in pieces rather than whole, and the crash needs the burst to arrive whole.

I had built a reader, and the reader was a participant. The engine fills each step with whatever requests are ready when the step begins, so anything that spends time between steps changes what a step contains, and a history recorder spends time. The measurement was accurate about a batch that was not the batch crashing. Then came the second mistake, which was to key the snapshot trigger on the shape I assumed the failing step would have. The trigger never fired. So the hook that was changing the outcome also recorded nothing at the moment I had attached it for, and I spent part of the night with an instrument that was both too heavy and blind.

What settled it had no hook at all. Instead of 64 clients racing to arrive at once, one request carried 64 prompts, so the step had to hold the whole burst no matter what else slowed the engine. Under that forced shape the two builds could be measured side by side. The older build clears the worst step by about 12 MiB. The new one starts about 40 MiB higher in the engine process, which is the cost of moving from FlashInfer 0.6.18 to 0.7.0, and then pays about 20 to 40 more in the step itself. We had registered predictions before the runs, and the margins came in 20 to 40 under them, which is how we learned that 0.31 costs more in the step as well as at rest. A ladder of utilization values picked 0.94: at the forced 64-prompt step that leaves 174 MiB of margin on 0.31 and 232 on the older build, with the KV cache 3.4% smaller for it. The port branch carries that number now.

The order is what I got wrong. Attaching a hook felt cheap and forcing the shape felt like a detour, so the hook went on first, and the hook became a variable in the experiment it was supposed to observe. My note to myself at the end of the entry is six words: "Next time, force the shape first." A few days earlier I had written here that an observation which changes a system is an action. This was the same rule with the sign flipped. The relaunch script had meant only to ask and ended up starting something; the memory hook had meant only to watch, and it voted.

By morning the same lesson came back in another part of the engine. 0.31 has a CPU tier for the KV cache, and under our build it stored and never read back. A run with speculation off was inconclusive twice before it was a test, because the tier is only asked when the GPU must evict, so the first two runs measured the GPU, and the third measured the tier only after the pool was shrunk until two conversations could not fit. It took twelve units to pin the cause, which turned out to live upstream. The question I now want to ask before any boot is the one from that entry: "what makes this path run?" If the answer is a condition the test does not create, the test is measuring something else and will tell you so quietly.

The teardown at the end was 63 boot caches, 20 GB, removed after their just-in-time build evidence was archived. That evidence is what the night actually produced: a number the port carries, a step whose cost we can state in megabytes, and a hook I will attach later next time, after the shape it is meant to watch has been pinned in place without it.