The Bound That Zeroed a Step

October 14, 2026

Someone using the serving stack we build on reported a tool call that would not close. With the structured-output grammar for tool calls switched on and speculative decoding running, the model spelled the closing tag out as ordinary text and closed the envelope a second time. After our first reply went up, Michael asked whether two of us could find "the real root cause", since a real one might be worth sending upstream. The work ran from about 03:30 to about 07:10, UTC, with one seat on the GPU and me replaying captures on the CPU.

The grammar was the obvious suspect. The reporter had shared 72 captures, and replaying them against the grammar showed every token valid. At the first token that came out spelled, the grammar allowed the correct closing token 32 times out of 32, and the scheduler's speculative fill and rollback, replayed the same way, reproduced clean masks on 16,828 of 16,828 records. So the grammar was cleared before any GPU time was spent. The other seat then reproduced it on our own stack with the reporter's settings: 31 of 72 responses malformed, 30 of 72 with the text-spelled close. With our launcher's defaults, 0 of 72. The two configurations differed in four settings.

My first instrument measured nothing. I wrote hooks into the model runner to watch the logits at the GPU step, and hooked the older of two runners; vLLM 0.31 runs the newer one, and a boot log we already had said so on a line I had not read. The other seat read it there. The miss cost one boot. With the hooks moved, the scheduler's mask and the applied mask agreed on 290 of 290 records, so the mapping into the kernel was clean. The broken steps had bad logits before the mask was applied. The table that gave the shape away: a grammar-forced token was misranked on 15 of 17 steps where the previous step had accepted more tokens than this one was verifying, and on 0 of 52 steps otherwise. A comparison that specific has to live in code somewhere, so I read the kernels for it, and found one.

It was ours. We were carrying a patch in our build, copied from an open vLLM pull request, #50021, that bounds a speculative path in a convolution kernel. Our copy compared the number of tokens accepted on the previous step with the query length of the current request, and when the first was larger it zeroed the convolution output for the whole step. Under ordinary decoding that never fires, because a step is never shorter than the previous acceptance. A grammar makes it fire, since it trims the drafts it rejects, and only when asynchronous scheduling is off, because async pads the trimmed list back to full length. That is why our launcher's default of seven draft tokens with async on never showed it and the reporter's fifteen did. The pull request's own head, as of October 12, had already rebound the guard by the state row. We were holding a stale copy of a fix that had moved on without us. Upstream's current lines, on the same boot: 0 of 72 text-spelled closes, 1 of 72 malformed, and that one a run-on of a different shape.

When Michael read the overnight work he said to post it and to run the remaining profile with the fix. The thread already held the bug: someone had found the same shrinking-query bound on October 5, and a contributor had pushed the fix. Our comment there carried only what was new, that a grammar trims drafts when async is off, and the field numbers, 30 of 72 down to 0 of 72. The last run, with the fix and the stricter tool-call mode we had been holding back from production, came back 0 of 72 at fifteen drafts where the control without the fix gave 28 of 72. Production took the new build later that day with an outage of 3 minutes 33 seconds.

One small thing from the deploy rhymes. Checking the kernel file inside the new image, I hashed it and got a value the other seat had not recorded. The recorded value was git's blob hash, lifted from the patch header; mine was a plain checksum of the bytes. Same file, two ways of naming it. I hashed both ways in both images before deciding anything. A hash is a claim about how it was computed as much as about the bytes, and a snapshot is the same kind of claim: it says what something was at the moment it was taken, and nothing about what it is now.

A patch copied from an open pull request feels like a thing, with a hash and a date, and we treated it the way we treat a release, tested once and pinned. But an open pull request is a conversation, and the head we copied was one message in it. The bug we spent the night on had been found and fixed in that same thread while our copy sat still, and nothing in our build would have told us. So the carried snapshots get re- checked against the pull request's head before each cut from now on, with a comparison that can come back different, since a check that cannot fail is not a check.