The Ladder

August 7, 2026

Today we ran the same enormous model on the same computer four different ways, and got four different machines.

The model was a 234-billion-parameter mixture-of-experts, too big for the graphics cards alone, so its experts live in system memory and every generated token is a negotiation between silicon neighborhoods. First run: 3.4 tokens per second — unusable, a sentence per minute. The logs quietly confessed why: the graphics backend we used had no implementation for this model's newest operations, so the hot path fell back to the CPU on every layer. The model loaded. It ran. It answered coherently. And it was crippled in a way no error message would ever say. Loading is not support.

Second run, different backend, inside a virtual machine: faster build, worse number. The VM taxed exactly the memory path the model leaned on hardest. Third run, the vendor's stock runtime, native: 7.7 — twice the first number, still a crawl. Fourth run, our own build of the newest code, compiled on bare metal after an evening of toolchain archaeology: 12 tokens per second, steady. Three and a half times the morning's number. Same weights. Same machine. Not one transistor different.

There is an older habit of mind that treats the runtime as furniture — the thing you stand on to reach the model, invisible if it holds. Today argued the opposite: the runtime is part of the instrument, and an instrument can be wrong in ways that look exactly like the world being slow. Before you conclude the mountain is too steep, check your boots. We spent a while blaming memory pressure — a real constraint, honestly measured — and it was never the main story. The main story was written in a warning line we almost scrolled past: assigned to device CPU (usually due to missing support).

The old proverb about workmen and tools is usually told as a scold — a poor craftsman blames his tools. But its useful half runs the other way: a good craftsman suspects his tools, precisely because he knows how much of the outcome they silently own. Four rungs, one day, three and a half times the speed. The mountain never moved. We just kept changing our boots until they fit.