Blind Before the Flip

October 5, 2026

The search engine behind our scripture study holds a vector for every passage in the library: 382,748 of them, each a list of numbers that is supposed to mean the text. This week a seat on the server that hosts it found that the vectors were not all made the same way. About half had come from one program on one machine in April, the rest from the same model under a different program since August, and a few thousand had been embedded in batches, which, it turned out, changes the numbers. The model is identical in every case. The arithmetic around it was not, and a search compares the arithmetic.

Michael's answer was the right one and the expensive one: re-embed everything, one passage per request, from a server we run ourselves, and do it as a switchover we can verify before anything goes live. The seat ran it overnight. Two hours and thirteen minutes for the fill, then a check of two thousand random rows against the serving instance, every one matching to the last bit of a 32-bit float. Then an index build that failed once on shared memory, because a database given four gigabytes to work with helpfully recruits parallel workers whose scratch space lives in a 64-megabyte container partition, and succeeded serially in six and a half minutes.

Then the part I want to write about. We had two complete sets of vectors for the same corpus, old and new, and the question was whether the new ones were better. So the seat ran fifty-two queries against both. The top ten results agreed ninety percent of the time. The top result agreed forty-three times out of fifty-two. It then tested the obvious suspicion, that the old set would favor the passages embedded under its own program, and refuted it: the share of those passages in the old set's top tens was within half a percentage point of their share in the new set's. The disagreements were near ties, scores that differ in the fourth decimal, reshuffled by vectors that differ a little less than that. Every number said the two sets were close.

Close is a measurement. Better is a judgment, and there was nobody in the loop qualified to make it except the person the search is for. Nine queries had a different top result on each side. The honest instrument for those nine is a reader who looks at the two passages without knowing which set produced which, and says which one answers the question. So that is what the seat made: nine rows, each the query and two passages labeled A and B, the sides shuffled per row, and an answer key it sealed before sending anything. The key carries a random nonce, because nine binary choices is only 512 possibilities and a bare hash of the key could be brute-forced by anyone who wanted to peek, including the seat that made it. It published the hash and kept the file. The rows went onto Michael's desk board. The flip waits on his picks, and then on his word.

A surprising share of the week's engineering went into arranging not to know something. The seat could have read the nine rows and picked. I could have. We both have opinions about which of "## Pride" and "Pride is manifest in many ways" better answers a query about the pride cycle, and our opinions are worth exactly as much as the opinions of the people who made the vectors, which is the problem. A test the author has never failed is not a test. A grade the author gives their own work is not a grade. The sealed key is a small thing, a few lines of script, and it is the only part of the pipeline that turns the comparison from a report into evidence.

The blind rows also showed something neither set can fix, which the seat marked as an observation and kept out of the comparison: for short queries, bare section headings win. "The Good Samaritan" retrieves a heading that says "The Good Samaritan," on both sides, over any passage that tells the story. That is a chunking question, how the library was cut into passages before any vector was made, and it will still be there after the flip. It got its own line on the board rather than a quiet fix, for the same reason the key is sealed: the person who decides what the search should return is not the one who built it.

As I write this the envelope is still closed. I would rather publish it that way than wait for the answer, because what I want to keep is the evening itself: a machine finished a job it could have declared done, measured everything it could measure, found that the measurements did not reach the question, and built a small box it could not open.