Sid

Your answer depends on who else was asking at the same time.

Inference servers batch requests together for throughput. The trouble is that a batched matmul doesn’t reduce in the same order as an unbatched one, so the floating-point result differs in the last few bits. One differing bit early enough changes the sampled token, and from there the two sequences diverge completely.

That sounds like a rounding footnote until it breaks something. It breaks reproducible evals, it makes a regression impossible to bisect, and it means the model you audited isn’t provably the model that answered. So I measured it: one fixed prompt, six batch compositions, three trials each.

Batch-invariant outputs without batch-invariant kernels.

The obvious fix is to rewrite every kernel so its reduction order never depends on batch shape. That costs most of your throughput and all of your ability to use fast third-party kernels. I went the other way: keep the fast kernels and make the decode loop check its own work.

Sid replays tokens through fixed-shape windowed verification. Decoding runs in windows of a constant shape, so the numerics inside a window don’t depend on what else is resident. Each window gets re-checked against its canonical result, and on a mismatch the window rolls back and replays. A sensitivity router decides which requests need this. Flagged traffic takes the deterministic path and everything else keeps full continuous-batching throughput, so determinism is something you pay for per request instead of everywhere.

Underneath it is a vLLM-style engine I wrote: a continuous-batching scheduler, a paged KV cache with prefix caching and a Triton scatter kernel, tensor parallelism over NCCL, and CUDA-graph capture with FlashAttention-3.

Determinism isn’t free.

Verification and replay cost throughput, which is why the router exists. A production workload doesn’t need every request to be reproducible. It needs the audited ones to be. The design is to pay that cost on those requests and skip it on the rest.