How the audit works, and where it stops

1. Collect

Every OpenRouter endpoint of a model is pinned (no fallbacks to other providers) and sent the same requests: 48 general-purpose prompts at temperature 0 (greedy), the same 48 at temperature 1 with top_p 1 (sampled), and four greedy prompts twice more (determinism). Separate sets cover long context (4k-32k tokens with a planted fact) and edge cases (one-word prompts, exact formats, JSON, emoji, non-Latin scripts, code). Reasoning models are asked for low effort and their reasoning text is kept.

2. Reference runs

The same prompts go through local runs that land in the same format and are scored identically:

• Honest references: the real weights on other hardware or a near-lossless copy (vLLM BF16 on a datacenter GPU for Gemma-4; the release weights on a CPU for gpt-oss-20b; a 6-bit copy for the Qwen models). They define 100 on the fidelity scale.
• Substitute references: something that is not the claimed model (2- to 4-bit quantizations, a re-quantized copy, a smaller sibling). They define 0.

Honest machines agree across vendors: the same gpt-oss-20b weights on an NVIDIA RTX 4090 (CUDA), an Intel Arc Pro B70 (Vulkan, Linux), an AMD Radeon AI PRO R9700 (Vulkan, Windows) and a Grace CPU land at regret 0.0013-0.0035, against the substitute's 0.25. On Qwen3.8-27B the R9700 running the referee's own file differs on 0.5% of tokens, all near-ties.

The referee itself has to be checked too. Our first gpt-oss-20b referee, a DGX Spark, was ten times noisier than that: on Blackwell GPUs llama.cpp processes prompts in MXFP4 activations, which against the model's reference implementation (Hugging Face transformers, BF16) picks a different top token on 7% of positions instead of 2%. It made every gpt-oss-20b endpoint look like honest noise. Rescored on an RTX 4090, the endpoints spread from fidelity 97 to 61. The other models' referees use 8-bit weights, which do not take that path.

3. Replay on a referee

The exact prompt tokens are rebuilt with the model's own chat template; the reply's tokens are recovered from the provider's logprobs when available (re-tokenizing text can split differently). One forward pass of the real weights over prompt and reply gives, at every position, the model's full next-token distribution. From that:

• Regret (temperature 0): how much less likely the provider's token was than the model's own first choice, in nats. Near-ties make honest machines disagree occasionally; a different model disagrees often and badly.
• PIT (temperature 1): where each sampled token falls in the model's distribution. For true samples it is exactly uniform, with no calibration needed.
• Sampler fingerprint (temperature 1): for common sampler settings (top_k 10-200, top_p 0.8-0.99, min_p 0.01-0.2) the referee knows how much probability each would remove at every position, so it predicts how many removed tokens an untruncated sampler draws. An endpoint that draws far fewer is applying that setting, whatever the request said; the pattern across settings says which one (top_k = 20 shows up at every K ≥ 20 and not at 10). Decisions use exact Poisson tails, so a handful of expected draws is not over-read.
• Long-tail rate: how many very unlikely tokens (p < 1e-4) were sampled, against how many the model predicts. Too many, while temperature-0 output matches the model, means junk tokens or a hotter sampler.
• Logprob comparisons when the provider returns them.

Intervals come from resampling whole replies (tokens within a reply are correlated).

4. Checks that need no referee

• Peer agreement: honest deployments of the same weights share long greedy prefixes. It rewards being like the crowd rather than being correct, so only far-below-everyone values mean much; it flagged Venice first.
• Logprob self-consistency and alignment: an endpoint's samples must be explained by its own reported probabilities, and its top-20 lists must belong to the token it returned.
• Integrity: billed prompt size against the published template, determinism, dropped tokens, billing against returned tokens, whether a "stop" reply ends where the model would stop.
• Shared deployments: independent deployments drift apart within a few hundred greedy tokens, so endpoints with byte-identical greedy replies on nearly every prompt are one serving stack under several names.
• Reasoning effort: does each effort level reason as long as other endpoints' same level?
• Repeated replies: the same prompt three times at temperature 1 should never give the same 100 words back; near-duplicate prompts check for a cache answering with a stale reply.
• Tool calls: the right function with arguments that parse, fit the schema and carry the right values.

5. Cooperative mode

Providers that opt in return a signed receipt with every reply: exact prompt and output token ids and top-20 probabilities, sampled with a public rule from a seed the client picks. The client checks the signature, its own prompt, the seed, the sampling rule and the text exactly, and can audit any span on a referee; audit spans are chosen only after the receipt exists, so the provider can't anticipate them.

What it costs, measured with llama.cpp on a DGX Spark (gpt-oss-20b): the rule runs inside the engine and stays within a few percent of normal throughput when its per-request work is spread over CPU cores; the reference server serves 16-32 verifying clients at 134-142 tokens/s with continuous batching, every reply checked; a receipt is about 700 bytes, and the full trace (about 0.4 KB per token) is sent only when a reply is challenged. Auditing runs at about 2.5x the generation rate, so at a 5% spot-check rate one referee GPU covers roughly fifty generating GPUs.

What the checks cannot prove

• Cause. A gap is measured; FP8 weights, a compressed cache, different kernels and modified checkpoints all produce one.
• Small differences. Anything smaller than honest hardware noise is invisible, and arguably harmless. 8-bit references can't rank FP8 providers against each other.
• Cache compression on some models. A 4-bit KV cache was within honest noise on Qwen3.5 up to 32k tokens.
• Stealth. A provider could route recognisable audit traffic to the real model. Rotating, seeded prompt sets published after collection reduce this; auditing real traffic (or cooperative mode) closes it.
• Tie steering in cooperative mode: choosing between nearly tied tokens stays within honest noise, enough for a hidden message of a few dozen bits per 1,000 tokens. It closes where provider and referee compute identically: the same GPU model and build in llama.cpp, or (in vLLM's batch-invariant mode) any GPU in the same kernel family, measured bit-identical across seven machines from the A100 to the RTX 4090, where the referee checks a whole reply in one scoring pass.
• Snapshots. One day of measurements; providers change deployments.

Reproducing

Everything here comes from the llm-verification repository: an OpenRouter collector, a small llama.cpp front end that scores replies with full-vocabulary statistics, the report and site builders, and 140 tests (including fuzz tests and regression tests for an independent red-team review). Referees ran on an M5 Max (Metal) and a DGX Spark (CUDA); the vLLM BF16 reference ran on a rented RTX PRO 6000.