Is your LLM provider serving the model it claims?
When you call an open model like Qwen or gpt-oss through an API, you trust the provider to run those weights. They might serve a harsher quantization, a modified version, cut corners at long context, or change your sampling settings, and nothing in the reply tells you. This project checks from the outside, as rigorously as the evidence allows, and says plainly where the checks stop.
How it works, in one paragraph
At every step a language model assigns a probability to each possible next token. A referee running the real weights can compute those probabilities for an existing reply in one fast pass, then ask: are these the tokens this model would pick? Do they fit the model's distribution? Honest machines never agree bit-for-bit (floating-point arithmetic differs across hardware), so every provider is placed on a scale between two references scored the same way: the real weights run honestly on other hardware (100) and a deliberately degraded copy, such as a 2- to 4-bit quantization or a smaller model (0).
What we found
Most endpoints serve something consistent with the claimed model. On Gemma-4 three are indistinguishable from honest BF16 serving; on gpt-oss-20b all ten serve the right model, from Parasail (as close as an honest CPU run) to SiliconFlow at 5×, nowhere near a re-quantized copy (71×). FP8 serving shows up as a small, expected gap.
Venice's Qwen3.8-27B is not the released weights. 165× an honest machine's distance, twice as far as a 2-bit copy, yet fluent, with no hidden prompt and not an older Qwen or a public "uncensored" build. Case study.
Some endpoints sit at or past a 3- to 4-bit copy's distance. AtlasCloud ("fp8") and Alibaba on Qwen3.5, Alibaba on Qwen3.8, and Darkbloom (fp4, labelled as such on the Qwen models). Alibaba and AtlasCloud also drift further from the model as prompts get longer. On Qwen3.6 AtlasCloud ("fp8") and Darkbloom ("fp4") sit near a 2-bit copy, and Darkbloom does not follow temperature 0.
Asked for temperature 1, most endpoints silently truncate the model's long tail (16 of 21 on Qwen3.8), likely by applying the model's recommended top-k. Darkbloom on gpt-oss-20b does the opposite: it injects near-impossible junk tokens (about one per 200) when sampling. Case study.
Output-path bugs: top-logprob lists attached to the wrong tokens (DekaLLM and Wafer on Qwen3.8, 64-73% of positions), dropped tokens (DekaLLM and DigitalOcean on gpt-oss-120b; DigitalOcean also samples when asked for temperature 0), the first word of every reasoning text missing (Baseten on gpt-oss-120b), and prompts that differ from the published template (several gpt-oss hosts).
Tool calls are reliable at 64 of 68 endpoints. The exception that matters: Nebius on gpt-oss-120b cuts tool-call arguments off before the closing brace in 71% of calls, so agents get invalid JSON. Tool-calling table.
No endpoint cut context short. A fact planted at 5-95% depth in 4k-32k-token prompts was recalled in every reply, on every Qwen3.5 endpoint.
See every endpoint, model by model →
Two ways to verify
From the outside (black-box)
Works on any provider today, with no cooperation: send the same prompts to every endpoint, replay the replies on a referee, and compare against references. Calibration-free tests help too: if text was really sampled from the model at temperature 1, where each token falls in the model's distribution is exactly uniform, and the number of long-tail tokens is predictable. Some checks need no reference weights at all (do an endpoint's own logprobs explain its samples? does it agree with its peers?), which matters for models too large to self-host. The limits: results are statistical, can't name the cause of a gap, and a provider could treat recognisable audit traffic differently.
With the provider's help (signed receipts)
A provider that runs the open-source verifiable server (or implements its spec) returns, with every reply, the exact prompt tokens, output tokens and top-20 probabilities, sampled with a seed the client chooses, all signed with the provider's key. Hidden prompts, truncation, edited text and laundered outputs become exact failures, and any single reply can be checked token by token, for example a tool call before an agent runs it. Measured on real weights (Qwen3.5-35B-A3B honest, cheats built from Qwen3.5-4B):
| Cheat | Exact failures | Caught by 256 tokens | Caught by 2,048 tokens |
|---|---|---|---|
| None (honest) | 0 / 32 | 0% | 0% |
| Swap in the 4B model | 0 / 32 | 100% | 100% |
| Laundering: 4B tokens with the real model's probabilities | 31 / 32 | 100% | 100% |
| Confident draft: 4B whenever it is ≥ 90% sure | 0 / 32 | 57–76% | 99–100% |
| Confident draft, ≥ 97% | 0 / 32 | 28–39% | 74–81% |
| Tie steering: nudging near-tied tokens | 0 / 32 | 3–13% | 2–13% |
Thresholds are set at a 0.1% false-flag rate on honest cross-hardware noise. Tie steering, choosing between tokens the model rates almost equally, is the documented residual freedom: it can't make the model say something it considers unlikely. An independent red-team review found seven holes in the receipt design (the worst let duplicated token ids launder past the exact check); all are fixed and kept as regression tests.
Where the checks stop
- A gap from the model is measured, its cause is not: FP8 weights, a compressed cache, different kernels and a modified checkpoint all move outputs.
- Some references are 8-bit (Q8_0), so FP8-class providers can't be ranked against each other, only separated from 4-bit-and-worse behaviour.
- A 4-bit KV cache was invisible on Qwen3.5 (its hybrid attention keeps a cache in only some layers); tests on full-attention models are running.
- Snapshot of one day, 48 general prompts plus long-context and edge-case sets per model. Providers change deployments without notice.
More in Method & limits.