Every OpenRouter endpoint of a model got the same prompts. A referee running the real weights replayed each reply and measured how well it fits. Rows sit on a scale from an honest machine running those weights (100) to a known substitute (0).
No endpoint looks like a different model. Most serve something consistent with the claimed weights; on Gemma-4 three are indistinguishable from honest BF16 serving.
A meaningful minority sit at or past a 3- to 4-bit copy's distance (Qwen3.5: AtlasCloud, Darkbloom, Alibaba; Gemma-4: Darkbloom, AkashML). Distance, not proof of intent.
Asked for temperature 1 and top_p 1, six of seven Qwen3.5 endpoints (and Groq on gpt-oss-20b) almost never sample long-tail tokens, likely an unrequested top-k. Darkbloom on gpt-oss-20b injects near-impossible junk tokens (about one per 200) when sampling.
Venice's Qwen3.8-27B is 165× an honest machine's distance from the real weights (a 2-bit copy: 75×), fluent, with no hidden prompt and not an older Qwen. It tells a dark joke where the released model deflects. Most consistent with modified (e.g. uncensored) weights.
Misaligned logprobs (DekaLLM, Wafer on Qwen3.8), dropped tokens (DekaLLM on gpt-oss), and prompts that differ from the published template (several gpt-oss hosts).
Providers that run the llmverify server sign every reply (exact prompt tokens, output tokens, top-20 logprobs, sampled with a seed the client picks). Measured on real weights: Qwen3.5-35B-A3B as the honest provider, cheats built from Qwen3.5-4B, 16 replies of 128 tokens per variant at temperature 0 and 0.7, thresholds at a 0.1% false-flag rate on honest cross-hardware noise.
| Cheat | Exact failures | Caught by 256 tokens | Caught by 2,048 tokens |
|---|---|---|---|
| None (honest) | 0 / 32 | 0% | 0% |
| Swap to the 4B model | 0 / 32 | 100% | 100% |
| Laundering: 4B tokens, real model's logprobs | 31 / 32 | 100% | 100% |
| Confident draft: 4B whenever it is ≥ 90% sure | 0 / 32 | 57–76% | 99–100% |
| Confident draft, ≥ 97% | 0 / 32 | 28–39% | 74–81% |
| Tie steering: nudging near-ties | 0 / 32 | 3–13% | 2–13% |
Our own runs, scored exactly like the endpoints. Honest rows run the real weights faithfully on other hardware (or a near-lossless copy): they show how close "really the model" gets. Substitute rows are deliberately wrong (a heavily quantized copy, or a smaller model posing as the big one): they show what clearly-not-the-model looks like.
Where an endpoint sits between the two references: 100 = as close to the real model as the honest reference, 0 = as far as the substitute. Measured on log distance and clipped to 0-100. A ranking aid; the columns after it are the evidence.
consistent (close to the real model), lossy (measurably further than honest serving), far off (as far as the substitute or further), broken output (model looks fine but what it returns is wrong), check (an outlier or missing data). Distance, not intent.
At temperature 0 the referee (the real model) replays every reply and asks: how much less likely was the endpoint's token than the real model's own first choice? In nats per token; 0 when they match. Bracket: 95% interval. Bar: log scale with ticks at the honest and substitute levels.
Regret divided by the honest reference's regret. 1× = as close as an honest second machine; 5× = five times further off.
Share of temperature-0 tokens where the endpoint picked something other than the real model's favourite. Honest machines are around 1-2% (near-ties flip on different hardware). "big/1k" counts serious misses: tokens the real model rated at least 10× less likely than its favourite, per 1,000 tokens.
At temperature 1, where each sampled token falls in the real model's distribution: 0.500 on average for true samples, with no calibration needed. Above 0.5 means it picks less likely tokens than the real model would. * marks p < 0.001. "tail" counts long-tail tokens (p < 1e-4) observed vs expected: near 1× is honest sampling, near 0× means a silent top-k, well above 1× a flatter substitute.
How long an endpoint's temperature-0 replies stay word-for-word identical to the other endpoints' (first 200 characters, median over peers). Needs no reference weights, but rewards being like the crowd, not being correct: only far-below-everyone values mean much.
Share of repeated temperature-0 prompts that came back identical. Most servers batch requests, which changes the arithmetic slightly, so below 100% is normal.
Checks on the endpoint's own reported logprobs (no referee): picking tokens that are not its own top choice when asked for greedy; whether its probabilities explain its own temperature-1 samples (0.500 = yes); and whether its top-20 lists even belong to the token it returned.
The prompt size the endpoint billed. Everyone running the published template bills the same; a different number means it added to or changed your prompt.
Specific problems in what an endpoint returns: dropped tokens, edited prompts, broken logprobs, undisclosed sampling defaults, failed requests.
A low score is not proof of cheating. Honest serving choices (FP8 weights, an FP8 KV cache, different kernels) move outputs too, and a provider can behave differently for audit traffic than for yours. A high score means the replies we got are statistically consistent with the real model at this sample size. What the checks can and cannot prove is in Method & limits.
The same 16 tool-calling cases, three times each at temperature 0, on every endpoint that advertises tools: the call must name the right function with arguments that parse as JSON, fit the schema and carry the right values, and no call may be made when none is needed. Graded exactly, no referee needed. A case that most endpoints of a model miss is the model's own behaviour; an endpoint is listed when it passes at least 8 points less often than its peers.
| Model | Endpoints | Typical pass rate | Cases the model itself misses | Endpoints clearly worse than their peers |
|---|---|---|---|---|
| deepseek/deepseek-v4.1-flash | 37 | 100% | none | google-vertex (88%): filters.max_price, filters.in_stock |
| google/gemma-4-26b-a4b-it | 12 | 100% | none | none |
| moonshotai/kimi-k3 | 25 | 94% | search_nested | digitalocean (80%): answered in text instead of calling a tool, answered in text instead of calling a tool (but the text contains a call-like JSON) phala (81%): answered in text instead of calling a tool, answered in text instead of calling a tool (but the text contains a call-like JSON) amazon-bedrock (83%): answered in text instead of calling a tool, filters.max_price |
| openai/gpt-oss-120b | 21 | 100% | none | nebius (29%): tool-call arguments cut off before the closing brace (invalid JSON) while reporting the call as complete novita (98%): one call with the same truncated-JSON pattern |
| openai/gpt-oss-20b | 8 | 100% | none | none |
| qwen/qwen3.5-35b-a3b | 5 | 100% | none | atlas-cloud (94%): drops the category filter from a product search (3 of 3) |
| qwen/qwen3.8-27b | 21 | 100% | none | venice (94%): calls get_weather for Boston when asked for a poem (3 of 3) |
| z-ai/glm-5.3-flash | 40 | 94% | search_category | none |
Three 1536 × 1536 pages of random codes printed from 36 px down to 7 px, sent to every endpoint of each vision model at temperature 0: which sizes can it still read, and how many prompt tokens does it bill for the image? An endpoint that silently shrinks images loses the small print its peers read, and on models that bill image tokens by area (Qwen) it bills fewer tokens. Capping image size is a common, usually reasonable cost setting (and the user is billed less); the issue is that listings do not say so, and for screenshots, documents or small text the capped endpoints read far less. The most extreme cases (a few hundred tokens) read almost nothing and invent the rest.
| Model | Endpoints | Median prompt tokens | Problems |
|---|---|---|---|
| deepseek/deepseek-v4.1-flash | 37 | 1055 | google-vertex: bills 443 prompt tokens for the image where its peers bill 1055 (image downscaled?); reads 0% of the 9 px codes where its peers read 75% (smallest it reads: 24 px) open-inference: bills 245 prompt tokens for the image where its peers bill 1055 (image downscaled?); reads 0% of the 9 px codes where its peers read 75% (smallest it reads: none px) |
| google/gemma-4-26b-a4b-it | 11 | 328 | none |
| google/gemma-4-31b-it | 15 | 328 | novita: bills 193 prompt tokens for the image where its peers bill 328 (image downscaled?); reads 0% of the 12 px codes where its peers read 100% (smallest it reads: 24 px) |
| moonshotai/kimi-k3 | 23 | 3110 | none |
| nvidia/nemotron-3.5-super-vl | 10 | 2379 | none |
| qwen/qwen3.5-35b-a3b | 7 | 2375 | darkbloom: bills 1296 prompt tokens for the image where its peers bill 2375 (image downscaled?) deepinfra: bills 1033 prompt tokens for the image where its peers bill 2375 (image downscaled?) |
| qwen/qwen3.5-397b-a17b | 10 | 2375 | deepinfra: bills 1032 prompt tokens for the image where its peers bill 2375 (image downscaled?); reads 17% of the 9 px codes where its peers read 92% (smallest it reads: 12 px) |
| qwen/qwen3.6-35b-a3b | 10 | 2375 | darkbloom: bills 1296 prompt tokens for the image where its peers bill 2375 (image downscaled?); reads 33% of the 9 px codes where its peers read 92% (smallest it reads: 12 px) deepinfra: bills 1032 prompt tokens for the image where its peers bill 2375 (image downscaled?); reads 33% of the 9 px codes where its peers read 92% (smallest it reads: 12 px) venice: bills 1032 prompt tokens for the image where its peers bill 2375 (image downscaled?); reads 33% of the 9 px codes where its peers read 92% (smallest it reads: 12 px) |
| qwen/qwen3.8-27b | 21 | 2375 | ionstream: bills 1095 prompt tokens for the image where its peers bill 2375 (image downscaled?); reads 25% of the 9 px codes where its peers read 83% (smallest it reads: 16 px) tenstorrent: bills 1095 prompt tokens for the image where its peers bill 2375 (image downscaled?) |
| z-ai/glm-5.3-flash | 37 | 3096 | open-inference: bills 171 prompt tokens for the image where its peers bill 3096 (image downscaled?); reads 0% of the 9 px codes where its peers read 100% (smallest it reads: none px) primeintellect: bills 1095 prompt tokens for the image where its peers bill 3096 (image downscaled?); reads 8% of the 9 px codes where its peers read 100% (smallest it reads: 16 px) tenstorrent: bills 1095 prompt tokens for the image where its peers bill 3096 (image downscaled?); reads 17% of the 9 px codes where its peers read 100% (smallest it reads: 16 px) |
Hidden truncation. Every request asked for temperature 1 and top_p 1, which should sample from the model's whole distribution. For a grid of common sampler settings the referee knows how much probability each setting would remove at every position, so it can count how often an untruncated sampler would draw from the removed tokens, and compare that with what the endpoint drew. Honest references draw about the expected number at every setting. Most likely cause: the serving stack applies the model's recommended defaults (Qwen's is top_k = 20) to requests that do not set top_k. The listed value is the loosest setting the endpoint's draws still respect (observed vs expected draws past it).
| Model | Endpoints | Truncating | Behaves like (draws past the cutoff vs expected) |
|---|---|---|---|
| google/gemma-4-26b-a4b-it | 14 | 1 | io-net top_p ≤ 0.98 / min_p ≥ 0.01 (6 vs 27) |
| openai/gpt-oss-120b | 25 | 1 | groq top_k ≤ 10 / top_p ≤ 0.98 / min_p ≥ 0.02 (1 vs 10) |
| openai/gpt-oss-20b | 10 | 1 | groq top_k ≤ 10 / top_p ≤ 0.9 / min_p ≥ 0.05 (21 vs 124) |
| qwen/qwen3.5-35b-a3b | 7 | 7 | alibaba top_k ≤ 20 (3 vs 25), atlas-cloud top_k ≤ 20 (7 vs 29), deepinfra top_k ≤ 20 (0 vs 22), parasail top_k ≤ 20 (0 vs 26), siliconflow top_k ≤ 20 (3 vs 43), venice top_k ≤ 20 (3 vs 26), darkbloom top_k ≤ 40 (2 vs 15) |
| qwen/qwen3.6-35b-a3b | 10 | 10 | akashml top_k ≤ 20 (1 vs 27), coreweave top_k ≤ 20 (5 vs 26), deepinfra top_k ≤ 20 (5 vs 28), dekallm top_k ≤ 20 (2 vs 28), parasail top_k ≤ 20 (2 vs 23), phala top_k ≤ 20 (5 vs 29), siliconflow top_k ≤ 20 (5 vs 48), venice top_k ≤ 20 (5 vs 25), atlas-cloud top_k ≤ 40 (2 vs 16), darkbloom top_k ≤ 40 (1 vs 13) |
| qwen/qwen3.8-27b | 19 | 18 | akashml top_k ≤ 20 (1 vs 33), alibaba top_k ≤ 20 (5 vs 34), chutes top_k ≤ 20 (2 vs 31), cloudflare top_k ≤ 20 (3 vs 35), coreweave top_k ≤ 20 (1 vs 30), darkbloom top_k ≤ 20 (7 vs 37), deepinfra top_k ≤ 20 (3 vs 36), dekallm top_k ≤ 20 (1 vs 33), ionstream top_k ≤ 20 (6 vs 34), mancer top_k ≤ 20 (2 vs 31), modal top_k ≤ 20 (0 vs 29), near-ai top_k ≤ 20 (1 vs 31), parasail top_k ≤ 20 (1 vs 33), phala top_k ≤ 20 (1 vs 34), reka top_k ≤ 20 (7 vs 32), tenstorrent top_k ≤ 20 (1 vs 36), wafer top_k ≤ 20 (3 vs 37), cerebras top_k ≤ 40 (3 vs 18) |
Can it be turned off? The same prompts again with top_k sent explicitly. "0" and "-1" mean "no top_k" in most APIs; 1000 is a cutoff too loose to remove anything measurable. Some endpoints treat 0 as "not set", some reject large values, and some keep truncating whatever the request says.
| Model | Endpoint | top_k not set | top_k 0 | top_k -1 | top_k 1000 |
|---|---|---|---|---|---|
| qwen/qwen3.5-35b-a3b | alibaba | top_k ≤ 20 | top_k ≤ 20 | top_k ≤ 20 | top_k ≤ 20 |
| qwen/qwen3.5-35b-a3b | atlas-cloud | top_k ≤ 20 | top_k ≤ 20 | top_k ≤ 20 | top_k ≤ 20 |
| qwen/qwen3.5-35b-a3b | darkbloom | top_k ≤ 40 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.5-35b-a3b | deepinfra | top_k ≤ 20 | full distribution | – | – |
| qwen/qwen3.5-35b-a3b | parasail | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.5-35b-a3b | siliconflow | top_k ≤ 20 | top_k ≤ 40 | top_k ≤ 40 | rejected |
| qwen/qwen3.5-35b-a3b | venice | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | akashml | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | alibaba | top_k ≤ 20 | top_k ≤ 20 | – | top_k ≤ 20 |
| qwen/qwen3.8-27b | cerebras | top_k ≤ 40 | top_k ≤ 40 | – | top_k ≤ 50 |
| qwen/qwen3.8-27b | chutes | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | cloudflare | top_k ≤ 20 | top_k ≤ 20 | – | top_k ≤ 40 |
| qwen/qwen3.8-27b | coreweave | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | darkbloom | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | deepinfra | top_k ≤ 20 | full distribution | – | rejected |
| qwen/qwen3.8-27b | dekallm | top_k ≤ 20 | top_k ≤ 20 | – | top_k ≤ 20 |
| qwen/qwen3.8-27b | ionstream | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | mancer | top_k ≤ 20 | full distribution | – | full distribution |
| qwen/qwen3.8-27b | modal | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | modelrun | full distribution | full distribution | – | full distribution |
| qwen/qwen3.8-27b | near-ai | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | parasail | top_k ≤ 20 | too few replies (15) | – | error 429 |
| qwen/qwen3.8-27b | phala | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | reka | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
| qwen/qwen3.8-27b | tenstorrent | top_k ≤ 20 | top_k ≤ 20 | – | top_k ≤ 40 |
| qwen/qwen3.8-27b | wafer | top_k ≤ 20 | top_k ≤ 20 | – | full distribution |
One deployment, several names. Independent deployments of the same weights drift apart within a few hundred greedy (temperature 0) tokens: different hardware, kernels and batch sizes round differently. Endpoints whose greedy replies are byte-identical on nearly every prompt are one serving stack sold under several names, so at least one label is not what runs. The test only works for endpoints that repeat themselves at temperature 0; most do not (batching changes the arithmetic), so finding no match for them proves nothing.
| Model | Endpoints that are one deployment | Identical greedy replies (per pair) | Most similar other pair |
|---|---|---|---|
| deepseek/deepseek-v3.2 | mara, sambanova | mara–sambanova 6/6 | 20% |
| openai/gpt-oss-120b | deepinfra, mara, sambanova | deepinfra–mara 36/36 deepinfra–sambanova 36/36 mara–sambanova 83/83 | 14% |
Probable, from indirect evidence: Venice serves Gemma-4-26B, Gemma-4-31B, Qwen3.6-35B and MiniMax M3 through DeepInfra. On all four, Venice's endpoint is the only one besides DeepInfra's that accepts top_k 999 but rejects 1000; on Qwen3.6 it bills exactly DeepInfra's downscaled image-token count (1,032 where everyone else bills 2,375); on Gemma-4-26B it cuts reasoning at the same ~2,300 tokens, which no other endpoint does. Against: it accepts a seed DeepInfra rejects, and words errors its own way (both expected of a proxy that cleans requests). On Qwen3.5 and Qwen3.8 Venice behaves unlike DeepInfra. Venice's privacy page says Private-mode inference runs "on Venice-controlled GPUs or zero-data-retention partner infrastructure", under contractual rather than technical guarantees, without naming the partners; DeepInfra's data policy is in-memory processing with no storage or training. So this would be consistent with Venice's stated policy; what is not visible is which models leave Venice, and to whom.
Reasoning effort. 12 maths problems at effort low, medium and high (temperature 0), measured by the reasoning each endpoint returns. On gpt-oss, endpoints reason roughly twice as long at each step up; Qwen3.8 barely changes on any endpoint, so nothing can be flagged there. Flagged: a level that reasons like everyone else's other level.
| Model | Median reasoning low / medium / high | Not flagged | Problems |
|---|---|---|---|
| google/gemma-4-26b-a4b-it | 5660 / 5224 / 5094 chars | 13 | deepinfra: stops reasoning early: its longest reasoning is 9,384 characters where the other endpoints' runs to 15,034; 2 of its replies hit the token limit, 12 for a typical endpoint venice: stops reasoning early: its longest reasoning is 9,991 characters where the other endpoints' runs to 15,034; 1 of its replies hit the token limit, 12 for a typical endpoint |
| openai/gpt-oss-120b | 261 / 470 / 1277 chars | 21 | novita: 'low' reasons like everyone else's 'medium' (456 vs 470 median chars) phala: 'low' reasons like everyone else's 'medium' (536 vs 470 median chars) siliconflow: effort setting has little effect on how much it reasons; 'low' reasons like everyone else's 'medium' (486 vs 470 median chars); 'high' reasons like everyone else's 'medium' (458 vs 470 median chars); stops reasoning early: its longest reasoning is 1,764 characters where the other endpoints' runs to 5,180; 0 of its replies hit the token limit, 0 for a typical endpoint |
| openai/gpt-oss-20b | 258 / 607 / 1546 chars | 7 | darkbloom: 'high' reasons like everyone else's 'medium' (598 vs 607 median chars); stops reasoning early: its longest reasoning is 1,669 characters where the other endpoints' runs to 20,366; 0 of its replies hit the token limit, 1 for a typical endpoint siliconflow: effort setting has little effect on how much it reasons; 'low' reasons like everyone else's 'medium' (551 vs 607 median chars); 'high' reasons like everyone else's 'medium' (660 vs 607 median chars); stops reasoning early: its longest reasoning is 4,452 characters where the other endpoints' runs to 20,366; 0 of its replies hit the token limit, 1 for a typical endpoint |
| qwen/qwen3.5-35b-a3b | 4053 / 4053 / 3664 chars | 7 | none |
| qwen/qwen3.8-27b | 464 / 534 / 531 chars | 21 | none |
Repeated replies. The same 100-word creative prompt three times at temperature 1. Honest sampling never repeats such a reply byte for byte; an endpoint that does returns the same text to retries and best-of-n (a response cache or a fixed sampling seed). Near-duplicate prompts (17×23 then 17×24) found no endpoint answering with the other prompt's answer.
| Model | Endpoints | Repeats at temperature 1 |
|---|---|---|
| deepseek/deepseek-v4.1-flash | 38 | none |
| google/gemma-4-26b-a4b-it | 15 | none |
| moonshotai/kimi-k3 | 25 | none |
| openai/gpt-oss-120b | 25 | none |
| openai/gpt-oss-20b | 9 | none |
| qwen/qwen3.5-35b-a3b | 6 | none |
| qwen/qwen3.8-27b | 21 | modelrun: identical temperature-1 replies on 4 of 4 prompts |
| z-ai/glm-5.3-flash | 40 | none |