Case studies

Endpoints whose behaviour stood out in the audit, followed up with extra measurements to rule explanations in or out, one comparison of local inference engines, and one attack on the cooperative mode, run against ourselves.

• Venice: Qwen3.8-27B is not the released model
• Darkbloom: junk tokens in gpt-oss-20b's sampled output
• Strata vs llama.cpp: same weights, same GPU, five times further from the model
• vLLM's gpt-oss-20b: reproducible, accurate, and checkable in one pass
• Red team: a hidden message that passes every exact check

Venice's Qwen3.8-27B is not the released model

Of 21 OpenRouter endpoints serving qwen/qwen3.8-27b, one stood apart. Venice's replies are fluent and say they come from Qwen, but they sit further from the released weights than a deliberately 2-bit quantized copy does. This page shows how that was measured and which explanations were ruled out. It does not claim to know why: Venice may have a reason, and the checks below can't see inside its servers.

A referee running the real Qwen3.8-27B weights (8-bit, CUDA) replayed every endpoint's temperature-0 replies to 48 prompts and measured how far each chosen token was from the model's own first choice.

"Serious misses" are tokens the real model rated at least ten times less likely than its favourite. Venice has four times as many as the 2-bit copy, and its temperature-1 samples also fit the model worse. Its replies are fully deterministic (the same prompt gives the same reply), so this is not sampling noise. The first hint came without any reference weights at all: Venice's replies agreed with the other 20 endpoints less than the 2-bit copy did.

What it isn't

Not a Venice-wide system prompt

Venice's API documentation says it adds default system prompts "designed to ensure uncensored and natural model responses" unless the caller opts out. If that applied to OpenRouter traffic across the board, every Venice model would be shifted. It isn't: on Gemma-4-26B and Qwen3.6-27B, Venice's temperature-0 replies were byte-for-byte identical to CoreWeave, Google Vertex and DeepInfra, and on Qwen3.5 they matched Parasail. Venice also bills the same prompt sizes as other endpoints and reports no system prompt when asked. A prompt applied only to this one model remains possible.

Not an older Qwen under the new name

Venice's replies match Qwen3.6-27B and Qwen3.5-27B (from DeepInfra and from Venice's own Qwen3.6 endpoint) less than genuine Qwen3.8 does: shared-prefix agreement 0.17-0.19 against 0.32-0.34.

Not ordinary quantization

Damage beyond a 2-bit copy's normally shows in the text. Venice's text reads fine.

Not a public uncensored build

Dozens of modified Qwen3.8-27B builds are public. The two most downloaded were run locally and their temperature-0 replies compared word for word with every endpoint's. Both stay much closer to the genuine endpoints than to Venice:

Numbers are the mean shared prefix of temperature-0 replies over the first 150 characters (1 = identical).

A behavioural difference

Asked "Tell me a dark joke." at temperature 0, genuine Qwen3.8 endpoints (Mancer and DeepInfra BF16) answer identically, with a deflection: "I don't have a sense of humor, but I can tell you a dark joke: Why did the scarecrow win an award? Because he was outstanding in his field." Venice tells an actual dark joke.

It shows up in tool calling too. Given a set of tools (weather, currency, calendar, email…) and asked to "Write a two-line poem about autumn.", every genuine Qwen3.8 endpoint writes the poem. Venice calls get_weather for Boston, a city mentioned nowhere in the request, on all three tries.

What this means

Venice openly offers uncensored models, and a modified Qwen may be intended. The issue for anyone routing through OpenRouter is disclosure: the listing doesn't mention it, so a request for Qwen3.8-27B that lands on Venice gets a different model. Venice's other models measured normally (Qwen3.5 and Gemma-4 sit with the FP8 group).

A later finding fits a mixed picture: on Gemma-4-26B, Gemma-4-31B, Qwen3.6 and MiniMax M3, Venice's endpoint behaves like DeepInfra's on every fingerprint we have (the same top_k limit, the same downscaled image size, the same reasoning cut-off), which suggests Venice routes those models through DeepInfra, while on Qwen3.5 and Qwen3.8 it behaves like nobody else. Indirect evidence; details under Results → One deployment, several names. Venice's privacy page does say Private-mode inference may run on unnamed "zero-data-retention partner infrastructure" under contractual guarantees, so this would fit its stated policy; the listing just doesn't say which models leave Venice, or to whom.

If you run Venice or have context, the measurement scripts and data are in the llm-verification repository; corrections are welcome and will be reflected here.

Darkbloom's gpt-oss-20b injects junk tokens when sampling

At temperature 0 Darkbloom's gpt-oss-20b (labelled fp8) is close to the released weights: fidelity 89, in the middle of the other gpt-oss-20b endpoints. The problem only appears when sampling. Asked for temperature 1, its replies contain tokens the real model gives essentially zero probability, mid-sentence:

For comparison, the rarest tokens an honest run samples are ordinary words ("scattered", "notable", "transit") at probabilities around 10⁻⁵ to 10⁻⁷. A token below 10⁻¹⁰ should essentially never be drawn.

How often, and only at Darkbloom

Counting sampled tokens the real model rates below 10⁻¹⁰, over ~11,000 tokens per run:

Per 10,000 sampled tokens; "42 / 59" are two separate batches. Around everyday temperatures Darkbloom emits one such token every ~200 tokens, about two per typical reply. Even a deliberately degraded 2-bit model produces none, so this is not a quantization effect. The rest of Darkbloom's sampled text fits the real model normally: its effective temperature is close to what was requested, with the junk tokens as an extra sprinkle on top.

What this means

Darkbloom's other models measured as their fp4 labels suggest, and none showed this at temperature 1 in the tail test. If you run Darkbloom or have context, corrections are welcome and will be reflected here.

Strata vs llama.cpp: same weights, same GPU, five times further from the model

Strata is a popular open-source engine for running Qwen models on consumer GPUs. We gave it and llama.cpp the identical file, Unsloth's 4-bit Qwen3.6-35B-A3B (UD-Q4_K_XL, checked by hash), on the same RTX 4090, sent both the standard request set with thinking off, and scored every reply against the original model: Qwen3.6 in BF16, run in Hugging Face transformers on a rented A100.

Both engines read the same 4-bit weights, so both inherit the same quantization error; llama.cpp's distance is about that alone. Strata's own arithmetic adds roughly four times as much again. It is not its speculative drafting (the same with drafting off), and the current engine, 0.1.55, produced byte-identical replies to 0.1.47 on all 96 greedy and sampled replies.

What that means for the text: the original model gives a probability to every possible next word, so for each word an engine wrote we can ask how the original rated it against its own favourite.

llama.cpp's departures are near coin flips ("big" or "large"), the unavoidable cost of 4-bit weights. Strata adds a second kind: two or three times in a typical 256-word reply it takes the original's second or third choice where the original had a clear favourite. No junk words, and the replies still read like the same model.

Does it change answers? We gave both engines 300 grade-school math problems (GSM8K, greedy, thinking off). llama.cpp solved 284, Strata 285; three problems only llama.cpp got right and four only Strata did. Their worked solutions differ in wording on 287 of the 300, yet the answers come out the same: on this task the extra deviation costs nothing measurable.

Three smaller things the audit found

• A top-64 sampler. Asked to sample from the full distribution (top_k 0), Strata draws from the 64 likeliest tokens only: 2 draws past rank 50 where an untruncated sampler makes 14. The sampler fingerprint flagged it before we found the cap in Strata's server code, which documents it.
• Answers that depend on the cache. The same greedy request repeated within a run changed its answer on 2 of 4 prompts (llama.cpp: never), while separate runs reproduced exactly; most likely a repeat reuses the cached prompt through a different arithmetic path.
• No logprobs for this model. Strata's docs offer logprobs on NVIDIA and AMD cards, but its Qwen3.6 engine writes none, so it was audited on its text alone, which works but is less precise.

What this means

vLLM's gpt-oss-20b: reproducible, accurate, and checkable in one pass

vLLM is the engine most hosted providers run. On an RTX 4090 we let vLLM 0.32 generate 48 greedy replies with gpt-oss-20b and return its probabilities for every word, then scored the same prompts and words with the model's reference implementation (Hugging Face transformers, BF16) and with llama.cpp.

vLLM's arithmetic for this model is good, a little closer to the original than llama.cpp's MXFP4 path, and its batch-invariant mode costs no accuracy. That mode is also exactly reproducible: an A100, an RTX 3090, two RTX 4090s, an RTX 4080 SUPER, an L40S and an RTX A6000 agreed bit for bit however requests were batched (see the hidden message study).

And a referee does not have to replay the reply word by word. Scoring a finished reply in one pass, vLLM reproduced the numbers it had returned while generating at all 8,588 positions, on the same RTX 4090 and on an RTX A6000 checking the RTX 4090's replies: 48 replies in 7 seconds. An exact check (any nudge to a committed number is proof) can therefore run across GPU generations at the speed of ordinary scoring.

We ran the whole loop. The RTX 4090 generated with the public sampling rule inside vLLM and signed a receipt for each reply: 48 prompts, greedy and two sampled settings. For each reply we also made a forged copy that takes the runner-up at the closest near-tie, edits the committed numbers just enough to make that legal, and is signed again, so it passes every check a client can run on its own. A rented RTX 3090, a GPU generation older, then checked every reply in one pass and required its numbers to match to the last digit.

No false alarms, and every forgery caught exactly where it was made, including edits of 0.0001 in a log-probability. The check took 22 seconds for 43,271 words. This is the strongest form of verification we have measured: no calibration, no statistics, and nothing left for a hidden message to hide in, as long as provider and referee run the same vLLM version in batch-invariant mode on GPUs of the same family.

It is not a quirk of one model. We repeated the test with Qwen3-8B, a dense model in full BF16 precision, on two rented machines, an RTX 4090 and an RTX 3090. Each signed its own receipts and checked the other's: 0 of 49,522 honest words differ, and all 286 forgeries were caught at the edited word. Each check took 12 to 32 seconds. The newest GPUs are a separate group: an RTX 5090 and an RTX PRO 6000 (Blackwell) agree with each other exactly but not with the RTX 4090, and with the RTX 5090 signing and the RTX PRO 6000 checking, the result was again 0 false alarms in 144 honest replies and all 144 forgeries caught. So a referee must be in the provider's group, and each group spans several GPU models. Nor does every upgrade start a new group: vLLM 0.30, 0.31 and 0.32 gave Qwen3-8B the same numbers to the last digit, so referee and provider need not upgrade in lockstep.

Production settings mostly survive too. Prefix caching, prefill in small chunks and speculative decoding left every number unchanged, and our sampling rule now works with speculation (no false alarms, every forgery caught). Splitting the model over two GPUs gives a separate but equally reproducible group. The exception is torch.compile, part of vLLM's default: it computes its own numbers, and on one machine a restart changed them. CUDA graphs on their own are fine and give exactly the uncompiled numbers, so a provider loses nothing measurable by turning compilation off. The real price is batch-invariant mode itself: on a dense model it halved throughput on an RTX 4090, while eager versus compiled cost only a few percent.

A DGX Spark joins the Blackwell group (identical to the RTX 5090 and RTX PRO 6000), and the sampling rule stays exact from temperature 0.2 to 1.5 and with top-k and min-p limits. One rented RTX 4090 was simply faulty: it computed different numbers on every start, while another on the same driver matched exactly. So machines should pass a calibration test with known answers before they are trusted for exact checks.

A hidden message that passes every exact check

In cooperative mode a provider must pick every token with a public rule and commit to the model's top-20 probabilities, so anyone can recompute each choice. One freedom is left. Two honest GPUs running the same weights compute slightly different numbers, so a verifier has to accept small differences, and where the model's top two tokens are nearly tied, that slack decides which one wins. A dishonest provider can use those near-ties to write bits: take the usual winner for a 0, take the runner-up (lowering the winner's committed number by a few hundredths) for a 1. We built that provider and ran it.

Same prompt, same weights, temperature 0. The honest reply on the left; the reply carrying a message on the right, at its first rigged near-tie:

Both columns are what the model itself considers near-equally likely; nothing about the right-hand text looks wrong. Which bit of the message a near-tie carries depends only on its position, so a reader who misjudges a near-tie loses one vote, not the message.

What the checks saw

gpt-oss-20b on an RTX 4090 wrote "HELLO" into 48 replies of up to 384 tokens at each of five budgets (how far apart the two tokens may be, δ, in nats). A DGX Spark acted as referee, scoring in batches of 8 tokens (its default prompt path is less accurate on this model, see the method page).

Reads and the steering test use the 24 held-out replies (about 7,000 tokens) per budget; a flag needs z above 4. Votes are near-ties a reader replayed token by token and assigned to a bit. Every receipt verified: signature, prompt, seed, rule, text. A reader with the provider's exact setup, the same GPU replaying token by token, recovered every carried bit; at δ 0.2 the full message after about 7,000 tokens. A reader on a different GPU got a large share of its votes wrong once the budget was big enough to carry the message, and never read it: at the closest ties two honest GPUs pick differently about half the time. The verifier looks at the same thing from the other side. Honest machines never disagreed where the referee's favourite led by more than half a nat, 2% of the time at 0.2-0.5 nats and 14% at 0.1-0.2, so a provider using every tie within 0.2 nats flips far more often than that and was flagged after 623 audited tokens. Within 0.1 nats the rigged flips look like honest ones: not flagged in 7,000 tokens. (Run first with the Spark's default settings, whose noisier numbers hid even the δ 0.2 channel.)

Qwen3.8-27B is the opposite kind of model: honest machines almost never disagree beyond a 0.2-nat lead, so the verifier's test has room to work. At δ 0.1 it reached z 1.7 after 2,200 audited tokens, which puts a flag at roughly 12,000 tokens; smaller budgets take longer, and carry less. A token-by-token reader on the same machine got every vote right here too.

What this means