How to Fingerprint AI Models When Prompts Lie
When evaluating third-party API gateways, proxy ro…
When evaluating third-party API gateways, proxy routers, or anonymous arena models, prompt-based identification is essentially useless. A basic system prompt or lightweight fine-tune can make Claude claim it is GPT-4, make Llama sound like Mistral, or alter refusal behaviors entirely.
Persona and tone are malleable. But the underlying serving stack (tokenizers, merge tables, serving harnesses, and error validation boundaries) is rigid. To fake those, a proxy provider would need to intercept streams, translate tokens bidirectionally in real time, and rewrite metadata on the fly, which introduces severe latency and stream desyncs.
Here is how our client-side fingerprinting engine identifies models deterministically using infrastructure artifacts instead of conversational cues.
1. Tokenizer Vocabularies and Byte Fallbacks
Every lab trains a distinct tokenizer with custom vocabulary sizes, merge tables, and fallback rules. Because the tokenizer determines the exact input matrix shape of the model weights, a model cannot alter how it segments text without a full retrain.
We send small, fixed text inputs to the endpoint and inspect the returned usage.prompt_tokens :
Latin Pangrams: A fixed 212-character English string locks in baseline BPE merge behavior.
CJK Segmentation: Chinese text shows high divergence. Chinese frontier models (Qwen, GLM, DeepSeek) encode CJK with high vocabulary density (low token count), while Western models fragment the same string into 2 to 4 times more tokens.
Indentation and Code: Tests how 2-space vs 4-space whitespace blocks and code syntax brackets are compressed.
Emoji and Unicode: Tests whether multi-byte unicode sequences resolve to dedicated tokens, grapheme clusters, or raw byte fallbacks.
2. Template Offsets and Hidden Harnesses
Inference engines (vLLM, SGLang, TGI, TensorRT-LLM) wrap user prompts in chat templates (like ChatML or Jinja templates) and often inject default system prompts.
By sending an empty prompt or a single 1-token payload and comparing the returned usage.prompt_tokens against the raw payload token count, we calculate the static overhead:
offset = usage.prompt_tokens - payload_tokens
This delta exposes hidden guardrail templates, platform wrappers, and chat formatting artifacts applied upstream before the prompt hits the model.
3. Error Taxonomy and Validation Ceilings
Validating normal responses only tells part of the story. Feeding intentionally invalid parameters triggers backend validation written by specific engineering teams:
-
Temperature ceilings: Sending temperature: 2.5 causes some runtimes to fail at >1.0, others at >2.0, with distinct error strings.
-
Context refusals: Requesting max_tokens: 1000000000 forces the backend to reject the request and echo its true physical generation limit (e.g. "max_tokens must be ≤ 131072" ).
-
Vendor error codes: Safety filter triggers often return proprietary numeric codes (e.g. Zhipu AI internal safety code 1301 ).
4. Response Serialization Dialects
Minor implementation details in how the server formats JSON reveal the runtime:
-
The exact choices[0].finish_reason string (e.g. stop vs end_turn vs content_filter ).
-
How null or optional fields are serialized in the response body.
-
Server-Sent Events (SSE) framing quirks during streaming.
Summary of Probe Vectors
Vector Input Observed Artifact
Tokenizer Fixed Pangram / CJK block usage.prompt_tokens
Harness 1-token minimal payload Token delta (Δ offset)
Boundaries Extreme max_tokens / temp Status and error string
Error Code Triggered filter payload Vendor code (e.g. 1301)
Dialect Short completion choices[0].finish_reason
Client-Side Execution
Because probes are purely deterministic, we run them directly in the browser. Requests are dispatched straight from the client tab to the target endpoint without passing through any intermediate proxy or logging server.
Each probe exports a simple runner contract that returns a deterministic value:
export default { name: "tokenizer/cjk-density", description: "Measures CJK token segmentation count", async run(ctx) { const res = await ctx.chat({ messages: [{ role: "user", content: "人工智能模型基准评测体系" }], max_tokens: 1, temperature: 0, }); return { value: res?.usage?.prompt_tokens ?? "ERR_NO_USAGE", }; }, };
Why This Matters
As API wrappers and proxy routers become more prevalent, verification cannot rely on trust or conversational vibes. By testing deterministic infrastructure artifacts like tokenizers, template deltas, and boundary conditions, we can accurately verify what model is actually serving your traffic.
本条由桃子采集流水线（启发式模式）自动整理，原文见文末信源。