Same model name, different behaviour. Quantization, silent config changes and provider swaps can break tool calls and shift refusals without any notice. We test every major open-model provider daily, from the outside, and alert you when yours drifts.
Get early access| Provider | Score | Status | |
|---|---|---|---|
| Provider A | 99.8% | stable | |
| Provider B | 98.9% | stable | |
| Provider C | 95.4% | watch | |
| Provider D | 84.1% | drift |
Model makers and evaluators keep finding the same thing: one model, many providers, measurably different results.
Tool-call schema accuracy for the same Kimi K2 model across API vendors, in the model maker's own verifier.
Moonshot AI, K2 Vendor Verifier (test 2025-11-15)Refusal rates for Llama 3.3 70B across four providers serving the same model.
Surelio.ai (2026)Independent, continuous checks you can subscribe to today. Routers grade their own providers; labs build one-off verifiers for their own models.
We never sit in your request path and never need your API keys.
Every day we send a fixed battery of tests to each provider: tool-call schemas, refusals, long-context recall, encoding and output fingerprints against a reference.
Each provider and model gets a public daily score and a history, so you can see who is stable and who changed.
Tell us which providers your product depends on. When one drifts from its baseline, you hear about it in Slack or email before your users do.
We're choosing the first models and providers to cover based on who signs up.