Observability catches crashes.
dedrift catches character changes.
LLM systems fail differently: outputs stay well-formed while length, refusals, tool use and call order, latency tails, and meaning quietly shift. Nothing throws — so nothing pages you.
Silent model updates
The API alias stays the same. The model behind it doesn't. Nobody emails you.
Prompt & tool-schema edits
A colleague "improves" the system prompt. Refusal behavior shifts overnight. No test fails.
RAG & pipeline refreshes
New embeddings, new chunking, new retrieval order — same code, different agent.
Complementary tools, different jobs
Observability platforms are the right tool for crashes, latency, and spend. Drift monitors ship useful shift scores at telemetry scale. dedrift answers a different question: did my agent's behavior change, and can I trust the alert?
| Capability | Observability dashboards | Raw PSI/ADP monitors | dedrift |
|---|---|---|---|
| Crashes, exceptions, latency, spend | ✓ core strength | — | — not an APM |
| Behavioral distributions on frozen canaries | — | partial raw shift scores | ✓ one primary test per channel |
| Calibrated false-alarm rate at canary scale | — | ✗ flagged 100% of stable checks in our harness | ✓ 3.2% measured, enforced in CI |
| Multiplicity control (BH FDR) + materiality gates | — | — | ✓ |
| Anytime-valid mode for continuous watch | — | — | ✓ lifetime error budget |
| Config-fingerprint attribution | partial deploy markers | — | ✓ model / prompt / tools |
Factual, not tribal: use all three. dedrift deliberately does not do uptime, traces, or billing.
A designed experiment, not a dashboard
Frozen canary prompts, run N times per cycle, compared as distributions — because a single LLM run is noise.
Freeze the canaries
A versioned suite across six behavioral families, happy-path to adversarial.
Run N reps per cycle
Every record carries a config fingerprint of your model, prompt, and tools.
Compare distributions
One primary test per channel → BH FDR adjustment → observed-effect materiality gate.
Dual baselines
Sudden, vs a rolling window; cumulative, vs a frozen golden baseline.
Attribute
Alerts matched to fingerprint changes — "consistent with", never "caused by".
We swapped the model.
Nobody told the detector.
A scripted silent swap you can regenerate in four commands — 30 canaries × 7 reps, golden frozen at cycle 3, the swap at cycle 8 behind a changed model fingerprint.
- First post-swap check: DRIFT DETECTED on both baselines
- 21 signature groups co-shift, attributed to the fingerprint change at onset
- Same pipeline pages you on prompt edits and RAG refreshes
Measured, not asserted
Monitoring tools love p-values and hate publishing their false-alarm rates. We enforce ours in CI — a release cannot ship if the statistics fail their own audit.
- Every p-valued detector calibration-tested against a documented band, on every commit
- Pipeline null: 16/500 stable checks alerted (3.2%, Wilson upper 5.1%), family-wise over m≈336 primaries
- Anytime-valid mode: 2/500 stable runs ever alerted over 2,000 cycles (Wilson upper 1.5%)
- The guarantee's edge, published too: the e-value property is already unresolved at σ=0.10 wobble — before the alert rate shows anything is wrong
- Cross-implementation validation: two audit harnesses, built independently by different AI coding agents with no sight of our simulator, re-measure this build. Every finding they returned is locked behind a regression test
- Vendor cross-check: raw PSI flags 92.5% of stable windows at the conventional threshold — dedrift's guard emits none
No unmeasured claims
If a number carries a p-value, the simulation proving its false-alarm rate runs on every commit. A release that fails its own audit doesn't ship.
Alerts must earn it twice
Multiplicity-adjusted equality-test evidence and an observed effect worth acting on. Statistically real but practically trivial doesn't page you — alert fatigue kills monitoring.
We publish what we can't detect
Small suites have limited power, and the docs show that math instead of hiding it — including the shifts this battery will likely miss. Overclaiming is just drift of another kind.
A guarantee with a measured edge
Anytime-valid mode bounds the whole watch horizon, not each check: 2 of 500 stable runs over 2,000 cycles. And the edge is published as carefully as the claim: the e-value property is already unresolved at σ=0.10 wobble, well before the alert rate shows anything — it stays inside budget to σ=0.25, breaching only at 7.2% with persistent AR(1) offsets. Watching the alert rate alone would miss that. The trade-off, quantified.
Don't take our word for any of it — audit the statistics.
Open core, honestly priced
The statistics are free forever. The advanced tier funds the work.
Open source
- Full detector battery, BH FDR adjustment + observed-effect gating
- Anytime mode — lifetime-valid rate e-processes, assumptions and boundary reported
- Dual baselines, attribution, reports
- Calibration suite — audit us
- Runs on a laptop: JSONL + SQLite, zero ML dependencies
Design partner
- Hands-on setup on your agent
- Canary suite designed for your failure modes
- A statistician reading your drift reports
- Priority input on the roadmap
Every statistical guarantee — including anytime-valid mode — is in the open core, forever. A commercial hosted tier (managed monitoring, production-traffic drift) is in development — write to us to hear when it ships.