dedrift¶
Agents don't throw errors when they degrade.
They keep confidently producing worse outputs.¶
dedrift catches silent behavioral drift in AI agents — with calibrated false-alarm rates, multiplicity-adjusted alerting, and config-change attribution. Every statistical claim is enforced by simulation tests that run in CI.
Get started The statistics GitHub
A silent model swap, caught in one canary cycle with correct attribution — reproducible in four commands. see the demonstration.
It doesn't catch character changes.
Model updates, prompt edits, tool-schema changes, RAG refreshes, provider-side silent updates — all shift agent behavior without a single error in your logs.
-
Calibrated, not vibes-based
Every detector's false-alarm rate is validated by simulation tests that run in CI on every commit — and inside the release pipeline. A release cannot ship if the statistics fail their own audit.
-
Two gates before any alert
Benjamini–Hochberg adjustment across every primary test in the check, then an observed materiality gate on effect size. Statistically significant but practically trivial? You don't get paged.
-
Frozen canaries, N repetitions
LLM outputs are random, so single-run comparisons are meaningless. Canaries run N times per cycle; dedrift compares distributions — against a rolling window and a frozen golden baseline.
-
Attribution, honestly framed
Every record carries a config fingerprint. When behavior shifts, the report correlates onset with the nearest config event — "consistent with the model change 5h earlier", never "caused by".
-
Runs on a laptop
JSONL logs + SQLite. No servers, no docker-compose, no SaaS account. Core installs with zero ML dependencies; embeddings are an optional extra with a pinned-forever model.
-
Honest about power
Small canary suites have limited detection power. The docs show the math — including which shifts you won't detect at default scale — instead of hiding it.
A scripted silent model swap — same prompt, same canaries, nothing announced. The first post-swap check fired on both baselines: 124 gated alerts, 21 co-shifting signature groups, attributed to the config fingerprint change at onset.
See the demonstration Reproduce it yourself
Every detector's false-alarm rate is measured against a documented acceptance band by simulation tests that run in CI on every commit. The full pipeline's null alert rate: 16 of 500 stable-agent checks (3.2%, Wilson 95% upper bound 5.1%), measured at m ≈ 336 primaries — the any-alert rate is family-wise and grows with battery size even under valid per-test FDR, so the headline number is always stated at its battery size. Opt-in wobble-robust modes for hosted models are measured in both directions. The parts that aren't calibrated — the diagnostic flag channel — carry their measured rate too, because honesty is the product.
Try it in two minutes¶
No API keys: the built-in simulator plays an agent whose model version is swapped mid-history.
pip install dedrift
mkdir drift-demo && cd drift-demo
dedrift init
dedrift sim --cycles 8 --change-cycle 7
dedrift baseline set cycle-0000 cycle-0001 cycle-0002
dedrift check
Current cycle: cycle-0007
Sudden (vs rolling 4 cycles): DRIFT DETECTED
Cumulative (vs golden 3 cycles): DRIFT DETECTED
Alerts: 168 (BH-adjusted equality tests q=0.05, observed-effect gated)
[golden] adversarial/refusal two_proportion_z: effect=+21.0 pp, p_adj=0.0009
[golden] adversarial/format_valid two_proportion_z: effect=-21.9 pp, p_adj=0.0023
[golden] edge_case/output_words ks: effect=D=0.71, p_adj=1.4e-20
# dedrift report --out report.md → attribution: nearest config event,
# model fingerprint change, 0.0 h before onset — consistent with the swap.
With your own agent¶
One function and a YAML file of frozen canary inputs:
# myagent.py
def agent_fn(input: dict) -> dict:
response = my_agent.run(input["text"])
return {"text": response.text, "structured": response.json}
dedrift canary run --suite canaries.yaml --agent myagent:agent_fn \
--model 'anthropic/claude-sonnet-5@2026-05-01'
dedrift check && dedrift report
Run it on a schedule. When behavior shifts, you'll know what moved, by how much, since when — and what changed in your stack at the same time. Full walkthrough: quickstart · writing canary suites · configuration.
We're onboarding a small number of design partners: teams running production agents who want drift monitoring they can actually trust. You get hands-on setup, a canary suite designed for your failure modes, and a statistician reading your reports. We get the thing no simulation can give — real drift, in the wild.
dedrift Pro¶
A commercial tier with advanced inference is in development and lives outside this repository — contact support@dedrift.ai to hear when it ships.