Use case · LLM eval regression detection
Catching an LLM eval regression early means telling a real drop in judge scores from the wobble every nightly run shows — exactly what a fixed threshold on a noisy average can't do. ValidAnytime watches your eval streams with anytime-valid e-processes and fires one alarm the day the drop becomes real evidence, with a false-alarm budget across every eval you run.
Free to start · no credit card
You ship a prompt change on Monday. Tuesday's eval scores a point lower — within noise, you tell yourself. Wednesday, same. By the next week the average is down four points and tickets are climbing, but no single run ever crossed a line big enough to alarm. The regression didn't hide because you weren't watching. It hid because a threshold on a noisy score can't tell a slow, real drop from ordinary variance.
A fixed threshold on a noisy eval trips on nothing — check a fixed 5%-error test 5 times and the real false-positive rate is already ~23%; at 20 looks it passes 64% — and scheduled evals re-roll the dice every run, so you miss the slow regressions or drown in false ones. Eval and tracing frameworks (LangSmith, Langfuse, Braintrust) are great at producing scores; the alarm on top has no valid-under-peeking guarantee.
Judge scores, task success, format validity, groundedness, latency, cost — send the numbers your evals already produce. No prompts or outputs leave your stack.
The backtest gate replays your past runs and shows whether, and on which day, it would have caught your last regression — and stays quiet on the runs that were fine.
An e-process accumulates evidence across runs and fires once, with a certificate, the day the drop is real. Online FDR keeps false discoveries bounded across every eval in the suite.
Load a drifting LLM-judge series and watch the evidence line stay flat through the noise, then cross the alarm the day the drift is real — with nothing tripped before it. Nothing is uploaded. Then watch the same engine adjudicate “did Claude or GPT change?” live in the Observatory.
LLM judge score — slow regression
Synthetic, illustrative — runs in your browser, nothing uploaded.
From the seeded demo — synthetic stream, real engine. Load it in the live dashboard →
Keep your eval framework — this isn't one, and we don't run your evals or host judges. ValidAnytime is the production alarm on the scores it emits: any numeric eval stream, through the API.
Point ValidAnytime at one stream and it replays your own history — free, no credit card.