Alarms you can trust — no matter how often you look.
The regression that shipped on a Tuesday sat green for weeks, because fixed-threshold dashboards false-alarm so often you learn to ignore them. ValidAnytime watches every stream with anytime-valid e-processes — a guaranteed false-alarm budget across your whole fleet, so you catch the real one early and can trust every alarm you get.
Two tiers ship together: the classical detectors you already run — CUSUM, EWMA, threshold charts — as fast, unbudgeted warnings, plus a page tier whose stated false-alarm budget holds however often you look.
Free tier · your data stays yours · see the live Observatory
Fixed threshold — ≈ 7 false alarms
The more often you look, the more it cries wolf on stable noise.
ValidAnytime — 0 alert, 0 false
Watching — no evidence of a real change yet.
Roll 20 genuinely random healthy streams and watch nearly all of them stay quiet — the false-alarm bound, re-runnable yourself.
Illustrative run on a synthetic stream, scored by the same anytime-valid detector we run on your data. The metric labels above are just framing — the numbers and the detector’s behavior are identical. In onboarding, ValidAnytime replays your history and shows whether — and on which day — it would have caught your regression.
Designed to ingest from the tools you already use
- OpenAI
- LangChain
- OpenTelemetry
- MLflow
- SageMaker
- Databricks
- Postgres
- Webhooks
Not affiliations — these are input sources. If it emits a number, we can watch it.
0
false alarms on the healthy stretch
across 84 healthy days of the demo stream — no threshold to trip by chance
13 days after onset
caught the injected drift
flagged the injected regression the moment the evidence was undeniable
∞
looks, still valid
check once or a thousand times — the error budget holds either way
Type-I error controlled at every stopping time — a consequence of Ville’s inequality, the same guarantee behind the demo detector above. Numbers taken from that synthetic run (log-e ≥ 7.60).
The same always-valid inference behind modern A/B-testing platforms.
Built on 70 years of sequential-analysis theory — the demo above is the cited Robbins normal-mixture e-value.
- Ville 1939
- Robbins 1952
- Howard, Ramdas, McAuliffe, Sekhon 2021
The peeking problem
Your monitoring dashboard is lying to you
Every fixed-threshold alert makes the same statistical mistake: it tests the same hypothesis over and over, then acts surprised when noise crosses the line. Watch long enough and a false alarm is guaranteed.
after 20 looks at a fixed-threshold rule. Continuous monitoring inflates error with every peek.
Armitage; Millerfrom repeated significance testing on a live stream. The classic peeking problem — known since 1952.
Robbins, 1952metrics times looks. The more you watch, the more it cries wolf — so teams mute it and miss the real failure.
alert fatigueAnytime-valid statistics fix this at the root. E-processes and confidence sequences keep their guarantee under continuous monitoring, optional stopping, and as many looks as you like. See how it works
The wall
90 minutes. 20 services. Two alerting policies.
17 of these p99-latency streams are healthy; 3 carry a real injected incident. Both policies watched every stream, minute by minute. Every stream is synthetic and labeled, reproducible from a frozen seed — and every alarm below was verified against the production engine.
…in 90 minutes, on data where nothing was wrong. And it still caught only 2 of the 3 real incidents — one sailed through untouched.
…on the same 20 streams, and all 3 incidents caught — each page sent inside a stated false-alarm budget, with the evidence attached.
15 pages on nothing is how alert fatigue starts: the channel gets muted, and the incident that matters sails through. An alarm you can act on has to be rare on healthy data — that is the property the e-process guarantees, and the threshold rule can’t.
To be fair to the threshold: on the 2 incidents it did catch, it was faster — in minutes, +4 vs our +16, +4 vs our +17 (and it never caught the third; we paged that one +20 minutes after onset). The same sensitivity that catches in 4 minutes is what pages 15 times on nothing. We sell the quiet, not the sprint.
How it works
From raw stream to trustworthy alarm in three steps
The rigor is the engine; the outcome is what you ship. Setup is minutes, and nothing goes live until it’s proven on your data.
- 01
Connect a stream
Send any metric — model error, an LLM judge score, CPA — through the Python SDK or a single HTTP call. No agents, no infra to run.
- 02
We prove it on your history
Auto-config replays your own data through a backtest gate: it must stay quiet on normal history and fire on a real degradation before it ever goes live.
- 03
Trust every alarm
Live e-processes watch each stream with a guaranteed false-alarm budget; online FDR bounds false discoveries across the whole fleet.
from validanytime import Client
va = Client()
monitor = va.create_monitor(
name="llm-answer-quality",
config={"template": "llm_quality"},
)
# stream any metric — judge score, model error, CPA…
va.ingest(monitor.id, value=0.91, event_id="evt_1042")The activation event
“This is the day it would have caught it.”
Before a single live event, the backtest gate replays your history and shows whether, and on which day, your regression would have been flagged — on your data, with zero setup calls.
Try the backtest on your own history →The product
Watch it catch what a fixed threshold misses
An LLM-judge quality score starts slipping, a little every hour — never enough in any one look to trip a static alert. The seeded demo monitor runs both alarm tiers: a sensitive classical warning that is model-calibrated and unbudgeted, and an e-process page delivered inside a budget of roughly one false page per stream-year.
Quiet on every tier for the 64 healthy hours. The unbudgeted warning flagged the slide 4 hours in; the budgeted page came at hour 78 — 14 hours after onset, with the evidence attached.
The stream is synthetic — the regression was injected on purpose. The calibration, alarm, e-value, certificate, and fleet review are real engine output.
Why it’s different
Rigor as a feature, not a footnote
The incumbents alert on thresholds and eval snapshots. ValidAnytime is built on the one capability they lack: calibrated, anytime-valid decisions under drift — with fleet-wide error control.
Anatomy of an alarm
- monitor
- llm-answer-quality
- fired_at
- 2026-06-24T14:02:11Z
- statistic
- log-e = 8.38 (≥ 7.60)
- guarantee_tag
- average_run_length_e_detector
- theorem_ref
- Howard et al. 2021, Thm 1
- message
- quality dropped — caught on day 97
An example alarm payload — the exact shape every real alarm ships. The tag and theorem are the receipt: what was guaranteed, and the math that backs it.
Anytime-valid by construction
E-processes and confidence sequences keep their guarantee under optional stopping. Look once or a thousand times — the error budget holds.
Fleet-wide false-discovery control
Online FDR (e-LOND / e-LORD) bounds false alarms across every monitored stream at once — not one threshold at a time.
Every alarm is explainable
Each alert ships a guarantee_tag and theorem_ref stating exactly what is guaranteed, and under what assumptions.
Backtest-gated configs
Nothing goes live that false-alarms on your normal history. The gate is non-bypassable, even for AI-suggested configs.
Built for messy reality
Delayed and missing labels, drift, and changepoints are first-class — not edge cases you bolt on later.
Featherweight at scale
O(1) math per event with a tiny carry state. Millions of events on cheap compute — no per-token inference tax.
- Valid no matter how often you look?
- Fleet-wide false-alarm control?
- Proof it works on your own data?
The benchmark
756 false warnings vs 2 false pages — on the same healthy fleet.
On a synthetic healthy fleet — 200 streams of 90 points each, no break anywhere — the ARL-2000-tuned two-sided CUSUM raised 756 false warnings across 177 of 200 streams, while the budgeted page tier raised 2 false pages against its stated budget of ≈9. The CUSUM also caught 239 of 240 breaks injected into the same texture — genuinely faster than the page tier — which is exactly why the suite ships both tiers instead of picking one.
Frozen seeds, replayed through the production engine — every number regenerates from one command.
One engine, every stream
Point it at the signals that matter
The same anytime-valid core watches any metric you produce. Lead with the pain you feel today; expand to the rest from one dashboard.
LLM & agents
Catch silent provider regressions in days, not weeks.
- Judge scores
- Task success
- Format validity
- Latency & cost
ML models
Valid drift detection — migrate off threshold dashboards.
- Prediction error
- Delayed labels
- Feature drift
- Calibration
Ads & KPIs
Tell a real CPA spike from a broken pixel.
- ROAS / CPA
- Conversion rate
- Spend
- Measurement breaks
Recently shipped
We build in the open.
A steady stream of things you can try right now — no login required.
- Jul 2026The detector benchmark
Every detector we ship, graded honestly on labeled synthetic fleets — including where the classical charts beat the budgeted tier. Frozen seeds, machine-readable results, one regen command.
- Jul 2026Classical detector guides + in-browser playgrounds
CUSUM, EWMA, static thresholds, rolling bands — when each wins, when it lies, with a line-for-line, golden-tested port of each running right in your browser.
- Jul 2026The Observatory goes live on the US power grid
The same engine, pointed at a real deployed model: each US grid's day-ahead demand forecast. We monitor error = actual − forecast and page only on a genuine regime change, inside a stated budget — with a dated ledger back to 2021.
- Jul 2026Docs for agents
A machine-readable guide so coding agents can wire a monitor end-to-end — endpoints, working examples, and the full reference in a single fetch at /llms-full.txt.
- EarlierFleet-wide online FDR (e-LOND)
Alarms from e-process monitors now enter a pooled false-discovery budget — confirmed discoveries control FDR across your whole fleet, under arbitrary dependence.
- EarlierIn-browser backtest
Paste your own history and watch the anytime-valid detector run — it shows whether, and at which exact point, it would have caught your last regression. No signup.
- EarlierSolutions pages — by problem and by team
Dedicated pages for LLM, MLOps, and SRE teams, plus eval regression, model drift, and alert fatigue — each with a live proof block and a one-click demo.
Pricing
Start free. Pay only for volume.
No credit card to start. You’re billed on committed events — one data point accepted into a monitor — with a generous free tier and the public Observatory free for everyone.
Open SDK
The open-source client for the API — integrate in a few lines.
Free & open-source client
- Typed Python client (MIT)
- Talks to the hosted API
- Pairs with any cloud plan
- Backtest examples · community support
Free cloud
Kick the tires on your own data.
100k committed events / mo
- 1 live monitor
- Both alarm tiers — warnings + budgeted pages
- Backtest gate on your history
- Webhook alerts
- Email & Slack delivery — at launch
Pro
For a solo dev or a small team.
5M committed events / mo
then $20 per additional 1M · set a spend cap
- Up to 10 monitors
- Webhook alerts
- Email & Slack delivery — at launch
- Alarm certificates (verifiable exports)
- 90-day history
- Optional monthly spend cap
Team
Fleet-scale, with false-discovery control.
50M committed events / mo
volume discounts beyond — talk to us
- Unlimited monitors
- Online-FDR across streams
- Priority support
- SSO (soon)
Prices in USD. Metered on committed events; volume discounts at scale. Coming at launch: monitors whose verdict page you make public are free forever. See full pricing & estimate your cost →
FAQ
Questions, answered
Stop guessing whether it broke.
Point ValidAnytime at one stream and see whether — and on which day — it would have caught your last regression. On your own data, in minutes, free.
The backtest is free and runs in your browser — no signup, nothing uploaded.