Why did my adaptive anomaly detector miss a regression it watched happen?

Because adaptation cuts both ways: a trailing-window band re-fits its notion of normal at every point, so a regression that arrives slower than the window's memory is folded into the baseline instead of flagged. The detector did not fail at its job — its job, judged honestly, is spikes against the recent past, not slow slides.

What window size should a rolling band use?

It is a lose-lose dial: a short window adapts fast (absorbs regressions faster, noisier σ̂, more false alarms), a long window is steadier but treats benign level changes as incidents for longer. There is no window that both adapts to legitimate change and refuses to adapt to illegitimate change — that distinction needs a detector with an explicit null, like an e-process.

Is a rolling band better than a static threshold?

They fail in opposite directions: the static line misses nothing that crosses it but goes stale the moment the world legitimately moves; the rolling band survives legitimate movement but absorbs slow regressions. A monitoring suite that must be honest runs both — and neither carries a false-alarm budget, which is why they warn rather than page in ValidAnytime.

All detectors

Rolling band

A rolling band judges each point against a mean ± k·σ band re-fitted on the trailing window just before it — the rule behind most “adaptive” anomaly detectors and dynamic-threshold alerts. Because the band follows the data, it stays useful when the metric's level legitimately moves around.

WARN

Also known as: trailing-window anomaly band / adaptive threshold

That adaptivity is also its structural failure: a slow regression re-trains the band point by point, so the rule learns the degraded level and goes quiet — it absorbs exactly the incident you most need to catch. In ValidAnytime it runs in the warning tier, unbudgeted.

The rule, exactly

window_t = the previous 30 points (exclusive of x_t)
μ̂_t, σ̂_t  re-fit on window_t at every point
warn when x_t newly leaves [μ̂_t − k·σ̂_t, μ̂_t + k·σ̂_t]

k = 3 and a 30-point trailing window in the cloud suite. "Newly leaves" is an upcrossing event, as with the static threshold. The first 30 points have no complete window and cannot fire.

Live playground — synthetic data, labeled as such; runs entirely in your browser

break size3σ

trailing window30 ptsband width k·σ3.0

0 warnings on the 80 calm points · catches the break 9 points after onset.

Synthetic standard-normal noise (seeded, 160 uncadenced points — points, not minutes), with a labeled break at point 81 when one is enabled. The detector code running here is the same TypeScript implementation the /try bake-off runs, parity-tested against the cloud suite.

WARN

What is — and is not — guaranteed

Its calibration is the same Gaussian tail story as the static threshold, applied to a moving fit: with independent normal data and a well-estimated local baseline, band exits are rare.

It is not anytime-valid, and worse, the moving fit makes its miss behavior structural: a drift slower than the window's adaptation is invisible in principle, not just in practice — the band converges to the degraded level and reports normal. Our benchmark shows this openly.

Warning tier: classical control charts — sensitive and fast, but calibrated against a model rather than a guarantee, so warnings arrive unbudgeted. See how it scores in the benchmark.

When it wins

The anomaly is sudden and local: a spike against the recent past is exactly what a trailing band is built to flag, even when the metric's absolute level has legitimately wandered.
The metric has slow benign level changes: the band follows daily and weekly wander that would keep a frozen threshold permanently firing.
You need adaptivity with zero modeling: no seasonality model, no training pipeline — the window is the model.

When it lies

The regression is slower than the window: the band re-learns the degraded level as it arrives and never fires — the silent-regression failure, by construction.
The window is small: σ̂ estimated on 30 points is noisy, so the band breathes, and its effective false-alarm rate can sit far from the textbook number.
An incident partially recovers: after absorbing a shift, the band treats the *return to normal* as the anomaly — alarms that point in exactly the wrong direction.

Where it comes from

The rolling band has no single inventor — it is the folk descendant of moving-average control charts, rediscovered in every monitoring stack as “dynamic thresholds” or “anomaly bands”. ValidAnytime ships it in the warning tier because it is what most teams actually run today, which makes it the comparator every honest bake-off must include — the playground runs a line-for-line TypeScript port of that warning-tier rule, golden-tested against the cloud implementation's own outputs.

Questions engineers ask

Related detectors

In the glossary: Changepoint detection · Conformal monitoring

Run it on your own history — free, in your browser.

The /try bake-off replays your metric history through this detector and the rest of the suite side by side — false alarms and catch lag, honestly graded, nothing uploaded.

Backtest your history Start free