Is a 3σ threshold really one false alarm per 370 points?

Only under the textbook assumptions: independent, Gaussian, stationary data with a perfectly estimated baseline. Production metrics violate all four — heavy tails and autocorrelation can multiply the real crossing rate many times over, and repeated checking compounds the problem. Treat one-in-370 as the model's opinion, not a property of your data.

Why does my threshold alert miss slow regressions?

Because a static line only sees level, not trend: a metric that drifts a fraction of a sigma per day can accumulate a large real regression while every individual point stays inside ±3σ. Detectors with memory — CUSUM, EWMA, or an anytime-valid e-process — accumulate that evidence instead of discarding it every point.

Should I just widen the threshold to stop false alarms?

Widening to 4σ or 5σ trades false alarms for misses — the slow-regression blind spot gets strictly worse, and on heavy-tailed data even wide limits keep firing. The structural fix is not a wider line but a detector whose false-alarm rate is controlled by construction, which is what the e-process page tier provides.

All detectors

Static threshold

A static threshold is the simplest alerting rule there is: fit a mean and standard deviation on a training window once, draw the line at mean ± k·σ, and fire whenever the metric crosses it. It is the rule behind most hand-set dashboard alerts, and the direct descendant of Shewhart's 3σ control chart.

WARN

Also known as: fixed threshold / Shewhart-style limit

In ValidAnytime, the static threshold runs in the warning tier: it is the honest baseline every other detector must beat, and its false-warning calibration is model-based — a Gaussian tail probability — rather than guaranteed.

The rule, exactly

μ̂, σ̂  fit once on the first 30 points, then frozen
upper = μ̂ + k·σ̂     lower = μ̂ − k·σ̂
warn when x_t newly crosses outside [lower, upper]

k = 3 is the textbook choice. "Newly crosses" means an upcrossing event — a point outside whose predecessor was inside — so a stretch parked over the line counts as one warning, not one per point.

Live playground — synthetic data, labeled as such; runs entirely in your browser

break size3σ

threshold k·σ3.0

0 warnings on the 80 calm points · catches the break 1 point after onset.

Synthetic standard-normal noise (seeded, 160 uncadenced points — points, not minutes), with a labeled break at point 81 when one is enabled. The detector code running here is the same TypeScript implementation the /try bake-off runs, parity-tested against the cloud suite.

WARN

What is — and is not — guaranteed

Its calibration is a Gaussian tail bound: with independent normal data, a 3σ limit is crossed by roughly one point in 370 — the number every SPC textbook quotes.

That number is not a guarantee about your data: heavy tails make 3σ crossings routine, autocorrelation makes them arrive in bursts, and the one-in-370 story is not anytime-valid — the more often the rule is checked against fresh data, the more chances it has to cry wolf. Our benchmark measures the real rates openly.

Warning tier: classical control charts — sensitive and fast, but calibrated against a model rather than a guarantee, so warnings arrive unbudgeted. See how it scores in the benchmark.

When it wins

The failure is a large, abrupt spike: nothing beats a fixed line for catching a metric that jumps far outside its history — the very first bad point fires.
You need zero sophistication: no state, no tuning beyond k, explainable to anyone in one sentence.
The baseline is genuinely stable and the tails are genuinely thin — the closer your data is to iid Gaussian, the closer the textbook rate is to the truth.

When it lies

The regression is slow: a metric can degrade for weeks inside ±3σ and never cross the line — the silent-regression failure this product exists to catch.
The tails are heavy: on t-distributed or bursty data, 3σ crossings are dramatically more common than the Gaussian story promises, and the alert becomes noise people mute.
The world legitimately moves: a frozen threshold treats every level change — a traffic pattern, a seasonality — as an incident, forever, until a human re-fits it.

Where it comes from

The fixed control limit goes back to Walter A. Shewhart's work at Bell Labs in the 1920s, where the 3σ chart founded statistical process control. The static threshold on a dashboard is that chart, minus the discipline around it. ValidAnytime ships it in the warning tier both because it is genuinely useful on spikes and because it is the baseline any honest benchmark has to include — the playground runs a line-for-line TypeScript port of that warning-tier rule, golden-tested against the cloud implementation's own outputs.

Questions engineers ask

Related detectors

In the glossary: The peeking problem · Anytime-valid inference

Run it on your own history — free, in your browser.

The /try bake-off replays your metric history through this detector and the rest of the suite side by side — false alarms and catch lag, honestly graded, nothing uploaded.

Backtest your history Start free