CUSUM's false-alarm promise, measured

A control chart’s calibration is a promise about how often it cries wolf. CUSUM’s is usually stated as an average run length: tune it for ARL 2,000 and you are promising roughly one false alarm per 2,000 healthy points. We measured what that promise delivers — on 100 synthetic healthy streams of 2,000 points each, labeled and regenerated from frozen seeds — and the answer is one false warning per ~189 points on textbook data and one per ~13 on realistic texture. This post is the decomposition of that gap, because none of it is exotic and all of it is measurable.

The setup, precisely

The chart is the two-sided CUSUM at the product’s default parameters: k = 0.5 on standardized residuals, threshold h solved for an iid-Gaussian ARL of 2,000, mean and sigma estimated from the first 30 points. Two worlds, both healthy end to end: iid Gaussian — the world the calibration is priced for — and a realistic texture of slow AR(1) wander plus heavy-tailed spikes. Both are synthetic; that is what makes “zero real breaks” a fact rather than a hope.

What was observed

In the iid world CUSUM raised 1,040 false warnings — one per ~189 points, on 90 of 100 streams — already ~11× its stated rate on the exact distribution it was priced for. On the textured world it raised 15,089: one per ~13 points, on 99 of 100 streams — about 154× the nominal rate. The other classical charts (EWMA, static threshold, rolling band) show the same shape of shortfall; the calibration table has every row.

The decomposition

The iid-world gap factors into two mundane pieces. First, the ARL-2,000 threshold is solved for a one-sided chart, and the shipped default runs two-sided because regressions go both ways — that roughly halves the spacing. Second, the textbook pricing assumes mean and sigma are known, while practitioners standardize on the ~30 training points they actually have — estimation error costs roughly another 4×. Multiply those and you have essentially the observed one-per-~189 spacing before the data misbehaves at all. Everything beyond that is the texture: autocorrelated wander and heavy tails are off-model for an iid-Gaussian calibration, and model-based calibration evaporates off-model. That is a property of the math, not of any vendor’s implementation — a hand-rolled CUSUM in a notebook inherits every factor.

The same honesty, applied to our own tier

The budgeted page tier gets measured with the same ruler. The coverage e-process held its budget exactly: 0 of 100 streams fired in both worlds against a stated 1-in-100 cap. The e-detector logged no events at all in the iid world; on the textured 2,000-point streams its observed spacing was one per ~663 points against its 2,000-point target — its run-length promise is conditional on the conformal miss probability staying at or below its null of 0.1, and this texture’s fastest excursions transiently push the genuine miss probability above that null, which is the precise condition it exists to flag. On the 90-point fleets that match the product’s monitoring horizon it stayed well inside budget: 2 events across 18,000 points, where the target allows ≈9.

What to do with this

Not “stop using CUSUM” — on sudden steps it is genuinely faster than anything carrying a guarantee, which is why the suite ships it. The conclusion is narrower: treat a model-calibrated chart’s alarms as unbudgeted early warnings, not as events with a known error rate, and put the budget on a tier built to hold one. The CUSUM guide shows the rule on a live playground, and the benchmark write-up covers where each tier wins.