Your monitoring dashboard is lying to you

Fixed-threshold monitoring dashboards lie by design: every look at a healthy metric is another chance for noise to cross the line, so checked continuously they all but guarantee false alarms — while the slow drifts that actually cost you never trip the threshold at all.

The model regressed on a Tuesday. Answer quality slid a few percent — not a cliff, just a drift. Your dashboard stayed green for two weeks. By the time anyone noticed, support tickets were up and a customer was drafting a churn email. Not because the dashboard was broken — because fixed-threshold alerting is built to fail one of two ways, and you only get to choose which.

The peeking problem

Set the threshold tight and you get pinged constantly. Most of it is noise, your team learns to ignore the alerts, and the one real page scrolls past at 2am. Set it loose and the slow drifts — the ones that actually cost you — never trip it at all.

Here’s the part nobody tells you: the more often you look, the worse it gets. Every time you check a fixed threshold, a perfectly healthy metric has some small chance of poking past the line by luck. Check once a day and you’ll see the occasional false alarm. Check every request, around the clock, and you are guaranteed to trip it — on data where nothing is wrong. (The rule behind most of those dashboard alerts is the static threshold — its guide spells out exactly what k·σ alerting promises, and what it doesn’t.)

The fix: evidence, not thresholds

There’s a class of methods built for exactly this — call them anytime-valid. The idea in one sentence: accumulate evidence over time, and only alarm when the evidence is strong enough that it couldn’t plausibly be luck. Because it’s evidence rather than a fresh coin-flip on every look, you can check it as often as you want — continuously — and your false-alarm budget still holds.

In the demo above, the drift starts on day 84. The evidence line stays flat through the noise and crosses its alarm level on day 97 — 13 days after the injected change began — with zero false alarms on that run. That number isn’t dressed up for a marketing page; it’s what the detector did on this stream, and you can reproduce it on your own data. For how the classical rules fare on the same kinds of breaks — including where they warn earlier than the budgeted tier — the honest drift-detector benchmark publishes the full grid.

What this buys you

• Alarms you can act on. When it fires, it’s backed by evidence on a stated false-alarm budget — so on-call stops triaging noise and starts fixing things.
• Catch drifts early. Slow regressions accumulate evidence and can trip the alarm while they’re still a drift, not an incident.
• Watch continuously, without the fatigue. Looking more often makes you safer, not noisier. That’s the whole point.