Glossary
The ideas that make an alarm you can trust, explained in plain English — what each one means for someone watching a metric, and why it matters. The rigor is real; the jargon is optional.
also: always-valid inference
Anytime-valid inference is a way of testing that stays statistically valid no matter how often you look at the results.
Read the definitionalso: e-value / test martingale
An e-process is a running score of evidence against 'nothing has changed'; its value at any moment is an e-value, and it stays valid at every look.
Read the definitionalso: anytime-valid confidence interval
A confidence sequence is a sequence of confidence intervals that stays valid at every point in time, so you can read it whenever you like.
Read the definitionalso: false discovery rate control
Online FDR control is a way to bound the fraction of false alarms across many streams that you are testing continuously over time.
Read the definitionalso: conformal prediction for monitoring
Conformal monitoring is the practice of turning a model's outputs into calibrated evidence of change without assuming how the data is distributed.
Read the definitionalso: optional stopping / repeated significance testing
The peeking problem is the reason a metric can look 'significant' just because you checked it too many times.
Read the definitionalso: sequential analysis
Sequential testing is the practice of testing a hypothesis as data arrives, deciding to stop as soon as the evidence is conclusive.
Read the definitionalso: anytime-valid p-value
An always-valid p-value is a p-value you are allowed to read at any moment — it never gets less trustworthy the more often you check.
Read the definitionalso: Ville's maximal inequality
Ville's inequality is the theorem that makes 'valid no matter how often you look' true rather than wishful.
Read the definitionalso: martingale / test supermartingale
A test martingale is a running evidence score that, if nothing has changed, is not expected to grow — the honest core of an e-process.
Read the definitionalso: FDR
The false discovery rate is the fraction of fired alarms that turn out to be false — the number you actually want to control across a fleet.
Read the definitionalso: change detection
Changepoint detection is the task of spotting the moment a metric's behavior genuinely shifts — as opposed to normal noise wobbling around.
Read the definitionalso: statistical process control for machine learning
SPC for ML is the practice of putting machine-learning metrics under statistical process control — control charts with explicit rules, not eyeballed dashboards.
Read the definition