ValidAnytime
  • Try it free
  • Observatory
  • Pricing
  • Docs
Sign inStart free
ValidAnytime

A guaranteed false-alarm budget across your whole fleet, valid no matter how often you look.

Made by Compiled Intelligence — a frontier AI lab working on quantitative finance from first principles; ValidAnytime is the monitoring we built for our own model fleets, productized.

Product

  • How it works
  • Pricing
  • Observatory
  • Try it free
  • Dashboard

Solutions

  • Who it's for
  • For LLM engineers
  • For MLOps engineers
  • For SRE & platform
  • LLM eval regression
  • Model drift detection
  • Alert fatigue

Developers

  • Docs
  • Quickstart
  • Python SDK
  • API reference

Learn

  • Benchmark
  • Detector guides
  • Glossary
  • Compare
  • Why dashboards lie
  • Blog
  • Anytime-valid 101
  • Winter Storm Uri case study
  • Incident ledger
  • Observatory methodology

Company

  • Contact
  • Privacy
  • Terms

Observatory alarms & product updates, by email

Rare emails only. We store your address and the tag “footer” — nothing else. Unsubscribe anytime.

© 2026 ValidAnytime. All rights reserved.

Every alarm ships with its guarantee_tag and theorem_ref.

Glossary

E-process

An e-process is a running score of evidence against 'nothing has changed'; its value at any moment is an e-value, and it stays valid at every look.

Also known as: e-value / test martingale

An e-process is a running score of evidence that your metric has genuinely shifted — think of a pile of chips that grows as the evidence mounts. It starts at one. If nothing is wrong, it drifts around and tends to stay small; if a real change begins, it climbs. You raise an alarm when it crosses a level you set in advance — and because of how it is built, checking it constantly never cheats you.

The practical payoff is a single, honest dial. A large e-value means strong evidence; a small one means keep watching. Every alarm ValidAnytime raises carries the e-value that triggered it, so you can see exactly how much evidence stood behind the call.

Formally, an e-process (in the simplest case a test martingale) has expected value at most one under the null hypothesis. Ville's inequality then bounds the chance it ever exceeds 1/α by α — which is precisely why an e-process is safe to monitor continuously.

Go deeper

  • The math behind the demo (docs)
  • Compare against threshold dashboards
  • The e-detector — guide and in-browser playground

Related terms

  • Anytime-valid inferenceAnytime-valid inference is a way of testing that stays statistically valid no matter how often you look at the results.
  • Confidence sequenceA confidence sequence is a sequence of confidence intervals that stays valid at every point in time, so you can read it whenever you like.
  • Sequential testingSequential testing is the practice of testing a hypothesis as data arrives, deciding to stop as soon as the evidence is conclusive.
  • Online FDR controlOnline FDR control is a way to bound the fraction of false alarms across many streams that you are testing continuously over time.
  • The peeking problemThe peeking problem is the reason a metric can look 'significant' just because you checked it too many times.
  • Conformal monitoringConformal monitoring is the practice of turning a model's outputs into calibrated evidence of change without assuming how the data is distributed.
  • Always-valid p-valueAn always-valid p-value is a p-value you are allowed to read at any moment — it never gets less trustworthy the more often you check.
  • Ville's inequalityVille's inequality is the theorem that makes 'valid no matter how often you look' true rather than wishful.
  • Test martingaleA test martingale is a running evidence score that, if nothing has changed, is not expected to grow — the honest core of an e-process.
  • False discovery rateThe false discovery rate is the fraction of fired alarms that turn out to be false — the number you actually want to control across a fleet.
  • Changepoint detectionChangepoint detection is the task of spotting the moment a metric's behavior genuinely shifts — as opposed to normal noise wobbling around.

Put the theory to work.

ValidAnytime turns these ideas into a live alarm you can trust — valid no matter how often you look. Prove it on your own data, free.

Start freeTry the detector