# Normal Is a Moving Target: four conditions for a valid anomaly score > Every anomaly score compares a reading with a baseline. Four conditions decide whether it holds: signal identity, stationarity, peer validity and observer bias. - URL: https://blog.asama.ai/blog/normal-is-a-moving-target/ - Author: Manish Gautam - Published: 2026-07-04 - Tags: Frameworks ## Executive summary Anomaly detection, drift analysis and health scoring are all downstream of one claim: what this system should look like when nothing is wrong. If that claim is wrong, everything built on it is wrong too, and almost no monitoring system states it explicitly. Normal is not the absence of problems. It is a statistical characterisation of expected behaviour under expected conditions, and it is a property of a metric, in a context, at a point in time. Remove any of those qualifiers and the concept is undefined. Three forces keep it moving: workloads that change what the hardware is doing, fleets that change what the hardware is, and deliberate operating choices that change what counts as acceptable. None are data quality problems. They are structural properties of the systems being observed. The useful reframe is to stop asking along which axes normal varies and start asking what conditions must hold for two observations to be validly comparable. That question resolves into four: are we measuring the same thing, is the generative process unchanged, is the comparison group genuinely exchangeable, and how much of the relevant signal was visible at all. They are not four parallel models. They are a pipeline: signal identity gates stationarity, stationarity gates peer validity, and observer bias qualifies the result. The output is not a score. It is a score with an envelope describing how well each condition held. --- Is this node degraded? Has this configuration drifted? Is this metric anomalous? All three are comparisons, and none can be answered without a reference point. In most estates that reference point has never been written down. It is implicit in a threshold set four years ago, or in a baseline function quietly averaging the last fourteen days, and it carries more weight than anything else in the detection stack. > "Every detection is a comparison. The interesting question is never the measurement. It is what the measurement is being compared against, and whether that thing is still true." For a single static server running a fixed workload, this is tractable. Observe it long enough, fit a distribution, flag deviations. Fleets are not static. They are heterogeneous, continuously evolving systems in which the conditions that define normal are themselves in motion. --- ## Why normal moves Three forces drive that motion, and none of them is a modelling failure. **Workload evolution.** Infrastructure is purpose-built to be repurposed. The same cluster may run large-scale training one week and low-latency inference the next. Those are not variations, they are different operational regimes. Power draw, thermal profile, memory access pattern and PCIe utilisation are determined more by what the hardware is doing than by what it is. A baseline built during training is actively misleading during inference. **Fleet and system evolution.** [Firmware updates change power management behaviour](https://blog.asama.ai/blog/issue-10-asama-the-firmware-layer-nobody-owns). New nodes arrive with slightly different silicon. Topology changes alter traffic patterns. OS upgrades shift how software touches hardware. Each can move the distribution of normal for part or all of the fleet without any degradation occurring at all. **Operational and economic context.** You redefine acceptable behaviour deliberately. Cost optimisation may mean running with less thermal headroom than a conservative baseline allows. SLA tiering may mean some nodes are intentionally operated differently. From a naive baseline this looks like drift. It is intent, and a system that cannot tell the difference generates noise that erodes trust in everything else it says. That intent belongs in policy, where you can see it, not in the statistical baseline. A baseline that learns your cost trade-offs is right on average and wrong in every specific case. The answer to "what is normal for this system" therefore cannot be a static number, a static distribution, or even a slowly updated model. --- ## When two readings are comparable The obvious framing is to ask along which axes normal varies, and then to model each axis. That framing describes how normal varies, not how to model the variation, detect a change or tell legitimate change from degradation. The better question is narrower: **what conditions must hold for two observations to be validly comparable?** That turns a modelling problem into a set of preconditions. An anomaly score is a comparison between an observation and a baseline. If the two are not comparable, the score is not weak, it is meaningless, and no amount of statistical sophistication downstream repairs it. Four conditions have to hold. | Condition | The question | If it fails | | --- | --- | --- | | **Signal identity** | Are these observations drawn from the same operational regime? | The baseline fits the data and the scoring is systematically wrong. Degradation in one regime is masked by healthy behaviour in another | | **Distributional stationarity** | Is the generative process behind the metric unchanged? | Real anomalies get absorbed into the baseline, or legitimate evolution gets flagged forever | | **Peer validity** | Is this comparison group a genuine equivalence class? | Fleet-wide problems become invisible, because every node looks normal against neighbours that share the problem | | **Observer bias** | How much of the relevant signal is actually being seen? | High-confidence scores are produced in exactly the cases where confidence is least warranted | These are not four models running in parallel. They form a pipeline, and the order matters. Signal identity gates stationarity: you cannot assess whether a distribution has changed until you know which regime's distribution you are looking at. Stationarity gates peer validity: peer comparison is only meaningful once the baselines being compared are current. Observer bias gates nothing and qualifies everything, because a score is only as good as the fraction of relevant signal that was visible when it was computed. --- ## What each condition requires **Signal identity** makes workload regime a first-class dimension of the data model rather than a metadata tag applied afterwards. Regimes can be discovered from telemetry alone, which matters because workload metadata and hand-entered labels are not reliably available. A Gaussian mixture model finds the regimes by [clustering multivariate signals](https://blog.asama.ai/blog/clustering-for-infrastructure-observability) such as utilisation, memory bandwidth, PCIe throughput, power and thermals, and its soft membership shows when a node sits between two. A classifier trained on those clusters then labels each reading in near real time. Every reading carries a regime label and baselines are partitioned by regime. The hard part is transitions. Workload changes are not instantaneous, and mid-move a node's metrics belong cleanly to neither regime. Scoring has to be suspended through the transition, and the detector has to be conservative, requiring sustained evidence before committing to a new label. A jumpy classifier produces anomalies that are artefacts of its own indecision. **Distributional stationarity** means separating three kinds of change that look identical on a chart. Gradual drift, from a changing thermal environment or incremental load, should be absorbed into the baseline slowly, in versioned steps, and only once peers, or an outside anchor for a whole group, show it is legitimate. Ageing that slides towards a failure looks exactly like legitimate drift. A structural break, from a firmware update or hardware replacement, needs a new baseline version rather than slow absorption. A transient deviation should be flagged as an anomaly and kept out of the baseline entirely. [Bayesian online change-point detection](https://arxiv.org/abs/0710.3742), run per metric, per node and per regime, returns [a posterior over the time since the last change](https://blog.asama.ai/blog/determinism-probability-and-the-limits-of-observability), rather than a binary flag, which makes this principled instead of heuristic. On detection the right move is not immediate recalibration but a provisional window: collect evidence on whether the new distribution is stable. A stable one triggers baseline versioning. A continuing shift means a transient or an ongoing degradation event. Distinguishing a legitimate break from degradation needs a side channel. That is a **confound register**: a log of discrete fleet events such as firmware updates, scheduled maintenance, hardware replacements and topology changes. A change-point coinciding with a known confound is tagged as likely-legitimate. One that does not is escalated. **Peer validity** is the strongest condition of the four, because exchangeability is a strong claim. Two nodes are exchangeable when, conditional on group membership, the variation between them is noise and individual difference rather than systematic difference in what they are doing or how they are configured. Two nodes with the same SKU fail that test if they run different workloads, sit at different firmware versions, occupy different thermal neighbourhoods, or differ substantially in age. Peer groups therefore have to be [built on several axes at once](https://blog.asama.ai/blog/issue-11-asama-peer-comparison-is-the-baseline), and membership has to be fluid. Taking each node's nearest neighbours on those axes as its peers does that, because a regime shift moves the node and changes its neighbours. Below a minimum group size the system should fall back to broader priors rather than produce an unreliable peer-relative score. **Observer bias** is the least intuitive and arguably the most important. A baseline never represents the true state of the infrastructure. It represents the infrastructure as captured by a specific set of sensors, at a specific sampling rate, with a specific set of failure modes in mind. A node with full thermal telemetry and no memory error counters has a known gap. A gauge sampled once a minute catches a fault that clears in thirty seconds only when a sample happens to land inside it. These gaps are encoded silently into the baseline and shape normal without ever being stated. The response is a **sensor coverage map**: a matrix indexed by node and failure mode class, recording what fraction of the relevant signal is observed at sufficient resolution. Every score is annotated with a coverage confidence derived from it. --- ## Two failures that need an outside anchor Two failures sit outside what the pipeline can resolve on its own. **Collective degradation.** If every node in a peer group degrades together, from a fleet-wide firmware issue, a systemic thermal problem or a shared environmental factor, peer comparison finds nothing. Each node looks normal relative to its peers precisely because they are all affected. The group's centroid has moved, and the centroid is the reference point. Detecting it requires tracking how each peer group's centre of mass moves over time, with an anchor that sits outside the group. In practice that means one of three things: a pinned historical snapshot, a cross-site reference cohort running comparable work, or a physics-based prior derived from what the hardware is specified to do. Without an external anchor, a fleet quietly normalises its own decline. **Systematic blind spots.** Individual instrumentation gaps show up in the coverage map. Gaps shared across the whole fleet do not, because they reflect design choices in the monitoring architecture rather than per-node failures. A fleet where every node is missing the same sensor class develops a baseline that is coherent, stable, and wrong in a way that is invisible from inside the data. These cannot be found from the fleet's own telemetry. The mechanism is periodic cross-fleet audit: comparing this fleet's coverage map and baseline structure against fleets with richer instrumentation, to find signal classes that are absent here and informative elsewhere. That is an audit function, not a real-time model, and it belongs on a calendar rather than in a pipeline. --- ## A firmware update the baseline learned to accept Run the hot node from [The Four Levels of Detection Maturity](https://blog.asama.ai/blog/four-levels-of-detection-maturity) through the four conditions. A firmware update resets its fan profile from performance to optimised, and six days later, during a heavy load run, its clocks throttle and a chassis temperature alert fires. Signal identity holds: the node runs the same work throughout. Stationarity finds the break on the day of the update, fan RPM down and CPU package temperature up. The break coincides with a logged firmware update, so the confound register tags it likely-legitimate, the provisional window sees a stable distribution, and the baseline is versioned. Against its own history, the hotter node is now normal. Peers still catch it while the node is alone on the new firmware. Its group falls below the minimum size, so the score falls back to a broader prior, its SKU on any firmware, and against that its fan RPM is low. That is how The Four Levels found it. Once the whole cohort takes the update, every peer shares the fault. Then only an anchor that judges the new state catches it: a physics-based prior, since a CPU at that load and inlet temperature should not throttle, or a check of the fan profile, read as state, against the profile the cohort should run. A logged change explains why a distribution moved, not whether the new one is acceptable. Treat likely-legitimate as explained, not accepted, until the new state is [checked against intent](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer). --- ## The output is a score with an envelope Because every condition qualifies the score, the honest output of the pipeline is not a single number. Two nodes flagged at identical severity can be in completely different epistemic positions. One sits in a large, stable peer group, in a confidently classified regime, with comprehensive coverage and no recent confounds. The other is a rare SKU with six peers, mid-transition between workloads, on a chassis missing a sensor class relevant to the failure mode in question. Both may be correct. Only one should be acted on without a second look, and the number alone does not say which. So a score should arrive with all four conditions attached. Not "this node is anomalous" but "this node is anomalous, the peer group is valid and large, the regime is confidently assigned, coverage is comprehensive." Or, equally usefully: "this looks anomalous, but the peer group is thin, the regime classifier is uncertain, and coverage of this failure mode class is partial." The goal is not confident scores. It is accurate ones, with [honesty about where accuracy is limited](https://blog.asama.ai/blog/trust-is-the-real-interface). Confidence estimates get acted on while they are trusted, and once they prove unreliable, the correct ones are ignored along with the rest. --- ## From Asama > ### Built around the four conditions > Asama works this way. > > **Baselines are versioned, not continuous.** Every baseline carries a version, a timestamp, the event that triggered it, and the peer group composition in force at the time. Every score is computed against an identifiable baseline version, so any past detection can be audited after the fact. > > **Peer groups are explicit and auditable.** The composition of the group used to score an observation is visible, and you can see why two nodes are or are not treated as peers. Where your domain knowledge exceeds what the model captures, you can pin membership. > > **Confidence is a first-class output.** Low-confidence scores are surfaced distinctly rather than blended in. Scores derived from partial sensor coverage carry that annotation explicitly. > > **The system models its own blind spots.** It can report not only that a node looks healthy, but that it looks healthy given the sensors available, and that a named failure mode class has limited coverage on that node. --- ## Food for thought > When your monitoring says a node is fine, what is it comparing against, and when did anyone last check that the comparison is still valid? **Update, October 2026.** Asama now reads every layer of every node, detects against each node's peers and its own history, traces each finding to a root cause, and plans, runs and verifies the fix. Asama samples critical signals every second, so a fault that clears in thirty seconds spans thirty samples. CMDB entries, change requests and golden configs come in as intent and change, the input a confound register and a check against intent both need. To see what your current baselines miss, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.