Field notes
6 posts
The correctable error trap: when PCIe correctable errors start to matter (Problem Specifics 2/5)
Why a climbing PCIe correctable error rate can signal a failing link, how to read AER counters in sysfs, and why peer comparison beats a fixed replay threshold.
What SMART data isn't telling you: which attributes predict drive failure (Problem Specifics 1/5)
Why a drive's SMART status can say PASSED right up to failure, which attributes predict it (5, 187, 188, 197 and 198), and how to trend them against peers.
GPU health checks on AMD: amd-smi telemetry and RAS, plus RDC and RVS diagnostics (Part 2)
How amd-smi reads health, telemetry and RAS errors from AMD Instinct GPUs, which commands to run, and where RDC and RVS take DCGM's role for diagnostics.
Clustering for infrastructure observability: finding outlier servers with HDBSCAN
How HDBSCAN finds servers that behave unlike their peers, how to judge the clusters, and how to compare each outlier's infrastructure state with its peers'.
GPU health checks with NVIDIA DCGM: health watches and dcgmi diag run levels (Part 1)
How NVIDIA DCGM checks GPU health: health watches for PCIe, memory, thermal, power and NVLink faults, and what each dcgmi diag level tests and how long it runs.
node_exporter pcidevice collector: NUMA, SR-IOV and PCIe power state metrics
PCIe link width, NUMA node, SR-IOV virtual functions, power state and vendor names as Prometheus metrics: what each one reports and how to turn them on.