Fleet problems
13 posts
The firmware layer nobody owns: infrastructure state your monitoring can't see (Problem Specifics 5/5)
Infrastructure state is what a server is, not what it does. Why a BIOS setting or firmware change surfaces only as a symptom, days later, and how to read it.
The heterogeneous fleet problem nobody admits: what Redfish doesn't normalise (Problem Specifics 4/5)
Why Redfish gives Dell, HPE and other BMCs a common API but not common BIOS settings or event messages, and what a normalisation layer has to do across vendors.
Bring-up is where schedules go to die: what new servers need before production (Problem Specifics 3/5)
Bring-up is the work between racked and first workload accepted. What a slip costs while purchased capacity waits, and a five-step gate for every new server.
The correctable error trap: when PCIe correctable errors start to matter (Problem Specifics 2/5)
Why a climbing PCIe correctable error rate can signal a failing link, how to read AER counters in sysfs, and why peer comparison beats a fixed replay threshold.
What SMART data isn't telling you: which attributes predict drive failure (Problem Specifics 1/5)
Why a drive's SMART status can say PASSED right up to failure, which attributes predict it (5, 187, 188, 197 and 198), and how to trend them against peers.
The maintenance queue is a CapEx problem: what idle servers cost while they wait (Problem Scope 5/5)
Servers in the maintenance queue keep depreciating while they wait. Worked arithmetic for a 1,000-server fleet, and the share of the queue a digital fix clears.
Your automation has an expiry date: how playbooks go stale as the fleet changes (Problem Scope 4/5)
Why automation written for last year's hardware skips or misapplies fixes on today's fleet, and how to build each remediation plan from the node's actual state.
The tribal knowledge tax: what walks out when your senior engineers leave (Problem Scope 3/5)
Why a pinned BIOS setting or held-back firmware version often lives in one engineer's head, and what that costs in investigation time and deferred maintenance.
Configuration drift is the silent killer: the settings nobody tracks (Problem Scope 2/5)
Configuration drift hides in the BIOS, firmware and kernel settings no one told config management to check. Why no alert fires, and how to see it on every node.
Uptime is not reliability: the gray failures green dashboards miss (Problem Scope 1/5)
A server can pass every health check and still run below spec. Why uptime misses gray failure, and the three properties that make infrastructure reliable.
What happens when the loop stops: why remediation is the mission (3/3)
Finding faults faster does not end the firefighting loop. One BIOS incident, told two ways, shows why the fix itself has to be planned, approved and verified.
Where the loop always starts: the hardware signals your monitoring misses (2/3)
PCIe replays, BIOS drift and disks that SMART calls healthy: the hardware signals behind silent degradation, and what 35,000 servers at Flipkart taught us.
Your team is trapped in a loop: why infra teams keep firefighting (1/3)
Hardware telemetry split across tools, runbooks that age with every firmware change, servers stuck in maintenance: the loop that keeps infra teams firefighting.