The layer underneath, watched properly.

Nearly every outage was predictable

In our experience the majority of server failures were announced in advance — in a log nobody read, a disk usage graph nobody plotted, or an expiry date nobody tracked. The failure is rarely the machine. It is that nobody was looking at it.

This is the work we started in, before the design side of the business existed. Apache and nginx configuration, load balancing, log analysis, and the unglamorous discipline of reading warnings before they become incidents.

Certificates are the classic one

A TLS certificate expires on a specific known date and takes the site down with a browser warning that frightens customers. It is entirely preventable and it still happens constantly, because renewal depended on somebody remembering. Here it is tracked.

Backups are only real if restored

A backup nobody has restored is a hypothesis. Verification — actually restoring and confirming the result — is part of maintenance rather than something discovered during a disaster. See automated backups.