We All Have Downtime

I recently came down with a mild flu, which reminded me of a moment at work. My team once had a client who asked for 100% service availability. Not 99.999%, not 99.999999%, a flat 100%. When one of our senior technical leads read that line in the spec, he just said, half in anger: “Even humans have downtime.”

Downtime

For anyone non-technical: downtime is the time a service is down. It is when an app, website, or service stops working, whether that was planned or not.

If you can’t connect to Discord (or any chat app) and there is no notice about maintenance, that is unplanned. If it is a maintenance window they warned you about ahead of time, that is planned.

Availability

In formal docs, downtime is usually framed as the flip side of availability: what percentage of the time, over some period, the service has to be up. You will often see it written in “nines”, from 99.000% (two nines) up to 99.999% (five nines).

Splunk has a blog post on it if you want to read more.

This normally lives in a Service Level Agreement (SLA) with your users or clients. Depending on the organization and the agreement, planned maintenance may or may not count against the availability number.

100% availability

“Anything that can go wrong will go wrong.” (Murphy’s Law)

100% is effectively impossible. There is always some factor outside your control:

A spreadsheet on your computer? Your computer can blue-screen.
The crew at McDonald’s? A flu wave can go through them.
The self-service checkout at McDonald’s? Someone having a bad day might smash every machine.
An AWS data center? Amazon recently declared multiple zones “hard down” after a missile strike hit its sites in Bahrain and Dubai.
Your run-of-the-mill software service? The servers can just fall over.

And if none of those land, I can always curse your setup with a meteorite strike to make the point.

Whatever the service, object, or person, there is always a “bruh moment” waiting to take it down. The goal isn’t to guard against every possible bruh moment. It is to guard against them to a reasonable degree, at a reasonable cost.

Availability and cost

You can absolutely reach 99.9999999% availability. You could:

Keep a second computer on hand for when the first one blue-screens.
Hire extra crew to stand by.
Build the self-service checkout like a tank.
Park a missile interceptor at your local data center.
Just throw more servers at your software.

But at what cost, and is it worth it? Some organizations chase absurdly high availability for very little real benefit. Do the users even need that many nines? Often the money is better spent on incident response.

Incident response

You are going to hit some unexpected downtime eventually, so why not plan for it? That might look like:

Sending your computer to a repair shop.
Calling in off-duty staff or emergency temps.
Getting a repair crew out to the self-service checkout.
Telling customers about the data center outage while a crew works on it.
Telling customers about the service outage and restarting the server, with a quiet prayer.
Even NASA keeps an incident response team on hand for bugs in space. The latest one I saw was them fixing a Microsoft Outlook bug in orbit (lol).

The point: if you can’t afford to prevent it, be ready for it as best you can. I couldn’t stop myself catching a flu, but I could at least see a doctor and get some medication 🥲.