Home / System Design / Concepts / Availability and the Nines Concepts
CONCEPT SHEET · Concepts
Availability and the Nines
One-liner: Availability is the fraction of time a system serves correctly; each extra nine is 10x less downtime and roughly 10x more cost.
Article Concepts © Arthur Bernier Jr 528 words 1 code block
Say this#
SAY THIS
SLI — Service Level Indicator: the measured metric (success rate, p99 latency).
SLO — Service Level Objective: internal target on the SLI (99.9% of requests < 300 ms).
SLA — Service Level Agreement: external contract with penalties; looser than SLO.
Error budget — 1 − SLO; how much failure you are allowed to spend on releases.
MTBF / MTTR — mean time between failures / mean time to recovery. Availability = MTBF / (MTBF + MTTR).
Redundancy — N+1, 2N; no single point of failure (SPOF).
Failover — switching to a standby; active-passive (hot/warm/cold standby) vs active-active.
Graceful degradation — serve partial functionality instead of nothing.
Fault tolerance vs high availability — FT: no interruption; HA: fast recovery.
Blast radius — scope of impact of one failure; cells/AZs/regions limit it.
How it works#
TEXT Copy
Serial dependencies multiply: A(0.999) -> B(0.999) = 0.998 (worse) Parallel redundancy: 1 - (1-0.999)^2 = 0.999999 (better)
Define SLIs from the user's view (request success, latency), not CPU.
Set SLO below SLA; the gap is your safety margin.
Remove SPOFs: redundant LBs, multi-AZ DB with automatic failover, multiple app instances.
Detect fast (health checks, heartbeats), fail over fast (DNS TTL low, VIP move, leader election via etcd/Zookeeper/Consul).
Spend the error budget on deploys; freeze when exhausted.
Active-passive: standby idle, promoted on failure; risk of stale replica and split brain. Active-active: all nodes serve; needs conflict handling or stateless nodes.
Trade-offs#
Choice Pro Con Active-passive Simple, no conflicts Idle capacity; failover delay; possible data loss if async Active-active Full use of capacity, instant failover Conflict resolution, data sync complexity Multi-AZ Survives datacenter failure Cross-AZ latency (~1-2 ms) and transfer cost Multi-region Survives region failure Replication lag, much higher complexity/cost Chasing more nines Happier users Each nine ~10x cost and ops burden
Numbers#
Availability Downtime / year Downtime / month 99% 3.65 days 7.3 h 99.9% 8.76 h 43.8 min 99.95% 4.38 h 21.9 min 99.99% 52.6 min 4.38 min 99.999% 5.26 min 26.3 s
Two 99.9% services in series ≈ 99.8%. Two 99.9% in parallel ≈ 99.9999%.
S3 designed for 99.99% availability, 99.999999999% (11 nines) durability.
In the interview#
SAY THIS
State it in non-functional requirements: "Target 99.99% for the read path, 99.9% for writes; that is ~52 min/year of read downtime."
Say: "Every tier has at least two instances across AZs; the DB is a leader with a synchronous standby and automatic failover."
Follow-up: "What does 99.99% cost you?" → "Multi-AZ everything, automated failover under a minute, no manual steps on the critical path, and an error budget that gates deploys."
Follow-up: "Why SLO below SLA?" → "Headroom: you want to be paged before you owe customers money."
Creator
Written by Arthur Bernier Jr (Big Poppa Code). Arthur Bernier Jr — Big Poppa Code — is a software engineer, educator and entrepreneur: a former CTO at VC-backed startups and engineering lead at a billion-dollar company, mentor to 800+ engineers, now Senior Director of AI, Computer Programming and Cloud Computing at Per Scholas, and the host of the Deeper Than Code podcast. Everything on this page is his original teaching material.
bigpoppacode.io
License © Arthur Bernier Jr all rights reserved