Latency vs Throughput

One-liner: Latency is time per request; throughput is requests per time; they are independent, and the tail of the latency distribution is what users feel.

Say this#

SAY THIS

Latency — time for one operation (request → response). Measured as a distribution, never an average.
Throughput — operations completed per second (QPS, RPS, TPS, MB/s).
Bandwidth — maximum data rate of a link (bits/s); throughput ≤ bandwidth.
p50 / p95 / p99 / p99.9 — percentiles: 99% of requests are faster than p99.
Tail latency — the slow percentiles; dominated by GC pauses, queueing, cache misses, retries.
Tail amplification / fan-out — a request touching 100 services hits someone's p99 almost always.
Little's Law — L = λ × W: concurrency = arrival rate × time in system.
Queueing / utilization — latency explodes as utilization → 100% (M/M/1: W ∝ 1/(1−ρ)).
Saturation — the point where added load only adds queueing.
Hedged requests — send a duplicate after p95 and take the first response.
Head-of-line blocking — one slow item blocks the ones behind it (HTTP/1.1 pipelining, TCP, single partition).
Apdex / SLO latency target — e.g. "p99 < 300 ms".

How it works#

TEXT

Little's Law:  L = λ W  1,000 req/s × 0.2 s per request  = 200 requests in flight -> need ≥200 concurrent workers/connections  pool of 50 connections, 20 ms per query -> max 50 / 0.02 = 2,500 queries/s Fan-out tail:  P(at least one of 100 backends is in its p99) = 1 - 0.99^100 ≈ 63%Little's Law:  L = λ W
  1,000 req/s × 0.2 s per request  = 200 requests in flight -> need ≥200 concurrent workers/connections
  pool of 50 connections, 20 ms per query -> max 50 / 0.02 = 2,500 queries/s

Fan-out tail:  P(at least one of 100 backends is in its p99) = 1 - 0.99^100 ≈ 63%

Measure latency as a histogram; report percentiles per endpoint.
Throughput capacity = concurrency / service time; add workers or cut service time.
Keep utilization under ~70-80% to keep queueing latency bounded.
Attack the tail: timeouts, hedging, avoiding fan-out, warming caches, tuning GC, isolating noisy neighbors.

Averages lie: mean of [1,1,1,1,1,1,1,1,1,100] ms is ~11 ms; p50 is 1 ms, p90 is 100 ms. The user who hit 100 ms doesn't care about the mean.

Trade-offs#

Choice	Pro	Con
Batching	Higher throughput	Higher per-item latency
Pipelining / async IO	Throughput without more threads	Complexity; HOL blocking if ordered
Hedged requests	Cuts tail latency	Extra load (cap at ~5% of requests)
Running at high utilization	Cheaper	Latency spikes; no headroom for bursts
Caching	Cuts median and tail	Staleness; cold-start misses
Bigger machines	Lower latency per op	Cost; single-node ceiling

Numbers#

Same-DC RTT ~0.5 ms; cross-continent ~150 ms; human-perceptible ~100 ms; "instant" < 100 ms.
1 Gbps link ≈ 125 MB/s; 10 Gbps ≈ 1.25 GB/s. Transferring 1 GB over 1 Gbps ≈ 8 s regardless of latency.
Queueing: at 50% utilization wait ≈ 1x service time; at 90% ≈ 9x (M/M/1).
Commonly quoted web SLOs: p99 < 200-500 ms for APIs; p50 < 100 ms.

Say this#

How it works#

Trade-offs#

Numbers#

In the interview#