Someone asks how fast your app is, and you have to give them a number. The average is the tempting answer, and it's the wrong one — it describes a user who doesn't exist. p95 and p99 are the numbers that describe what people actually feel, and the gap between them tells you whether your slowest requests are rare bad luck or a real problem with your name on it.
The definitions
Sort every request's duration from fastest to slowest. The value at the Xth position from the bottom, where X is your percentile, is pX.
- p50 (median) — half your requests are faster, half are slower
- p95 — 95% of requests finished under this; 5% were slower
- p99 — 99% of requests finished under this; 1% were slower
- p99.9 — typical SLO for mission-critical services
Why averages lie
Take an endpoint where 99% of requests finish in 50ms and 1% take 30 seconds. The average is around 350ms — a number that describes neither population. The fast users don't see 350ms; the slow users see 60,000x worse.
Averages mask outliers. Percentiles are the outliers — which is what you care about.
The gap is the signal
p95 tells you the story for most users. p99 tells you the story for your most frustrated users. The gap between them tells you how frustrated those users are.
Healthy service
p50: 80ms
p95: 180ms
p99: 240ms
Gap p95 → p99: 60ms (small — consistent tail)Service with a bad tail
p50: 85ms
p95: 220ms
p99: 3400ms
Gap p95 → p99: 3180ms (huge — 1% of users hit something broken)Common causes of a big p95 → p99 gap: database contention on specific rows, cache misses, garbage collection pauses, one slow downstream service affecting only some requests, uncached file reads, or — classic in Laravel — an N+1 query that fires on some pages but not others.
A real example
A dashboard endpoint looked fine on the average: 140ms, comfortably inside the SLO. Its p95 was 190ms. Its p99 was 2.8 seconds.
That gap was the whole story. One in a hundred requests was doing something the other ninety-nine weren't. Pulling up a handful of the slow ones showed it: they all belonged to accounts with thousands of records, and one widget loaded those records without a limit. For a normal account it returned in 40ms. For the biggest tenants it scanned the table.
The average never showed it. The p95 barely did. The p99 made it impossible to miss, and the individual slow requests behind that p99 named the accounts and the query. That's the workflow worth internalizing: the percentile tells you a tail exists, the slow requests behind it tell you why.
How to pick which to optimize
In typical user-facing web apps:
- p95 — your reliable SLO. Alert when it regresses; make it the number you promise.
- p99 — your canary. Watch the gap, not the absolute value. A widening gap is the earliest signal of degradation.
- p99.9 — for payment, auth, health-critical paths. Worth the operational work.
How many samples you need
Percentiles need volume before they mean anything. A p99 computed from 200 requests jumps every time one slow request lands — that's noise, not signal. The rough rule is 100× the reciprocal of (1 − percentile): about 2,000 requests before p95 settles, about 10,000 before p99 does.
In practice that makes percentiles a per-route, per-window measurement. p99 across your whole app over a day is solid. p99 for one rarely-hit admin route over an hour is mostly random. So when a percentile looks alarming, check the sample count behind it before you act on it.
How NightOwl surfaces percentiles
NightOwl stores request duration in microseconds and computes p50, p95, and p99 across any time range — per route, per query pattern, per job class. Because the data lives in your PostgreSQL, calculations use a straightforward ORDER BY duration OFFSET ... LIMIT 1 query instead of lossy histograms or sketches.
RELATED
- What is APM?
- Apdex score — another way to summarize latency
- Guide — monitor slow SQL queries