Latency metrics
The numbers that describe how fast a model feels, on one page.
Definitions
| Metric | Definition | Formula | Mostly driven by |
|---|---|---|---|
| TTFT time to first token · ms | Request sent → first non-empty token received | measured directly | queueing + prefill + network |
| ITL inter-token latency · ms | Average gap between consecutive tokens, first token excluded | (E2E − TTFT) / (N − 1) | one decode step |
| Decode TPS per user · tok/s | Speed of the stream after the first token | 1000 / ITLms | decode speed |
| E2E end-to-end latency · ms | Request sent → last token received | TTFT + (N − 1) × ITL | everything, including client and network |
| Throughput system TPS · tok/s | Output tokens per second across all concurrent requests | total output tokens / wall time | batching, concurrency (see Under load) |
Naming in this course: say ITL, not TPOT ("time per output token", same idea). Say decode TPS for the per-user stream speed and throughput for the system total. NVIDIA's "TPS per user" is N ÷ E2E, which includes TTFT, so always state your definition.Definitions and the caveat "tool implementations vary, so compare results only when definitions align": NVIDIA, "Metrics".
Formulas you'll use constantly
TPS = 1000 ÷ ITL(ms) · ITL(ms) = 1000 ÷ TPS
E2E ≈ TTFT + (N − 1) × ITL
largest affordable ITL = (deadline − TTFT) ÷ (N − 1)
per-user TPS ≈ throughput ÷ concurrent users (if shared evenly)
What moves what
| You change… | TTFT | ITL | E2E | Why |
|---|---|---|---|---|
| Longer prompt | ↑↑ | ≈ (slight ↑) | ↑ | more prefill before the first token |
| Longer answer | = | = | ↑ linear | more decode steps, same cost each |
| Cold model | ↑↑↑ | = | ↑ | weights must load into memory first |
| Repeated prompt prefix | ↓↓ | = | ↓ | the server reuses cached prefill work (Lesson 19) |
| More users, one slot (queueing) | ↑↑↑ | = | ↑↑ | requests wait in line; throughput flat |
| More users, more slots (batching) | ↑ | ↑ | ↑ | each step serves several users; throughput ↑ |
Measuring honestly
- Warm up first, unless cold start is what you're measuring. The first request may include loading the model.
- Bust the prefix cache when measuring prefill (
stream_probe.pydoes this by default), or measure cache hits on purpose. - Count tokens with the server's usage numbers, not stream chunks. A chunk isn't always exactly one token.
- Client clock vs server clock: the gap between them is network and client overhead.
- Report percentiles, not just means: p50, p95 and p99, with enough samples (over 100 for a real p99).
- Write down how you measured: model, quantization, hardware, prompt and output lengths, concurrency, tool.
Under load
Latency is what one user feels; throughput is what the server delivers in total. Serving more users from the same hardware usually raises the first and lowers the second's cost per token.
| Term | Meaning |
|---|---|
| p50 / p95 / p99 | the value that 50% / 5% / 1% of requests exceed. Means hide the tail; with N requests, p99 needs N well over 100. |
| SLO | a target written as a percentile, e.g. "95% of requests have TTFT below 200 ms" |
| Goodput | requests per second completed while meeting the SLO |
| Queueing | one slot: extra requests wait. TTFT ↑↑, ITL =, throughput flat |
| Batching | several slots decode in the same step. Throughput ↑, every user's ITL ↑ |
| Closed loop | C users, each waits for its reply before sending again (--concurrency). Load backs off when the server slows: coordinated omission. |
| Open loop | requests arrive at a set rate regardless (--rate). Like real users; shows honest tails. |
| Online / offline / async | a person waits now (latency) / nobody waits (throughput, cost) / a queue + webhook with a deadline |
Little's law: L = λ × W (in system = arrival rate × time in system)
queued TTFT ≈ (requests ahead) × (time per request) + own prefill
P(at least one of n calls is slower than p99) = 1 − 0.99n
Teacher's Lab 2 run on your Mac (llama3.2 3B, Ollama 0.6.8, 128 output tokens, 24 requests per level, closed loop):
| Slots | Load | TTFT p50 | TTFT p99 | ITL | E2E p50 | Throughput |
|---|---|---|---|---|---|---|
| 1 | C=1 | 145 ms | 181 ms | 18.9 ms | 2.5 s | 50 tok/s |
| 1 | C=8 | 17,553 ms | 18,813 ms | 19.0 ms | 19.9 s | 50 tok/s |
| 4 | C=4 | 527 ms | 647 ms | 51.3 ms | 7.0 s | 73 tok/s |
| 4 | C=8 | 7,131 ms | 7,261 ms | 49.1 ms | 13.3 s | 77 tok/s |
On a datacenter GPU, batching is far cheaper: Databricks saw batch 64 give 14× throughput for 4× latency. Run it yourself: python3 labs/tools/load_test.py --label np1 --csv.
Your baseline
Your own Lab 1 numbers (M1 Pro, llama3.2 3B Q4_K_M via Ollama 0.6.8, 15 Oct 2026, from portfolio/LOG.md).
| Condition | TTFT | ITL | Decode TPS |
|---|---|---|---|
| ~80-token prompt, warm | ≈ 145 ms | ≈ 18.5 ms | ≈ 55 tok/s |
| ~1,420-token prompt, warm | ≈ 2,620 ms | ≈ 22.9 ms | ≈ 44 tok/s |
| Same long prompt, repeated (cached) | ≈ 36–49 ms | ||
| Cold start (load 2,019 ms + prefill 531 ms) | ≈ 2,570 ms |
Sources
- NVIDIA, "Metrics", NIM LLM Benchmarking Guide: precise definitions used by benchmark tools such as AIPerf.
- Databricks, "LLM Inference Performance Engineering: Best Practices": TTFT and TPOT in practice.
- Baseten, Inference Engineering §1.4 "Measuring latency and throughput": percentiles, end-to-end metrics, TPS as latency or throughput.
- Google SRE book, "Service Level Objectives": why percentiles beat the mean.
- Modular handbook, "LLM inference metrics": SLOs and goodput.
- Ollama, "Usage": the server-side timing fields
stream_probe.py --server-timingsreads.