Lesson 2 of 30 · Part 1 · Outside in

Latency vs throughput

Load-test your own server, plot its latency–throughput curve, and see the one trade-off that shapes almost every decision in inference engineering.

Week 1~90 minutesLab on your Mac with OllamaNo downloads needed

Roadmap nodes: Latency vs. ThroughputLatency PercentilesEnd-to-End MetricsOnline vs. Offline InferenceAsynchronous Inference

By the end of this lesson you can

tell latency from throughput, and say which one an online, offline or async workload cares about;
read p50, p95 and p99, and explain why an average hides the users who suffer;
load-test a server, plot its latency–throughput curve, and tell queueing from batching by looking at it;
check a load test with Little's law, and say what a closed-loop test can hide.

Why this matters for your mission: "How would you load-test this, and what would you report?" is a standard inference interview question. After today you can answer it with a curve you made yourself, and explain its shape.

Warm-up: Lesson 1 from memory

Three quick questions using your own Lesson 1 numbers. Answer before scrolling back. Recalling it now, three days later, is what makes it stick.

Your long prompt (~1,420 tokens) had a TTFT of about 2,620 ms. With --no-cache-bust it fell to about 45 ms, and the server clock said prefill took only 24 ms. Why?

The server reused work from the identical earlier prompt
The model produced fewer tokens on the second run
The decode steps became faster after the model warmed
The network cached the HTTP response from the server

The server kept the results of prefilling that exact prompt, so it skipped almost all of the prefill on the repeat. Ollama's prompt_eval_duration only counts the uncached prompt tokens, which is why it said 24 ms for 1,412 tokens. This is prefix caching. Lesson 6 shows what is being kept (the KV cache) and Lesson 19 shows how production systems share it between users.

Your short-prompt ITL was 18.5 ms. What decode speed is that?

1000 ÷ 18.5 ≈ 54.1 tokens/s. You measured 54.9, which matches. ITL and decode TPS are the same fact seen two ways.

Your short-prompt run: TTFT 144 ms, ITL 18.5 ms, 69 output tokens. Estimate the end-to-end latency.

TTFT + (N − 1) × ITL = 144 + 68 × 18.5 = 144 + 1,258 = 1,402 ms. You measured 1,397 ms, within 0.4%. You skipped Experiment B, but your own data already shows the formula works.

Two customers, two numbers

In Lesson 1 there was one user: you. A real server has many, so there are now two kinds of numbers to watch.

Latency is what one user feels: TTFT, ITL, end-to-end time.
Throughput is what the server delivers in total: output tokens per second across all users, or requests per second.

They pull against each other. Baseten's book puts it bluntly: "One of the primary tradeoffs in inference engineering is latency versus throughput."Baseten, Inference Engineering, §1.2 "About your app". Serving more users from the same hardware usually makes each of them wait a bit longer. Which side you favour depends on who is waiting.

Workload	Who waits	Optimize for	App-dev analogy
Online: chat, code completion, voice agents	a person, right now	latency, especially the tail (p95/p99)	a synchronous API endpoint
Offline: overnight summaries, evals, backfills	nobody, until the job finishes	throughput and cost per token	a nightly batch job
Async: long tasks whose result is used later	a system, not a screen	throughput, but with a deadline	a background job queue with a webhook

The book says "most AI applications – code completion, chat, voice agents – are online applications that run in real time", while "offline jobs are better served by high-throughput model deployments".Same section. It also notes that B2B customers "require high availability and consistently low latency", so "consistently" is about the tail, not the average. Asynchronous inference sits between them. Baseten describes it as "a fire and forget pattern": you get a request ID back straight away, the request waits in a queue, and the result is POSTed to your webhook when it's ready. Its docs also say that "sync requests take priority", so async work fills the gaps between live traffic.Baseten docs, "Async inference". Recommended for long-running tasks, batch processing and priority queuing. If you've used BullMQ or Sidekiq, you already know this shape.

End-to-end means what the user sees

Your probe measured from the client, so its numbers include the network and any time spent queued. The book calls this "an end-to-end measurement that accounts for network latency and any queue time", and gives a rule worth memorizing: "When inference time is fast but end-to-end time is slow, turn your attention to infrastructure rather than model performance optimization."Baseten, Inference Engineering, §1.4 "Measuring latency and throughput". You saw a tiny version in Lesson 1: the server clock (load + prefill) was a little below the TTFT your client measured. Today the gap stops being tiny, because under load most of TTFT becomes waiting.

Percentiles: why averages lie

"Inference engineers measure latency in percentiles."Book §1.4 again. Its definitions: at P50, "1 in every 2 requests is slower"; at P90, "1 in every 10"; at P99, "1 in every 100 requests is slower". Sort all your measurements. The p50 (median) is the middle one, p95 is the value only 5% of requests exceed, and p99 is the value only 1% exceed.

Google's SRE book explains why the mean isn't enough: "A simple average can obscure these tail latencies", with an example where "although a typical request is served in about 50 ms, 5% of requests are 20 times slower!"Google, Site Reliability Engineering, "Service Level Objectives": "We generally prefer to work with percentiles rather than the mean", and "a high-order percentile, such as the 99th or 99.9th, shows you a plausible worst-case value". Those 5% are real people, and they're the ones who complain.

Three habits follow:

Count your samples. With 24 requests, "p99" is basically your slowest request. You need well over 100 requests before p99 means much. Today's tool prints a reminder.
Tails multiply. If one page makes 5 independent model calls, the chance that at least one of them lands in the slowest 1% is 1 − 0.99⁵ ≈ 4.9%. Agents that chain many calls feel the p99, not the p50.
Write targets as percentiles. A service-level objective (SLO) reads like "95% of chatbot interactions should have a TTFT below 200 milliseconds". Goodput is then the number of requests per second "the LLM successfully completes while meeting your defined service-level objectives".Both quotes from the Modular/BentoML handbook, "LLM inference metrics". Goodput is the honest throughput number: requests that were too slow don't count.

When users pile up: queueing or batching

Send a server more requests than it's working on and one of two things happens.

Queueing. If the server works on one request at a time, the others wait in line. Each request still decodes at full speed once it starts, so ITL doesn't change. But TTFT now includes the wait for everyone ahead of you, and total throughput doesn't move, because the server was already as busy as it can get.

Batching. If the server has several slots, it decodes several requests in the same step. One step now produces several tokens, one for each user, so throughput goes up. Each step costs a bit more, so every user's ITL rises. Databricks measured both effects on a GPU: "if we maximize throughput with a batch size of 64, latency increases by 4x while throughput increases by 14x."Databricks, "LLM Inference Performance Engineering: Best Practices": "Request latency increases with batch size." Their Figure 7 is a throughput–latency curve like the one you'll draw today. That deal is so good on GPUs because decode is limited by memory reads, not by arithmetic (Lesson 8). Engines like vLLM go further with continuous batching, which slots new requests in between decode steps (Lesson 12).

Four requests arrive together. One slot makes them take turns. Four slots let them share every step, so they all finish sooner, but each one streams more slowly than it would alone. The times are from a test run of today's lab on your Mac.

Ollama lets you choose. The environment variable OLLAMA_NUM_PARALLEL sets how many requests one loaded model works on at once. With 1 you get pure queueing; with 4 you get batching across up to four requests.Ollama FAQ, "How does Ollama handle concurrent requests?": "The maximum number of parallel requests each model will process at the same time". Today's docs give the default as 1, but your version (0.6.8) defaults to "auto", which picked 2 on your Mac. That's why the lab sets it explicitly.

Little's law: a free sanity check

One equation links the three things a load test measures:

L = λ × W requests in the system = arrival rate × time each one spends there

It's Little's law, proved by John Little in 1961, and it holds no matter how requests arrive or in what order they're served.Wikipedia, "Little's law": the relationship "is not influenced by the arrival process distribution, the service distribution, the service order, or practically anything else". It only assumes a stable system (long-run averages). Two ways to use it:

Check your load test. With 8 users always waiting on a reply, L = 8. If the server completes 0.4 requests per second, each request must take about 8 ÷ 0.4 = 20 s end to end. If your tool reports something very different, the measurement is broken.
Plan capacity. 0.5 requests per second arrive, and each takes 6 s. On average 3 are in flight, so you need at least 3 slots, or a queue.

Closed loop vs open loop

There are two ways to generate load:

Closed loop: C simulated users, each sending the next request only after the last one finished. This is what --concurrency does. It's simple and it's what most benchmarks report.
Open loop: new requests arrive at a set average rate, whether or not earlier ones have finished. Real users behave like this, since they don't wait for strangers' requests to complete. This is --rate.

The trap with closed loop: when the server slows down, the simulated users slow down too, so the load backs off exactly when things get bad. Gil Tene named this coordinated omission. The tester quietly skips the requests it would have sent during the stall, so the worst moments never get measured.Gil Tene, "How NOT to Measure Latency" (talk, ~1 h). The classic on percentiles, coordinated omission and why your monitoring may be lying. Closed loop is fine for drawing a curve. To find out what users will feel at a given traffic level, use open loop.

Predict first

You know the direction of each effect now. These predictions are about size, which is where intuition usually fails. Lock them in before you run anything.

One slot (OLLAMA_NUM_PARALLEL=1). Concurrency goes from 1 to 8 users, and each answer is 128 tokens. What happens to the median TTFT?

Same setup. What happens to total throughput (output tokens per second)?

Four slots (OLLAMA_NUM_PARALLEL=4). Concurrency goes from 1 to 4. What happens to throughput?

Same setup. What happens to each user's ITL?

Lab: your server's latency–throughput curve

Lab · ~50 minutes · your Mac

The new tool is labs/tools/load_test.py. It reuses stream_probe.py's streaming code, runs many requests at once with threads, and reports percentiles at each load level. Every answer is asked to run to exactly 128 tokens, so all requests do the same amount of work. No new installs.

Quit the Ollama app (click the llama icon in the menu bar, then Quit Ollama). The lab runs its own server with settings you choose, and two servers can't share port 11434. If you skip this, the next step fails with address already in use.
Start a one-slot server in a new terminal tab, and leave it running:
```
OLLAMA_NUM_PARALLEL=1 ollama serve
```
Look for OLLAMA_NUM_PARALLEL:1 in the startup lines and Listening on 127.0.0.1:11434 at the end. This tab is now your server log.
Skim the tool (5 minutes). In load_test.py, find run_closed_loop (C threads that each send a request, wait for it to finish, then send the next) and summarize_level (where the percentiles and throughput come from). Note that throughput is total output tokens ÷ wall-clock time.
Run the one-slot sweep in your first tab (about 5 minutes):
```
cd ~/Desktop/learning
mkdir -p portfolio/data
python3 labs/tools/load_test.py --label np1 --csv --md --json portfolio/data/0002-np1.json
```
It runs 24 requests at each concurrency level: 1, 2, 4 and 8. While C=8 runs, look at the server tab. Requests still finish one at a time.

Switch to four slots. In the server tab, press Ctrl+C, then:

OLLAMA_NUM_PARALLEL=4 ollama serve

Run the sweep again, with a new label (about 4 minutes):

python3 labs/tools/load_test.py --label np4 --csv --md --json portfolio/data/0002-np4.json

Read the server's own setting. In the server log, find the line where the model loads. It contains --parallel 4 and --ctx-size 16384. Ollama gave every slot its own 4,096-token context, so 4 slots used 4× the memory for context. The Ollama FAQ says it directly: "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH." Keep that in mind for Lesson 6: concurrency costs memory.
Plot and check. Paste both CSV blocks (each starts with label,load,…) into the box below and press Plot and check. The checker compares your curves with your predictions and runs Little's law on your data.
Stretch (optional, ~6 minutes): open loop. Restart the server with OLLAMA_NUM_PARALLEL=1. Your C=1 row's req/s is the server's capacity (about 0.39 req/s). Now send random arrivals at about half of that, then at about 90%:
```
python3 labs/tools/load_test.py --label open --rate 0.2 --requests 16 --csv
python3 labs/tools/load_test.py --label open --rate 0.35 --requests 16 --csv
```
Paste those CSV lines into the box too. Compare TTFT p50 with p99 at each rate. The server is never "full" on average, yet the tail is ugly. Random arrivals bunch up, and bunches queue.
Log it. In portfolio/LOG.md, paste the two markdown tables and write one sentence in your own words for each of these: what NP=1 did, what NP=4 did, and what you'd tell a product manager who asks "can we just add more users?"
Clean up. Ctrl+C the server tab, then reopen the Ollama app from Applications if you want it back.

Compare with your teacher's run (open after you've plotted your own)

The same lab, run on your Mac while this lesson was being written (llama3.2 3B Q4_K_M, Ollama 0.6.8, 128 output tokens, 24 requests per level).

One slot (np1) shoots straight up: same throughput, longer and longer waits. Four slots (np4) moves right first (more throughput, ~1.5×) and then shoots up once C=8 overflows the 4 slots. On a GPU the np4 curve would stretch much further right before turning up.

Open loop at one slot (16 requests): at 0.2 req/s (~51% of capacity) TTFT was p50 256 ms but p99 3,180 ms. At 0.35 req/s (~90%) it was p50 1,627 ms and p99 5,396 ms. Same server, same requests, and the only change was how busy it was.

Why only 1.5× from batching here? This older llama.cpp build on Apple's GPU doesn't get the near-free batching a datacenter GPU gets, and its ITL tail spikes whenever another request's prefill cuts in (max gap at C=4 was 456 ms). Lesson 12 repeats this lab with vLLM on a free T4, so you can compare the two curves.

Exercises

From memory first. Then check yourself against your own data.

Your dashboard shows a mean TTFT of 300 ms, but users say the bot "sometimes hangs for seconds". What should you look at?

The p99 TTFT over the same window
The mean ITL over the same window
The median TTFT over a longer window
The total throughput over the same window

"Sometimes" is a tail problem. A mean of 300 ms is compatible with 1 request in 50 taking 5 seconds. The p99 (or p95) shows the slow requests directly. The median would hide them even more than the mean does.

Ollama with OLLAMA_NUM_PARALLEL=1: closed-loop concurrency goes from 1 to 8. Which number grows the most?

Time to first token, because requests wait in line
Inter-token latency, because each decode step gets much slower
Total throughput, because the server works on more requests
Decode speed, because the model warms up under load

With one slot, extra users can only wait. The wait happens before the first token, so TTFT soaks up all of it (about 120× in the test run), while ITL and throughput stay flat.

A client wants 1 million documents summarized by tomorrow morning. Nobody reads the results live. How should you serve it?

Offline async jobs tuned for maximum throughput
Online sync requests tuned for minimum TTFT
Online streaming responses tuned for low ITL
Single requests in sequence for lowest latency

Nobody is waiting on any one document, so per-request latency barely matters. Big batches and a queue (async: submit, get an ID, collect results by webhook) get the most tokens per GPU-hour, which means the lowest cost. The only latency that matters is the deadline for the whole job.

You raise the batch size so a GPU serves 8 users at once instead of 1. What do you usually give up?

Some speed per user for more total tokens
Some total tokens for more speed per user
Some accuracy in answers for more total tokens
Nothing, because batching on GPUs is completely free

Each decode step now has 8 users' work, so it takes a bit longer, and every user's ITL rises. The step produces 8 tokens instead of 1, so total throughput rises much more. Answers don't change. Cheap isn't the same as free: Databricks saw 4× latency for 14× throughput.

Why can a closed-loop load test make an overloaded server look healthier than it really is?

Its clients wait for replies, so load drops when latency rises
Its clients send bigger prompts, so the server caches more work
Its clients use streaming, so the server skips the full queue
Its clients measure the mean, so percentiles are never even computed

Each simulated user waits for its reply before sending again, so a slow server automatically gets fewer requests. Real users keep arriving. That's coordinated omission, and it's why open-loop (fixed arrival rate) tests show uglier and more honest tails.

Requests arrive at 0.5 per second, and each one takes 6 seconds end to end. On average, how many requests are in the system at once?

Little's law: L = λ × W = 0.5 × 6 = 3 requests. If your server only has 2 slots, a queue will build and TTFT will climb.

One slot. Each request takes 2.5 s from start to finish. Eight requests arrive at the same moment. Roughly how long does the eighth one wait before its first token? (Ignore its own short prefill.)

It waits for the seven ahead of it: 7 × 2.5 = 17.5 s. The test run's TTFT p50 at C=8 was 17.6 s. When requests queue, TTFT ≈ (requests ahead) × (time per request), which is why it explodes while ITL stays flat.

An agent makes 5 independent model calls per user action. What's the chance that at least one of them is slower than the p99?

1 − 0.99⁵ = 1 − 0.951 = ≈ 4.9%. So about 1 action in 20 hits a "1 in 100" delay. With 20 calls it's 18%. The more calls a feature chains, the more your p99 becomes everyone's experience.

Alone, a request decodes at 50 tokens/s. At batch size 64 the GPU's total throughput is 14× higher. Spread evenly, what decode speed does each of the 64 users get?

Total = 50 × 14 = 700 tokens/s. Per user: 700 ÷ 64 ≈ 10.9 tokens/s, a bit under 5× slower per user, in line with the "latency increases by 4x" Databricks reports. Whether that's acceptable depends on the workload: fine for an offline job, too slow for a chat UI.

In two to four sentences, using your own lab numbers: why did one slot keep throughput flat while TTFT exploded? And why did four slots raise throughput but slow each user down?

Model answer. With one slot the server was already fully busy at C=1, so extra users couldn't add work, only waiting. The wait comes before the first token, so TTFT grew with the queue (~145 ms → ~17.5 s) while ITL and throughput (~50 tok/s) didn't move. With four slots, each decode step serves up to four users at once. That produces more tokens per step (throughput ~50 → ~73 tok/s), but each step takes longer, so every user's ITL rose (~19 → ~51 ms). Past four users, the extra requests queue again.

Make it stick

On day 3, open Lesson 1, press Reset exercises and redo its exercises, then do the same here. Mixing the two lessons is deliberate: telling "this is a prefill problem" from "this is a queueing problem" is the skill. On day 7, redo just this lesson's exercises. The latency metrics reference now has a section on load, with today's formulas.

Primary source · read after the lab (~25 min)

Baseten, Inference Engineering, §1.4 "Measuring latency and throughput" and §1.2 "About your app". These are the two sections the roadmap's nodes for this lesson come from. While reading §1.4, find the sentence that says TPS can be two different kinds of metric, and decide which kind each column of your load-test table is.

If you have an hour for a talk: Gil Tene, "How NOT to Measure Latency". It isn't about LLMs, but every load-testing mistake it describes applies to them.

Talk to practitioners

Post your two curves to r/LocalLLaMA and ask how OLLAMA_NUM_PARALLEL behaves on other Apple chips, or on a newer Ollama. People there benchmark constantly, and a real answer to "why is batching only 1.5× on Metal?" is worth more than another blog post. If you'd rather lurk, search the sub for "concurrent requests ollama" first.

Ask your teacher

Ask anything that's unclear, for example "Why does batching make each decode step slower at all?" or "How would I set an SLO for a voice agent?" When you finish, paste your two CSV blocks and your one-sentence takeaways (from Lesson 1 too) in the terminal. Lesson 3 gets written from that.