Lesson 1 of 30 · Part 1 · Outside in

Your first served model

Serve a model on your Mac and measure the four numbers every inference engineer works with: time to first token, inter-token latency, tokens per second, and end-to-end latency.

Week 1~90 minutesLab on your Mac with OllamaNo downloads needed

Roadmap nodes: IntroductionInference vs. TrainingThe Inference StackTime to First TokenInter-Token LatencyTokens Per Second

By the end of this lesson you can

say how inference differs from training, and name the three layers of the inference stack;
measure TTFT, ITL, decode speed and end-to-end latency on a real model, with a tool you can read;
predict how each number moves when the prompt or the answer gets longer, and explain why.

Why this matters for your mission: every technique in this course, from quantization to autoscaling, is judged by its effect on these numbers. Today you set the baseline that every later lab tries to beat.

Training builds the model, inference serves it

Inference engineering is the practice of making generative AI models faster, cheaper and more reliable to serve in production.The definition used by the roadmap, which is built around Baseten's book Inference Engineering. It exists because two very different jobs happen to a model.

Training learns the model's weights from data. It's a huge batch job that runs for days or weeks and then ends. Inference takes those frozen weights and uses them to answer requests, around the clock, while users wait. Training success shows up in loss curves. Inference success shows up in time to first token, tokens per second, and uptime.NVIDIA's classic explainer "What's the Difference Between Deep Learning Training and Inference?" puts it simply: inference is where capabilities learned during training are put to work.

If you think in app terms, training is the build step that produces an artifact (a file of weights), and inference is running that artifact as a production service. A model is trained once per version. Inference happens on every single request.

The inference stack

A production inference system has three layers:This three-layer split (runtime, infrastructure, tooling) is the roadmap's "The Inference Stack" node, taken from Baseten's book.

Most of this course lives in the runtime layer. Today you'll measure that layer from the outside, the way a user experiences it.

Anatomy of a streamed response

With streaming turned on, a chat API sends the answer as a stream of small chunks, usually one token each. A token is a word or a piece of a word.Hugging Face's LLM course explains how text becomes tokens: "Tokenizers". You'll build intuition for this in Lesson 5. Four numbers describe the stream:

The shape of every streamed answer. A long wait up front, then a steady rhythm.

Metric	What it measures	What mostly drives it
TTFT, time to first token	request sent → first token arrives	queueing + prefill (reading the whole prompt)
ITL, inter-token latency	the gap between two consecutive tokens	one decode step (producing one token)
TPS, decode tokens per second	how fast tokens arrive for one user, after the first	1000 ÷ mean ITL (in ms)
E2E, end-to-end latency	request sent → last token arrives	all of the above, plus network and client

E2E ≈ TTFT + (N − 1) × ITL where N = number of output tokens

Two phases hide behind those numbers. Prefill processes your entire prompt before the first token can appear. Decode then generates the answer one token at a time, each step building on the last. Lesson 6 opens both phases up. Today you only watch their footprints.

One naming note: some sources call ITL time per output token (TPOT). This course always says ITL.NVIDIA defines TTFT, ITL, E2E and throughput precisely, including how benchmarks measure them, in "Metrics" (NIM benchmarking guide). Databricks' "LLM Inference Performance Engineering: Best Practices" uses TPOT for the same idea. Also note that TPS here is per user. A server handling many users at once also has a total tokens per second. That's a throughput number, not a latency number, and the tension between the two is Lesson 2.

Predict first

Lock in these predictions before you run anything. They aren't graded. The lab results checker will tell you whether your data agrees.

The prompt grows from about 80 tokens to about 1,400 (roughly 17×), and the answer length stays the same. What happens to TTFT?

Same change. What happens to ITL?

Now the prompt stays short, but the answer grows from 40 to 320 tokens (8×). What happens to end-to-end latency?

Lab: measure your own model

Lab · ~45 minutes · your Mac

You'll use labs/tools/stream_probe.py, a tool written for this course. It sends a streaming request to any OpenAI-compatible server and timestamps every chunk as it arrives. Skim it before you start. It's plain Python with no dependencies, and later lessons point it at llama.cpp and vLLM.

Start the model server. Open the Ollama app, or run ollama serve in its own terminal tab. Check that it's up:
```
curl localhost:11434/api/version
```
Tip: on macOS, any ollama command can start the Ollama menu-bar app, which keeps a server running in the background. Quit it from the menu-bar icon after the lab to free the memory.
Meet your model. Run ollama show llama3.2. Write down its parameter count, quantization and context length. You'll use them in later lessons.
Watch a raw stream. This is exactly what a chat UI receives:
```
curl -N localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2","stream":true,"messages":[{"role":"user","content":"Name three planets."}]}'
```
Each data: {…} line is one Server-Sent Event carrying a small piece of text.MDN: "Using server-sent events". Ollama mirrors OpenAI's API, see "OpenAI compatibility". Notice the pause before the first line, then the steady rhythm. The pause is TTFT, and the rhythm is ITL.
Measure one request.
```
cd ~/Desktop/learning
python3 labs/tools/stream_probe.py --runs 3
```
The first line is an unmeasured warm-up. Compare its TTFT with the three measured runs.

Experiment A: prompt length. This tests your first two predictions:

python3 labs/tools/stream_probe.py --experiment prompt-length --runs 3 --md

Experiment B: answer length. This tests your third prediction:
```
python3 labs/tools/stream_probe.py --experiment output-length --runs 2
```
Use the formula with your numbers: does TTFT + (N − 1) × ITL land close to the measured E2E?
Experiment C: cold start, and the server's own clock.
```
python3 labs/tools/stream_probe.py --experiment cold-start --server-timings
```
The first request forces Ollama to load the model into memory again. The "server clock" line is Ollama's own split of the request into load, prefill and decode time.These come from the timing fields in Ollama's native API responses (load_duration, prompt_eval_duration, eval_duration, in nanoseconds). See Ollama's API docs, "Usage". Compare the server's load + prefill with the TTFT your client measured. The small gap is HTTP and client overhead.
A mystery. Run Experiment A again, but let the server see the same prompt repeatedly:
```
python3 labs/tools/stream_probe.py --experiment prompt-length --runs 2 --no-cache-bust
```
Look at the long prompt's TTFT on the repeated runs. Write a one-line guess for why it changed. Lesson 19 explains it, or ask your teacher now.
Hint (only after you've guessed)
Add --server-timings to the same command and compare the server's prefill time with the prompt's token count. Then read how Ollama's Usage page defines prompt_eval_duration: the time "to evaluate the uncached prompt tokens".
Check your results in the form below.
Log it. Paste the markdown table from Experiment A into portfolio/LOG.md, add your cold and warm TTFT, and write a one-sentence takeaway in your own words. Optionally, save the raw data by re-running Experiment A with --json portfolio/data/0001-prompt-length.json added.

Exercises

Answer from memory first. Wrong answers are useful here: they show you exactly what to re-read.

Users say your chatbot "thinks for ages before it starts typing, then types smoothly." Which number is the problem?

Time until the first token is high
The gap between tokens is too high
Decode tokens per second is too low
The answers are too many tokens long

"Thinks before typing" is the wait for the first token: TTFT. "Then types smoothly" means ITL is fine. TTFT is mostly queueing plus prefill, so the first suspects are a long prompt, a busy server or a cold model. You saw all three effects in the lab.

Which of these is inference, rather than training?

Serving fixed weights to answer live requests
Updating weights from data in large batches
Tracking loss curves to judge learning progress
Collecting and labeling examples for the model

Training learns the weights. Inference uses the frozen weights to serve requests continuously, under latency targets. Loss curves and labeled data belong to training.

Your team turns on quantization and prefix caching on a single GPU server. Which layer of the inference stack is that?

The runtime layer
The infrastructure layer
The tooling layer
The application layer

The runtime layer makes one model fast on one machine, through batching, caching, quantization and speculation. Infrastructure deals with many machines. Tooling is how engineers control both.

You double the length of the prompt and keep the answer length the same. What do you expect?

TTFT rises; inter-token latency stays similar
Inter-token latency rises; TTFT stays similar
Both rise roughly in proportion together
Neither changes; output length is fixed

A longer prompt means more prefill work before the first token, so TTFT rises. Each decode step costs about the same, so ITL barely moves. That's the pattern from Experiment A.

Right after a deploy, the very first request is several times slower than every request after it. What's the most likely cause?

Model weights are loading into memory
The prompt is longer than usual
Decode steps get slower over time
Network bandwidth limits the token stream

A cold server has to load gigabytes of weights into memory before prefill can even start. On your Mac that was about a second for a 2 GB model on a fast local SSD. In production, weights may first have to come over the network, so cold starts can be far worse. Lesson 25 is about fighting them.

A response's first token arrives after 350 ms. After that, a token arrives every 20 ms. The answer is 401 tokens long. What is the end-to-end latency?

E2E ≈ TTFT + (N − 1) × ITL = 350 + 400 × 20 = 350 + 8,000 = 8,350 ms. It's N − 1 because the first token's time is already inside TTFT.

Your mean ITL is 18.3 ms. What decode speed is that, in tokens per second?

TPS = 1000 ÷ ITL in ms = 1000 ÷ 18.3 ≈ 54.6 tokens/s. That's the decode speed measured when this lab was tested on your Mac.

Product requirement: a 300-token answer must finish within 5 seconds. TTFT is 600 ms. What's the largest mean ITL you can afford?

The decode budget is 5,000 − 600 = 4,400 ms, spread over N − 1 = 299 gaps: 4,400 ÷ 299 ≈ 14.7 ms per token, about 68 tokens/s. Your Mac's ~18 ms ITL wouldn't meet this. A faster GPU, a smaller or quantized model, or speculative decoding might (Parts 3–5).

A server streams 2,000 tokens per second in total, shared evenly by 40 users at once. Roughly what decode speed does each user see?

2,000 ÷ 40 = 50 tokens/s per user. The 2,000 is throughput, a property of the server. The 50 is per-user TPS, a latency number each user feels. Adding users usually raises the first and lowers the second. Lesson 2 measures that trade-off on your own server.

From memory, in two or three sentences: why did TTFT grow with the prompt while ITL barely changed? Use the words prefill and decode.

Model answer. Before the first token can appear, the server runs prefill over the whole prompt, so a longer prompt means more work up front and a higher TTFT. After that, each token comes from one decode step, which costs about the same however long the prompt was, so ITL stays nearly flat. (It creeps up slightly, because each step still looks back at all earlier tokens. Lesson 6 explains how.)

Make it stick

Come back on day 3 and again on day 7. Press Reset exercises and redo them from memory, without re-reading first. The effort of recalling is what moves this from "I followed along" to "I know it". The one-page latency metrics reference has every formula from this lesson.

Primary source · read after the lab (~20 min)

"Metrics", NVIDIA NIM LLM Benchmarking Guide. These are the precise definitions of TTFT, ITL, E2E latency and throughput that industry benchmark tools implement, including details like whether ITL includes the first token. Compare its definitions with what stream_probe.py measures. Can you spot one difference?

Answer (after you've read it)

NVIDIA's TPS per user divides the output length by the whole end-to-end latency, so TTFT is included. stream_probe.py's decode tok/s only counts the time after the first token, so it reports a higher number for the same request. NVIDIA's headline TPS is something else again: total output tokens per second across all simultaneous requests, which is system throughput. The page itself warns that "tool implementations vary, so compare results only when definitions align". That's a habit worth stealing: always say how you measured.

Ask your teacher

Your teacher is in the terminal where this course was built. Ask anything that's unclear or that you're curious about, for example "Why is prefill so much faster per token than decode?" or "What would these numbers look like on a cloud GPU?" When you finish, tell your teacher how the lesson went: what clicked, what didn't, and your lab numbers. Lesson 2 gets written from that.