# The Art of LLM Inference: Serving Language Models from First Principles to Production

A language model is a file of numbers. Serving it means answering thousands of strangers at once, each wanting the first word quickly and the next words steadily, on machines that cost more per hour than most people earn in a week, without leaking one stranger's words to another. The file does not change. Everything that makes serving hard is about *where the numbers are kept, how often they are moved, and who is waiting.*

## The story: Lantern

Lantern is a start-up of four people running its own dense 70-billion-parameter model on rented GPUs. On day one it has fifty users; in six months, fifty thousand, an agent product that calls the model forty times per task, a security review and a bill. Each part begins with something that goes wrong for it.

| When | What goes wrong | The idea that explains it | Where |
| --- | --- | --- | --- |
| Day 1 | One user, and the answer comes at 70 tokens a second on four GPUs that could do thousands | A decode step is limited by bytes, not by arithmetic | A |
| Day 3 | Sixty-four users cost under a third more time per step than one | Batching spreads the weight reads over many tokens | A |
| Day 5 | "Out of memory" at 54 conversations | The cache of past keys and values | A |
| Week 2 | The GPU is idle while a long answer finishes, and everyone freezes when a long prompt arrives | Continuous batching, paging, chunked prefill | B |
| Week 3 | The first bill | Quantisation, speculative decoding, parallelism | B |
| Month 2 | Two nodes, and a router that must decide | Prefix caching, disaggregation, scheduling, queueing | B |
| Month 3 | The agent product: forty calls a task, each re-reading the whole conversation | Cache economics, session routing, tiered cache | D |
| Month 4 | A customer asks who else can see their prompts | Threat model, injection, isolation | E |
| Month 5 | Moving from one node to a fleet in a cloud account | Reference architecture, build against buy | F |
| Month 6 | Operating it | SLOs, load tests, rollouts, failure | F |

Part C asks what the labs and open frameworks do in public.

## How to read the notes

<figure class="mfig ds-figure ds-wide" id="inf-legend" aria-label="Colour key: blue memory, amber arithmetic, purple shared, green user, pink control, red attacker">
<div class="ds-plot dg-host" data-h="227"></div>
<figcaption><b>One colour per role, in every figure.</b> Equations use the same blue, copper and violet.</figcaption>
</figure>

| | Assumption |
| --- | --- |
| Times | floors: 0.5 of peak arithmetic, 0.8 of peak memory bandwidth; real engines land below |
| Model | dense decoder: 80 layers, width 8192, 64 query and 8 key-value heads of size 128, feed-forward 28672, vocabulary 128256; 2-byte weights and cache unless stated |
| Machine | H100 SXM: 80 GB at 3.35 TB/s, 989 teraFLOPS dense 16-bit (NVIDIA's 1979 is *with sparsity*); four linked at an assumed 450 GB/s each way, 8 microseconds per collective |
| Money | an illustrative three dollars per GPU-hour, not a price list |
| Sources | a claim about a company is tied to the reading list; "probably" marks an inference |

**Notation.** A derivation defines any other symbol where it is used.

| Symbol | Meaning |
| --- | --- |
| $\ell,\ d,\ d_h,\ n_q,\ n_{kv},\ f$ | layers, width, head size, query heads, key-value heads, feed-forward width |
| $P$ | number of parameters |
| $b,\ b_{kv}$ | bytes per weight, bytes per cached number |
| $B$ | batch: sequences decoded together |
| $n,\ c$ | prompt length in tokens; context length (tokens in the cache) |
| $C$ | chunk size of chunked prefill, in tokens |
| $g$ | number of GPUs that share one model replica (tensor-parallel degree) |
| $F,\ \mathrm{BW}$ | arithmetic throughput in FLOPs per second; memory bandwidth in bytes per second |
| $L,\ \tau$ | bandwidth of the link between GPUs; fixed latency of one collective operation |
| $s_{kv}$ | bytes of cache per token |
| $\rho,\ \lambda,\ \mu$ | server utilisation; arrival rate; service rate |
| $\alpha,\ k,\ \delta$ | speculative decoding: chance a draft token is accepted; drafted tokens per round; cost of one draft token as a fraction of a large-model step |
| $\Delta$ | tokens added to an agent's context by one turn |
| $u^\star$ | break-even utilisation of a self-hosted fleet |

**Boxes.** **Check** states a numerical confirmation; **Question** holds something unsettled; derivations are folded.

## A. The cost of one token

### A1. A request is a prompt, a pause and a stream

Lantern's first user pastes a 2000-token error log. A *token* is a piece of text a little shorter than a word on average. The request runs in two phases.

<figure class="mfig ds-figure ds-wide" id="inf-phases" aria-label="A request as a prompt, a prefill pass, a first token, a decode loop and an answer">
<div class="ds-plot dg-host" data-h="250"></div>
<figcaption><b>A request is a compute-bound prefill, then a memory-bound decode.</b> Hover a box, or press Play.</figcaption>
</figure>

Four numbers describe how a request feels.

| Metric | Definition | What the user feels |
| --- | --- | --- |
| Time to first token (TTFT) | from sending to the first token arriving | the pause before anything appears |
| Time per output token (TPOT), or inter-token latency | gap between consecutive tokens after the first | how smoothly the text streams |
| End-to-end latency | TTFT plus (output length − 1) × TPOT | how long the whole answer takes |
| Throughput | tokens per second the system produces, across all users | what the hardware earns |

An *SLO* is a promise such as "95th-percentile TTFT under two seconds, TPOT under 50 ms"; *goodput* is the throughput of requests that met it. Judge by percentiles: a good mean can hide a bad product.

Day one: 2000 tokens of prefill, 400 of decode. How long, and why?

### A2. Counting FLOPs: two per weight per token

A *FLOP* is one floating-point addition or multiplication. A weight matrix of shape $m\times n$ times a vector takes $2mn$ FLOPs and holds $mn$ weights: *each weight is used once per token, for two FLOPs.* Summed over the model:

$$
\text{FLOPs per token}\;\approx\;2P_{\text{used}}+\underbrace{4\,\ell\,n_q\,d_h\,c}_{\text{attention over } c \text{ cached tokens}},
$$

Here $P_{\text{used}}$ is the number of weights multiplied, $\ell$ the layers, $n_q$ the query heads, $d_h$ the head size and $c$ the context.

<details class="more"><summary>Derivation: the FLOPs of one token, matrix by matrix</summary>

The claim is that a token costs two FLOPs for each weight it touches, plus a term that grows with the context.

**Step 1: the matrices of one layer.** With width $d=8192$, $n_q=64$ query heads, $n_{kv}=8$ key-value heads of size $d_h=128$ and feed-forward width $f=28672$, a layer holds these matrices (rows × columns):

$$
\begin{aligned}
W_q&:\;d\times n_qd_h,&\qquad W_k,W_v&:\;d\times n_{kv}d_h\ \text{each},\\
W_o&:\;n_qd_h\times d,&\qquad W_{\text{gate}},W_{\text{up}}&:\;d\times f\ \text{each},\qquad W_{\text{down}}:\;f\times d.
\end{aligned}
$$

**Step 2: count the entries.** Attention has $8192\cdot8192+2\cdot8192\cdot1024+8192\cdot8192=150{,}994{,}944$ entries. The feed-forward block has $3\cdot8192\cdot28672=704{,}643{,}072$. One layer holds $855{,}638{,}016$ weights, and 80 layers hold $68{,}451{,}041{,}280$.

**Step 3: the two tables.** The input embedding is a table of $128256\times8192=1{,}050{,}673{,}152$ entries, and the output projection (the "head") is a matrix of the same size. Looking up a row of the input table is not arithmetic. The head is a matrix product, so it counts. Total parameters: $68.451+2\times1.0507=70.55$ billion. Weights multiplied per token: $68.451+1.051=69.50$ billion.

**Step 4: the matrix FLOPs.** Two FLOPs per weight used gives $2\times69.50\times10^9=139.0$ GFLOPs per token.

**Step 5: attention against the cache.** The score of a new query against one cached key is a dot product of length $d_h$ in each head: $2d_h$ FLOPs, over $n_q$ heads. The weighted sum of the cached values is the same again. For $c$ cached tokens and $\ell$ layers, $2\cdot2\cdot n_qd_hc\,\ell=4\ell n_qd_hc$. At $c=8192$ this is 21.5 GFLOPs, so a token costs 160.5 GFLOPs there, 15 percent more than at an empty context.

**Step 6: prefill.** Token $i$ of the prompt attends to $i$ cached positions (the prompt tokens before it and itself), so the prompt costs $n\cdot139.0\text{ GFLOPs}+\sum_{i=1}^{n}4\ell n_qd_h\,i=n\cdot139.0\text{ GFLOPs}+2\ell n_qd_h\,n(n+1)$. The second term grows with the square of the prompt.

</details>

**Keep:** inference costs $2P$ FLOPs per token, training about $6P$. Lantern's model costs 139 GFLOPs per generated token; a 2000-token prompt costs 0.28 petaFLOPs. At 0.5 of an H100's 989 teraFLOPS, that is 495 per GPU and 1980 for four.

| | FLOPs | On 4 GPUs at 0.5 of peak |
| --- | --- | --- |
| Prefill of 2000 tokens (arithmetic only) | 0.28 × 10¹⁵ | 143 ms |
| One decode token | 139 × 10⁹ | 0.07 ms |

Arithmetic alone would finish the 400-token answer in 28 ms (the table leaves out the all-reduce, B6). Lantern sees nearly six seconds, two hundred times longer: the arithmetic is not the problem.

<aside class="chk">

The model counts 70,552,387,584 parameters (against a second count of named matrix shapes), 139.0 GFLOPs per token at an empty context and 160.5 at 8192, and the prefill attention sum against an explicit loop.

</aside>

### A3. Counting bytes: the roofline, and why decode waits

Every weight travels from *HBM* (80 GB on an H100, read at 3.35 TB/s) to the arithmetic units each time it is used. Lantern's 141 GB of weights do not fit one GPU, so four share a replica.

<figure class="mfig ds-figure ds-wide" id="inf-replica" aria-label="Four GPUs holding a quarter of every matrix each, joined by an all-reduce">
<div class="ds-plot dg-host" data-h="250"></div>
<figcaption><b>Four GPUs act as one.</b> Each holds a quarter of every matrix; the quarters are added after every layer (B6).</figcaption>
</figure>

A step has two costs, and they overlap in time:

$$
t_{\text{compute}}=\frac{\text{FLOPs}}{F},\qquad t_{\text{memory}}=\frac{\text{bytes moved}}{\mathrm{BW}},\qquad t_{\text{step}}\;\ge\;\max(t_{\text{compute}},\,t_{\text{memory}}).
$$

The larger is the *bound*. What decides it is the *arithmetic intensity* (FLOPs per byte moved) against the machine's *ridge point* $F/\mathrm{BW}$, here $495/2680=184.5$ FLOPs per byte: above it a step is compute-bound, below it memory-bound.

<figure class="mfig ds-figure ds-wide" id="inf-bound" aria-label="Decode at batch 1 against prefill of 2000 tokens: arithmetic intensity against the ridge point">
<div class="ds-plot dg-host" data-h="248"></div>
<figcaption><b>One request, two regimes.</b> The ridge is 184.5 FLOPs per byte.</figcaption>
</figure>

A decode step streams every weight once whatever the batch, so batch-1 decode sits 185 times below the ridge and the arithmetic units wait almost entirely for memory. A 2000-token prompt reuses each weight read 2000 times: compute-bound, 143 ms of arithmetic plus 19 ms of communication, 162 ms on four GPUs.

<details class="more"><summary>Derivation: intensity and the ridge batch</summary>

The claim is that decode intensity equals the batch size when weights take two bytes, and that the crossover batch is the machine's ridge point.

**Step 1: bytes per step.** A decode step streams all matrices once, whatever the batch: $b\,P_{\text{used}}$ bytes, with $b=2$ bytes per weight. (The cache adds more, see A4.)

**Step 2: FLOPs per step.** Each of the $B$ sequences contributes $2P_{\text{used}}$ FLOPs (ignoring attention for now), so $2BP_{\text{used}}$ in all.

**Step 3: the ratio.**

$$
I(B)=\frac{2BP_{\text{used}}}{b\,P_{\text{used}}}=\frac{2B}{b}=B\quad\text{FLOPs per byte for } b=2.
$$

**Step 4: the crossover.** Compute time equals memory time when $I(B)=F/\mathrm{BW}$, that is $B^\star=\frac{b}{2}\cdot\frac{F}{\mathrm{BW}}$, which for $b=2$ is $F/\mathrm{BW}=184.5$.

**Step 5: what it means.** The same weight read serves every sequence in the batch. Doubling $B$ doubles the useful work and leaves the bytes moved unchanged, so up to $B^\star$ a step costs almost the same however many sequences ride in it.

</details>

A decode step costs the time to read the model, and sharing that read is nearly free. Drag the ring to change the batch.

<figure class="mfig ds-figure ds-wide" id="inf-roofline" aria-label="Time of one decode step and tokens per second against batch size, for four GPUs">
<div class="ds-plot"></div>
<figcaption><b>A decode step costs the same to read the model for one sequence or two hundred.</b> Top: step time on four GPUs at a context of 2000 against batch; blue is bytes, copper arithmetic, violet the step including communication; the dashed line is the largest batch the cache allows. Bottom: tokens per second. Drag the ring: 64 sequences take 18.7 ms against 14.3 ms for one, and return 49 times the tokens.</figcaption>
</figure>

- Throughput climbs almost with the batch while the step time barely moves: the economic engine of serving.
- Arithmetic never crosses the memory line before the dashed line: the cache grows with the batch and fills memory first (A4).
- The lines meet later than $B^\star=184.5$, because cache bytes add to the memory side.

**Lantern, day 1 and day 3.** One sequence takes 14.3 ms on four GPUs: 13.0 ms to move 139 GB, 1.3 ms of communication, the 0.07 ms of arithmetic hidden. That is 70 tokens a second and 5.7 s for 400 tokens. At 64 sequences the step takes 18.7 ms and delivers 3419 tokens a second: the bill per token fell 49-fold without buying anything.

<aside class="chk">

One formula, $\max(t_{\text{compute}},t_{\text{memory}})+t_{\text{comm}}$, agrees with an independent recomputation for batches 1 to 256 and 2, 4, 8 GPUs, to $10^{-9}$ relative; the ridge batch is exactly 184.5.

</aside>

### A4. The cache of keys and values, and why memory decides who gets served

Each token's vector is projected to a *query* (what it looks for), a *key* (what it offers) and a *value* (what it hands over). Earlier tokens never change what they offer, so the engine *keeps* keys and values and not queries: the *KV cache*, one key and one value per key-value head, for every layer and earlier token.

<figure class="mfig ds-figure ds-wide" id="inf-kvlayout" aria-label="One layer of a transformer: a new token makes a query that is scored against the cached keys and values of all earlier tokens">
<div class="ds-plot dg-host" data-h="300"></div>
<figcaption><b>The cache keeps what earlier tokens offered, not what they asked.</b></figcaption>
</figure>

For one token the cache grows by

$$
s_{kv}=2\times\ell\times n_{kv}\times d_h\times b_{kv}=2\times80\times8\times128\times2=327{,}680\ \text{bytes}=320\ \text{KiB}.
$$

One conversation of 8192 tokens holds 2.5 GiB; at 128k tokens, 40 GiB. *The cache is the model's working memory, paid per token per user.*

**Grouped-query attention is a cache decision.** 64 query heads share 8 key-value heads, so the cache is eight times smaller than the 2.5 MiB per token of a pair per query head. GQA (Ainslie et al. 2023) reports quality close to full multi-head attention. DeepSeek's multi-head *latent* attention compresses further, storing a latent from which keys and values are rebuilt.

**Memory is a budget.** Four GPUs hold 320 GB: 10 percent reserve, 141.1 GB of weights, 146.9 GB of cache pool, divided into whole conversations:

$$
N_{\max}=\left\lfloor\frac{0.9\,g\,M_{\text{gpu}}-P\,b}{c\;s_{kv}}\right\rfloor.
$$

At 8192 tokens that is 54 conversations on four GPUs and 162 on eight; at 2000, 224; at 128k, three. A full-head cache would give 6 at 8192.

<figure class="mfig ds-figure ds-wide" id="inf-kv" aria-label="Memory of four GPUs divided into weights, reserve and cache, and how many conversations fit, for a chosen context length">
<div class="ds-plot"></div>
<figcaption><b>After the weights, memory is a pool of cache, and the context divides it into conversations.</b> Each bar is the 320 GB of four GPUs: blue weights, grey reserve, violet one conversation each. Top: the grouped-query cache; bottom: a pair for every query head. Drag the ring along the context.</figcaption>
</figure>

**Lantern, day 5.** At 8192 tokens the fifty-fifth user meets "out of memory" while the arithmetic idles. Three remedies, all used: paging (B2); a smaller cache (1-byte values, B5; old cache moved elsewhere, D3); more GPUs.

<aside class="chk">

The pool and the conversation counts (54 and 162 at 8192; 224 and 663 at 2000; 3 and 10 at 131072) agree with an independent recomputation. A 2-GPU replica leaves 2.9 GB after the weights: one conversation of 8192 tokens, or 4 at 2000.

</aside>

### A5. The GPU, in the detail serving needs

<figure class="mfig ds-figure ds-wide" id="inf-hierarchy" aria-label="Four levels of memory from on-chip SRAM to the network, each larger and slower, with what an engine keeps there">
<div class="ds-plot dg-host" data-h="380"></div>
<figcaption><b>An engine moves data down this hierarchy as little as it can.</b></figcaption>
</figure>

| Fact | What it means for serving |
| --- | --- |
| FlashAttention | exact attention computed in tiles that stay in SRAM, so fewer trips to HBM (Dao et al. 2022) |
| Low precision | 8-bit halves the bytes and doubles the tensor rate (3958 teraFLOPS is with sparsity, so 1979 dense): both phases gain |
| MFU and MBU | arithmetic, or bytes, a step needs over what the GPU could do in its time; prefill is judged by MFU, decode by MBU. The 0.5 and 0.8 here are the assumed values; an engine at MBU 0.4 has a factor of two to recover (B8) |
| A newer GPU | an H200 has the H100's arithmetic, 141 GB and about 4.8 TB/s (from memory; check the datasheet): larger $N_{\max}$, lower decode floor, lower ridge |
| Links | NVLink through NVSwitch inside a machine (900 GB/s per GPU, NVIDIA's page), a far slower network between machines (EFA on AWS): keep a replica on one machine |

The machines Lantern could rent (from the pages opened for this note):

| Instance | Accelerators | Memory | Network | Note |
| --- | --- | --- | --- | --- |
| p5.48xlarge | 8 × H100 | 640 GB HBM3 | 3200 Gbps EFA | 900 GB/s GPU interconnect inside |
| p5e / p5en.48xlarge | 8 × H200 | 1128 GB HBM3e (141 GB each) | 3200 Gbps EFA | more memory per GPU |
| trn2.48xlarge | 16 × Trainium2 | 1.5 TB | 3.2 Tbps EFA | 46 TB/s total memory bandwidth, up to 20.8 petaFLOPS in FP8 |

Per Trainium2 chip: 96 GB, about 2.9 TB/s, 1.3 petaFLOPS of FP8. It is programmed through AWS's Neuron software, not CUDA: the main cost of choosing it.

<aside class="q">

Real kernels overlap memory and arithmetic imperfectly, so the true time sits between $\max(t_c,t_m)$ and $t_c+t_m$. How far above the floor do real engines run on Lantern's shape, and does the gap close with batch size? This note computes floors; it does not measure engines.

</aside>

### A6. What Part A leaves Lantern with

A request is a compute-bound prefill plus a memory-bound decode; a decode step costs one read of the model, shared by the batch; the cache grows 320 KiB per token and the memory after the weights sets how many conversations fit. Serving is *keeping the batch large and the cache small, and not letting prefill and decode hurt each other.*

**Exercises.** Each answer is folded.

**1.** A model has 8 billion parameters stored in 16 bits, 32 layers, 8 key-value heads of size 128. On one GPU with 3.35 TB/s and a 0.8 efficiency, what is the batch-1 decode step time, and how many bytes per token is the cache?

<details class="answer"><summary>Worked example: an 8-billion-parameter model</summary>

The weights are $8\times10^9\times2=16$ GB, so the step is $16\times10^9/(0.8\times3.35\times10^{12})=6.0$ ms, about 168 tokens a second. The cache per token is $2\times32\times8\times128\times2=131{,}072$ bytes, 128 KiB. A smaller model reads fewer bytes per token, which is the first reason small models are fast and cheap.

</details>

**2.** At what context length does the cache of one conversation exceed the weights, for Lantern's model?

<details class="answer"><summary>Worked example: when the cache outweighs the model</summary>

Weights are 141.1 GB and the cache is 327,680 bytes per token, so the crossing is at $141.1\times10^9/327{,}680\approx430{,}000$ tokens. Beyond that, one conversation's cache is larger than the model. The more useful comparison is at a batch: a batch of 64 conversations of 2000 tokens reads $64\times2000\times327{,}680=41.9$ GB of cache per step against 139 GB of weights, so the cache is 23 percent of the bytes moved.

</details>

## B. Serving many users: the engine

An *inference engine* (vLLM, SGLang, TensorRT-LLM; C1) turns requests that arrive at random times with unknown lengths into *steps*, each one pass of the model over some tokens.

<figure class="mfig ds-figure ds-wide" id="inf-engine" aria-label="The engine loop: waiting requests, a scheduler with a block manager, a forward pass and tokens out, with finished requests leaving and new ones joining">
<div class="ds-plot dg-host" data-h="330"></div>
<figcaption><b>An engine is one loop.</b> Every step: who rides, does the cache fit, one forward pass, one token each. Press Play to build it.</figcaption>
</figure>

### B1. Continuous batching: do not wait for the slowest

*Static batching* takes $B$ requests and runs them until all finish. A request that wants 12 tokens idles until the one that wants 500 is done, and a step costs the same full or half-empty (A3): every empty slot is throughput lost.

*Continuous batching* (Orca, Yu et al. 2022; iteration-level scheduling) decides membership at every step: a finished sequence's slot is refilled at the next iteration, without waiting for the batch to drain.

Counted exactly: 512 requests with output lengths from a fixed generator (shortest 8, mean 182.7, longest 516), every step the same time.

| Slots $B$ | Static: steps | Continuous: steps | Ratio | Slots busy, static | Slots busy, continuous |
| --- | --- | --- | --- | --- | --- |
| 8 | 27,023 | 11,880 | 2.27 | 43% | 98% |
| 16 | 14,894 | 6,138 | 2.43 | 39% | 95% |
| 32 | 7,863 | 3,236 | 2.43 | 37% | 90% |
| 64 | 4,048 | 1,771 | 2.29 | 36% | 83% |

With 32 slots static batching keeps 37 percent of the slots busy and continuous 90: 2.4 times the tokens per second. The ratio is a property of the *length distribution*: the more lengths vary, the more static wastes.

<figure class="mfig ds-figure ds-wide" id="inf-batching" aria-label="Slots over time under static and continuous batching for a chosen number of slots">
<div class="ds-plot"></div>
<figcaption><b>Static batching leaves slots empty while the longest request finishes; continuous batching refills them at once.</b> Rows are slots, time runs right in decode steps, a violet block is one request. The first 300 steps are drawn; the percentages are for all 512 requests. Drag the ring to change the slots.</figcaption>
</figure>

<details class="more"><summary>Derivation: why the gain grows with the spread of lengths</summary>

The claim is that static batching spends, per group, as many steps as the longest request, while continuous batching spends, in the long run, only the average length per slot.

**Step 1: static batching.** A group of $B$ requests with lengths $m_1,\dots,m_B$ occupies the $B$ slots for $\max_im_i$ steps, and does $\sum_im_i$ steps of useful work. The share of slot-steps that are useful is $\sum_im_i/(B\max_im_i)$.

**Step 2: the average of the share.** If the lengths were all equal, the share would be 1. For lengths with mean $\bar m$ the share is about $\bar m/E[\max_im_i]$, and the expected maximum of $B$ draws grows with $B$ for any distribution that has a tail. For the generator used here the mean is 182.7 and the longest of 512 is 516, so a group that contains the longest request uses about $182.7/516=35$ percent of its slot-steps, and the average over all groups at $B=32$ is the measured 37 percent.

**Step 3: continuous batching.** A slot is idle only when the queue is empty. With 512 requests and $B$ slots, the total work $\sum m=93{,}526$ slot-steps is divided over $B$ slots, so the run takes at least $\lceil93{,}526/B\rceil$ steps (2,923 for $B=32$), and the measured 3,236 is 11 percent above that bound, the price of the tail at the end of the run when fewer than $B$ requests remain.

**Step 4: the ratio.** The ratio of the two run lengths is therefore about $E[\max]/\bar m$, corrected at both ends by the tail: static batching also wastes the last partial group, and continuous batching wastes the slots that idle at the end of the run. The two corrections grow with $B$, which is why the measured ratio is flat (2.27, 2.43, 2.43, 2.29) and not rising for $B$ between 8 and 64.

</details>

<aside class="chk">

The step counts agree with a second simulation written as an event queue; the drawn schedules have no overlap within a slot.

</aside>

Limits: this counts steps of equal time; a step that also prefills takes longer (B3). Orca reports 36.9 times the throughput of FasterTransformer on a 175-billion-parameter model, combining this effect with its other changes.

### B2. Paging the cache: stop reserving for the worst case

The naive cache is one contiguous block per request, sized for the longest answer it *might* give: 4096 tokens whether it uses 9 or 4000. On the 512 requests above 93,526 of $512\times4096=2{,}097{,}152$ token-slots are used: **95.5 percent of the reservation is empty**, about twenty times the concurrent users lost (A4).

The cure, from vLLM (Kwon et al., SOSP 2023), is borrowed from operating systems: fixed *blocks* of, say, 16 tokens, given to a request as it grows, anywhere in memory, with a per-request table. This is *PagedAttention*. The abstract reports near-zero waste and 2 to 4 times the throughput of FasterTransformer and Orca.

<figure class="mfig ds-figure ds-wide" id="inf-paging" aria-label="Contiguous reservation against paged blocks: the empty share of the cache for each, and two requests sharing one block of prefix">
<div class="ds-plot dg-host" data-h="330"></div>
<figcaption><b>Blocks turn a worst-case reservation into a table.</b></figcaption>
</figure>

What is left is about half a block per request, set by the block size.

<figure class="mfig ds-figure ds-wide" id="inf-paged" aria-label="Memory reserved against memory used for 512 requests, for contiguous reservation and for paging with a chosen block size">
<div class="ds-plot"></div>
<figcaption><b>Reserving for the longest answer wastes 95 percent of the cache; blocks of 16 tokens waste 4.</b> Top: twenty of the 512 requests; dashed is the reservation, violet what was used, copper the unfilled tail of the last block. Bottom: the empty share over all 512. Drag the ring to change the block size from 1 to 256.</figcaption>
</figure>

Paging brings two more things, both important later.

**Sharing.** Requests that begin with the same text have identical keys and values for it, so they can *point at the same blocks*: *prefix caching*. SGLang's *RadixAttention* keeps all cached blocks in a tree keyed by token sequence (abstract: up to 6.4 times the throughput on structured workloads). APIs sell it as *prompt caching* (C2); for agents it is the most valuable feature (D2).

<figure class="mfig ds-figure ds-wide" id="inf-radix" aria-label="A tree of cached blocks: a shared system prompt and tools, then a branch per conversation">
<div class="ds-plot dg-host" data-h="286"></div>
<figcaption><b>Prefix caching is a tree.</b> A new request walks the tree to its longest cached prefix and computes only the rest.</figcaption>
</figure>

**Preemption.** When memory runs out an engine evicts a request's blocks and recomputes them later, or copies them to CPU memory. A block table makes both cheap.

<details class="more"><summary>Derivation: the waste of paging is about half a block per request</summary>

The claim is that with blocks of $\beta$ tokens and request lengths spread over many values, the expected empty space per request is close to $(\beta-1)/2$ tokens.

**Step 1: the blocks used.** A request of $m$ tokens needs $\lceil m/\beta\rceil$ blocks, so it holds $\beta\lceil m/\beta\rceil$ token-slots and wastes $\beta\lceil m/\beta\rceil-m$.

**Step 2: the remainder.** Write $m=q\beta+r$ with $0\le r<\beta$. If $r=0$ the waste is 0; otherwise it is $\beta-r$, which takes each value $1,\dots,\beta-1$ when $r$ takes each value $1,\dots,\beta-1$.

**Step 3: average over remainders.** If $r$ is equally likely among the $\beta$ values, the mean waste is $\frac1\beta\sum_{j=1}^{\beta-1}j=\frac{\beta-1}{2}$ tokens.

**Step 4: compare.** For $\beta=16$ the mean is 7.5 tokens per request, against a mean length of 182.7: about 4 percent, which is the measured 3.97 percent. For $\beta=256$ it is 127.5 tokens, 70 percent of the mean length, and the measured waste, 46 percent of the allocated memory, is a little above the 41 percent that 127.5 wasted tokens per 182.7 used would give, because many requests are shorter than one block and waste most of it.

</details>

Small blocks cost gather speed and table space; engines default to about 16 tokens.

<aside class="chk">

Allocated memory for blocks of 1, 16, 64 and 256 tokens is 93,526, 97,392, 111,168 and 173,568 token-slots; the empty fractions are 0, 4.0, 15.9 and 46.1 percent against 95.5 for the 4096 reservation.

</aside>

### B3. When a long prompt arrives, everyone else freezes

Thirty-two users stream a token every 18 ms when one pastes an 8000-token document. Prefilled in one step it takes 678 ms on four GPUs (derivation below), 37 times a normal 18.4 ms step, and none of the 32 gets a token. The mean looks fine; the percentile does not.

<figure class="mfig ds-figure ds-wide" id="inf-collision" aria-label="One 8000-token prefill step that stalls 32 decoders, against chunks of 512 that ride along with the decode steps">
<div class="ds-plot dg-host" data-h="270"></div>
<figcaption><b>A big prefill freezes everyone; chunks keep the stream moving.</b> Four GPUs, 32 decoders at 4000 tokens of context.</figcaption>
</figure>

**Chunked prefill** (SARATHI, Agrawal et al. 2023) processes at most $C$ prompt tokens per step *alongside* the decodes. Decode is memory-bound (A3), so up to about 128 tokens are nearly free and a few hundred stretch the step 1.4 to 2.6 times.

Lantern: 32 decoders at 4000 tokens of context and an 8000-token prompt.

| Chunk $C$ | Step time | Steps for the prompt | TTFT of the new prompt |
| --- | --- | --- | --- |
| none (one step) | 678 ms | 1 | 678 ms |
| 2048 | 177 ms | 4 | 707 ms |
| 512 | 47 ms | 16 | 754 ms |
| 256 | 26 ms | 32 | 818 ms |
| 128 | 19.6 ms | 63 | 1232 ms |

A chunk of 512 holds the running users' pause to 47 ms instead of 678 and delays the newcomer's first token by about 10 percent. A chunk of 128 is almost free for the others (19.6 ms) but the newcomer waits 1232 ms. The chunk size is a dial between the two groups; the figure moves it.

<figure class="mfig ds-figure ds-wide" id="inf-chunk" aria-label="Pause seen by running users and time to first token of a new 8000-token prompt, for a chosen chunk size">
<div class="ds-plot"></div>
<figcaption><b>The chunk size trades the pause of running users against the wait of the new one.</b> Top: the gap a running user sees. Bottom: time to the new prompt's first token. Dashed: no chunking (678 ms each); a normal step is 18.4 ms. Drag the ring from 128 tokens to the whole prompt.</figcaption>
</figure>

**Disaggregation** takes the opposite stance: *prefill and decode run on different GPUs*, and a prompt's cache is sent over the network between them.

<figure class="mfig ds-figure ds-wide" id="inf-disagg" aria-label="A router sending a prompt to a prefill pool, the KV cache moving to a decode pool, and tokens streaming out">
<div class="ds-plot dg-host" data-h="200"></div>
<figcaption><b>Disaggregation separates the phases and pays in transfer.</b> Each pool is sized for its own bottleneck; every prompt's cache crosses the network once.</figcaption>
</figure>

<details class="more"><summary>What three papers report</summary>

- **DistServe** (Zhong et al., OSDI 2024): colocating the phases causes interference and couples their parallelism; it chooses each phase's parallelism from the TTFT and TPOT targets and reports 7.4 times more requests, or a 12.6 times tighter SLO, in its own evaluations.
- **Splitwise** (Patel et al., 2023): token generation does not need the newest GPUs; the abstract reports 1.4 times the throughput at 20 percent lower cost, or 2.35 times at equal cost and power.
- **Mooncake** (Qin et al., 2024), the Kimi service: separate clusters, CPU memory and SSDs pooled as cache, early rejection when overloaded; up to 525 percent more throughput in some simulations, 75 percent more requests in real workloads.

</details>

Each design has a price.

| | Chunked prefill | Disaggregation |
| --- | --- | --- |
| Idea | mix, but in small pieces | separate the phases onto different GPUs |
| Cost | the new prompt is slower; one engine must serve both well | the cache must cross the network after every prefill |
| Pays off when | prompts are moderate and one pool of GPUs is simplest | prompts are long, SLOs on TTFT and TPOT are both tight, and the network between pools is fast |
| Failure mode | a chunk too big stalls decoders; too small starves prompts | a slow link makes the prefill-to-decode handoff the new bottleneck |

The cache in flight is not small: Lantern's 8000-token prompt leaves 2.4 GiB to ship (A4). A four-GPU replica on a p5 has 200 GB/s of network, 13 ms at line rate and several times that in practice: about two percent of a 700 ms TTFT at best.

<details class="more"><summary>Derivation: the time of a step that carries a prefill chunk</summary>

The claim is that a step with $B$ decoders and a chunk of $C$ prompt tokens takes $\max(t_{\text{compute}},t_{\text{memory}})+t_{\text{comm}}$ with the arithmetic of $B+C$ tokens and the memory traffic of the decoders alone.

**Step 1: the arithmetic.** Every token that goes through the model, decoder or chunk, costs 139.0 GFLOPs for the matrices plus $4\ell n_qd_h$ FLOPs per cached position it attends to. The step has $B+C$ tokens. For the table the attention of every token is evaluated at a context of 4000 tokens, which is the middle of the 8000-token prompt and also the decoders' context; this is the convention, and a chunk at the start of the prompt is a little cheaper and one at the end a little dearer.

**Step 2: the bytes.** The weights are read once for the whole step, 139.0 GB, and the decoders' caches once each, $B\,c\,s_{kv}$. The chunk's own keys and values are produced in the step, and the model counts no extra reads for them.

**Step 3: the time.** $t_{\text{compute}}=(B+C)\,\text{FLOPs per token}/(g\,F_{\text{eff}})$ and $t_{\text{memory}}=\text{bytes}/(g\,\mathrm{BW}_{\text{eff}})$. The all-reduces of B6 carry $B+C$ activations. For $B=32$, $c=4000$, $C=512$: the 544 tokens need 81.3 TFLOPs (149.5 GFLOPs each at this context) and take $81.3/1978=41.1$ ms; the memory traffic is $139.0+41.9=180.9$ GB and takes $180.9/10.72=16.9$ ms; communication takes 6.0 ms. The step is 47.1 ms: compute-bound, because the chunk is large enough to cross the ridge.

**Step 4: the prompt.** An 8000-token prompt takes $\lceil8000/C\rceil$ steps, and its first token appears at the end of the last one.

</details>

<aside class="chk">

Step times for chunk sizes 128, 512, 2048 and 8000: 19.6, 47.1, 176.7 and 678.5 ms.

</aside>

### B4. Scheduling and queueing: who waits, and how long

At every step the engine decides whom to admit, how many prompt tokens to process and whom to evict. The arithmetic behind these *scheduling* decisions is queueing theory.

**Utilisation is the lever, and the curve bends sharply.** With arrivals at rate $\lambda$ and service at rate $\mu$, utilisation is $\rho=\lambda/\mu$. In the simplest queue (M/M/1) the mean time in the system is $\frac{1}{\mu}\cdot\frac{1}{1-\rho}$, and the 95th percentile is $\ln20=3.0$ times the mean.

<figure class="mfig ds-figure ds-wide" id="inf-queue" aria-label="Mean time in the system as a multiple of the service time, at four utilisations">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>The last tenth of utilisation costs the most.</b> Mean time in system, in service times.</figcaption>
</figure>

<details class="more"><summary>Derivation: the mean time in an M/M/1 queue</summary>

The claim is that with Poisson arrivals at rate $\lambda$ and exponential service at rate $\mu$ with $\rho=\lambda/\mu<1$, the mean time in the system is $1/(\mu-\lambda)$.

**Step 1: the balance.** Let $\pi_n$ be the long-run chance that $n$ requests are in the system. In steady state the rate of moving from $n$ to $n+1$ equals the rate of moving back: $\lambda\pi_n=\mu\pi_{n+1}$.

**Step 2: the distribution.** Hence $\pi_n=\rho^n\pi_0$, and since the $\pi_n$ sum to 1, $\pi_0=1-\rho$ and $\pi_n=(1-\rho)\rho^n$.

**Step 3: the mean number.** $E[N]=\sum_nn(1-\rho)\rho^n=\rho/(1-\rho)$.

**Step 4: Little's law.** The mean number in a stable system equals the arrival rate times the mean time, $E[N]=\lambda E[T]$. So $E[T]=\frac{\rho}{\lambda(1-\rho)}=\frac{1}{\mu(1-\rho)}=\frac{1}{\mu-\lambda}$. $\square$

</details>

LLM serving is not M/M/1 (many requests are served at once, times are far from exponential), but the *shape* survives: delay explodes near full, so a TTFT SLO needs headroom (this note uses 65 percent). Batching softens it, since a busier engine is more efficient per token (A3).

**What a scheduler chooses.**

<figure class="mfig ds-figure ds-wide" id="inf-scheduler" aria-label="A scheduler choosing among admission, order, preemption and load shedding">
<div class="ds-plot dg-host" data-h="300"></div>
<figcaption><b>Four decisions, one loop.</b> Admission and order are everyday; preemption and shedding fire under pressure. Priorities let interactive traffic jump the queue and starve the rest.</figcaption>
</figure>

Admission takes a request only if its cache fits; preemption pauses one. Across replicas a request goes to the longest matching cache unless that replica is overloaded: how a fleet turns prefix caching into savings (D2).

**Open-loop testing.** A load test that sends the next request when the last returns slows as the system saturates. Real users do not wait for each other: send at a fixed rate and watch the percentiles pass the knee.

<figure class="mfig ds-figure ds-wide" id="inf-openloop" aria-label="A closed-loop test that slows down as the server saturates, against an open-loop test at a fixed rate">
<div class="ds-plot dg-host" data-h="248"></div>
<figcaption><b>Only an open-loop test finds the knee.</b></figcaption>
</figure>

### B5. Making each token cheaper

Three families of tricks attack the bytes, the number of steps and the overheads.

**Quantisation: fewer bytes per number.** Weights in 8 or 4 bits halve or quarter the bytes a step reads. *Weight-only* schemes (GPTQ, AWQ, LLM.int8(): unverified here) expand to 16 bits in the arithmetic units; FP8 *weight-and-activation* schemes also use the faster mode; *KV-cache* quantisation shrinks the cache.

<figure class="mfig ds-figure ds-wide" id="inf-quant" aria-label="Weight storage at 16, 8 and 4 bits, with the bytes, the batch-1 step and the batch-64 throughput of each">
<div class="ds-plot dg-host" data-h="290"></div>
<figcaption><b>Half the bytes, nearly half the step.</b> Four GPUs, weight-only, context 2000.</figcaption>
</figure>

For Lantern, four GPUs, context 2000 (weight-only: arithmetic, activations and all-reduces stay 16-bit):

| Storage | Weights | Step, batch 1 | Tokens/s, batch 1 | Step, batch 64 | Tokens/s, batch 64 | Conversations at 8192, 4 GPUs |
| --- | --- | --- | --- | --- | --- | --- |
| 16-bit weights and cache | 141.1 GB | 14.3 ms | 70 | 18.7 ms | 3419 | 54 |
| 8-bit weights | 70.6 GB | 7.8 ms | 128 | 12.2 ms | 5231 | 81 |
| 4-bit weights | 35.3 GB | 4.6 ms | 218 | 9.0 ms | 7116 | 94 |
| 8-bit cache | 141.1 GB | 14.3 ms | 70 | 16.8 ms | 3818 | 109 |
| 8-bit weights and cache | 70.6 GB | 7.8 ms | 128 | 10.3 ms | 6226 | 162 |

Weights set the step time at small batches; the cache sets how many conversations fit and the step at large batches. Both at 8 bits: 162 conversations at 8192 against 54, and 1.8 times the tokens per second at batch 64. The price is accuracy: run Lantern's own evaluations, since a benchmark loss is not a promise about a product.

**Speculative decoding: more tokens per run of the big model.** A memory-bound step that checks *several* tokens costs almost the same as one that makes one. A small *draft* model guesses $k$ tokens and the large model scores all in one step (Leviathan et al., 2022; Chen et al., 2023, unverified here).

<figure class="mfig ds-figure ds-wide" id="inf-spec-seq" aria-label="A draft model proposing four tokens, the large model checking them in one step, and a prefix of them being kept">
<div class="ds-plot dg-host" data-h="330"></div>
<figcaption><b>The large model verifies; it does not generate.</b> A guess is kept with probability $\min(1,p/q)$; the first rejected one is replaced and the round ends. The output has exactly the large model's distribution.</figcaption>
</figure>

If a draft token is accepted with probability $\alpha$, a round yields $\frac{1-\alpha^{k+1}}{1-\alpha}$ tokens; if a draft token costs a fraction $\delta$ of a large-model step, a round costs $1+\delta k$, so

$$
\text{speedup}(\alpha,k,\delta)=\frac{1-\alpha^{k+1}}{(1-\alpha)(1+\delta k)}.
$$

<details class="more"><summary>Derivation: expected tokens per round of speculative decoding</summary>

The claim is that a round yields on average $1+\alpha+\alpha^2+\dots+\alpha^k$ tokens.

**Step 1: the structure of a round.** The large model checks the $k$ drafts in order. Draft $i$ is kept if all earlier drafts were kept and draft $i$ itself passes the test (probability $\alpha$). The first rejected draft is replaced by a token drawn from the large model's leftover probability, and if all $k$ pass, the large model's pass also yields one extra token from the position after the last draft. Either way the round ends with one token the large model produced itself.

**Step 2: the number of tokens.** If the first $j$ drafts pass and the next fails, the round yields $j+1$ tokens. This happens with probability $\alpha^j(1-\alpha)$ for $j<k$, and $\alpha^k$ for $j=k$ (all pass, and the bonus token follows).

**Step 3: the mean.** $E=\sum_{j=0}^{k-1}\alpha^j(1-\alpha)(j+1)+\alpha^k(k+1)$.

**Step 4: simplify with a telescoping sum.** Using $P(\text{at least }i\text{ drafts pass})=\alpha^i$, the number of tokens equals $1+\sum_{i=1}^{k}\mathbf 1[\text{at least }i\text{ pass}]$, so $E=1+\sum_{i=1}^k\alpha^i=\frac{1-\alpha^{k+1}}{1-\alpha}$. $\square$

**Step 5: the cost.** A round runs the draft model $k$ times at cost $\delta$ each and the large model once at cost 1, so it costs $1+\delta k$ in units of a large-model step. The tokens per unit cost, relative to the 1 token per unit of ordinary decoding, is the speedup above.

</details>

At $\alpha=0.8$ and $\delta=0.05$ (about 0.7 ms per token; the draft must fit one GPU, since split over four it pays B6's 1.3 ms per token), $k=4$ yields 3.36 tokens per round and a speedup of 2.8; the best $k$ is 8, with 3.1. A worse draft ($\alpha=0.6$) or a slower one ($\delta=0.2$) peaks at 1.9.

<figure class="mfig ds-figure ds-wide" id="inf-spec" aria-label="Speedup of speculative decoding against the number of drafted tokens for three acceptance rates">
<div class="ds-plot"></div>
<figcaption><b>Drafting more helps until the drafts stop being accepted.</b> Speedup against $k$ for a draft costing 5 percent of a step, at three acceptance rates; the dot is the best $k$. Drag the ring to read tokens per round and the speedup at $\alpha=0.8$.</figcaption>
</figure>

- At large batches the check of $k$ extra tokens is no longer free: speculation helps *latency at small and moderate batch* most.
- Acceptance depends on the text: boilerplate often, creative text less.
- Variants: extra heads (Medusa), a light head on the large model's features (EAGLE), n-gram matches from the prompt; all unverified here.

**Overheads.** *Kernel fusion* keeps intermediates out of HBM (FlashAttention is the famous case); *CUDA graphs* record a step as one launch. Both close part of the gap between this note's floor and a measured engine.

### B6. Parallelism: when one GPU is not enough, and when more is not better

Lantern's 141 GB of weights need two GPUs to exist and four to leave room for a cache. There are four ways to spread a model.

<figure class="mfig ds-figure ds-wide" id="inf-layouts" aria-label="Four ways to spread a model: tensor, pipeline, replicas and expert parallelism, with what flows between GPUs">
<div class="ds-plot dg-host" data-h="380"></div>
<figcaption><b>Four layouts differ in what is split and what crosses between GPUs.</b></figcaption>
</figure>

**Tensor parallelism and its cost.** Each of $g$ GPUs holds $1/g$ of every matrix, so the step floor falls as $1/g$. After attention and after the feed-forward block of every layer the pieces are added, an *all-reduce*: $2\ell=160$ per step, each a small message with a fixed latency that does not fall with $g$.

<details class="more"><summary>Derivation: the cost of the all-reduces in one step</summary>

The claim is that the communication of tensor parallelism per step is $2\ell\left[\frac{2(g-1)}{g}\cdot\frac{B\,d\,b}{L}+\tau\right]$, where $L$ is the link bandwidth and $\tau$ the fixed latency of one collective.

**Step 1: the message.** After a layer's attention each sequence has one activation vector of $d$ numbers of $b$ bytes, partial on each GPU, to be summed. For $B$ sequences the message is $Bdb$ bytes.

**Step 2: a ring all-reduce.** In a ring, each GPU sends and receives $\frac{g-1}{g}$ of the message twice (once to sum, once to share), so $\frac{2(g-1)}{g}Bdb$ bytes pass through each GPU's link, and the time is that divided by the bandwidth $L$, plus the latency $\tau$.

**Step 3: how many per step.** Two per layer (after attention, after the feed-forward block), $2\ell$ in all.

**Step 4: the numbers.** With $d=8192$, $b=2$, $\ell=80$, $L=450$ GB/s per direction (assumed) and $\tau=8\ \mu$s (assumed), a step with $B=1$ at $g=4$ pays $160\times\left[\frac{6}{4}\cdot\frac{16384}{450\times10^9}+8\times10^{-6}\right]=1.29$ ms, which is almost entirely the latency term: the message is 16 KB.

</details>

Communication is a fixed 1.3 ms while memory time shrinks as $1/g$. Lantern at a context of 2000:

| GPUs per replica | Step, batch 1 | Of which communication | Most conversations that fit | Tokens/s per GPU at the largest batch up to 64 |
| --- | --- | --- | --- | --- |
| 2 | 27.3 ms | 1.3 ms | 4 | 72 |
| 4 | 14.3 ms | 1.3 ms | 224 | 855 |
| 8 | 7.8 ms | 1.3 ms | 663 | 771 |

- 4 to 8 GPUs cuts batch-1 latency by 45 percent (14.3 to 7.8 ms) but *lowers* throughput per GPU at batch 64 (855 to 771 tokens/s): the fixed communication is a bigger share.
- Two GPUs cannot batch: four conversations of 2000 tokens, under a tenth of the four-GPU throughput per GPU.
- *More GPUs per replica buy latency and cache space, and cost throughput per GPU.* Use the smallest degree that fits the model with a useful cache and meets latency; scale throughput with *replicas*.

<figure class="mfig ds-figure ds-wide" id="inf-par" aria-label="Step time at batch 1, throughput per GPU and conversations that fit for a chosen number of GPUs per replica">
<div class="ds-plot"></div>
<figcaption><b>More GPUs per replica make one user faster and each GPU less productive.</b> Context 2000. Left: batch-1 step, memory (blue) and fixed communication (copper). Middle: tokens per second per GPU at the largest batch up to 64. Right: conversations that fit. Drag the ring from 2 to 8 GPUs.</figcaption>
</figure>

**Pipeline parallelism** sends one activation per stage boundary but leaves GPUs idle while the first token crosses (the *bubble*) unless micro-batches overlap. It matters when a replica spans machines, whose network is far slower than the interconnect inside one.

**Mixture of experts.** The feed-forward block becomes many *experts*, and each token goes to a few. DeepSeek-V3's abstract: 671 billion parameters, 37 billion activated per token. Memory follows the total, compute the active ones.

<figure class="mfig ds-figure ds-wide" id="inf-moe" aria-label="A router sending each token to two of eight experts: all experts stored, two computed">
<div class="ds-plot dg-host" data-h="360"></div>
<figcaption><b>Memory follows the stored experts; compute follows the chosen ones.</b></figcaption>
</figure>

- *Memory:* 671 billion parameters in 8 bits is 671 GB. It does not fit a p5.48xlarge (640 GB) but fits a p5e (1128 GB), leaving 457 GB for cache.
- *Compute:* 37 billion active, about 74 GFLOPs per token, half of Lantern's dense model.
- *Bytes per step:* one token reads its experts, but a large batch touches nearly all, so batching helps even more.
- *Communication:* experts on different GPUs need an all-to-all each layer; the network matters more.

Tensor parallelism is capped by the 8 key-value heads: beyond 8 GPUs a head is copied, wasting cache. **Sequence and context parallelism** split one very long sequence across GPUs (search terms in G).

<aside class="chk">

Step times at batch 1 for 2, 4, 8 GPUs (27.34, 14.32, 7.80 ms, 1.29 ms of communication each) and throughput per GPU (72, 855, 771 tokens/s) agree with the reference to $10^{-9}$. The quantised rows of B5 use the same formula with the width changed.

</aside>

### B7. Reading a real benchmark

Everything above is a floor; a measurement is what an engine really does, and the gap is where the engineering is.

<figure class="mfig ds-figure ds-wide" id="inf-bench" aria-label="Measure at fixed rates, convert to utilisation, find where the gap went, profile">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>A benchmark is a loop, not a number.</b> The figure of merit is the highest rate at which the SLO holds.</figcaption>
</figure>

**Measure.** Match the product's length distribution, send open loop at a series of rates (B4), discard a warm-up, record TTFT, TPOT and tokens per second at p50, p95 and p99 with errors. Tools, named from memory: vLLM's scripts, GenAI-Perf, GuideLLM.

**Read.** At batch 1, MBU is bytes per step over $t\cdot g\cdot\mathrm{BW}$. An engine at 25 ms per step on four GPUs moves 139.7 GB: MBU 0.42 against the 0.8 assumed, so half the floor is on the table. Where it went:

- unfused kernels, one launch per operation: idle gaps (B5)
- a collective that does not overlap: the 1.3 ms of B6
- long prompts: the pauses of B3
- wasted blocks: a smaller batch than A4 predicts

**Profile before guessing** (PyTorch's profiler, Nsight Systems; from memory). The floor says which resource to blame, the most you could gain, and when to stop: MBU 0.8 on decode and MFU 0.5 on prefill is as good as this model says it can be.

### B8. What Part B leaves Lantern with

By month three Lantern runs vLLM-style engines on 4-GPU replicas with continuous batching, paged cache, chunked prefill at 512, 8-bit weights and cache, and a router in front.

<figure class="mfig ds-figure ds-wide" id="inf-lantern-b" aria-label="Lantern on day 5 against month 3: conversations at 8192, step time and throughput at batch 64">
<div class="ds-plot dg-host" data-h="358"></div>
<figcaption><b>Part B, in Lantern's numbers.</b> Floor model, four GPUs.</figcaption>
</figure>

Batching is the economic engine, the cache its limit, paging removes the waste. Chunking softens the prefill-decode collision, disaggregation removes it, each at a price. Quantise (and verify), speculate while memory-bound, buy latency with parallelism, scale throughput with replicas.

**Exercises.** Each answer is folded.

**1.** A queueing server runs at utilisation 0.9 with a mean time in the system of 1.0 s. What is its 95th percentile, and what utilisation keeps the 95th percentile under 2 s?

<details class="answer"><summary>Worked example: the knee of the queue</summary>

The mean is $\frac{1}{\mu}\frac{1}{1-\rho}=10$ service times, so a service time is 0.1 s. The 95th percentile is $\ln20=3.0$ times the mean: 3.0 s. To keep it under 2 s we need $\frac{0.1\times3.0}{1-\rho}\le2$, that is $1-\rho\ge0.15$, $\rho\le0.85$. Backing off from 0.9 to 0.85 cuts the tail from three seconds to two, which is why capacity planners watch the last ten percent.

</details>

**2.** A draft model has acceptance rate 0.7 and costs 10 percent of a step. Which of $k=2,3,4,5$ is best?

<details class="answer"><summary>Worked example: choosing the number of drafts</summary>

The speedup is $\frac{1-0.7^{k+1}}{0.3(1+0.1k)}$. For $k=2$: $(1-0.343)/(0.3\times1.2)=1.82$. For $k=3$: $(1-0.2401)/(0.3\times1.3)=1.95$. For $k=4$: $(1-0.168)/(0.3\times1.4)=1.98$. For $k=5$: $(1-0.1176)/(0.3\times1.5)=1.96$. So $k=4$ is best, at about 1.98 times.

</details>

## C. Frameworks and the big providers

Who implements Part B's ideas, and what the labs make public. Product statements come from the reading-list pages as of 3 November 2026; the field changes in months.

### C1. The engines

An engine implements Part B's loop on top of kernels, with an API server. The ones to know:

| Engine | What it is | Worth knowing |
| --- | --- | --- |
| vLLM | open-source inference and serving library, from UC Berkeley's Sky Computing Lab | PagedAttention, continuous batching, chunked prefill, prefix caching; FP8, INT8, INT4, GPTQ, AWQ, GGUF; EAGLE speculation; tensor, pipeline, data, expert, context parallelism; OpenAI-compatible server; NVIDIA, AMD, Intel, CPUs |
| SGLang | a frontend language plus a runtime (the paper); RadixAttention | built around cache reuse across calls: a natural fit for agents (an inference from its design, not a benchmark); abstract: up to 6.4 times the throughput of other systems |
| TensorRT-LLM | NVIDIA's open-source library for NVIDIA GPUs | custom attention, matrix and MoE kernels; disaggregation, wide expert parallelism, speculation; FP8 and FP4 |
| NVIDIA Dynamo | orchestration above engines | disaggregated pools, routing by load and cache overlap, cache offload to CPU, SSD or remote; works with SGLang, TensorRT-LLM, vLLM (support differs; check its README) |
| llm-d | distributed serving stack for Kubernetes, a CNCF sandbox project | cache- and load-aware routing, disaggregation, tiered offload, SLO-aware autoscaling |
| llama.cpp, Ollama, MLC | local and small-scale engines | unverified here: laptops and single machines, not fleets |
| Together AI, Fireworks, Baseten, Modal, Groq, Cerebras | hosted platforms that run open-weights models | named from memory, not examined; between a lab's API and your own fleet (F2) |

<figure class="mfig ds-figure ds-wide" id="inf-layers" aria-label="Three layers: an orchestration layer above engines, engines above kernels">
<div class="ds-plot dg-host" data-h="444"></div>
<figcaption><b>The layers are separating, and the work is moving from the kernel to the fleet.</b> Every serious engine has the same feature list: the answers to Part B.</figcaption>
</figure>

An engine runs one replica well; the layer above decides *which* replica gets a request, how many prefill and decode workers to run and where cache lives.

<figure class="mfig ds-figure ds-wide" id="inf-path" aria-label="The usual path: provider API, then a hosted platform or Bedrock, then your own fleet for the steady core">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>The usual path of a start-up.</b> Lantern is at the first step on day one; by month five its agent product has the volume and prefix reuse that make the third worth costing (F2).</figcaption>
</figure>

**Choosing an engine.** A defensible default is vLLM or SGLang behind an OpenAI-compatible API, so a later switch is configuration; measure Lantern's real traffic on both; look at TensorRT-LLM when ten percent on NVIDIA pays for the engineering. A recommendation, not a measurement.

### C2. What the biggest providers say in public

<figure class="mfig ds-figure ds-wide" id="inf-public" aria-label="Four kinds of public evidence: the API surface, postmortems, open-weights reports, and inference">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>What is public falls into four kinds.</b> Solid fill is stated by the source; dashed is inferred.</figcaption>
</figure>

**1. The API surface exposes the architecture.** A price list is a cost model made visible.

*Prompt caching.* The two documents read for this note describe the same idea with different knobs.

| | Anthropic | OpenAI |
| --- | --- | --- |
| How it is switched on | a top-level marker, or markers on blocks in tools, system or messages (up to four) | automatic, with an explicit mode on the newest models |
| What is cached | the whole prefix, in the order tools, then system, then messages, up to the marked block | the rendered prefix up to a breakpoint, the longest matching prefix found first |
| Matching | exact: any change at or before a breakpoint breaks the hit | exact prefix |
| Minimum length | 512 to 4096 tokens, depending on the model | 1024 tokens on the newest models; varies on earlier ones |
| Lifetime | 5 minutes by default, refreshed free on each read, or 1 hour at extra cost | 30 minutes by default or 12 hours on the newest models; earlier models use in-memory or 24 hours |
| Price on reads | 0.1 times the input price by default, lower on some models | 0.1 times on the newest models ("up to 95 percent" off is the general statement) |
| Price on writes | 1.25 times for 5 minutes, 2 times for 1 hour | 1.25 times for 30 minutes, 2.5 times for 12 hours (newest models) |

On a lab's API the cache is a price; on your own fleet it is prefill you do not pay GPU time for (D2). Anthropic's page lists what breaks a cache:

- changing tool definitions invalidates everything after them
- changing the system prompt invalidates system and messages
- changing images, the thinking configuration or tool choice invalidates the messages
- a request looks back at most 20 blocks: put the breakpoint on the last block identical across requests, not on a timestamp

These are B2 made visible: a changed token near the start throws the rest away.

*Batch and tiers.* APIs sell latency-tolerant work at a discount and priority work at a premium: B4's scheduler as a price, since a request that can wait fills a gap in the batch. Discounts are not examined here.

**2. Postmortems, the best source on how production serving fails.** Anthropic's write-up of three bugs in August and September 2025 says Claude is served through its own API, Bedrock and Vertex AI, on Trainium, NVIDIA GPUs and TPUs. None of the three is about the model.

<figure class="mfig ds-figure ds-wide" id="inf-postmortem" aria-label="Three production bugs: a routing error, a misconfiguration and a compiler bug">
<div class="ds-plot dg-host" data-h="195"></div>
<figcaption><b>None of the three is the model.</b> Figures are from the post.</figcaption>
</figure>

Why they were hard to see, per the write-up: benchmarks, safety evaluations and canaries missed the degradation; privacy controls limit engineers' access to user interactions; each bug looked different on each platform. Its remedies are the ones F3 ends with. The transferable lesson: **a serving stack has numerical behaviour as well as latency behaviour**, and a change in batching, precision, kernels or hardware can degrade answers without an error.

**3. Open-weights reports.** DeepSeek-V3 (671 billion parameters, 37 billion active; latent attention shrinks the cache) and Moonshot AI's Mooncake (separate prefill and decode, cache over CPU memory and SSDs) are the most detailed public views of production scale. D and F build on them.

**4. What can only be inferred.** Closed providers do not publish engines, layouts or hardware counts. The API surface, price ratios and papers support inferring continuous batching, paged and shared cache, disaggregation or chunking, quantisation, speculation and some cache-aware routing; not the combination or the scale. Anything more specific is a claim to check.

Amazon Bedrock lists models from many providers behind one API, with cross-Region inference for throughput. For Lantern, part of F2 is whether to run the model at all.

### C3. The mental model, and a worked sizing

<figure class="mfig ds-figure ds-wide" id="inf-five" aria-label="Five questions to ask of any serving system">
<div class="ds-plot dg-host" data-h="202"></div>
<figcaption><b>Everything in A and B reduces to five questions.</b></figcaption>
</figure>

**The equation of serving cost.** The cost per million output tokens of a fleet busy a fraction $\rho$ of the time:

$$
\text{cost per million tokens}=\frac{\text{dollars per GPU-hour}\times g}{3600\times\text{tokens per second per replica}\times\rho}\times10^6.
$$

Part B lifts the denominator (tokens per second per replica) or the utilisation reachable under the SLO; Part D lowers the work per request (a cache hit is prefill not paid for).

**Lantern's sizing.** Traffic: 20 requests a second, 2000-token prompts, 400-token answers (context 2400), 4-GPU replicas at an illustrative three dollars per GPU-hour. A replica holds 186 conversations and decodes 6301 tokens a second; a prompt prefills in 162 ms. Decode needs 1.27 replicas and prefill 3.24: **72 percent of the GPU time is spent reading prompts.**

<figure class="mfig ds-figure ds-wide" id="inf-sizing" aria-label="Three plans for 20 requests a second: raw capacity, with headroom, and with a prefix cache">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>The cached plan with headroom is cheaper than the uncached plan without it.</b> Dollars are per hour, then per million output tokens, prompts included.</figcaption>
</figure>

The figure sweeps the traffic.

<figure class="mfig ds-figure ds-wide" id="inf-capacity" aria-label="GPUs needed and dollars per million output tokens against requests per second, for three plans">
<div class="ds-plot"></div>
<figcaption><b>The fleet is lumpy at small traffic, and the prefix cache saves more than the headroom costs.</b> 2000-token prompts, 400-token answers, 4-GPU replicas at an illustrative three dollars per GPU-hour. Rows: raw capacity, planned for 65 percent utilisation, and the same with 1500 prompt tokens from a prefix cache; bar is GPUs, number is dollars per million output tokens. Drag the ring from 1 to 100 requests a second.</figcaption>
</figure>

At low traffic the fleet is a *minimum unit*: one request a second costs 8.33 dollars per million output tokens, twenty cost 2.08. That is why a start-up's cost per token is worse than a large provider's by more than the hardware price explains, and why managed APIs and pooled hosting (F2) exist.

<aside class="chk">

Replica counts for the three plans at 20 requests a second are 5, 7 and 4; with headroom and the cache, 1, 1, 1, 2, 4, 9 and 17 at 1, 2, 5, 10, 20, 50 and 100. Three dollars per GPU-hour is a parameter.

</aside>

<aside class="q">

The sizing adds decode and prefill on one pool and saturates a replica at batch 112. Under chunked prefill they share steps; under disaggregation they are sized apart. How much of the 72 percent survives each, and does the best prefill-to-decode ratio move with prompt length? This note derives the additive floor and does not simulate schedules.

</aside>

### C4. What Part C leaves Lantern with

Pick an engine for its interface and for its cache and routing features, since the cost is going to the layers above it. The public record is thin on internals and rich on API semantics, postmortems and open reports.

**Exercises.** Each answer is folded.

**1.** If the shared prefix is 1000 tokens instead of 1500, with the same traffic, how many replicas does the plan with headroom need?

<details class="answer"><summary>Worked example: a shorter shared prefix</summary>

Prefill of the remaining 1000 tokens takes 81 ms, so prefill demand is $20\times0.081=1.62$ replicas. With decode at 1.27 the total is 2.89, and divided by 0.65 it is 4.4, so 5 replicas: 20 GPUs, 60 dollars an hour. The cache of 1500 tokens needed 4 replicas, so 500 more tokens of stable prefix save one replica, 12 dollars an hour.

</details>

**2.** A cache read costs 0.1 of an input token and a write 1.25. How many uses of a prefix, within its lifetime, make caching cheaper than not caching?

<details class="answer"><summary>Worked example: the break-even number of uses</summary>

Without caching, $n$ uses of a prefix of $p$ tokens cost $np$. With caching the first use is a write, $1.25p$, and each later use a read, $0.1p$: $1.25p+0.1p(n-1)$. The two are equal when $n=1.25+0.1(n-1)$, that is $0.9n=1.15$, $n=1.28$. So a prefix used twice within its lifetime is already cheaper cached: $1.25+0.1=1.35$ against 2. The risk is the lifetime: a prefix that is written and never read within five minutes costs 25 percent more than not caching.

</details>

## D. The agentic era

Lantern's agent product loops: call the model, run the tool it asked for, append the result, call again. This part is what that does to serving.

<figure class="mfig ds-figure ds-wide" id="inf-agent-seq" aria-label="An agent loop calling a model, receiving a tool request, running the tool and calling again with a longer context">
<div class="ds-plot dg-host" data-h="360"></div>
<figcaption><b>Each turn re-sends everything before it.</b> The tool runs for seconds to minutes while the session waits.</figcaption>
</figure>

### D1. The shape of an agent's traffic

A session of 20 calls starts at 6000 tokens (instructions, tools, task) and every turn appends 1500, so the context at call $t$ is $6000+1500t$ tokens.

<figure class="mfig ds-figure ds-wide" id="inf-chat-agent" aria-label="A chat request against a 20-call agent session: calls, input read, output written, idle time">
<div class="ds-plot dg-host" data-h="358"></div>
<figcaption><b>An agent reads far more than it writes, and waits.</b> Session of 20 calls, final context 34,500 tokens.</figcaption>
</figure>

- **Processed many times.** The 20 calls present 405,000 input tokens, **11.7 times** the final context: without reuse, work grows with the *square* of the length.
- **Input large, output small.** About 70 to 1; *prefill*, not decode, is the larger share (72 percent at a 2000-token prompt in C3, more here).
- **Pauses, bursts, variance.** The cache idles while a tool runs; subagents make one action a burst of ten sessions on one prefix; tasks end in 3 calls or 300, so averages mislead.

### D2. Cache economics: the prefix is the product

An append-only conversation is the best case for prefix caching: if the previous call's cache is alive, only the suffix is new. Priced as a customer of a lab's API at C2's ratios (read 0.1 of a fresh input token, write 1.25); on your own fleet a miss costs GPU time, the prefill of the whole context (3.5 s at 34,500 tokens, D3). The cost of call $t$, with context $c_t$ and $\Delta=1500$ added:

$$
\underbrace{0.1\,(c_t-\Delta)}_{\text{read the old prefix}}+\underbrace{1.25\,\Delta}_{\text{write the new part}}\quad\text{if the cache is alive},\qquad 1.25\,c_t\quad\text{if it was lost},
$$

where the first term reads the old prefix and the second writes the new part.

<details class="more"><summary>Derivation: the cost of a session with and without a cache</summary>

The claim is that with no cache loss a 20-call session costs 19.8 percent of the uncached cost, that a lost cache at the last call costs 8.3 times a hit, and that losing it at every call is worse than not caching.

**Step 1: the uncached cost.** Each call pays full price for its context: $\sum_{t=0}^{19}(6000+1500t)=20\times6000+1500\times190=405{,}000$ token-prices.

**Step 2: the cached cost with no loss.** The first call writes its 6000 tokens at 1.25: 7500. Each of the 19 later calls reads $c_t-\Delta$ at 0.1 and writes $\Delta=1500$ at 1.25, which is $0.1(c_t-1500)+1875$. With $c_t-1500=6000+1500(t-1)$, the reads sum over $t=1,\dots,19$ to $0.1\times(19\times6000+1500\times171)=0.1\times370{,}500=37{,}050$, and the writes to $19\times1875=35{,}625$. The total is $7500+37{,}050+35{,}625=80{,}175$: **19.8 percent** of 405,000.

**Step 3: a lost cache.** At a call where the cache is gone (here the last one, where it costs most), the whole context is written again at 1.25, instead of the $0.1(c_t-\Delta)+1.25\Delta$. At the last call ($c=34{,}500$) that is $43{,}125$ against $0.1\times33{,}000+1875=5175$: a miss costs 8.3 times a hit.

**Step 4: the limit.** If every call misses, the cost is $1.25\times405{,}000=506{,}250$, **25 percent worse than not caching at all**, because every call pays the write premium for a cache nobody reads.

</details>

The sensitivity to misses is the part to remember. Same session, $m$ of the 19 later calls missing the cache (spread evenly, the last always one):

| Misses | Cost relative to no caching |
| --- | --- |
| 0 | 19.8% |
| 1 | 29.2% |
| 2 | 34.3% |
| 5 | 49.6% |
| 10 | 75.2% |
| 19 | 125% |

A miss costs more the later it falls, since the context is larger: 1.7 points of the uncached bill at the second call, 9.4 at the last. One late miss raises a cached session's cost by nearly half.

<figure class="mfig ds-figure ds-wide" id="inf-agent" aria-label="Cost of each call of a 20-call agent session with and without a cache, for a chosen number of cache misses">
<div class="ds-plot"></div>
<figcaption><b>A cache makes an agent session five times cheaper, and every lost cache takes back two to nine points.</b> One bar per call of a 20-call session whose context grows from 6000 tokens by 1500. Grey outline: no cache; blue: cache alive; copper: cache lost, whole context written again. Misses are spread evenly and the last call is always one. Drag the ring: at 19 misses the session costs 25 percent more than with no cache.</figcaption>
</figure>

**How the cache is lost.** Four mechanisms, all avoidable in design.

<figure class="mfig ds-figure ds-wide" id="inf-misses" aria-label="Four ways a cache is lost: time, a changed prefix, rewritten history and routing, each with its fix">
<div class="ds-plot dg-host" data-h="213"></div>
<figcaption><b>Everything stable first, everything that changes last, never edit what came before.</b></figcaption>
</figure>

- **Time.** A tool slower than the cache lifetime finds the entry expired; the one-hour option costs 2 times to write and pays only if gaps are long.
- **Changing the prefix.** The match is exact: a timestamp, a reordered tool list, a changed instruction or a mid-session tool breaks the entry from that point.
- **Rewriting history.** Summarising shrinks the context but rewrites the prefix; it pays only when rare (below).
- **Routing.** The next call must reach the machine that holds the cache.

<details class="more"><summary>Derivation: when is it worth compacting the context?</summary>

The claim is that replacing a context of $c$ tokens by a summary of $\sigma$ tokens pays back after $m>12.5\,\sigma/(c-\sigma)$ further calls.

**Step 1: continuing as is.** Each further call reads the prefix at 0.1 and writes the new suffix; the prefix part costs $0.1c$ per call (the suffix cost is the same in both plans).

**Step 2: compacting.** Compaction writes the summary once at 1.25, $1.25\sigma$, and afterwards each call reads $\sigma$ tokens at 0.1, so $0.1\sigma$ per call.

**Step 3: the break-even.** Compaction pays when $1.25\sigma+0.1\sigma m<0.1cm$, that is $1.25\sigma<0.1m(c-\sigma)$, so $m>\frac{12.5\,\sigma}{c-\sigma}$.

**Step 4: examples.** Shrinking 30,000 tokens to 5000 pays after 2.5, so 3 calls. Shrinking 30,000 to 10,000 pays after 6.25, so 7. Shrinking 10,000 to 5000 pays after 12.5, so 13. The model's accuracy on a long context, which is a reason to compact that this cost model does not include, is a separate question.

</details>

**What to do with this.** Keep the beginning of the prompt identical across calls and sessions (every Lantern agent shares the same 1500 tokens of instructions and tools, C3); put volatile text last; log the cache-read and write counts and watch their ratio as a health metric; on your own engine, route by session (D3).

### D3. Where does the cache live while the tool runs?

A session at 34,500 tokens holds $34{,}500\times327{,}680=11.3$ GB of cache (A4), so **12 sessions fill** a 4-GPU replica's pool. Thousands are open, almost all waiting for a tool, so most caches cannot stay in GPU memory. For each waiting session the choice is the one for a preempted request (B2): drop the cache and recompute, or move it somewhere cheaper.

<figure class="mfig ds-figure ds-wide" id="inf-recompute" aria-label="Recompute a waiting session's cache against reloading it from CPU memory">
<div class="ds-plot dg-host" data-h="248"></div>
<figcaption><b>Reloading is at least eight times faster than recomputing.</b> Host-link rates of 100 and 25 GB/s are assumptions; the path from host to GPU is much slower than the GPU's own memory.</figcaption>
</figure>

That is the case for a **cache hierarchy**:

<figure class="mfig ds-figure ds-wide" id="inf-tiers" aria-label="A router in front of four cache tiers: GPU memory, CPU memory, local SSD and a remote store">
<div class="ds-plot dg-host" data-h="330"></div>
<figcaption><b>Keep the cache where the session will be needed soonest.</b> Each tier down holds one or two orders of magnitude more, and moves bytes as much slower.</figcaption>
</figure>

Mooncake pools CPU memory and SSD as cache, Dynamo's README lists offload to CPU, SSD and remote storage, llm-d lists tiered offload. Which tier a session sits in is a scheduling problem like B4: predict when it returns, which the tool latency tells you.

**Routing is the other half.** A *cache-aware* router sends a request to the replica with the longest matching prefix, unless that queue is long enough that recomputing elsewhere is faster: *affinity* (hits) against *balance* (even load).

<figure class="mfig ds-figure ds-wide" id="inf-routing" aria-label="A router choosing between a replica that holds the session cache and an idle replica that does not">
<div class="ds-plot dg-host" data-h="216"></div>
<figcaption><b>Affinity against balance.</b> The main reason frameworks above the engine exist.</figcaption>
</figure>

**Disaggregation matters more.** Agent traffic is prefill-heavy and bursty; with separate pools (B3) a burst of ten subagent sessions loads the prefill pool, not other users' decoders.

### D4. Latency and failure compound

An agent task of 40 calls is a chain. Time per token is the point to get right, because long contexts shrink the batch (D3): a replica that decodes 64 conversations of 2000 tokens in 18.7 ms holds only 12 of 34,500, steps in 27.0 ms and delivers 444 tokens a second against 3419. One call with a hit, 300 output tokens and a 3-second tool:

$$
0.4\ \text{s}+300\times0.027\ \text{s}+3\ \text{s}\approx11.5\ \text{s per call},\qquad 40\times11.5\ \text{s}=7.7\ \text{minutes}.
$$

(The 0.4 s TTFT is assumed, for a hit that prefills a short suffix and waits its turn.)

<figure class="mfig ds-figure ds-wide" id="inf-callbudget" aria-label="Time of one agent call: first token, decoding, tool; and the 40-call task">
<div class="ds-plot dg-host" data-h="195"></div>
<figcaption><b>Decoding is 8.1 of the 11.5 seconds.</b> So speculative decoding (B5) and shorter outputs are the first levers.</figcaption>
</figure>

The others: smaller models for easy steps; **parallel tool calls**; *streaming* a tool call as the model writes it; caps on the *reasoning budget*, since hidden tokens decode at the same price as visible ones.

**Failure compounds.** At 0.99 per call a 40-call task completes with probability $0.99^{40}=0.67$; at 0.999, 0.96. The per-call target is set by the *longest task* a user will run. Two twists on ordinary distributed systems:

- **Retries must be safe.** A tool that sends an email must not run twice because the call that asked for it was retried: idempotency keys, and a record of which results the loop has.
- **A broken stream is a broken state.** If a response stops half-way, is the partial message kept? The policy must keep cache and conversation consistent.

<figure class="mfig ds-figure ds-wide" id="inf-idem" aria-label="A retried tool call that runs twice without an idempotency key and once with one">
<div class="ds-plot dg-host" data-h="316"></div>
<figcaption><b>Retries need idempotency keys.</b></figcaption>
</figure>

**Tokens, not requests, are the unit of fairness.** One agent can use a hundred times a chat user's tokens. Count *tokens* (input and output, cached and uncached) per tenant per minute, and make the scheduler tenant-aware (B4). Also a security matter (E3).

<figure class="mfig ds-figure ds-wide" id="inf-fair" aria-label="Counting requests lets one agent starve other tenants; counting tokens does not">
<div class="ds-plot dg-host" data-h="248"></div>
<figcaption><b>Fairness is counted in tokens.</b></figcaption>
</figure>

**An agent is evaluated on its trajectory.** A 1 percent degradation per call is invisible in one answer and compounds over 40 calls: the point of the continuous production evaluations C2's postmortem calls for.

### D5. What Part D leaves Lantern with

Agent traffic is long contexts, many calls, small outputs and idle gaps. Work grows with the square of the length unless the prefix is reused: append-only prompt, stable part first, route by session, log hits. A waiting session's cache belongs in a hierarchy. Budget latency per call, make retries idempotent, count tokens.

<aside class="chk">

The session figures (405,000 token-prices uncached, 80,175 with a cache that never fails; 19.8, 29.2, 34.3, 49.6, 75.2 and 125 percent for 0, 1, 2, 5, 10 and 19 misses) agree with a recomputation that lists the miss turns. One session's cache is 11.3 GB, a replica holds 12, rebuilding one costs 3.5 s.

</aside>

<aside class="q">

The session uses a fixed growth and hit pattern. Real agents have heavy-tailed turn counts and tool latencies, so misses depend on the gaps against the cache lifetime. What do production traces look like, and is a one-hour lifetime or a tiered store better for the tail? This note has no traces.

</aside>

**Exercises.** Each answer is folded.

**1.** A subagent session has a stable prefix of 20,000 tokens and 10 calls each adding 500 tokens. Ten such sessions share the prefix. How much does a shared cache save on the prefix alone, at the C2 prices, if the sessions run one after the other within the cache lifetime?

<details class="answer"><summary>Worked example: ten sessions on one prefix</summary>

Without a cache each call reads at least the 20,000-token prefix at full price, so the prefix part of all 100 calls costs $100\times20{,}000=2{,}000{,}000$. With a cache the very first call writes it, $1.25\times20{,}000=25{,}000$, and the other 99 calls read it, $99\times0.1\times20{,}000=198{,}000$, a total of 223,000. The saving on the prefix is 89 percent, and it is worth having even if only the system prompt is stable. The growth of the sessions themselves (500 tokens a call) is the part that needs each session's own cache.

</details>

**2.** One call takes 11.5 s, 8.1 s of it decoding. Speculative decoding speeds the decoding by 2.0. How long is the 40-call task?

<details class="answer"><summary>Worked example: the effect on a whole task</summary>

Decoding falls from 8.1 s to 4.05 s, so a call takes 7.45 s and the task $40\times7.45=298$ s, 5.0 minutes against 7.7: a 35 percent reduction of the task from a factor 2 on one part. The tool time (3 s) and the first-token time (0.4 s) are now nearly half of the call, which is why the next lever is parallel tools and not a faster decoder.

</details>

## E. Security

In month four a Lantern customer asks: *who else can see my prompts?* The answer has three layers, because an LLM service has three attack surfaces that do not look alike.

<figure class="mfig ds-figure ds-wide" id="inf-threat" aria-label="Three attack surfaces: the model, the serving infrastructure and the agent">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>Three surfaces, three kinds of failure.</b> This part names each and ends with a list of what to build.</figcaption>
</figure>

### E1. What is being protected, and from whom

| Asset | Who wants it | How it leaks or is damaged |
| --- | --- | --- |
| Customers' prompts and outputs (often confidential or personal) | other tenants, outsiders, curious insiders | caches shared between tenants, logs, telemetry, an injected agent sending data out |
| The model's weights | competitors, thieves | a copied file, a leaky storage bucket, a malicious dependency |
| The credentials and permissions the agent holds | anyone who can influence the agent | prompt injection used as a confused deputy |
| GPU time and the bill | abusers, buggy clients | huge prompts, unbounded generation, runaway agent loops |
| The integrity of answers | anyone who can poison inputs or the supply chain | tampered model files, poisoned retrieved documents, a corrupted numerical path |

<details class="more"><summary>The OWASP Top 10 for LLM Applications, 2025</summary>

As listed on the project's page: prompt injection; sensitive information disclosure; supply chain; data and model poisoning; improper output handling; excessive agency; system prompt leakage; vector and embedding weaknesses; misinformation; unbounded consumption. This part is organised by the *mechanism* behind these, because a design can address mechanisms.

</details>

### E2. The model cannot tell instructions from data

Every token of the context, whether from Lantern's system prompt, the user or a page the agent fetched, is a row of the same matrix read by the same attention. There is no separate channel for commands, and nothing in the architecture makes the model weight some tokens more. That is the root of **prompt injection**.

<figure class="mfig ds-figure ds-wide" id="inf-context" aria-label="Four sources of text in one context window feeding a model whose actions carry the user's rights">
<div class="ds-plot dg-host" data-h="360"></div>
<figcaption><b>The model reads one stream, so anyone who writes into it can steer the actions.</b> Green is Lantern's text, red is text an attacker can write.</figcaption>
</figure>

- **Direct:** the user types "ignore your instructions and …"; the damage is bounded by what *that user* may do.
- **Indirect:** a third party writes instructions into something the agent reads (a page, an email, a code comment, a tool's output) and the agent obeys *with the user's privileges*. The person at the keyboard never sees it.

**An attack needs three ingredients**; defence is making sure no session has all three.

<figure class="mfig ds-figure ds-wide" id="inf-trifecta" aria-label="Private data, untrusted text and an outward channel meeting to let data leave">
<div class="ds-plot dg-host" data-h="280"></div>
<figcaption><b>Remove any one ingredient and the attack has nothing to use.</b></figcaption>
</figure>

The outward channel can be an HTTP request, an email, a commit or an image link that loads from the attacker's server. (The "lethal trifecta" is from the practitioner literature; unverified here.)

**Why filtering is not enough.** A detector lowers the rate, not to zero: 99 percent effective fails once in a hundred against an attacker who tries a thousand times. The reliable defences *limit what a success can do*.

**Limit the blast radius.** Assume the model can be compromised and bound the damage; none of this changes the model.

<figure class="mfig ds-figure ds-wide" id="inf-blast" aria-label="Model output passes validation, confirmation, a sandbox and an allow-listed egress, with credentials kept outside the model">
<div class="ds-plot dg-host" data-h="203"></div>
<figcaption><b>Every action crosses four gates, and the credentials are never in the prompt.</b> The seven defences are folded below.</figcaption>
</figure>

<details class="more"><summary>The seven defences</summary>

1. **Least privilege per session.** A credential that does the task and nothing else: read-only, one repository, valid for minutes, never in the prompt (a prompt can be echoed); tools use credentials *outside* the model.
2. **Separate readers from actors.** The agent that reads untrusted content gets no write or send tools; the one that acts sees only vetted inputs.
3. **Confirm irreversible or outward actions** with the human, showing the *actual* action and arguments.
4. **Control egress.** No network by default, an allow-list when needed; block remote images, links, DNS.
5. **Treat model output as untrusted input** (OWASP's *improper output handling*): never unchecked into a shell, SQL, a template or a browser.
6. **Sandbox execution** in a container or micro-VM with limited CPU, memory, time and filesystem, away from credentials.
7. **Log what the model asked for and what ran.**

</details>

### E3. The infrastructure shares more than you think

**Caches are a side channel.** A prefix cache shared between customers makes a matching prompt *faster*, so a customer can measure that someone else sent it. *Auditing Prompt Caching in Language Model APIs* (Gu et al., 2025) reports in its abstract global cache sharing across users at seven providers, OpenAI among them. It does not say how many were tested or what changed afterwards, and neither does this note.

<figure class="mfig ds-figure ds-wide" id="inf-sidechannel" aria-label="Tenant A sends a prompt, tenant B sends the same prefix and a fast answer reveals that A sent it">
<div class="ds-plot dg-host" data-h="512"></div>
<figcaption><b>Tenant A and tenant B share one cache; any shared cache with observable timing leaks membership.</b></figcaption>
</figure>

**Scope the cache to the tenant** and the leak closes, at a lower hit rate: tenants stop sharing common prefixes. Lantern has the same 1500-token prefix in every agent, so share only what is *public* (its own instructions and tools) and scope everything with customer text.

<figure class="mfig ds-figure ds-wide" id="inf-surfaces" aria-label="Five things tenants share, how each leaks, and what contains it">
<div class="ds-plot dg-host" data-h="301"></div>
<figcaption><b>What is shared leaks unless something contains it.</b></figcaption>
</figure>

**Memory and compute are shared.** A freed cache block may be reused by the next sequence. A correct engine never reads before writing, so the risk is a *bug*: isolate where cheap (a replica for tenants that require it), test, and treat a cross-tenant leak as severity one. GPU partition guarantees are in the vendor's documentation only, not examined here.

**Logs and telemetry are where prompts end up.** Decide what is logged (metadata and counts, not text), where, for how long, who reads it, and how deletions reach it. C2's postmortem shows the other side: strong privacy controls made a quality bug harder to diagnose. Settle the balance *before* the incident.

**Weights are an asset and a supply chain.** Pull from a trusted source, verify a hash recorded at review, encrypt with a key service, restrict the bucket, prefer formats that cannot execute code on load (safetensors over pickle; unverified). The engine and its dependencies get the same care.

**Unbounded consumption is an attack and a bug.** Prefill cost grows with the square of the prompt (A2): one 128k-token prompt costs 21.7 seconds of four GPUs, **134 times** the 162 ms of C3's 2000-token prompt (144 times its arithmetic) and 706 times the arithmetic of a 400-token answer; attention is 55 percent of it. A few per second can saturate a fleet serving a thousand users. Defences, on the right unit: limits on prompt length and `max_tokens`; token quotas per tenant per minute (D4); concurrency caps; early rejection (B4); a spend alarm; a cap on calls per agent task.

<aside class="chk">

A 131072-token prompt is 144 times a 2000-token prompt by A2's FLOP formula (40.7 petaFLOPs against 0.28), 706 times a 400-token answer at a context of 2000, 55 percent attention; with B6's communication, 21.7 s against 162 ms.

</aside>

### E4. A security checklist for Lantern

| Layer | Do | Because |
| --- | --- | --- |
| Identity and tenancy | authenticate every call; carry the tenant id to the engine and the cache key | caches and quotas are per tenant |
| Cache | scope by tenant except for public content; short lifetimes for sensitive tenants | timing side channel |
| Network | engines in a private network, reached through one gateway; no public endpoints on GPU hosts | smaller attack surface |
| Secrets | in a secret manager, per-session scoped credentials, never in prompts | injection can echo the prompt |
| Tools | sandboxed, no network by default, allow-listed egress, human confirmation for outward or irreversible actions | blast radius of injection |
| Output | validate and escape model output before any interpreter | improper output handling |
| Limits | prompt and output caps, token quotas, concurrency, loop caps, budget alarms | unbounded consumption |
| Data | log metadata, not text; retention and deletion policy; encryption at rest and in transit | the easiest leak is a log |
| Supply chain | verified weights and images, pinned dependencies, scan | tampering |
| Quality | production evaluations, per hardware and per engine version | silent degradation is also an integrity failure |

<aside class="q">

An attacker who influences only the *time* a request takes (by sharing a batch with the victim) might learn something; the cache channel is the known case. How large are the others (batch composition, speculative acceptance, streamed lengths through packet sizes), and what do providers do? One audit abstract informs this note.

</aside>

**Exercises.** Each answer is folded.

**1.** An agent summarises emails and can also send emails and read a private contacts file. Which of the three ingredients does it have, and what is the smallest change that removes the risk?

<details class="answer"><summary>Worked example: breaking the trifecta</summary>

It has all three: emails are untrusted content an attacker can write, the contacts file is private data, and sending email is an outward channel. The smallest effective change is to remove one. Split it into a reader (can read emails, can summarise, cannot send, cannot read contacts) and an actor (can send, sees only the summary after the user confirms it). If the product needs one agent, then require the user to confirm the full recipient list and body of any message sent, before it is sent.

</details>

**2.** Lantern scopes its cache by tenant. Its shared prefix is 1500 tokens and the tenant-specific part is 500 tokens. What does scoping cost in prefill work at 20 requests a second, if 100 tenants send equal traffic?

<details class="answer"><summary>Worked example: the price of isolation</summary>

If the 1500-token public prefix is shared across tenants (and only the tenant part is scoped), nothing is lost, since the public prefix is the same for everyone and holds no customer text. If *everything* is scoped by tenant, each tenant pays the 1500-token prefill when its entry is cold, and how often that happens depends on how often the tenant returns within the lifetime. With 0.2 requests a second per tenant and a cache that refreshes on each read (C2), the entry never expires and only the first request pays: a negligible loss. With one request every ten minutes, longer than a five-minute lifetime, every request pays the whole 1500 tokens, and scoping removes the entire benefit of the cache for that tenant. The cost of isolation is set by how often each tenant returns within the lifetime.

</details>

## F. Architecture and production on AWS

By month five Lantern decides what to run: the reference architecture, when not to build it, what operating it takes. Service facts come from the AWS pages in the reading list as of 3 November 2026; re-check names and limits.

### F1. The reference architecture: eight blocks on the path of one request

The path of one request, left to right; the dashed blocks sit beside it.

<figure class="mfig ds-figure ds-wide" id="inf-aws" aria-label="The path of one request through edge, load balancer, gateway, router and inference pools on AWS, with weights, tool plane and observability beside it">
<div class="ds-plot dg-host" data-h="330"></div>
<figcaption><b>A request crosses four blocks before the one that holds GPUs.</b> Pink is Lantern's control code, blue holds the model, red is where the agent acts, kept apart from the GPUs. Press Play for what breaks at each.</figcaption>
</figure>

| Block | What it does | Built from |
| --- | --- | --- |
| 1. Edge | terminates TLS, filters floods by rate, IP reputation and request shape, before cheap requests become expensive generations | Route 53, CloudFront, WAF |
| 2. Load balancer | spreads connections over gateway tasks in two zones. **The default idle timeout is 60 seconds**: a silent stream (long prefill, long hidden reasoning) is closed unless the timeout is raised or keep-alive data is sent | ALB |
| 3. Gateway (your code) | authenticates, attaches the tenant, checks quotas *in tokens*, validates prompt and output limits, runs input and output checks, meters usage, attaches an idempotency key; all of E4's limits live here | ECS or EKS; ElastiCache or DynamoDB |
| 4. Router | picks the replica by tenant, session, longest cached prefix and queue length (B4, D3); owns the queues: latency-sensitive straight through, batch work through a queue to fill gaps; often C1's orchestration layer | custom, Dynamo or llm-d; SQS |
| 5. Inference pools | engine pods, one per replica (a 4-GPU replica asks for four GPUs); prefill and decode pools if disaggregated (B3). P5 has 3200 Gbps of EFA per instance, shared by eight GPUs; instances in a Capacity Block sit close in an UltraCluster | EKS GPU nodes, Karpenter |
| 6. Weights and images | weights in S3 under a KMS key, copied to local NVMe or FSx for Lustre when a pod starts; images from ECR | S3, KMS, FSx, ECR |
| 7. Tool plane | everything the agent *does* runs outside the pool: sandboxes with no route out except an allow-listed egress proxy, a per-session role of minutes, secrets injected outside the model (E2) | Fargate or Lambda, Network Firewall, STS, Secrets Manager |
| 8. Observability | GPU health, engine and gateway metrics, traces from gateway through router into the engine, usage records for billing. The autoscaler listens to *queue depth and cache pressure*, not CPU | DCGM exporter, CloudWatch or Prometheus, OpenTelemetry |

Under all of it: IAM, KMS, PrivateLink (traffic to S3, ECR and model APIs stays off the public internet), CloudTrail, GuardDuty. GPU hosts have no public addresses (E4).

**Start-up time is a design parameter.** Each pod of a 4-GPU replica loads 35.3 GB in 16 bits (half in 8): 71, 18 or 3.5 seconds at an assumed 0.5, 2 or 10 GB/s, before the engine compiles and warms up.

<figure class="mfig ds-figure ds-wide" id="inf-coldstart" aria-label="Cold start of a replica: launch a node, pull the image, load weights, compile and warm, serve">
<div class="ds-plot dg-host" data-h="194"></div>
<figcaption><b>A cold start of minutes defeats a reactive autoscaler.</b> Weight loading alone is 71 s, 18 s or 3.5 s depending on where the bytes come from.</figcaption>
</figure>

<aside class="q">

The figure shows one region and one path. A second region for disaster recovery doubles the GPU bill unless its fleet may be smaller. How much should a standby region hold for a service whose cold start takes minutes? This note has no data on GPU availability across regions.

</aside>

### F2. Build or buy

Lantern does not have to run the fleet. There are three places to get tokens.

<figure class="mfig ds-figure ds-wide" id="inf-buy" aria-label="Who runs the product, gateway, engine and hardware for a provider API, Amazon Bedrock and a self-hosted fleet">
<div class="ds-plot dg-host" data-h="420"></div>
<figcaption><b>The three options differ in how far down the stack you run things.</b> Green is yours. A provider API for the most capable models or when small; Bedrock for several providers behind one AWS-native interface; self-hosting for control of the weights, the data path or the latency.</figcaption>
</figure>

Two Bedrock features answer questions of this note. Neither replaces retries and fallback in the application.

<figure class="mfig ds-figure ds-wide" id="inf-bedrock" aria-label="Provisioned Throughput reserves capacity for an hourly price; cross-Region inference routes to another Region when the home one is busy">
<div class="ds-plot dg-host" data-h="284"></div>
<figcaption><b>Reserved capacity, and borrowed capacity.</b> Geographic profiles keep data in a geography; global profiles trade that for about ten percent lower price (per the page) and wider capacity.</figcaption>
</figure>

**The cost comparison is a function, not a number.** A fleet costs a *flat* amount per replica-hour; an API costs in proportion to tokens. The question is the utilisation at which flat undercuts per-token. The unit here is dollars per million tokens, input and output together (C3's 2.08 per *output* token is the same fleet, rounded up to five whole replicas). For Lantern's traffic a 4-GPU replica at the floor handles 4.4 requests a second, 38.3 million tokens an hour, at an illustrative 12 dollars an hour.

<figure class="mfig ds-figure ds-wide" id="inf-cost" aria-label="Cost per million tokens of one replica: full use, 65 percent, 65 percent at half the floor, and one request a second">
<div class="ds-plot dg-host" data-h="195"></div>
<figcaption><b>Cost per token is a function of utilisation and of how near the engine gets to the floor.</b> Dollars per million tokens, prompts and answers together.</figcaption>
</figure>

The break-even rule is

$$
u^\star=\frac{\text{replica dollars per hour}}{\text{API dollars per million tokens}\times\text{million tokens per hour at full use}},
$$

and self-hosting wins above $u^\star$. With a placeholder API price of 2.0 dollars per million blended tokens, $u^\star=12/(2.0\times38.3)=0.16$: the replica must be busy 16 percent of the time.

The real answer is less favourable than the floor, and the comparison is unfair in one way:

- real engines run below the floor
- the fleet needs headroom and spares for failures
- the people who run it are a cost
- an open 70-billion-parameter model is not a frontier API model: compare at *equal quality on Lantern's task*, by its own evaluations

At fifty users the traffic is far below $u^\star$ of one replica, and the answer is an API.

<aside class="chk">

A replica handles 4.4 requests a second, 10.6 thousand tokens a second and 38.3 million tokens an hour; the costs are 0.31, 0.48 and 0.96 dollars per million tokens, and 1.39 at one request a second; weight-load times are 71, 18 and 3.5 s for a 35.3 GB shard; $u^\star$ is 0.16 at the placeholder price. Twelve dollars an hour is a parameter.

</aside>

### F3. Running it

**Say what good means: the golden signals of an LLM service.**

| Signal | Why |
| --- | --- |
| TTFT and TPOT at p50, p95 and p99, per model and per tenant tier | the SLO itself (A1) |
| Queue time (waiting for admission) and running and waiting request counts | utilisation is the lever (B4) |
| Cache usage (percent of blocks in use) and preemption count | memory sets the batch (A4); preemptions mean cache pressure |
| Prefix-cache hit rate, and for APIs the cache-read and cache-write token counts | the cost of an agent (D2) |
| GPU utilisation, memory, temperature, error counters | hardware health |
| Tokens per minute per tenant, input and output, cached and uncached | quotas, fairness, billing |
| Errors by kind (timeouts, rejections, truncations) | the user-visible failure rate |
| Dollars per million tokens (blended) | the number the business cares about |

**Scale on the right signal.** CPU says nothing about a GPU service: scale on queue depth and cache pressure. Scale-up takes minutes (F1), so a reactive autoscaler has lost the spike.

<figure class="mfig ds-figure ds-wide" id="inf-scale" aria-label="Queue depth and cache pressure drive the autoscaler, which draws on a warm pool first and new nodes later; shedding covers the gap">
<div class="ds-plot dg-host" data-h="286"></div>
<figcaption><b>Headroom, a warm pool, predicted load, and shedding for the minutes between.</b></figcaption>
</figure>

**Reserve capacity on purpose.** On-demand GPUs can be unavailable. EC2 Capacity Blocks for ML reserve GPU instances (P5, P5e, P5en, P6, Trainium; several Regions) for a fixed period starting up to eight weeks ahead, up to 64 instances, no cancellation; they suit bursts and planned peaks.

<figure class="mfig ds-figure ds-wide" id="inf-reserve" aria-label="Cover the base with commitments, the peak with on-demand or blocks, latency-tolerant work with interruptible capacity">
<div class="ds-plot dg-host" data-h="290"></div>
<figcaption><b>Three kinds of capacity for three kinds of load.</b> Terms are on the pricing pages.</figcaption>
</figure>

**Test as the users will use it.** Open loop (B4), with the real prompt and output lengths and prefix reuse, and the SLO percentiles read at each rate until the knee. A test with random short prompts measures a service Lantern does not run.

**Roll out like a database, not a web page.** A new engine version, quantisation, kernel, GPU type or model can alter *answers*, not only latency. C2's postmortem is the cautionary story: sticky routing kept affected users affected, and benchmarks and canaries missed it. For Lantern:

<figure class="mfig ds-figure ds-wide" id="inf-rollout" aria-label="A change is replayed on a fixed prompt set, canaried on a slice of traffic, compared on quality, then ramped, with rollback">
<div class="ds-plot dg-host" data-h="180"></div>
<figcaption><b>A rollout compares quality as well as speed.</b></figcaption>
</figure>

1. a **fixed set of prompts replayed after every change**, compared by exact match where decoding is deterministic, by log-probabilities and a task score where not;
2. **sampled production evaluation**, scored automatically on traffic customers allowed;
3. scores **per hardware and per engine version**, so a regression on one slice shows.

**Expect the hardware to fail.** A node with eight GPUs has eight times the parts. Health checks run a real generation, not a port check; failed nodes are drained, replaced and removed from the router; in-flight sessions follow D4's retry rules. One failed node stalls a multi-node group, which argues for small groups.

<figure class="mfig ds-figure ds-wide" id="inf-failover" aria-label="A health check that runs a real generation fails a replica; the router drains it and falls back to another replica or an API">
<div class="ds-plot dg-host" data-h="290"></div>
<figcaption><b>Failure path.</b> A fallback to an API keeps users served while the replacement boots.</figcaption>
</figure>

**Incidents have a shape.** Each corresponds to a signal in the table above, which is why the table is built first.

| Symptom | First look |
| --- | --- |
| Bad answers | the last change, then the slice |
| Latency spike | queue or cache pressure; a long-prompt storm; a degraded replica |
| Out of memory | a long-context request, or the cache fraction set too high |
| Hit-rate collapse | a changed prompt prefix or routing |
| Cost spike | a looping agent, a missing cap, lost caching |

### F4. What Part F leaves Lantern with

The architecture is eight blocks, one with GPUs; the others decide security, fairness, cost and reliability. Build or buy is a utilisation question at equal quality; a provider API or Bedrock is the default until volume or control justifies a fleet. Operating it is signals, headroom, quality-checked rollouts and an expectation of failure.

**Exercises.** Each answer is folded.

**1.** A replica takes 90 seconds from node launch to serving, and traffic can double in 60 seconds. How should headroom be set?

<details class="answer"><summary>Worked example: headroom against slow scale-up</summary>

During the 90 seconds the existing fleet must absorb the surge. A fleet planned for 50 percent utilisation lands at 100 percent when traffic doubles, which is saturation: the queue grows without bound until the new replicas arrive. A fleet planned for 65 percent lands at 130 percent and is overloaded by a third. To land at 80 percent, where the queue factor of B4 is 5 and the delay is bad but finite, the base utilisation must be 40 percent. The alternatives are a warm pool of started replicas, predictive scaling for known peaks, and shedding load for the minute and a half.

</details>

**2.** A tool sandbox needs to download a Python package from the public internet, and the session also holds a read token for the customer's repository. What is the risk and the fix?

<details class="answer"><summary>Worked example: egress and credentials together</summary>

An injected instruction could use the open egress to send the token or the repository's contents out. The fix is to split the two powers: package installs go through a caching mirror or a proxy with an allow-list of package indexes (no other destinations), and the repository token is never present in the sandbox but is used by a separate service that returns only what the task needs, or the sandbox gets a token scoped to read the one repository for a few minutes and has no route to any host except that mirror.

</details>

## G. Curriculum: what to study, in what order, and what to search for

Lantern is six months old, and so is the reader's first pass. This part turns the note into a path: the ideas, the open questions, a study plan, the terms to search, the reading list.

### G1. The core ideas, and where each one goes deeper

| # | Idea | One line | Where | Search for |
| --- | --- | --- | --- | --- |
| 1 | Prefill and decode | compute-bound pass over the prompt, memory-bound pass per token | A1, A3 | "LLM inference prefill decode phases" |
| 2 | FLOPs per token | $2P$ plus attention | A2 | "transformer inference FLOPs", "MFU model FLOPs utilization" |
| 3 | Roofline and arithmetic intensity | bytes versus arithmetic, the ridge point | A3 | "roofline model GPU", "memory-bound decode", "arithmetic intensity LLM" |
| 4 | KV cache | 320 KiB per token here; memory decides the batch | A4 | "KV cache size calculation", "KV cache memory" |
| 5 | GQA, MQA, MLA | fewer cache bytes by design | A4 | "grouped-query attention", "multi-query attention", "multi-head latent attention" |
| 6 | FlashAttention | exact attention with fewer HBM trips | A5 | "FlashAttention IO-aware", "kernel fusion" |
| 7 | Continuous batching | refill slots every step | B1 | "iteration-level scheduling", "continuous batching" |
| 8 | PagedAttention | the cache in blocks, as an OS pages memory | B2 | "PagedAttention", "KV cache fragmentation" |
| 9 | Prefix caching | share identical prefixes by block | B2, D2 | "RadixAttention", "automatic prefix caching" |
| 10 | Chunked prefill | mix prefill and decode in small pieces | B3 | "chunked prefill", "decode-maximal batching" |
| 11 | Disaggregation | prefill and decode on separate pools | B3 | "prefill decode disaggregation", "DistServe", "Splitwise" |
| 12 | Scheduling and queueing | utilisation sets tail latency | B4 | "LLM scheduling goodput", "Little's law", "M/M/1" |
| 13 | Quantisation | fewer bytes per number | B5 | "weight-only quantization", "FP8 inference", "KV cache quantization" |
| 14 | Speculative decoding | draft cheaply, verify in one step | B5 | "speculative decoding", "EAGLE", "Medusa" |
| 15 | Tensor, pipeline, expert parallelism | splitting a model, and what it costs | B6 | "tensor parallelism all-reduce inference", "expert parallelism MoE serving" |
| 16 | Cache hierarchy and routing | HBM, CPU, SSD, remote; send work to the cache | D3 | "KV cache offloading", "cache-aware routing", "Mooncake" |
| 17 | Agent cost model | prefix reuse makes the loop affordable | D1, D2 | "prompt caching agents", "context compaction" |
| 18 | Prompt injection and blast radius | instructions and data share a channel | E2 | "indirect prompt injection", "OWASP LLM top 10", "confused deputy problem" |
| 19 | Cache side channels and isolation | shared caches leak by timing | E3 | "prompt caching timing attack", "multi-tenant isolation LLM" |
| 20 | Serving on AWS | eight blocks, one with GPUs | F | "EKS GPU inference", "Bedrock provisioned throughput", "Capacity Blocks for ML", "SageMaker large model inference" |
| 21 | MFU and MBU | how close a measurement is to the floor | A5, B7 | "model FLOPs utilization", "model bandwidth utilization", "LLM benchmark methodology" |
| 22 | Mixture of experts serving | memory follows total parameters, compute follows active ones | B6 | "MoE inference memory", "expert parallelism all-to-all", "DeepSeek-V3 serving" |
| 23 | Sizing and cost per token | demand in replicas, headroom, minimum unit | C3 | "LLM capacity planning", "cost per million tokens self-hosted" |
| 24 | Prompt-cache API semantics | exact prefixes, lifetimes, read and write prices | C2, D2 | "Anthropic prompt caching", "OpenAI prompt caching", "cache breakpoints" |
| 25 | Build or buy | utilisation above which a fleet beats an API, at equal quality | F2 | "Bedrock provisioned throughput", "self-hosting LLM break-even" |
| 26 | Rollouts, evals and failure | quality regressions are silent; hardware fails | F3 | "LLM canary deployment", "production LLM evaluation", "GPU Xid errors" |

### G2. A study plan with things to build

Eight weeks, a few hours each. Every week has something to *compute* or *run*, because the aim is the instinct, not the vocabulary. A rented single GPU and a 1 to 8 billion-parameter model cover everything but the multi-GPU parts; otherwise use the simulator below.

| Week | Read | Build or compute |
| --- | --- | --- |
| 1 | A of this note; the FlashAttention abstract | Do A2 and A3 for a model you care about: parameters, FLOPs per token, bytes per token, the ridge point of your GPU, the batch-1 step time |
| 2 | B1 and B2; the Orca and vLLM papers | Run an open engine on a small model. Measure TTFT, TPOT and tokens per second at batch sizes 1 to 64, and plot against the roofline |
| 3 | B3 and B4; the SARATHI, DistServe and Splitwise abstracts, then the papers | Send a long prompt into a running stream and measure the stall; turn chunked prefill on and compare |
| 4 | B5; the speculative decoding paper; one quantisation paper | Compare 16-bit, 8-bit and 4-bit versions of one model on speed *and* on a small task of your own; try a draft model |
| 5 | B6; the DeepSeek-V3 report; a tensor-parallel paper | Compute where a mixture-of-experts model fits on a given node; sketch its all-to-all traffic |
| 6 | C and D; the prompt caching documents; the Mooncake paper | Build a small agent loop against an API with caching, log read and write tokens, and break the cache on purpose three ways. Compare two engines' documentation on the five features of C1 |
| 7 | E; the OWASP list; the cache auditing paper | Write an injection into a document your agent will read, and watch which of E2's defences stops what it tries to do |
| 8 | F; the AWS pages in the reading list | Deploy an engine on a managed Kubernetes or SageMaker endpoint, put a gateway in front, run an open-loop load test, and find the knee |

**A hardware-free simulator.** Replay A3's step-time model with a queue of random requests (arrivals at a rate, lengths from a distribution, a batch limit from A4's memory) in about 100 lines, and watch the latency knee appear. **The capstone** shows p95 TTFT and TPOT against request rate, the rate at which the SLO breaks, the cost per million tokens there, and one change from Part B that moves the knee.

### G3. The questions this note left open

1. **Floor and measurement.** How far above the floor of A3 do real engines run on this model shape, and does the gap close with batch size? (A5)
2. **Chunking against disaggregation.** How much of the prefill share of C3 survives each design, and how does the best prefill-to-decode ratio move with prompt length? (C3)
3. **Real agent traffic.** What are the distributions of turn counts, tool latencies and cache gaps in production, and is a long cache lifetime or a tiered store the better answer for the tail? (D5)
4. **Other side channels.** How large are the timing and size channels beyond the prefix cache, and what do providers do? (E4)
5. **Standby capacity.** How much GPU capacity should a second region hold for a service whose cold start takes minutes? (F1)

### G4. What the note did not cover

Each is a field of its own; the terms start a search.

| Field | Terms to search |
| --- | --- |
| Training and fine-tuning infrastructure | ZeRO, FSDP, Megatron-style parallelism, mixed precision, checkpointing |
| Multimodal serving | image and audio encoders, their cost and cache |
| Sampling and decoding | temperature, top-p, log-probabilities, determinism |
| Structured output | grammar masks, JSON schema; agents use it for tool calls |
| Several LoRA adapters served together | multi-LoRA, adapter batching |
| Reasoning models and hidden tokens | their cost in decode time |
| Long-context techniques | ring attention, context parallelism, sparse and linear attention, state-space layers |
| Retrieval-augmented generation | embeddings, vector stores, rerankers, OWASP's vector weaknesses |
| Evaluation | benchmarks, LLM judges, regression suites |
| Hardware beyond NVIDIA | TPUs, Trainium and Neuron, AMD, custom accelerators |
| Edge and local inference | llama.cpp, quantisation formats for CPUs |

### G5. Reading list

Entries with a link support the claims made from them, at the level of the abstract, README or page that each link points to. The entries without links are unverified here.

**Sources.**

- Kwon et al. 2023, *Efficient Memory Management for Large Language Model Serving with PagedAttention*, [arXiv:2309.06180](https://arxiv.org/abs/2309.06180) (SOSP 2023).
- Yu et al. 2022, *Orca: A Distributed Serving System for Transformer-Based Generative Models*, [USENIX OSDI 2022](https://www.usenix.org/conference/osdi22/presentation/yu).
- Agrawal et al. 2023, *SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills*, [arXiv:2308.16369](https://arxiv.org/abs/2308.16369).
- Zhong et al. 2024, *DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving*, [arXiv:2401.09670](https://arxiv.org/abs/2401.09670) (OSDI 2024).
- Patel et al. 2023, *Splitwise: Efficient generative LLM inference using phase splitting*, [arXiv:2311.18677](https://arxiv.org/abs/2311.18677).
- Qin et al. 2024, *Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving*, [arXiv:2407.00079](https://arxiv.org/abs/2407.00079).
- Zheng et al. 2023, *SGLang: Efficient Execution of Structured Language Model Programs*, [arXiv:2312.07104](https://arxiv.org/abs/2312.07104).
- Dao et al. 2022, *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*, [arXiv:2205.14135](https://arxiv.org/abs/2205.14135).
- Ainslie et al. 2023, *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*, [arXiv:2305.13245](https://arxiv.org/abs/2305.13245).
- Leviathan, Kalman and Matias 2022, *Fast Inference from Transformers via Speculative Decoding*, [arXiv:2211.17192](https://arxiv.org/abs/2211.17192).
- DeepSeek-AI 2024, *DeepSeek-V3 Technical Report*, [arXiv:2412.19437](https://arxiv.org/abs/2412.19437) (abstract only).
- Gu et al. 2025, *Auditing Prompt Caching in Language Model APIs*, [arXiv:2502.07776](https://arxiv.org/abs/2502.07776) (abstract only).
- Anthropic engineering, [*A postmortem of three recent issues*](https://anthropic.com/engineering/a-postmortem-of-three-recent-issues), September 2025.
- Anthropic, [*Prompt caching*](https://platform.claude.com/docs/en/docs/build-with-claude/prompt-caching) (API documentation, 3 November 2026), and OpenAI, [*Prompt caching*](https://developers.openai.com/api/docs/guides/prompt-caching) (API documentation, 3 November 2026).
- OWASP, [*Top 10 for LLM Applications 2025*](https://genai.owasp.org/llm-top-10/).
- NVIDIA, [H100 data-center GPU page](https://www.nvidia.com/en-us/data-center/h100/).
- AWS: [EC2 P5 instances](https://aws.amazon.com/ec2/instance-types/p5/), [EC2 Trn2 instances](https://aws.amazon.com/ec2/instance-types/trn2/), [Amazon Bedrock overview](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html), [Bedrock Provisioned Throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html), [Bedrock cross-Region inference](https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html), [EC2 Capacity Blocks for ML](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-blocks.html), [Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html), [SageMaker AI large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference.html) (index page only).
- Project pages: [vLLM](https://github.com/vllm-project/vllm), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [llm-d](https://github.com/llm-d/llm-d).

**Further reading, unverified here.** Shoeybi et al. 2019 (Megatron-LM, tensor parallelism); Frantar et al. (GPTQ); Lin et al. (AWQ); Dettmers et al. (LLM.int8()); Chen et al. 2023 (speculative sampling); Cai et al. (Medusa); Li et al. (EAGLE); the safetensors format; the "lethal trifecta" framing of prompt-injection risk from the practitioner literature; Kubernetes, Karpenter and the NVIDIA DCGM exporter documentation; queueing theory textbooks for M/M/1 and Little's law.