One request,
one process, one reply.
The simplest endpoint in the repo already has the whole skeleton of a serving system. Read it with the clever parts switched off.
A request arrives. FastAPI parses the JSON into a typed object and hands the prompt to the engine. The route knows nothing about tensors.
Depends(get_llm) returns one engine for the whole process — built once, before the server starts listening.
The prompt becomes a Sequence — a uuid, the text, an empty output list. The unit every scheduler in this article moves around.
Then it goes straight to the executor, wrapped in a one-element list. A batch of one.
It crosses into another process. task_queue.put pickles the list and pushes it through a pipe; result_queue.get blocks until something comes back.
The model lives on the far side of that pipe. The web server never touches it directly.
why? → after the lab belowFlo generates. Tokenise, one call to model.generate, decode. Up to 50 new tokens, greedy defaults.
Note what comes back: batch_decode returns prompt and completion glued together. Nobody slices the prompt off.
The reply is a tagged tuple. The child's loop wraps every result as ('complete', […]) or ('stream', […]) so the parent can tell them apart.
That is why basic_generate reads results[1][0]: index 0 is the tag, index 1 is the payload.
What it cost. One forward pass for one sequence. A GPU does almost the same work for sixteen — Flo is 23% busy and bored.
Ten users, ten passes, ten times the wall-clock. That gap is step 02.
Blast radius. A CUDA OOM or a hung kernel kills the child. The API keeps answering health checks and restarts it.
The GIL. The forward pass releases it; tokenising, sampling and the Python loop don't. Out of process, they stop competing with request handling.
Placement. One worker per GPU is the natural shape. Once it's a process, "one" becomes "eight across two nodes" without the API noticing — this is vLLM's Executor / Worker split.
async def basic_generate calls llm.basic_generate, which blocks on result_queue.get(). Inside a coroutine that freezes the event loop: no new connections, no health checks, until the pass ends.
Fix: drop the async (FastAPI moves it to a threadpool) or await asyncio.to_thread(...). Same fault on /generate and /generate_vllm. Only the stream is written correctly.
One forward pass,
four requests.
Decoding is memory-bound: a pass over sixteen sequences costs the GPU about what one does. Batching is the single highest-leverage idea in serving, and not a line of it is about neural networks.
Requests land in a queue. add_request mints a Sequence, drops it in incoming_queue, files it in sequence_map by id. Nothing touches the GPU yet.
Two lanes exist — one for blocking requests, one for streams. Same shape.
The scheduler tops up a standing batch. Not "build a batch" — refill one: while there is room under batch_size and the queue is not empty, admit. Then return the whole active list.
It returns the list itself, not a copy. Callers hold a live handle on scheduler state — which is why results are removed one by one downstream.
One pass for all of them. The batch crosses the pipe as a list; the worker pads every prompt to the longest and runs model.generate once.
Flo does the same work for four as for one, and is four times as useful.
Whose prompts? The comment in the source is the honest one: the batch you execute may hold other users' prompts. Two callers arrive 50 ms apart, one pass serves both, each picks its own results out by id.
That is what a shared scheduler is. The loop's exit condition only tracks your ids; the work it drives is global.
The batch waits for its slowest member. model.generate is atomic from the scheduler's view: it returns when the last sequence finishes. A four-token answer sits next to a fifty-token one and the GPU computes padding for 46 steps.
Static batching pays for the longest member twice: wasted steps, and queueing delay for everyone still waiting to board.
And it polls. An empty queue means time.sleep(0.1) — inside an async def handler, so the whole server sleeps with it.
A real engine puts this loop in its own thread and hands the handler a future. Step 03 does exactly that.
A token crosses
three worlds.
Humans judge a model by time-to-first-word. So the token produced deep inside a child process has to find one specific open connection in the parent, while fifteen others do the same. Two boundaries, of two different kinds.
World 1 — the event loop. The handler makes an asyncio.Queue, registers it with the scheduler together with the loop that owns it, then awaits. Every token it gets becomes one SSE frame: data: {…}\n\n.
The queue and the loop are a return address. They travel inside the Sequence.
World 2 — a thread that never sleeps. Started in __init__, before the first request. Its body is the decode loop every engine has: admit, one step for everyone, deliver, repeat.
The batch is re-derived every iteration, so a request that arrives between steps joins the next one. That is continuous batching — reached almost by accident.
Strip to data before the pipe. A streaming Sequence carries a live queue and a live event loop — neither pickles, and a copy of an event loop in another process is meaningless.
So only prompt and request_id cross. The return address stays home, where it works. Rule for every process boundary: send data, never handles.
World 3 — one token per pass. Not generate(): a raw forward, the logits at the last position, temperature 0.7, one sample. One token for every sequence in the batch, then back to sleep on task_queue.get().
Bridge A. You cannot touch an asyncio.Queue from another thread. run_coroutine_threadsafe(queue.put(tok), loop) moves no memory — it schedules the put onto the loop that owns the queue.
Bridge B was the pipe (a pickle). Bridge A is free. Get it wrong and you meet got Future attached to a different loop.
The prompt grows — and so does the bill. Each step appends the token to sequence.prompt; the next step re-tokenises all of it and runs a full forward with use_cache=False.
Generating n tokens attends 1+2+…+n positions. Quadratic. At the repo's 20 tokens you never feel it; at 512 it is 256× the work of a KV cache. This one flag is why vLLM exists.
Termination. A None on the queue closes the generator; finally drops the sequence — so a client that hangs up stops costing tokens. That is your cancellation story.
In practice only the token_count > 20 half ever fires. The other half can't — see the faults.
Right-padding. OPT pads on the right; logits[:, -1, :] for a short sequence reads the logits after PAD tokens. Fix: padding_side = "left".
EOS never fires. decode(…, skip_special_tokens=True) turns EOS into "", compared against "</s>". Compare ids: next_token[i] == eos_token_id.
Bare except. A None from get_sequence (routine, one step after cleanup) raises, the loop prints and restarts — dropping every other sequence's token that step.
generate_vllm is the shortest method in the file: SamplingParams, .generate(prompts), done. Everything above still exists, inside vLLM. What you get for the trade, each item mapped to a fault you just met:
| vLLM | removes | order of magnitude |
|---|---|---|
| Paged KV cache | use_cache=False re-attending the prefix each step | O(n²) → O(n) |
| Continuous batching | static generate() holding slots for finished sequences | 2–4× throughput |
| PagedAttention | reserving max_len of contiguous KV per sequence | ~4% waste vs 60–80% |
| Prefix caching | re-prefilling a shared system prompt per request | workload-dependent |
| Correct batched sampling | right-padding, decoded-string EOS | correctness |
| CUDA graphs, fused kernels | launch overhead dominating small steps | 10–30% at low batch |
One caveat: vllm.LLM.generate() is the offline API. It blocks, admits nothing while it runs, and is called from an async def route. The serving API is AsyncLLMEngine — or just vllm serve with your gateway in front.
Forty models,
two slots.
The second service in the repo answers the opposite question: not one model for many people, but many models on a box that fits two. Say "cache with an eviction policy" out loud and the design writes itself.
A registry, not a model. models.json becomes typed metadata. Clients hold an opaque id; the loader uses name; framework is the key the factory switches on.
Split id from name and you can repoint a uuid at a new checkpoint, a new framework, or a remote server, and nobody's code changes.
Two slots. An OrderedDict, oldest first. Python's dict already is an LRU if you promise to move things to the end when you touch them.
Miss → load. Unknown id: 404. Known id: build a worker, which pulls weights onto the device. Seconds of loading that nobody's request can hide behind.
Hit → move to end. One call, move_to_end, is the entire policy. Whatever sits at the front now is the least recently used.
Note the hit reads the worker from the engine's dict, not the cache — two records of residency that happen to agree.
Full → evict the oldest. popitem(last=False) takes the front; delete_worker drops the engine's reference. Then load the new one into the gap.
Dropping a reference is not freeing a GPU. See the fault below.
The factory and the strategies. framework → class. Three workers share one abstract base — _load_model, predict — and nothing else.
The base calls _load_model() from its own __init__, so a subclass must set its attributes before super().__init__(). That ordering will bite whoever adds the fourth worker.
Triton — someone else's depot. The fourth worker loads nothing. It asks a Triton server to, via its repository API, and unloads in __del__. Your LRU keeps the policy; Triton keeps the mechanism.
What Triton brings: many framework backends behind one protocol, server-side dynamic batching, versions, instance groups, ensembles.
Eviction frees a dict entry, not a GPU. The caching allocator keeps freed blocks; "max 2 models" becomes an OOM at model 3. Real eviction: drop the module, gc.collect(), torch.cuda.empty_cache(), then admit.
Two sources of truth. Residency lives in manager.model_cache and engine.workers; a hit reads the second. And ModelEngine.workers = {} at class scope is a shared mutable default waiting to be found.
TorchVisionWorker ignores metadata.name and always builds MobileNet. Add a second torchvision model and you serve the wrong one under its name.
A queue in front of
every resident model.
The repo's /predict runs one input per call. Put a small queue in front of each loaded worker, flush it when it is full or when the oldest request has waited long enough, and you have Triton's dynamic batcher — in forty lines, on the repo's own classes.
The problem. worker.predict(input) — one tensor through the model, per HTTP request. Three models resident, each getting a trickle: Flo runs a pass for every single input.
A queue per resident model. submit parks the input with a timestamp and hands back a Future — the caller awaits that, not the model.
Two triggers. Flush when the batch is full, or when the oldest request has waited max_delay_ms. The second trigger is what keeps a quiet model from starving its one user.
Flush = one pass per model. Take up to max_batch, run them together, resolve each future. Flo sees three batches instead of eleven inputs.
The manager gets a clock. submit still goes through the LRU (it may load, it may evict); tick flushes whatever is ready. The serving loop calls tick every few milliseconds.
A queue is bound to a specific worker object, so an evicted-and-reloaded model gets a fresh queue rather than a stale one.
The knob. max_delay_ms is latency you deliberately add to buy throughput. Triton calls it max_queue_delay_microseconds. Set it below your model's step time and batching barely happens; above your latency budget and users notice.
Evicting a model
mid-sentence.
Streaming and an LRU cache do not get along. A stream holds a model for seconds; the cache wants to evict it in the middle. The fix is a pin count — and an honest 503 when every resident model is busy.
A worker that streams. Same base class as the classifiers; predict becomes a generator. Each step feeds only the newest token id and the past_key_values from last time — the KV cache step 03 was missing.
The stop check compares ids, so it actually fires.
Pin while streaming. Increment on entry, decrement in finally — so a client that disconnects mid-stream still unpins. A generator's finally runs when the consumer closes it.
The LRU meets a pinned model. Walk oldest-first and skip anything with an active stream. The victim is the least recently used idle model.
When everyone is busy: refuse. No idle victim means the new model cannot load. Returning 503 with a retry hint is the correct answer; the alternative is killing someone's half-finished sentence.
This is backpressure. The repo has none — its queues are unbounded and nothing ever says no.
Same shape as step 04. Hit, miss, evict, load — one extra rule in the evict branch. Every extension in this article is the repo's class plus one idea; that is what a good skeleton buys you.
Draw a service.
Run it.
Every part from steps 01–06 is a box. Drag them onto the paper, wire them, open any box to change its Python, then run two hundred requests through the thing you drew and read the numbers.