Skip to content
Mum, I wanna be an AI Engineer

Interview angle: serving and platform

Prerequisites: lessons 01 to 06 of M6, which this page assumes you have read.

This page collects what lessons 01 to 06 prepare you to answer out loud. Each answer points back to the lesson that covers it in depth.

What they ask

Section titled “What they ask”

1. Your company is based in the EU and wants to self-host an open model. How do you shortlist one? Licence first. Qwen3.5, Gemma 4 and gpt-oss ship under Apache 2.0 and DeepSeek V4 under MIT; neither licence asks anything of you for running a model on your own servers, and their notices apply when you distribute the weights or a derivative. Llama 4’s Acceptable Use Policy withholds the Section 1(a) rights for its multimodal models, Scout and Maverick among them, from companies with their principal place of business in the EU. Then memory at a quantisation you have evaluated, context, features such as tool calling, and finally your own eval set on the two or three that remain. An open-weight release publishes no training data or training code, so you cannot audit how the model was trained (lesson 01).

2. Will this model fit on one 80 GB GPU, and at what concurrency? Weights, overhead and the KV cache of every concurrent sequence must fit under --gpu-memory-utilization, 0.92 by default. Lesson 02’s arithmetic for Qwen3.5-27B, with 5 GB of overhead assumed: bf16 weights leave 222,778 tokens of cache, less than one sequence at the model’s native 262,144, which is what an unset --max-model-len asks for; FP8 weights leave 634,765 tokens, and an fp8 KV cache doubles that. At 8K tokens per conversation the same GPU holds 27 to 205 sequences depending on the weight format and the KV dtype. Set --max-model-len to what the workload needs, and run the eval set after quantising (lesson 02).

3. Define TTFT, TPS and throughput, and say what counts as a token. TTFT runs from sending the request to the first chunk with a non-empty delta.content, measured at the client, so it includes network, queue and prefill. Per-request TPS is output tokens over the time between the first and the last token; throughput is all successful output tokens over the run’s wall time. The role-only first chunk, the finish chunk, the usage chunk and [DONE] are not tokens, and reasoning deltas are not tokens a user sees. A stream that ends without [DONE] is a failure. Percentiles by nearest rank need enough requests: with eight, p90 and p99 are both the slowest request (lesson 05).

4. A laptop benchmark of a 0.8B model shows a TTFT p50 of 2.3 s. What do you check? The course’s stage 02 record has that figure: 8 requests at concurrency 2 on Ollama, p50 2,281.5 ms and p90 2,526.7 ms, with no warm-up request. The same model in the stage 08 record returned whole answers of 31 to 44 tokens in 548 to 748 ms, one request at a time. So check for a cold model (Ollama unloads one after five idle minutes by default), the runtime’s parallel setting against the test’s concurrency (OLLAMA_NUM_PARALLEL defaults to 1), and where the clock starts. Rerun warm at concurrency 1 and record the setting. The run shows that the client and the routing work against a real model; throughput and cost need the target GPU (lesson 03).

5. How do you route requests across vLLM replicas, and between a self-hosted model and the API? Across replicas, a plain Service balances connections without looking at the pods. An InferencePool behind the gateway hands the choice to an endpoint picker that reads each pod’s queue and KV-cache use and can prefer the replica that holds a conversation’s prefix, so with prefix caching on and append-only prompts later turns skip most of their prefill (lesson 04, lesson 02). Between a self-hosted model and the API, a gateway with explicit rules: a request marked require: local, such as one carrying customer data, never reaches the API even when the local route is down; a local 5xx, timeout or connection error may fall back for unrestricted requests and a 4xx does not; every attempt’s cost counts. The course’s stage 08 run sent six prompts, each landed on its expected route, and the run cost 0.056779 USD, all of it on the three API calls (stage 05 of the exercise, lesson 03).

6. How do you autoscale vLLM on Kubernetes? On the queue. vllm:num_requests_waiting grows when users start waiting for a first token, while CPU says nothing about a GPU-bound process and GPU utilisation stays high whenever any batch runs. A KEDA ScaledObject with a Prometheus trigger divides the queue by threshold, the waiting requests one replica should carry; activationThreshold only decides between zero and one replica, and cooldownPeriod only covers the step to zero. Between other counts the HPA’s stabilisation windows decide, taking the lowest recommendation in the window going up and the highest going down. A new replica needs minutes to load its weights, so a one-minute spike should change nothing: on stage 04’s invented day, a 120-second scale-up window and a 300-second cooldown give two decisions, up to 3 at 12:02 and down to 1 at 13:05 (lesson 04, stage 04 of the exercise).

7. When is self-hosting cheaper than the API? The cost per million output tokens is the GPU’s price for the billed seconds divided by the output tokens produced, in millions. On the course’s invented H100 run at concurrency 16 and $2.99 an hour, that is $1.40 with the GPU busy, $5.58 at 25% busy and $19.78 serving one stream at a time. Opus 5.5 lists at $4 per million input tokens and $20 per million output tokens, costs nothing while idle and bills input separately, while the self-hosted figure already includes the GPU time spent on prefill. Self-hosting is cheaper when the GPU stays busy at high concurrency and an eval shows the open model meets the bar for the task. In the sample, concurrency moves the cost by a factor of 14 and the range of hourly prices by about 4.6 (lesson 05).

8. When would you fine-tune instead of prompting or adding retrieval? For behaviour a prompt cannot hold reliably: a strict output format, a house style, a narrow classification, or a task moved from a large model to a small one. Facts that change and must be cited stay in retrieval: in M4’s live run all three answers cited a chunk holding the query’s anchor, the pipeline refuses on a score it can see, and a changed document is a re-embed (all 685 chunks took 215.3 s on a laptop). Start with LoRA or QLoRA (a rank-16 adapter trains 0.58% of a layer’s weights in the lesson’s example), a 60-step smoke run, then a full run, and ship only if M2’s gate passes against the base model with its best prompt on the same held-out set (lesson 06).

System design prompt

Section titled “System design prompt”

“Serve a 30B open model for 2,000 daily chats and a nightly batch of 1M tokens in the EU; where does it run, how does it scale, what does it cost?”

A short model outline:

  • Ask first. Turns per chat and tokens per turn, the hours chats arrive in and the busiest hour’s share, the TTFT target, whether the batch’s 1M tokens are input or output and by when it must finish, and which 30B model. The licence check comes before hardware: an EU-based company cannot take Llama 4’s multimodal models, so the shortlist is Apache 2.0 or MIT models of about that size, such as Gemma 4 31B or Qwen3.5-27B, compared on the workload’s eval set.
  • Where it runs. vLLM on H100-class GPUs in a Kubernetes cluster in an EU region, which keeps prompts and outputs on infrastructure the company controls. A hosted API for the same model is a different contract and needs its own EU processing terms. Ollama on a laptop serves for developing the client and the gateway.
  • One replica. 30B parameters are about 60 GB in bf16 and 30 GB in FP8; at the default 0.92 of an 80 GB GPU that leaves about 14 GB or 44 GB for overhead and the KV cache. Gemma 4’s launch post says its 31B fits one H100 in bf16, and lesson 02’s arithmetic shows how little cache such a fit leaves. FP8 weights, possibly an fp8 KV cache, and --max-model-len set to the chat’s real length buy the concurrency; the eval set has to pass on the quantised deployment. Prefix caching stays on.
  • How many replicas. Peak output tokens per second is chats × turns × output tokens per turn, divided by the active seconds and multiplied by the peak-hour factor. Divide it by one replica’s throughput at the TTFT target, measured on the target GPU with an open-loop rate sweep. The course has no GPU measurement: its invented sample run gives 595 output tokens per second at concurrency 16, about 2.1 million an hour, which is the scale to compare 2,000 chats a day against. Run two replicas for availability whatever the arithmetic says, so that a node drain, a rollout or a crash leaves chat up, with a PodDisruptionBudget to keep one running.
  • Scaling. KEDA on vllm:num_requests_waiting with a threshold per replica, a scale-up window of a couple of minutes, a longer scale-down window, a minimum of two and a maximum bounded by the GPU quota. Scaling to zero overnight would make the first chat of the morning wait for weights to load (the chart’s startup probe allows about ten minutes), and the batch runs at night anyway. Plan GPUs for rollouts: the chart’s default surge starts a full new set of pods before it stops the old ones. Weights go on a ReadWriteMany volume or one claim per replica, and the Hugging Face token comes from a Secret.
  • The nightly batch. It runs on the same replicas when chat traffic is lowest, from a client that holds a fixed concurrency high enough to fill the batch slots without preemption, checkpoints each item and retries failures. At the sample’s 595 output tokens per second, 1M output tokens take about 28 minutes on one replica; if the 1M are mostly input tokens, measure prefill throughput instead. The batch also fills GPU hours that are paid for anyway, which lowers the cost per token.
  • Routing and data. An InferencePool with an endpoint picker in front of the replicas, for queue- and prefix-aware routing. If the platform also has an API route, customer chats carry require: local, so no fallback sends them out of the region.
  • What it costs. One replica around the clock is 720 GPU-hours a month: $2,153 at $2.99 an hour and $4,954 at $6.88, two of the U.S. on-demand H100 prices lesson 05 cites; EU regions need their own quotes. Two replicas double it. The bill is fixed whatever the traffic, so the cost per chat is the bill over 60,000 chats a month, about $0.072 a chat for two replicas at $2.99, and every idle hour raises it. Compare with the API per token on the measured token counts, keeping in mind that the API bills input separately and needs a contract that keeps the data in the EU.
  • What to measure. Client TTFT p50 and p90 from the first content delta; per-request TPS and throughput, reported separately; vllm:num_requests_waiting, vllm:kv_cache_usage_perc, the prefix-cache hit rate and preemptions per replica; the batch’s completion time; GPU-hours against output tokens for the cost per million; and the eval pass rate by category on the deployed quantisation, rerun whenever the model or the quantisation changes.

Close with the trades: a second replica buys availability with a GPU that idles at night unless the batch fills it; FP8 weights and an fp8 cache buy concurrency at an accuracy risk the eval has to clear; a longer scale-down window keeps users from waiting and pays for idle replicas; and every throughput figure in the outline is a placeholder until the load generator has run on the target GPU at the concurrency the chats need.

Take-home shape

Section titled “Take-home shape”

A good 2 to 4 hour take-home on this topic hands the candidate a fake OpenAI-compatible server whose token rate, queueing delay and slot count are settable, a vendored Helm chart with offline JSON schemas, and a stubbed API client, and asks for three things:

  1. A load generator with tests. Streaming requests at a bounded concurrency, TTFT from the first content delta, TPS and throughput kept apart, a stream without [DONE] scored as a failure, nearest-rank percentiles tested on known lists, and a test that more requests than server slots raise p90 TTFT.
  2. A routing gateway. A local route and an API route, a health probe and a context-length check on the local one, fallback on a 5xx, a timeout or a connection error but not on a 4xx, a require: local constraint that no failure can bypass, and cost and latency per route.
  3. A sizing memo. Manifests for one GPU replica that validate offline and pass a checklist (GPU request, readiness probe, PodDisruptionBudget, memory limit, model volume, token from a Secret, --max-model-len), and a page of arithmetic: replicas for a stated load from a stated throughput, the cost per million output tokens at two utilisations, and the comparison with an API price.

Grade on the edges (a role-only first chunk, a reasoning model that spends max_tokens thinking and returns an empty content, a connection cut mid-stream, a 404 from the local route, an unhealthy local route under require: local), on whether the GPU price and the utilisation are inputs rather than constants, and on honesty about the numbers: a memo that sizes production from a laptop run, quotes a p99 from ten requests or presents server throughput as one user’s speed fails that criterion.