Serving open models inside the building: sizing, quantisation and the API shape

§ 01Why the model runs here at all

The reason to serve a model on your own hardware is almost never price and almost always the wall: a data class that may not leave the building or a regulator who will ask where the prompt went. That decision belongs to data classification and regulation, not to preference, and once it is taken you inherit everything the provider used to do for you: hardware, sizing, patching, the boundary, and the queue.

The host is a product, not a workstation. I treat the GPU box the way I treat any controlled environment: a minimal image with the driver, the container runtime and the serving container, model weights pinned by content hash the same way packages are pinned, and nothing listening that I cannot name. The rest is the arithmetic that decides what fits and the contract that keeps applications ignorant of the answer.

§ 02The memory arithmetic

Two things occupy a card: the weights, which are fixed, and the KV cache, which grows with every token that is resident. Weights are parameters times bytes per weight: an 8-billion-parameter model at BF16 is 16 GB, at 8-bit about 8 GB, at 4-bit a little over 4 GB plus whatever is left at higher precision. The KV cache is where the surprises live. For every token in every sequence the model keeps a key and a value vector per layer, so the cost per token is 2 × layers × kv_heads × head_dim × bytes. All four numbers are in the model's config.json; grouped-query attention is why num_key_value_heads is smaller than num_attention_heads, and the reason a Llama-3-class model is cheaper to serve than a Llama-2-class one of the same size.

import json

def kv_bytes_per_token(config_path, kv_bytes=2):
    c = json.load(open(config_path))                # the model's config.json
    layers   = c["num_hidden_layers"]
    kv_heads = c.get("num_key_value_heads", c["num_attention_heads"])
    head_dim = c.get("head_dim", c["hidden_size"] // c["num_attention_heads"])
    return 2 * layers * kv_heads * head_dim * kv_bytes   # K and V, one token

def resident_tokens(weights_gib, kv_per_tok, card_gib=24, reserve_gib=2.0):
    budget = (card_gib - reserve_gib - weights_gib) * 2**30
    return int(budget // kv_per_tok)

# Llama-3.1-8B: 32 layers × 8 KV heads × 128 dims → 131 072 B per token at 16-bit
# Llama-3.1-70B: 80 × 8 × 128 → 327 680 B.  Qwen2.5-7B: 28 × 4 × 128 → 57 344 B
resident_tokens(14.96, 131072)   # BF16 weights on 24 GiB: ~57 700 tokens
resident_tokens(5.5,   131072)   # AWQ 4-bit, embeddings kept 16-bit: 135 168
resident_tokens(5.5,    65536)   # … and FP8 KV cache: 270 336

So a Llama-3.1-8B holds 128 KiB of cache per token at 16-bit, and a 32k-token conversation costs 4 GiB before a second user arrives; the 70B holds 320 KiB per token, so the same conversation costs 10 GiB. That is why context length is a capacity parameter and not a feature toggle. vLLM pre-allocates the KV pool at start-up from whatever memory is left after the weights and prints the number of cache blocks it obtained; that line in the log is the true capacity of the service, and I check it before I believe any throughput number.

§ 03What quantisation costs and buys

Quantisation stores the weights in fewer bits and dequantises on the fly. The formats that matter in practice are 8-bit (INT8 or, on hardware that has it, FP8), which halves the weights and, in published evaluations and in what I have measured, sits inside benchmark noise; and 4-bit, either AWQ or GPTQ with 128-weight groups at roughly 4.25 bits per weight once scales are counted, or GGUF's Q4_K_M at about 4.85. At 7–8B parameters the 4-bit loss is typically a point or two on knowledge benchmarks and larger on tasks that need exact formatting, long structured output or arithmetic; at 70B it is smaller. Below 4 bits the curve steepens quickly and the loss becomes task-specific. None of these numbers replace an evaluation set of your own; they tell you which candidates are worth running through it.

What quantisation buys is not only that the model fits. Decoding at low batch is memory-bound, every generated token reads every weight once, so fewer bytes per weight is directly more tokens per second per stream. At high batch the arithmetic dominates and dequantisation is a tax, which is what the Marlin kernels in vLLM exist to reduce. Quantising the KV cache to FP8 is a separate lever: it halves the per-token cost at a quality cost that is usually small, but I check it specifically on the longest prompts in the eval set, because that is where the error accumulates.

§ 04A sizing worked example

The assumptions: one 24 GiB card; 2 GiB reserved for the CUDA context, activations and fragmentation; a Llama-3.1-8B-class model, whose KV cost is 128 KiB per token at 16-bit and 64 KiB at FP8; 4-bit weights of 5.5 GiB because the embeddings and the output head stay at 16-bit; and requests of 4,096 tokens, prompt and answer together, which is what a retrieval assistant with a few passages typically spends. "Resident tokens" is the whole pool, shared between every sequence in flight; the last column divides it by the request size.

Two things the table does not say. First, sixty-six resident sequences is a memory figure, not a throughput one: an 8B on that card is compute-bound well before that, so the number to find with a load generator is the batch at which p95 time-to-first-token crosses your budget. Second, prefix caching changes the arithmetic for the better in exactly the systems I build: a retrieval assistant sends the same long system prompt with every request, and vLLM shares those cache blocks across sequences instead of recomputing them, so the effective cost per request is the passages plus the answer.

§ 05vLLM or Ollama

They solve different problems and the choice is rarely close. Ollama wraps llama.cpp: it pulls GGUF files, runs on a CPU, an Apple laptop or a single consumer GPU, offloads layers when the model does not fit, and exposes an OpenAI-compatible endpoint under /v1 with one command. Its concurrency is a fixed number of parallel slots per loaded model, its scheduling is simple, and it does not export Prometheus metrics on its own. That fits a developer's machine, an edge box with one user, or an internal utility used a few times a day.

vLLM is a serving engine: paged KV cache, continuous batching so a finished sequence frees its slot mid-batch, prefix caching, tensor parallelism across cards, safetensors plus AWQ, GPTQ and FP8 checkpoints, an OpenAI-compatible server built in, and a /metrics endpoint that already speaks Prometheus. It wants a proper GPU and a Python stack you will have to pin. The rule I use is the queue: if requests ever wait for each other, if more than a handful of clients share the model, or if anyone will ask for a p95, it is vLLM.

§ 06The API shape and the boundary

Every application talks to /v1/chat/completions, /v1/embeddings and /v1/models, and to nothing else. Both engines expose that shape, so the only things a client changes between the local service and a hosted provider are the base URL, the key and the model name: and the model name is an alias that a gateway resolves, so no application ever carries a vendor's model id. That is the whole portability story: a retrieval assistant or an agent's tool loop moves between local and hosted by configuration, which is what lets the data class decide where each workload runs.

import os
from openai import OpenAI
# The only difference between local and hosted is the base URL and the key.
client = OpenAI(base_url="https://llm.internal.example/v1",
                api_key=os.environ["LLM_GATEWAY_KEY"])
r = client.chat.completions.create(
    model="chat-default",          # alias → chat-8b (local) or a hosted model
    messages=[{"role": "system", "content": SYSTEM},
              {"role": "user",   "content": question}],
    temperature=0, max_tokens=600,
    extra_headers={"X-Request-Id": request_id, "X-Data-Class": "internal"})
print(r.usage.prompt_tokens, r.usage.completion_tokens, r.model)

The gateway is where the boundary lives, and it is a separate process from the engine on purpose. It holds one key per application, not per person, and turns each into a request-per-minute and a token-per-minute limit, because a single misbehaving batch job can otherwise starve everyone. It maps aliases to backends and can move an alias between the local engine and a hosted provider without a deploy. It logs a record per request, key, alias, resolved model and its hash, prompt and completion tokens, queue time, latency, request id, and it does not log bodies by default. This last point is where "local" fools people: a prompt that never left the building is still copied into a log file, and that copy inherits the data class of what it contains, with a retention rule and an access list of its own. Where bodies must be kept for audit, the X-Data-Class header routes them to a store with those rules.

services:
  vllm:
    image: vllm/vllm-openai:v0.10.1
    command: >
      --model /models/Llama-3.1-8B-Instruct-AWQ-INT4
      --served-model-name chat-8b
      --quantization awq_marlin --kv-cache-dtype fp8
      --max-model-len 16384 --gpu-memory-utilization 0.90
      --enable-prefix-caching --api-key ${VLLM_INTERNAL_KEY}
    volumes: ["/srv/models:/models:ro"]        # weights pinned by hash, read-only
    deploy: { resources: { reservations: { devices:
      [{ driver: nvidia, count: 1, capabilities: [gpu] }] } } }
    networks: [inference]                       # no published port: only the gateway
  gateway:
    image: ghcr.io/berriai/litellm:main-stable
    command: ["--config", "/etc/litellm/config.yaml", "--port", "4000"]
    volumes: ["./gateway.yaml:/etc/litellm/config.yaml:ro"]
    environment: [VLLM_INTERNAL_KEY, LITELLM_MASTER_KEY, DATABASE_URL]
    ports: ["127.0.0.1:4000:4000"]             # TLS terminates in front of this
    networks: [inference]
  prometheus:
    image: prom/prometheus:v2.54.1
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml:ro"]
    networks: [inference]                       # scrapes vllm:8000/metrics, gateway:4000/metrics
networks:
  inference: {}

§ 07What I measure, and what goes wrong

Five numbers on one dashboard, all from vLLM's own exporter. Time to first token at p95, which is what a person perceives as slowness. Time per output token at p95, which is what a stream feels like once it starts. Requests running against requests waiting, because a non-zero waiting count for more than a minute means the KV pool or the compute is saturated and the fix is capacity or a limit, not a restart. And gpu_cache_usage_perc together with the preemption counter, because when the pool fills vLLM evicts sequences and recomputes them later, which shows up to the user as a latency cliff with no error attached. Tokens per second, prompt and generation, are the capacity plan's feedback loop and the number that tells me when a second card pays for itself.

# p95 time-to-first-token over 5 minutes, per served model
histogram_quantile(0.95,
  sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

# saturation: anything waiting for a sustained minute is a capacity signal
avg_over_time(vllm:num_requests_waiting[1m]) > 0

# the pool is full and sequences are being evicted and recomputed
vllm:gpu_cache_usage_perc > 0.90 and rate(vllm:num_preemptions_total[5m]) > 0

# aggregate throughput, the capacity plan's feedback loop
sum(rate(vllm:generation_tokens_total[5m])) + sum(rate(vllm:prompt_tokens_total[5m]))

The failures are mostly memory failures wearing other clothes. A prompt longer than max-model-len is a clean 400, but a burst of prompts just under it fills the pool with a handful of sequences and everyone else queues; the fix is a lower per-key token limit at the gateway, not a bigger card. A driver or CUDA update that changes how much memory the runtime reserves turns a start-up that fit yesterday into an out-of-memory today, which is why the reserve is 2 GiB and not 1, and why the image is pinned. A card that throttles thermally halves throughput with no error in any log except the temperature sensor. And the quietest one: an alias repointed or a checkpoint re-downloaded, so that "chat-default" is no longer the model the evaluation ran on. I put the resolved model name and its weight hash on the response headers and in the request log, and the evaluation set from the sizing note runs against the alias on a schedule, so a silent change becomes a failed check the same day.