The right size of model for the task, decided by a number

§ 01Capability the task never uses

A frontier model earns its price on the work that needs it: long documents, several steps of reasoning, an instruction it has never seen. A classification with twelve labels, an extraction of six fields from a form that always looks the same, a routing decision that picks one of four queues: these use a sliver of that capability, and the rest is paid for, and waited for, on every call.

The waiting is the part people forget. A hosted frontier model answers a short classification in one to three seconds at the 95th percentile; a three-billion-parameter model on one mid-range GPU answers it in under 300 ms, and it sits inside the building, which matters when the input is a customer's email. On cost, the ratio between the smallest tier that clears the bar and the largest is somewhere between ten and a hundred, depending on the provider and the year, and it compounds with volume: a task that runs fifty thousand times a day is a budget line, not an experiment.

None of that argues for the small model. It argues for measuring. The right size is the smallest model that clears a quality bar the task itself sets, and “clears” has to mean a number on an evaluation set I can show, not an afternoon of promising examples. Where no small model clears it, the frontier model is the answer, and I write that down with the same care.

§ 02The evaluation set comes first

Before a single model runs, the task gets a gold set. For classification and routing that is a stratified sample from the real input distribution, 300 to 500 items, every class present, the rare classes over-sampled and re-weighted at scoring time, labelled independently by two people, with disagreements adjudicated and the adjudication recorded. For extraction the target fields are filled by hand and normalised (dates to ISO 8601, amounts to minor units, identifiers stripped of formatting), so that a match compares values rather than strings; the schema itself, with its confidence and provenance fields, is the subject of another note.

The set is versioned like code, split into a development half used to choose prompts and thresholds and a held-out half touched only when a decision is taken, and it carries a “hard” slice, the items the two labellers disagreed on first, because that is where models separate. It is never used for training: when I later distil or fine-tune, the training data is a different sample, and I check the overlap by hash before every run.

The metric is chosen per task and fixed before the ladder runs: macro-F1 for classification, because it refuses to let a common class hide a rare one; per-field exact match after normalisation for extraction, with a null counted correct only when the field is genuinely absent; for routing, accuracy weighted by the cost of each misroute, since sending an urgent case to the slow queue is not the same error as the reverse. The harness is small and dull on purpose, and a model that answers outside the schema is scored as wrong, not retried.

# eval.py: one task, one candidate, one number
import json, time, hashlib
from sklearn.metrics import f1_score

def run(candidate, gold_path, prompt_version):
    gold = [json.loads(l) for l in open(gold_path)]
    y, yhat, lat, toks = [], [], [], []
    for ex in gold:
        t0 = time.perf_counter()
        out = candidate.classify(ex["text"], prompt_version)  # -> {label, confidence, usage}
        lat.append(time.perf_counter() - t0)
        y.append(ex["label"])
        yhat.append(out.get("label") or "__parse_error__")   # outside the schema = wrong
        toks.append(out["usage"]["input"] + out["usage"]["output"])
    lat.sort()
    return {
        "candidate": candidate.name, "prompt": prompt_version,
        "gold_sha": hashlib.sha256(open(gold_path, "rb").read()).hexdigest()[:12],
        "n": len(gold),
        "macro_f1": round(f1_score(y, yhat, average="macro"), 4),
        "parse_errors": yhat.count("__parse_error__"),
        "p50_ms": round(lat[len(lat) // 2] * 1000),
        "p95_ms": round(lat[int(len(lat) * .95)] * 1000),
        "tokens_per_run": round(sum(toks) / len(toks)),
    }

§ 03The candidate ladder, measured the same way

Every task gets the same three rungs: a small open model that runs on one GPU inside the building (the 1–4 B class), a mid-size model (the 7–14 B open class, or a provider's smaller hosted tier), and a frontier model. Same prompt, same output schema, same decoding, temperature 0, a JSON schema or grammar constraining the output, so that the only variable is the model. Serving the local rungs is its own note.

The cost column is arithmetic, not a quote: cost per thousand runs = 1,000 × (input tokens × price per input token + output tokens × price per output token). For a hosted model the prices are on the list. For a local model the price is the GPU-hour cost divided by sustained throughput at the batch size the workload allows, for a 3 B model on one mid-range card and a 900-token prompt, in the low thousands of requests per hour, so it is small but not zero, and it does not fall while the card sits idle.

The worked example below is a twelve-class routing task: 900 input tokens (a 700-token static prefix with the label definitions plus about 200 tokens of input) and 40 output tokens. Prices are multiples of P, the blended price per million tokens of the small tier; I have set mid = 6 P and frontier = 50 P, inside the range I have seen, to be replaced with your own price list on the day.¹ Quality is macro-F1 on a 480-item held-out set. The numbers show the shape of the decision; they are not a benchmark.

Reading it: zero-shot, the small model is not good enough, and eleven items it could not even format. Two thousand labelled examples and an hour of LoRA move it to within a point of the frontier at a fiftieth of the cost and a tenth of the latency. The cascade recovers the last point by sending the twelve per cent it is least sure about upward: and note what it does not buy: its tail latency is the frontier's, because p95 sits inside the escalated fraction. Under a latency budget the cascade is a cost decision, not a speed one. Whatever the outcome, it is written where the next engineer will find it:

§ 04When the task is stable: fine-tune, distil

The move from 0.81 to 0.93 in the table is the one that decides most ladders, and it is only available when the task is stable: a label set that does not change monthly, an input distribution a sample can represent, and enough volume that a day of work is repaid. Under those conditions I fine-tune with LoRA on a few thousand examples rather than touching every weight: it trains in an hour on one card, the adapter is a file I can version and roll back, and the base model stays shared between tasks.

Where labelled data is short, the frontier model becomes the labeller. Distillation here is prosaic: run the frontier with the production prompt over five to twenty thousand unlabelled inputs, keep its label and its stated confidence, drop the least confident third, and fine-tune the small model on the rest. The student inherits the teacher's mistakes, so the gold set, human labels, never the teacher's, is the only score that counts, and I read per-class recall, because a teacher weak on one rare class produces a student silently blind to it. The gate at the bottom of the file below is the point of the exercise: an adapter that scores under the bar on the human set is not promoted, however good the training loss looked.

# distil-route.yaml: student run, adapter only
base_model: "<small-3b-instruct>"
adapter: lora
lora: { r: 16, alpha: 32, dropout: 0.05,
        target: [q_proj, k_proj, v_proj, o_proj] }
train:
  file: data/silver/route-inbound-teacher-v2.jsonl   # 14,200 rows, teacher confidence >= 0.80
  exclude_sha: gold/route-inbound.sha256             # refuse any row that is in the gold set
  epochs: 2
  lr: 1.5e-4
  max_len: 1024
eval:
  gold: gold/route-inbound.jsonl                     # human labels only, held-out half
  metric: macro_f1
  gate: 0.92                                         # below this the adapter is not promoted

§ 05A router with a calibrated threshold, a cache and a batch

The cascade needs a confidence, and a small model's stated confidence is not one until it is calibrated. I take the probability of the chosen label token from the logprobs where the serving stack exposes them, or the model's own confidence field where it does not, then choose the threshold on the development half of the gold set: for each candidate τ, compute the coverage (the fraction answered locally) and the accuracy of the covered items, and keep the τ with the widest coverage whose covered accuracy meets the target. The value is confirmed on the held-out half and written into the decision record. Chosen by feel, thresholds look prudent and escalate half the traffic.

def pick_threshold(dev, target=0.95, lo=0.50, hi=0.995, step=0.005):
    # dev: [(confidence, correct)] from the small model on the development half
    best, t = None, lo
    while t <= hi:
        covered = [ok for conf, ok in dev if conf >= t]
        if covered:
            cov, acc = len(covered) / len(dev), sum(covered) / len(covered)
            if acc >= target and (best is None or cov > best["coverage"]):
                best = {"threshold": round(t, 3), "coverage": round(cov, 3), "covered_acc": round(acc, 4)}
        t += step
    return best   # e.g. {"threshold": 0.86, "coverage": 0.88, "covered_acc": 0.957}

def route(x, small, frontier, tau):
    out = small.classify(x)
    if out["label"] and out["confidence"] >= tau:
        return {**out, "tier": "small"}
    return {**frontier.classify(x), "tier": "frontier", "escalated_from": out}

Two things sit beside the router. A cache, keyed on the hash of the normalised input together with the prompt version and the model identifier, because a change to either invalidates the answer; on classification workloads the exact-match hit rate is often 10–30 %, since the same document arrives twice, and I do not use a semantic cache here, because two inputs that embed close together are precisely the pair a classifier exists to tell apart. And batching, which is where a local model's cost actually falls: the interactive path serves one request at a time inside its budget, and everything that can wait, nightly reclassification, backfills, evaluation runs, goes through the batched path, where throughput per GPU-hour is several times higher. Hosted providers sell the same trade as a discounted batch tier; it is the same decision.

§ 06What I measure

The decision is right on the day it is written, so the same harness keeps running after it. Per release: macro-F1 on the held-out gold set, for the small model alone and for the cascade. Continuously: the escalation rate, because a rise means the input has moved or the model has been changed under me; the confidence histogram of the small model, whose shape shifts before the accuracy does; the parse-error rate, the first sign that a base-model update changed the output format; p50 and p95 latency per tier; cost per thousand runs from actual token counts, not the estimate; and the cache hit ratio, since a fall in it usually means the normalisation broke, not that the traffic changed.

For drift I keep a shadow sample: one per cent of production inputs also go to the frontier model, and the disagreement between the tiers is tracked as a series. Disagreement is not error, the frontier is wrong too, but a step in it is the reason to relabel a fresh gold sample and rerun the ladder. When the numbers move, the decision record gets a new line and the old one stays.

The failure modes I have met are all versions of the same thing, a number that stopped being true: a threshold calibrated on last year's distribution; a gold set that leaked into a training file through a shared export; a student that inherited the teacher's blind spot on the one class that mattered legally; a “cheaper” cascade whose retries on parse errors doubled its real token count. Each was found by a metric on the list above, and none by reading outputs.