What a non-technical team actually has to understand about a language model

§ 01What the model is doing when it answers

The first thing the room has to lose is the idea that the model looks things up. It does not. A language model is a function that takes a sequence of tokens and returns a probability for every token that could come next; the product samples one, appends it and calls the function again. That loop is the entire mechanism. Nothing in it consults a table of facts, and nothing in it knows whether the sentence it is halfway through is true. Fluency is what the training optimised. Correctness is a side effect that holds when the training data stated the same fact the same way often enough, and fails quietly when it did not.

# the whole mechanism, with everything except the loop removed
def generate(model, prompt_ids, max_new=400, temperature=0.7):
    ids = list(prompt_ids)
    for _ in range(max_new):
        logits = model(ids)            # one score per token in the vocabulary
        probs  = softmax(logits / temperature)
        nxt    = sample(probs)         # sampled, not argmax: same prompt, different answer
        ids.append(nxt)
        if nxt == EOS:
            break
    return ids
# nowhere in here: a lookup, a database, a check against a source

I put this loop in front of HR and legal teams and it works, because nobody needs to read Python to see that there is no lookup() in it. Hallucination stops being a mysterious defect and becomes what the loop does when the most likely continuation is a citation that does not exist: a case reference in the right format, a policy clause in the right tone, a specification with plausible figures. The shape was likely, so the shape was produced; the fact was never something the model held. Two corollaries the room has to hear twice. Confidence in the output is a property of the writing, not of anything behind it: a wrong answer and a right one come out of the same loop at the same speed. And because the next token is sampled, the same prompt gives different answers on different days, which is why a colleague “got something else” and neither of them is wrong about what they saw.

§ 02The context window, and why it forgets

The second fact is that the model has no memory. What looks like memory is the product re-sending the whole conversation on every turn: the system prompt, every earlier message, the document you pasted and the new question, as one sequence. That sequence has a hard maximum, the context window, measured in tokens; a token is roughly three quarters of an English word and closer to half a Spanish one, because the tokenisers were built mostly on English text¹. When the sequence would exceed the window the product drops something, normally the oldest turns, and the model “forgets” the instruction you gave at the start. It did not forget. On that call, it never saw it.

Tokens are also the unit of billing, and the two directions are not priced alike: at a price P_in per million input tokens, output typically costs three to five times as much, and every turn re-bills the whole context as input. The table gives the arithmetic for four tasks the room recognises, in multiples of P_in, assuming 1.35 tokens per English word and 1.8 per Spanish word, a 300-token system prompt, output at 4 × P_in, and a ten-turn conversation counted as ten re-sends of the document plus a growing history.

Two things the table teaches without a slide. A conversation over a long document costs an order of magnitude more than one summary of it, so the habit worth forming is to ask the question together with the document, not in the turn after it. And classification, the highest-volume task, is the cheapest per run and the one a small model measured on the task usually wins; that decision is N.06. Nobody in the room needs a price. They need the ratios.

§ 03Why arithmetic and dates fail

Two failure classes that non-technical users hit weekly, and both follow from the loop. Numbers are tokens: “4,187” is one token, two or four depending on the tokeniser, and what the model learned is the statistics of digit strings, not the algorithm for long multiplication. It writes a product the way it writes a sentence, and for small operands the training data contained the answer often enough that the writing is right; for an invoice total across forty lines it is not, and nothing in the output will say so. Dates fail differently. The model has a training cutoff, it does not know today's date unless the product injects it into the context, and “within thirty days of signature” needs the arithmetic it cannot do plus the date it does not have, plus a calendar of bank holidays it may have half-memorised for another country.

The rule I teach is short. The model may explain a calculation, set one up, or turn it into a spreadsheet formula; it may not perform one whose result someone will act on. Where a workflow needs the number, it comes from a tool the model calls or a deterministic layer beside it, with a test that every figure reaching a user arrived by that path: the same split as nutritional values in N.14, for the same reason: a plausible number is worse than a refusal.

§ 04The four checks

Verification is taught as a habit with four moves, and each move is a question you can type. The moves are in the list below; the one people find surprising is the passage. A model that has the document in its context can quote it verbatim, and you can search for the quote in the original; a model that does not will produce a paraphrase inside quotation marks, the search fails, and the failure is the answer. In a retrieval assistant this fourth check is a click on the citation, which is why those systems are built to return the paragraph rather than a paraphrase (N.03).

The exercises are role-specific, built on the team's own material, and each has the model wrong in a way one of the checks catches. HR asks about notice periods and receives the statutory default instead of the collective agreement. Legal asks for case law and receives a well-formed reference the database does not contain. Sales asks for a competitor's specification and receives a plausible figure the published sheet contradicts. Operations asks for a delivery date and receives one on a bank holiday. The exercise ends when the participant has found the error with a check, not when they have been shown the right answer; that is what makes it stick, and what makes it measurable later.

§ 05Four data classes, and what may leave the building

The policy that follows the session has to be enforceable by people who will not read the GDPR, so it uses four classes and one question per class: which tools may this go into? Public is what is already published, web copy, price lists, and may go anywhere. Internal is unpublished but not damaging, process drafts without names, anonymised notes, and may go into approved tools whose contract excludes training on the input. Confidential is client contracts, financials, source code, anything under NDA, and any personal data, which under the GDPR starts at a name next to a role: only tools with a data-processing agreement, hosted in the EU or in the building, never a consumer chat. Restricted is special-category personal data (health, union membership, biometrics), trade secrets and anything a regulator controls, such as an MDR technical file: an in-building model or nothing. When in doubt, one class up.

The distinction that matters most in practice is between a consumer chat product and an enterprise API for the same model. The first typically uses conversations to improve the service unless the user opts out, under a personal account the company cannot see; the second is typically covered by a contract that excludes training and bounds retention to a fixed abuse-monitoring window. Same model, different building. The policy exists in two forms: the one-page text people read, and a machine-readable file that the prompt library and the tool allow-list are checked against, so that a prompt tagged confidential cannot be pointed at a consumer tool by accident.

# policy/data-classes.yaml: the machine-readable half of the one-page policy
classes:
  public:       { may_use: [consumer_chat, enterprise_api, local_model] }
  internal:     { may_use: [enterprise_api, local_model],
                  requires: [no_training_clause] }
  confidential: { may_use: [enterprise_api, local_model],
                  requires: [dpa, eu_region, no_training_clause, retention_days_max_30] }
  restricted:   { may_use: [local_model], requires: [dpo_signoff] }
examples:
  public:       ["published price list", "web copy"]
  internal:     ["process draft without names", "anonymised meeting notes"]
  confidential: ["client contract", "a CV", "any name next to a role", "source code"]
  restricted:   ["sick-leave records", "union membership", "MDR technical file"]
tools:
  consumer_chat:  { trains_on_input: default_on, dpa: false }
  enterprise_api: { trains_on_input: false, retention_days: 30, dpa: true, region: eu }
  local_model:    { trains_on_input: false, retention_days: 0 }

Two legal anchors, and no more than two, because the audience will not retain a third. Under the GDPR the provider is a processor and needs a data-processing agreement, and personal data needs a lawful basis for the use it is being put to. Under the EU AI Act, Article 4 obliges providers and deployers to ensure a sufficient level of AI literacy among the staff who operate these systems, applicable since February 2025²; the session, its attendance record and the benchmark below are the evidence that the obligation was met, which is a better reason to keep them than habit.

§ 06The prompt library as a repository

A prompt library kept in a shared document dies within a month, because nobody owns any line of it and nobody knows which model a prompt was written against. I keep it as a git repository: one file per prompt, with front-matter for the id, an owner who works in the department rather than in IT, a version, the model it was last tested on, the highest data class it may be used with, and a test file beside it. Nothing enters main without an owner and three test cases, and when the model behind the tool changes the whole suite runs again, because a prompt is not portable across models the way a formula is across spreadsheets.

# prompts/hr/notice-period-check.md
---
id: hr/notice-period-check
owner: hr-lead
version: 3
tested_on: "provider/model-name@2026-05"
max_data_class: internal
tests: hr/notice-period-check.tests.yaml
---
You answer questions about notice periods using ONLY the collective
agreement supplied between <agreement> tags. If the text does not
cover the case, answer exactly: NOT IN AGREEMENT: ask HR.
Give the article number for every statement you make.

<agreement>{agreement}</agreement>
Question: {question}

The tests are properties, not exact strings: the output must contain an article number, must not contain a figure absent from the input, must abstain with the agreed sentence when the case is not covered. They run in CI against fixed inputs, and a change to a prompt is a pull request with the diff and the test results attached. The tests are also the second half of the training: an owner who has written three failing cases for their own prompt understands the model better than one who has sat through any number of slides.

# tests/test_prompts.py: one parametrised test over every prompt's cases
CASES = [(p, c) for p in load_prompts("prompts/") for c in load_yaml(p.tests)]

@pytest.mark.parametrize("prompt,case", CASES, ids=lambda x: getattr(x, "id", None) or x.get("name"))
def test_prompt(prompt, case):
    out = run(prompt, model=prompt.tested_on, **case["vars"])
    for s in case.get("must_contain", []):
        assert s in out
    for s in case.get("must_not_contain", []):
        assert s not in out
    if case.get("must_abstain"):
        assert out.strip().startswith("NOT IN AGREEMENT")
    for n in numbers_in(out):                 # no figure that is not in the input
        assert n in numbers_in(case["vars"]["agreement"])

# hr/notice-period-check.tests.yaml (one of three)
# - name: case-not-covered
#   vars: { agreement: "…art. 14: 15 days for technicians…", question: "and for interns?" }
#   must_abstain: true

§ 07What I measure

A satisfaction survey after the session measures how the session felt. It says nothing about whether anyone works differently on Monday, and it is what most programmes report. What I measure is task performance. Before the session, each participant does three tasks from their own role on their own material, timed, and the output is scored against a rubric by someone who did not attend. Two to four weeks after, the same design on held-out tasks, matched in difficulty, not identical, with the same rubric and the same blind scorer. The metrics: time to complete, uncaught errors per task, errors caught (the participant flagged something for verification), and policy violations (a confidential string in a prompt). The last one decides whether the programme worked at all; the others decide whether it was worth the time.

The table is a worked example of what the sheet looks like for a team of twelve with three tasks each; the numbers illustrate the design and are not a result I am reporting for a client. What I do not count is usage: prompts per week, minutes in the tool. Usage is not competence, and rewarding it teaches people to paste more.

If the after column does not move, the session was pleasant and the material was wrong for the team; the fix is different exercises, not a second session with the same ones. If errors caught rises while uncaught errors stay flat, the checks were learned but the holdout tasks were harder than the baseline, and the set needs rebalancing before anyone concludes anything. Both are readable from the sheet, which is the point of having one.