Scoring where a model belongs: a use-case ledger with a cost per case and a risk class

§ 01The walk-through produces candidates, not decisions

The assessment starts in the department, not in the tool. I spend an hour or two with each team that does the work and record tasks, not wishes: what arrives, what leaves, how often, how long it takes, who decides, and what happens when it is wrong. A task is a candidate when it is repetitive, when its input and output are text or structured data, and when it is light on judgement: a competent new hire could do it after a week with the written procedure. Anything that ends in a decision about a person, hiring, credit, discipline, a clinical call, is recorded too, but it enters the ledger flagged, because the Regulation treats it differently and so does the cost model.

The walk-through is deliberately shallow. It yields somewhere between twenty and sixty candidate rows for a mid-sized organisation, most of which will not survive scoring; the point is that they all sit on one sheet, described in the same fields, so the comparison is between numbers rather than between whoever argued best in the meeting. Only three fields are filled in the room: a one-sentence description with the verb first, a first estimate of volume, and where the input lives. Everything else is measured afterwards, and the row says so.

§ 02The scorecard is a schema, not a slide

I keep the scorecard in a spreadsheet during the assessment, because that is what the finance team will open, and I export it to a small database so the cost model and the classifier can run over it and run again when an assumption moves. The tool is irrelevant; the schema is the deliverable. Every column has a definition and a provenance: measured, audited, estimated or assumed.

-- one row per candidate task; the definitions travel with the schema
CREATE TABLE use_case (
  id               text PRIMARY KEY,           -- 'FIN-03'
  department       text NOT NULL,
  task             text NOT NULL,              -- one sentence, verb first
  input_kind       text NOT NULL,              -- pdf | email | erp_record | free_text
  output_kind      text NOT NULL,              -- label | fields | draft | decision
  volume_month     integer NOT NULL,           -- measured at handling, not at arrival
  minutes_case     numeric NOT NULL,           -- timed at the desk on >= 30 cases
  error_rate_base  numeric NOT NULL,           -- audited on a random sample
  error_cost_eur   numeric NOT NULL,           -- one error nobody catches
  error_cost_tail  numeric,                    -- the rare bad case, if it exists
  reversible       boolean NOT NULL,           -- can the output be undone cheaply
  tokens_in_p50    integer,  tokens_in_p95 integer,
  tokens_out_p50   integer,
  review_minutes   numeric,                    -- a person checks one output
  data_readiness   smallint CHECK (data_readiness BETWEEN 0 AND 3),
  affects_person   boolean NOT NULL,           -- output touches a natural person
  annex_iii_area   text,                       -- NULL | employment | essential_services | ...
  ai_act_class     text CHECK (ai_act_class IN
                     ('prohibited','high','transparency','minimal')),
  provenance       jsonb NOT NULL,             -- {"minutes_case":"timed","error_cost_eur":"assumed",...}
  owner            text NOT NULL,              -- who answers for it after rollout
  review_date      date NOT NULL               -- when the class is read again
);

Two columns do most of the work. volume_month and minutes_case are measured, not asked for: people count cases at arrival rather than at handling, and remember the task without its interruptions, and both errors flatter the business case. error_cost_eur is the cost of an error nobody catches, the invoice posted to the wrong account, the ticket that reaches the wrong engineer, the clause missed in a contract, because a review step will catch some errors and its cost has its own line. Where the tail matters, error_cost_tail holds the rare bad case: a mean of forty euros hides the one in a thousand that costs forty thousand, and reversible is what separates a task whose errors are annoying from one whose errors are final.

§ 03The cost model, with the assumptions written down

For each row I compute five numbers per case and one per month. The human cost per case is minutes × L/60, with L the fully loaded hourly rate of the role that does the task today. The model cost per run is (tokens_in × P_in + tokens_out × P_out) / 10⁶ plus the month's fixed cost, hosting, monitoring, the hour a week of the person who owns it, divided by the month's volume. The review cost is the minutes a person spends checking one output, times L. The expected error cost is the model's error rate after review, times the cost of an uncaught error. And the baseline error cost is the same product for the human process, because the people doing the task today also make mistakes and the model has to beat that figure, not zero. The net saving per case is the human cost plus the baseline error cost, minus the model, review and new error costs; per month, times volume.

from dataclasses import dataclass

@dataclass
class Assumptions:
    labour_eur_h: float = 32.0      # fully loaded, the role that does the task today
    p_in: float = 1.0               # per 1e6 input tokens: a parameter, not a quote
    p_out: float = 4.0              # output tokens are priced several times input
    fixed_eur_month: float = 400.0  # hosting, monitoring, an hour a week of an owner

def cost_per_case(r, a: Assumptions) -> dict:
    human   = r.minutes_case * a.labour_eur_h / 60
    tokens  = (r.tokens_in_p50 * a.p_in + r.tokens_out_p50 * a.p_out) / 1e6
    model   = tokens + a.fixed_eur_month / max(r.volume_month, 1)
    review  = (r.review_minutes or 0) * a.labour_eur_h / 60
    err_new = r.error_rate_model * (1 - r.review_catch) * r.error_cost_eur
    err_old = r.error_rate_base * r.error_cost_eur
    net     = human + err_old - model - review - err_new
    return dict(human=human, tokens=tokens, model=model, review=review,
                err_new=err_new, err_old=err_old,
                net_case=net, net_month=net * r.volume_month)

def break_even_error_rate(r, a: Assumptions) -> float:
    """Model error rate after review at which the row stops paying."""
    c = cost_per_case(r, a)
    return (c["net_case"] + c["err_new"]) / (r.error_cost_eur * (1 - r.review_catch))

def rank(rows, a: Assumptions):
    # readiness discounts the saving; high-risk rows leave the first wave
    w = {3: 1.0, 2: 0.7, 1: 0.35, 0: 0.0}
    scored = [(cost_per_case(r, a)["net_month"] * w[r.data_readiness], r) for r in rows]
    first  = [x for x in scored if x[1].ai_act_class in ("minimal", "transparency")]
    return sorted(first, key=lambda x: -x[0]), [x for x in scored if x not in first]

The table runs the model over four rows with the assumptions in the code: L = 32 €/h; P_in = 1 and P_out = 4 per million tokens as parameters, not prices: substitute yours, what matters is that output is priced several times input; a fixed cost of 400 € per month per row; token counts at p50 on real inputs; error rates from a two-hundred-case pilot; and a review catch rate estimated from the same pilot. Every figure a reader can dispute is on the sheet, which is the point.

Error rates behind the table: FIN-03 2 % baseline, 3 % model, review catches 80 %; OPS-07 5 % and 6 %, no review because a misroute is caught downstream at the cost in the column; LEG-01 1 % and 4 %, review catches 90 %; HR-02 8 % and 8 %, review catches half. Three things fall out. The token line is negligible in every row, under one per cent of the cost per case, so which provider is cheaper is the wrong debate for this kind of work; what moves the number is review time, fixed cost at low volume, and the two error terms. The row with the largest monthly saving, the CV screen, rests on the least measurable input in the table, a nine-hundred-euro guess for a wrong rejection, and it is the one row in an Annex III area, so it leaves the first wave whatever its total says. And the clause summary pays only because review catches nine errors in ten; at six in ten the row turns negative. Which model to run in each row, at what cost per thousand runs, is a separate decision with its own note.

§ 04Data readiness in four levels

A saving that depends on data nobody can reach is not a saving this year. Each row carries a readiness level, defined so that two people would assign the same one:

The readiness check is also where GDPR enters the row: which fields are personal data, under what lawful basis they are processed today, and whether they may leave the building at all. That answer decides hosted against self-hosted before any model is chosen, and it decides it per row: the same company can have a level-3 invoice case that runs on a hosted model and a level-3 HR case that may not.

§ 05One risk class per row, read from the Regulation

The EU AI Act classifies uses, not models and not companies, so the class is a column on the ledger and it is filled per row. Article 5 lists the prohibited practices, emotion recognition in the workplace, social scoring, untargeted scraping of faces, and any row that touches one is closed. Annex III lists the high-risk areas: biometrics, critical infrastructure, education, employment (recruitment and selection, task allocation, monitoring and evaluation of workers), access to essential services including credit and insurance, law enforcement, migration and justice. Employment is the area that catches most of what a company means by “AI in HR”, and essential services catches most of what a bank or an insurer means by anything. Article 50 adds transparency duties where people interact with a system or read generated content. Everything else is minimal risk, which is not the same as no obligations, since GDPR still applies to every row that carries personal data.

ANNEX_III = {"biometrics", "critical_infrastructure", "education", "employment",
             "essential_services", "law_enforcement", "migration", "justice"}
PROHIBITED = {"emotion_at_work", "social_scoring", "untargeted_face_scraping",
              "manipulative", "predictive_policing_by_profile"}

def ai_act_class(r) -> tuple[str, str]:
    # returns (class, reason); the reason is written into the row
    if r.practice in PROHIBITED:
        return "prohibited", "Art. 5"
    if r.annex_iii_area in ANNEX_III and r.affects_person:
        narrow = r.narrow_procedural or r.improves_prior_human_result \
                 or (r.detects_patterns and not r.replaces_assessment) or r.preparatory
        if narrow and not r.profiles_person:
            return "minimal", f"Annex III · {r.annex_iii_area} · Art. 6(3) derogation, documented"
        return "high", f"Annex III · {r.annex_iii_area}"
    if r.user_facing and r.output_kind in ("draft", "chat", "generated_media"):
        return "transparency", "Art. 50"
    return "minimal", "outside Annex III"

The derogation in Article 6(3) matters in practice: a system in an Annex III area is not high-risk when it performs a narrow procedural task, improves the result of a prior human activity, detects patterns without replacing the human assessment, or is preparatory: unless it profiles the person, in which case the derogation does not apply. That is what turns “rank the applicants” (high) into “move the fields from the CV into the form” (a documented derogation), and the redesign is often the better product anyway. The assessment behind the derogation is written down before the system is used and, for providers, registered¹; a company that builds a tool for its own use is both provider and deployer. A high in the column adds a risk-management system, data governance, logging, human oversight and technical documentation to the fixed-cost line, which is usually what moves the row off the first wave. The timetable for those duties has been the subject of amendment proposals since the text was adopted, so the row records which consolidated version was read, and when².

§ 06The pilot: baseline first, a holdout, and a log

The pilot exists to replace three guesses in the ledger, model error rate, review minutes, tokens per run, with measurements, and to test a fourth, the baseline, which was taken on a small sample. So the order is fixed. The baseline comes first: two to four weeks in which the human process is timed at the desk and a random sample of its outputs is audited by a second person, which turns minutes_case and error_rate_base into intervals rather than points. The success criterion is written before the first run, in the ledger's own terms, a net saving per month above a threshold, with an error rate after review no worse than the baseline's, and it is computed at the end by the same script, not by a slide. A holdout of twenty to thirty per cent of cases, chosen at random, stays human-only for the whole pilot, so a change in the shape of the workload during those weeks cannot be credited to the model, and the comparison is against the same weeks rather than against last quarter.

Every run is logged in a record that can be read a year later, which is a legal requirement in the high-risk case and plain sense in every other:

{
  "run_id":         "FIN-03/2026-03-12/0417",
  "case_id":        "inv-88213",
  "arm":            "model",                  // or "holdout"
  "model":          "…",
  "prompt_version": "fin03-v7",
  "input_sha256":   "…",
  "tokens_in":      2412,
  "tokens_out":     188,
  "latency_ms":     2140,
  "output":         { "supplier_id": "S-4471", "total": 1284.50, "iban": "…" },
  "confidence":     0.91,
  "review":         { "by": "u.412", "minutes": 0.4, "changed_fields": ["iban"] },
  "final_outcome":  "posted",
  "error_found_later": null            // filled from downstream, weeks after
}

Three fields carry the analysis. arm separates the pilot from the holdout. review.changed_fields and review.minutes are where the review cost and the model's real error rate come from: a field the reviewer changed is an error the model made, whatever its confidence said. And error_found_later is filled from downstream, weeks after the run, when the wrong account or the wrong queue surfaces; without it the pilot measures only the errors that were easy to see, and the error term in the ledger stays a guess.

§ 07Failure modes

The walk-through records the process on the diagram, not the one at the desk; when the manager's description and the timing sample disagree, the desk is right and the row is re-described. Volume is counted at arrival, so cases handled in batches, or quietly never handled, inflate the number and the saving. The baseline is asked for rather than timed, and self-reported minutes run short. Tokens are estimated from the demo input while production inputs are longer; p50 and p95 are measured on real files, and where p95 does not fit the context the readiness level drops.

The error cost has no tail, so a row of small reversible errors and a row of rare final ones score the same. The pilot runs on inputs someone selected, and the criterion is written after the results are in. The classification is done once: a minimal-risk drafting aid whose output later starts feeding a promotion decision becomes an Annex III case without anyone changing a line of code, which is why every row has an owner and a review date. And the rollout order follows the seniority of the sponsor rather than the column that says net saving times readiness: the ledger exists so that the column wins the argument.