Designing retrieval systems where the source remains authoritative

§ 01The contract, not the retriever

Three systems I have built share a rule that has nothing to do with how good the retrieval is. The document is the truth; the system's output is a pointer into it and a claim about what the pointer says. The normative RAG is not allowed to answer from memory: it returns the paragraph and a way to open it, or it says it does not know. the tender system ties every conclusion in a dictamen to the clause it came from and states when the evidence is insufficient. Erario puts a fit score next to a link to the official record and labels the score as a prioritisation signal, never as eligibility (N.12 is about why).

N.03 covers how the passages get found. This note is about what sits on top: what an answer may be made of, what the reader can check, and what happens when the system cannot honestly produce one. Written down, the contract has three clauses. Every claim carries a citation the interface can resolve. Every extracted value is a typed field with a confidence and a provenance, and null is a legal value. And a written boundary separates what the system may say by itself from what waits for a person, with a queue that is engineered, not improvised.

§ 02A citation is an id, not a decoration

A citation that is a title in italics is decoration. A citation the contract needs is a record the interface can open at the exact span, and that a program can check without a model in the loop. Mine carry the document id, the hash of the bytes the extractor read, a structural locator (clause path, article, table cell) from the parse, character offsets into the versioned text layer, and the quoted span. One extracted field on the wire:

{
  "field": "economic_solvency.min_annual_turnover",
  "value": { "amount": 450000, "currency": "EUR", "basis": "best_of_last_3_years" },
  "status": "present",            // present | absent | conflicting
  "confidence": 0.86,             // self-reported; meaningful only after calibration (§03)
  "provenance": [
    {
      "doc_id": "pcap:2026-08-14:e7c1a9",
      "doc_sha256": "3f9c…b21e",       // the bytes the extractor actually read
      "locator": "clause 12.1.a",      // from the structural parse, not from the model
      "page": 23,
      "char_start": 48211, "char_end": 48590,
      "quote": "…volumen anual de negocios … igual o superior a 450.000 €…",
      "quote_verified": true           // set by code, never by the model
    }
  ],
  "extractor": { "model": "…", "prompt_version": "ext-3.4", "run_id": "01J…" }
}

The hash matters more than it looks. Tender documents get corrected, standards get amended, and a link that opens the current version of a document the model read last month is a citation to something the model never saw. The doc_sha256 pins the claim to bytes; when the source changes, the record is not wrong, it is stale, and stale is a state the interface can show.

The quote_verified flag is not something the model sets. It is set by a check that runs on every provenance entry before anything reaches a screen:

import hashlib, re, unicodedata

def _norm(s):
    s = unicodedata.normalize("NFKC", s)
    return re.sub(r"\s+", " ", s).strip().casefold()

def verify(p, store):
    # 1. the same bytes the extractor saw
    raw = store.bytes(p.doc_id)
    if hashlib.sha256(raw).hexdigest() != p.doc_sha256:
        return False, "document changed since extraction"
    text = store.text(p.doc_id)   # the normalised text layer, versioned with the doc
    # 2. the quote is where the offsets say it is
    if _norm(text[p.char_start:p.char_end]) != _norm(p.quote):
        return False, "offsets do not contain the quote"
    # 3. the locator agrees with the structural parse
    if store.locator_at(p.doc_id, p.char_start) != p.locator:
        return False, "locator mismatch"
    return True, ""

The check is cheap and deterministic, and it turns the vaguest failure of these systems, a plausible quote that is almost what the document says, into a boolean. In the queue design below, a failed verification is a reason code, not a caveat.

§ 03Structured extraction, with null as an answer

Free generation asks the model to write about the document. Structured extraction asks it to fill a schema, and moves everything that is not language out of the model: which fields exist, which are enumerations, numbers with units, dates, or required. In Pliegos the conditions that decide admissibility, deadlines, solvency, classification, guarantees, award criteria, documentation, are separate fields, and each field's value is a typed object, not a sentence. Three properties of the schema do most of the work.

Status is explicit. A field is present, absent or conflicting. Absent means the extractor looked and found nothing; it is a finding, and it is what the interface renders as “not stated in the document”. Filling an absent field with a plausible default is the most common way these systems go wrong, so the schema makes null cheap and a fabricated value expensive: a non-null value without a verified provenance fails validation.

Confidence is a number the model emits, and it is only meaningful after calibration. On its own it is a self-report. What makes it useful is a reviewed sample: for each confidence bin, the fraction of fields the reviewer accepted unchanged. Once that curve exists, a threshold on confidence becomes a threshold on measured error, and I can set it per field rather than globally: a disqualifying condition earns a stricter one than an informational field.

Provenance is a list, not a string, because a value can rest on two passages (a clause and the annex it references), and because a conflict is two provenance entries with incompatible values, which is exactly the case a person should read. The extractor's own identity, model, prompt version, run id, travels with the record, so when a prompt change moves the calibration curve, the records say which curve they belong to.

§ 04The abstention rule

The rule that decides what the interface may do with a field is short enough to read on one screen, and it is code, not a prompt:

from enum import Enum

class Outcome(Enum):
    SHOW = "show"            # value + citation, no caveat
    SHOW_FLAGGED = "flag"    # value + citation + a "verify" badge
    REVIEW = "review"        # queued for a person; not shown as fact
    ABSTAIN = "abstain"      # rendered as "not stated in the document"

def decide(f, tau_show, tau_flag):
    if f.status == "absent" and not f.provenance:
        return Outcome.ABSTAIN          # null is a valid answer
    if f.status == "conflicting":
        return Outcome.REVIEW           # two passages disagree: a person reads both
    if not all(p.quote_verified for p in f.provenance):
        return Outcome.REVIEW           # a citation that does not resolve is a defect
    if f.confidence >= tau_show:
        return Outcome.SHOW
    if f.confidence >= tau_flag:
        return Outcome.SHOW_FLAGGED
    return Outcome.REVIEW

Two things about the order. Failed verification and conflict are checked before confidence, because a confident record with a citation that does not resolve is worse than a hesitant one. And ABSTAIN is a first-class outcome with its own rendering, not the absence of output. In the normative RAG this is the whole product: it answers with the clause or it says it does not know, and the second behaviour is the most important one it has.

§ 05The review boundary and its queue

The boundary between what the system may say by itself and what waits for a person is a product decision written as thresholds and reason codes. In these systems the model may show a passage, may show a field with a verified citation above threshold, and may say “appears to fit”; it may not decide legal eligibility, present generated text as official fact, or replace professional verification. Whether a company complies, or should bid, stays with the accountable person; a dictamen supports that person, it does not replace them.

The queue is where the design either holds or quietly fails. A review item is a record, with the candidate value untouched, the evidence in reading order, a reason code, and the deadline that actually matters: the tender's, not ours:

CREATE TABLE review_queue (
  id          bigserial PRIMARY KEY,
  doc_id      text NOT NULL,
  doc_sha256  text NOT NULL,
  field       text NOT NULL,             -- e.g. economic_solvency.min_annual_turnover
  reason      text NOT NULL CHECK (reason IN
               ('low_confidence','conflicting','unverified_citation',
                'schema_violation','sample_audit')),
  candidate   jsonb NOT NULL,            -- the record as extracted, untouched
  evidence    jsonb NOT NULL,            -- the passages, with offsets, in reading order
  hard        boolean NOT NULL DEFAULT false,  -- a disqualifying condition?
  due_at      timestamptz,               -- the tender's own deadline
  created_at  timestamptz NOT NULL DEFAULT now(),
  claimed_by  text, claimed_at timestamptz,
  decision    text CHECK (decision IN ('accept','correct','reject','abstain')),
  corrected   jsonb,                     -- what the reviewer says the field is
  decided_at  timestamptz,
  UNIQUE (doc_id, doc_sha256, field)     -- one item per field per version
);

-- Next item for a reviewer: hard conditions first, then the nearest deadline
SELECT id FROM review_queue
 WHERE decision IS NULL AND claimed_by IS NULL
 ORDER BY hard DESC, due_at ASC NULLS LAST, created_at ASC
 LIMIT 1 FOR UPDATE SKIP LOCKED;

Two choices carry the load. The uniqueness on document, version and field means a corpus re-extracted after a prompt change does not double the queue. And the ordering puts hard conditions and near deadlines first, so that when the queue is longer than the reviewers, what waits is the informational field of a tender that closes next month.

The reviewer's decision closes the loop twice: it corrects the record, and it becomes a labelled example: the calibration curve in §03 is built from nothing else. That is also where the threshold gets chosen. A worked example: 18 conditions per document (the count on the one real tender shown on the product page) and 200 documents a day (a number chosen to make the arithmetic visible), so 3,600 fields a day. Assume the confidence distribution is 55 / 20 / 12 / 7 / 6 % across the bins ≥ 0.95, 0.85–0.95, 0.75–0.85, 0.60–0.75 and below 0.60; assume calibration is honest (error rate ≈ 1 − bin midpoint); assume 90 seconds of reviewer time per item. The distribution is invented, the arithmetic is not, and it shows the shape of the trade:

The column that matters is the last one. Everything the queue does not absorb is shown as fact, and the apparatus of citations exists so that those errors are at least checkable. In practice τ is not one number: hard conditions sit at the bottom rows of the table and informational fields near the top, and the reviewers' hours go where a wrong value costs a bid.

§ 06Failure modes

The ones I design against, in the order they tend to appear.

The citation resolves to the wrong version. Without the hash, a corrected PCAP silently invalidates every claim made about the previous one, and the interface opens a document that no longer says what the record quotes.

The quote is almost right. Ellipses, a dropped negation, a number rounded in the paraphrase. The verifier catches the substring; it does not catch a quote that is verbatim and out of context, which is why provenance carries the locator and the interface opens the surrounding clause, not only the span.

The schema gets filled anyway. Given a required field and no evidence, a model produces a plausible value. The defence is structural: required means the field must be present in the record, not that the value must be non-null, and a non-null value without a verified provenance is a validation error, not a warning.

Confidence is uniformly high. Self-reported confidence clusters near the top; on the calibration curve that shows up as a bin at 0.95 with a 10 % error rate. Then the number is not a probability: set the threshold on the reviewed error rate directly, or swap the extractor for one whose scores spread.

Tables split across pages. Solvency thresholds and award criteria live in tables, and a passage boundary through a table yields two half-truths, both quotable. The structural parse has to keep the table as one unit together with the clause that introduces it: the same problem N.03 solves at retrieval time.

The queue becomes a rubber stamp. When items are accepted in ten seconds each, the label is worthless and the calibration built on it is fiction. I record time-to-decision per item and per reviewer, seed the queue with a small sample_audit stream of high-confidence records, and treat a run of instant accepts as an incident, not as throughput.

What I watch is short: citation resolution rate (target 100 %; less is a bug in the store, not the model), quote-verification pass rate before review, abstention rate against how often the reviewer agrees the answer was not there, per-bin error on the reviewed sample, and queue age against the tenders' own deadlines. None of these needs a model to compute, which is the point.