Hybrid retrieval that returns the paragraph, not a paraphrase

§ 01Parse along the structure, not the window

A fixed window of 512 tokens with a 64-token overlap is the default in every tutorial and the first thing I remove. A standard, a tender specification or a contract already has a structure, parts, sections, clauses, numbered paragraphs, tables, annexes, and that structure is the unit a reader cites. If a boundary falls in the middle of clause 7.3.2, the retriever finds half of it, the model quotes half of it, and the citation points at something that is not a thing in the document.

So the parser recovers the structure first and splits second. Headings are detected by numbering pattern and typography, tables are kept as tables with every row carrying its header row, lists stay with the sentence that introduces them, and footnotes are attached to the paragraph they annotate. Only then do I split, and I split at clause boundaries. A clause longer than about 400 tokens is divided into numbered parts that keep the clause id as a prefix; the ceiling sits well under the embedding model's limit because embedding quality decays before the limit does. That is the PDF structure recovery behind the tender system and the clause-level citation in the normative RAG: the same parser, different corpora.

Every chunk carries the path that led to it and an id derived from that path rather than from a byte offset, so a re-parse after a corrected PDF yields the same ids and the citations already stored in answers do not go stale. The record I keep per chunk:

§ 02One table, two indexes

Everything lives in PostgreSQL. A generated tsvector column with headings weighted above the body gives lexical search through a GIN index; a vector column from pgvector gives semantic search through an HNSW index. One table, two indexes, one transaction when a document is re-ingested, and a citation that is a primary key rather than a pointer into a second system.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
  doc_id      text PRIMARY KEY,   -- 'PCAP-2026-0412' or a standard's reference
  title       text NOT NULL,
  version     text NOT NULL,      -- edition or consolidation date
  source_uri  text NOT NULL,
  sha256      bytea NOT NULL      -- of the bytes that were parsed
);

CREATE TABLE chunks (
  chunk_id    text PRIMARY KEY,   -- doc_id || '#' || path
  doc_id      text NOT NULL REFERENCES documents,
  path        text[] NOT NULL,    -- {'III','12','12.3','p1'}
  heading     text,
  kind        text NOT NULL CHECK (kind IN ('clause','table','list','note')),
  page_from   int,  page_to int,
  body        text NOT NULL,
  n_tokens    int  NOT NULL,
  tsv         tsvector GENERATED ALWAYS AS (
                setweight(to_tsvector('spanish', coalesce(heading, '')), 'A') ||
                setweight(to_tsvector('spanish', body), 'B')) STORED,
  embedding   vector(1024) NOT NULL   -- dimension is the embedding model's
);

CREATE INDEX chunks_tsv_gin  ON chunks USING gin (tsv);
CREATE INDEX chunks_emb_hnsw ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 128);
CREATE INDEX chunks_doc      ON chunks (doc_id);

Two honesty notes on the schema. ts_rank_cd is not BM25: there is no term saturation and only the length normalisation you ask for through its third argument, and on a corpus of a few hundred thousand chunks the fusion step hides most of the difference; where it does not, a real BM25 index. ParadeDB's pg_search, or an external sparse index: takes the lexical leg and the rest of the pipeline stays as it is.¹ And the vector dimension is a property of the embedding model, so changing the model means re-embedding every chunk and re-running the evaluation set: a migration, not a configuration change.

§ 03Hybrid retrieval, fused by rank

The two legs fail in different places, which is the whole reason for having both. Lexical search wins on identifiers, “artículo 145”, “cláusula 12.3”, “IEC 61508-3”, a solvency figure, where an embedding sees only vaguely numerical text. Dense search wins on paraphrase: “can we bid without the ISO certificate” has to land on a clause about classification and solvency that shares no word with the question. Their scores live on different scales, so I do not add them. Reciprocal rank fusion discards the scores and keeps the ranks: each chunk gets the sum of 1/(k + rank) over the lists it appears in, with k = 60, the constant from the original paper and one I have never found a reason to tune.² A chunk near the top of both lists beats one that is first in one and absent from the other, and a leg that returns rubbish for a given question cannot poison the result, only fail to help.

-- $1 question text · $2 its embedding · $3 optional doc_id scope
SET LOCAL hnsw.ef_search = 100;

WITH q AS (
  SELECT websearch_to_tsquery('spanish', $1) AS tsq, $2::vector(1024) AS qv
),
lex AS (
  SELECT chunk_id,
         row_number() OVER (ORDER BY ts_rank_cd(tsv, q.tsq, 1) DESC) AS r
  FROM chunks, q
  WHERE tsv @@ q.tsq AND ($3::text IS NULL OR doc_id = $3)
  ORDER BY r LIMIT 50
),
sem AS (
  SELECT chunk_id,
         row_number() OVER (ORDER BY embedding <=> q.qv) AS r
  FROM chunks, q
  WHERE ($3::text IS NULL OR doc_id = $3)
  ORDER BY r LIMIT 50
),
fused AS (
  SELECT chunk_id, SUM(1.0 / (60 + r)) AS rrf     -- reciprocal rank fusion, k = 60
  FROM (SELECT * FROM lex UNION ALL SELECT * FROM sem) u
  GROUP BY chunk_id
)
SELECT c.chunk_id, c.doc_id, c.path, c.heading, c.kind, c.body, f.rrf
FROM fused f JOIN chunks c USING (chunk_id)
ORDER BY f.rrf DESC
LIMIT 30;                                          -- into the reranker

Fifty per leg, thirty out of fusion, and a scope filter when the question is about one document, which in a tender it almost always is. The filter matters for HNSW: pgvector applies it after the graph walk, so a tight filter can return fewer rows than asked; I set hnsw.ef_search to at least twice the limit for scoped queries, and on pgvector 0.8 turn on hnsw.iterative_scan so the walk continues until the limit is filled.

§ 04Rerank, then stop widening

Retrieval is a filter; the reranker is the judge. A bi-encoder compares two vectors that were computed without seeing each other. A cross-encoder reads the question and the passage together and produces one relevance score, and on clause-level questions it reorders the top thirty in ways that matter: the clause that answers moves from seventh to first, the one that merely mentions the topic drops. I use an open cross-encoder of the bge-reranker or MiniLM class, run locally, and keep the top six for the answer model. Six is a measured number, not a preference: the evaluation set decides it, and it usually lands between five and eight.

The budget below is what a question costs to answer with those parameters. The assumptions: a corpus of 100,000 chunks averaging 300 tokens, a 40-token question, a 1024-dimension embedding, the reranker on CPU, and typical latencies on a modest server, so read the times as orders of magnitude rather than measurements.

Two things the table makes visible. The reranker processes about four times as many tokens as the answer model and is still the cheapest stage per token, because it is a small local encoder; on a GPU its 300 ms become about 30. And the answer step is where the money goes: at a price P per million input tokens it costs roughly 2,300 × P × 10⁻⁶ per question, so widening the context from six passages to twelve doubles that line, and the evaluation set will usually show it bought nothing measurable. Which model answers, and how large it needs to be, is a separate note.

§ 05Cite the id or abstain

The answer model receives the six passages numbered, each with its chunk_id, and a prompt with three obligations: answer only from those passages, quote the passage verbatim and name its id for every claim, and where no passage supports the question return the abstention object rather than a helpful guess. The output is JSON against a schema, not free text, because the next step is a check the service runs before a person sees anything.

The check is mechanical on purpose. Every cited id must be one the model was shown: not merely one that exists in the corpus, because a model that has seen thousands of ids will happily produce a plausible one. Every quote must be a substring of that chunk's body after whitespace normalisation. An answer left with zero valid citations becomes an abstention regardless of how good the prose looks. What survives is an answer whose every citation the interface can open: id, then document, page and path, then the PDF at that page with the span highlighted. That is what a the tender system dictamen shows under each conclusion, and what the normative RAG shows instead of a summary.

import re
WS = re.compile(r"\s+")
def norm(s: str) -> str:
    return WS.sub(" ", s).strip().lower()

def check(answer: dict, shown: dict[str, str]) -> dict:
    """shown: chunk_id -> body of the passages the model was given."""
    if answer["status"] == "abstain":
        return answer
    kept = []
    for c in answer.get("citations", []):
        body = shown.get(c["chunk_id"])            # an id it was shown, not one that exists
        if body and norm(c["quote"]) in norm(body): # and the quote is really in it
            kept.append(c)
    if not kept:                                   # no verifiable support: abstain
        return {"status": "abstain", "reason": "unverified",
                "answer": None, "citations": []}
    answer["citations"] = kept
    return answer

Where the conclusion is a hard condition, a deadline, a solvency threshold, a required classification, the retrieved passage is the input to a rule and the rule decides; the model's job ends at finding and quoting. Why that split, and where the human-review boundary sits, is in the note on keeping the source authoritative and the one on eligibility.

§ 06What I measure

None of the parameters above, the chunk ceiling, fifty per leg, the RRF constant, thirty into the reranker, six out, the prompt wording, is defensible without a number, and the number comes from a question set written by people who know the corpus, before the pipeline is tuned. I aim for at least two hundred questions per corpus, a fifth of them unanswerable on purpose, each with the ids of the chunks that answer it. The set is versioned with the code and the model configuration, and it runs on every change.

Four numbers per run. Recall@6, is a gold chunk among what the answer model sees, measured at the reranker's output, with recall@30 at fusion's output logged beside it so a loss can be blamed on the right stage. MRR, because a gold chunk in first position and in sixth are not the same to a model that reads in order. Groundedness: the share of answers whose citations pass the check above, computed deterministically, plus a sampled judgement of whether the answer text is actually supported by the quotes, because a valid quote can still be misread. And the two abstention rates: on the unanswerable fifth, which should be high, and on the answerable rest, which should be near zero. Latency p95 rides along.

def evaluate(questions, retrieve, answer, k=6):
    m = dict(recall6=0.0, recall30=0.0, mrr=0.0, grounded=0,
             abstain_ok=0, abstain_bad=0, n_ans=0, n_unans=0)
    for q in questions:
        fused = retrieve(q["text"], scope=q.get("doc_id"))   # 30 after RRF, reranked
        top = fused[:k]
        ids30, ids = [c.chunk_id for c in fused], [c.chunk_id for c in top]
        out = check(answer(q["text"], top), {c.chunk_id: c.body for c in top})
        if q["answerable"]:
            gold = set(q["gold_ids"]); m["n_ans"] += 1
            m["recall6"]  += len(gold & set(ids))   / len(gold)
            m["recall30"] += len(gold & set(ids30)) / len(gold)
            hit = next((i for i, cid in enumerate(ids, 1) if cid in gold), None)
            m["mrr"] += 1 / hit if hit else 0
            m["grounded"]    += out["status"] == "answered"
            m["abstain_bad"] += out["status"] == "abstain"
        else:
            m["n_unans"] += 1
            m["abstain_ok"] += out["status"] == "abstain"
    a, u = max(m["n_ans"], 1), max(m["n_unans"], 1)
    return {"recall@6": m["recall6"] / a, "recall@30": m["recall30"] / a,
            "mrr": m["mrr"] / a, "grounded": m["grounded"] / a,
            "false_abstain": m["abstain_bad"] / a, "abstain_on_unanswerable": m["abstain_ok"] / u}

A change is real when it moves a metric by more than the run-to-run noise, and the noise is measured by running the same configuration twice; embedding and answer models are not perfectly deterministic, and an evaluation that forgets that will report improvements that are not there. Every run is stored with the commit, the model identifiers and the metrics, so the number a change was accepted on can be found a year later.

§ 07Failure modes

The ones I have had to design against, each with the fix that stuck.

• The orphaned table cell. A solvency threshold quoted from a table without its header row: the model cites “≥ 300,000 €” and cannot say for which lot. Fixed in the parser, never in the prompt: every table chunk repeats its header row.
• The stale citation. A consolidated text is re-issued and clause 12.3 now says something else under the same id. The stored citation keeps the document version and a hash of the body it quoted; when they no longer match, the interface says so instead of presenting the new text as the old.
• The one-hop definition. The retrieved clause says “as defined in 3.4” and the answer needs 3.4. Internal cross-references are resolved at parse time and stored as related ids; the answer step pulls one hop, never more.
• The vocabulary gap. The question says “fianza” and the document says “garantía definitiva”; both legs miss. The fix is a small, measured synonym table for the domain applied to the lexical leg: a larger embedding model helps less than the table and costs a re-index.
• The boilerplate twin. The same standard clause appears in hundreds of specifications; without a document scope the reranker cheerfully returns the right clause from the wrong tender. Scope is mandatory when the question is about one document.
• The split answer. The answer lives across two adjacent chunks and the model abstains on either alone. The false-abstention rate catches it; the fix is to include the neighbouring part when the top hit is one part of a divided clause.