Answering questions about 10,000 contracts without making them up

A legal team can ask thousands of contracts the same shape of question all day. The answers exist, inside the documents. The work is getting the right one out, with its source, fast enough to be useful and reliable enough to act on. That last word is where most of the engineering goes.

Where it stood at launch, against the targets agreed with the client:

• 92%answers correct and citedtarget 90%, judged on the eval set
• 1.3%answers with an unsupported claimtarget under 2%
• $0.012cost per answertarget under $0.020, at volume
• 3.1sp95 latencytarget 2.5s, still open

The job was answers, not search

A contracts team fields the same shape of question all day. What is the notice period on the Lyon lease. Which parties are named on contract 88. Who can approve a refund over a set amount. The answers already exist, sitting inside thousands of PDFs that nobody has time to reread. Keyword search finds documents, not answers, and it misses anything worded differently from the contract.

What the team wanted was narrower and harder than search: the answer, and the clause it came from, in seconds, without pulling a lawyer in for every lookup.

A demo of this is easy to build. You embed the documents, ask the handful of questions you picked, and it answers them. Production is the version that takes a question nobody rehearsed, phrased however the person happened to phrase it, and either answers it with a citation or admits it cannot.

What the demo got to ignore

The distance from demo to production is a list of things the demo got to skip:

• Exact terms matter. A party name, a clause number, a figure. Miss the string and the answer is wrong, not roughly right.
• Every answer needs a source. An answer a reader cannot trace to a clause is not one they can act on, no matter how fluent it reads.
• The system has to refuse. When the contracts do not cover a question, “I don’t know” is the correct answer, not a plausible guess.
• It runs at the team’s pace, inside their walls. A couple of seconds per question, a cost that holds up at volume, and the documents never leaving the client’s own cloud.

Retrieval is the system; the model is the last step

The language model is the final stage, and it is not the hard part. If the right clause is not in front of the model, no amount of prompting recovers the answer. The model will instead answer confidently from whatever it was handed. So most of the work is retrieval: getting the exact passage in front of the model every time, including for the rare question that matters most.

Two searches, then a reranker

Vector search understands meaning but misses exact strings. Ask for “contract 88” and an embedding model may hand you contract 8, or 188, because they sit near each other in meaning. Keyword search catches the exact string and misses anything paraphrased. Contracts need both.

So retrieval runs two searches over the same contract set and merges them. Semantic search uses embeddings in Postgres with pgvector and cosine distance. Keyword search uses Postgres full-text search over the same chunks. The two ranked lists are combined with reciprocal rank fusion, which scores a passage by its rank in each list rather than by raw scores that are not comparable across the two methods. That fused list of about twenty candidates then goes to a cross-encoder reranker, which reads the question and each passage together and orders them by how well they actually answer it. The top five go to the model.

Chunk on the contract, not on a token count

How the documents are split decides what can ever be retrieved together. The common default, splitting on a fixed token count, cuts straight through clauses: half the obligation lands in one chunk and half in the next, and neither one answers the question on its own.

We split on the contract’s own structure instead. Its sections and numbered clauses are the boundaries, so a clause stays in one piece, and each chunk carries metadata, the document, the section, and the clause number, that the model cites back.

Grounding, and a real “I don’t know”

The model answers only from the five passages it was handed, and every answer carries the clause it used. When those passages do not contain the answer, the system says so rather than reaching for something that sounds right.

This is not a nicety. A confident, wrong answer about a contract is the worst thing the system can produce, because someone acts on it. Refusal is set by the reranker’s score: when the best passage does not clear a calibrated threshold, the system refuses instead of answering, and the threshold is tuned against the eval set rather than guessed. Refusal is then measured in both directions, because a system that refuses too often is as useless as one that never does.

The eval set is the product

None of the above means anything without a way to measure it. Before launch we build an eval set from the real questions the team asks, each paired with a correct, cited answer. Every change to a prompt, a model, the chunking, or the fusion weights runs against that set, so “did this help?” is a number that moves, not an opinion.

The metrics fall into three groups, each defined so the number means something.

Retrieval quality asks whether the answer was even reachable. The main one is context recall at 10: for each question, did a passage that actually contains the answer make it into the top ten retrieved? If recall is low, no prompt will save the answer, so this is the first number we fix.

Answer quality asks whether the model used what it was given. Answer accuracy is scored by an LLM-as-judge against the written reference answer, with a rubric, and spot-checked by a person. The unsupported-claim rate, what gets loosely called hallucination, is the share of answers containing a statement that no retrieved passage backs up, scored by a separate faithfulness judge. Citation match is the share of answers whose cited clause genuinely contains the supporting text.

System behaviour asks whether it is usable. Refusal accuracy is measured on a held-out set of answerable and unanswerable questions. Cost per answer sums what it costs to embed the question, run the reranker, and generate the answer. Latency is the 95th percentile end to end, not the average, because the average hides the slow tail people actually feel.

Measure	Target	Sample run
Context recall at 10	95%	97%
Answer accuracy	90%	92.1%
Unsupported-claim rate	under 2%	1.3%
Citation match	95%	96.7%
Refusal accuracy	90%	88%, still open
Cost per answer	under $0.020	$0.012
p95 latency	2.5 s	3.1 s, still open

The numbers above were measured at launch on the client’s eval set. Two were still open, on purpose: an open number is a named piece of work you can see before real users do.

Spending the latency budget on purpose

Cost and latency are part of the spec, not an afterthought. We set a budget per stage up front and watch where the time goes. In the sample run, generation dominates, which is where the open latency work sits: a smaller model for easy questions, or a shorter answer format, buys most of it back.

Where it breaks, and what we watch

A system like this fails in a few known ways, and naming them is half of keeping them out:

• The clause was never retrieved. Low context recall is the root cause of most wrong answers. We watch recall per question type, because the rare, high-stakes questions are the ones the demo never tested.
• A near-duplicate clause. Two leases with almost identical wording, and the model cites the wrong one. The reranker and the document metadata are what keep these apart.
• The confident paraphrase. The answer reads perfectly and is subtly not what the clause says. The faithfulness check and the citation requirement are aimed squarely at this.
• The eval set drifts. The questions the team asks change over time. When refusal or accuracy slips in production, the fix is usually new eval cases, not a new model.

Every answer, its retrieved passages, its scores, and its latency are traced, so a wrong answer can be replayed and explained rather than argued about.

What a sprint like this leaves behind

The system, its eval set, and the documents live in the client’s own repository and cloud. No lock-in, and a number they can rerun themselves whenever something changes. The next question the team wants answered becomes the next line in the eval set, and the eval set is what makes the thing safe to change for the next year.