Finding Before Answering

Can you return those holiday boots, and until when? To answer from a shop's documents, software first has to find the relevant policy. Follow that question from search to reply below. The shop, its policies, and its replies are a fictional teaching example, not a recorded system.

The shop keeps a handful of policy documents. Among them are these passages:

A. Items can be returned within 30 days of delivery with a receipt.

B. Orders placed from November 15 to December 24 can be returned until January 31.

C. Sale items are final sale.

D. Standard shipping takes 3 to 5 business days.

A visitor asks: "I ordered boots on November 20 for the holidays. Does the holiday return window cover them, and until when?"

The system searches first and keeps a shortlist of passages that might help.

Keyword search looks for the words in the question. It finds "return" in passages A and B. It is good at exact terms such as an order number. Embedding search turns the question and each passage into numbers that represent meaning, then looks for passages close in meaning. It can connect "for the holidays" with "November 15 to December 24" even though they share no words. Anthropic's contextual retrieval article explains both, including how embeddings can miss exact matches.

Hybrid search runs both and merges the results into one ranked list. Elastic's hybrid search documentation describes this. For our question, a hybrid search might return A, B, and C, with D ranked lower.

Finding passages and then writing from them is often called retrieval-augmented generation, or RAG, after a 2020 research paper that combined a retriever with a language model.

A shortlist is not an answer. A choosing step, sometimes called ranking or reranking, puts the most useful passages first, and it can keep more than one. Here B fits the order date, A is the general rule that B extends, and D is about shipping. C matters too: if the boots were on sale, no return window applies, and the question does not say. Choosing from a fixed set is a different job from writing, and Writing an Answer or Choosing One covers when each one fits.

A generator then writes the reply from B and C: "Orders placed between November 15 and December 24 can be returned until January 31, so boots ordered on November 20 qualify. One exception: sale items are final sale. If your boots were on sale, they cannot be returned."

An unconditional "Yes" would have been wrong for some visitors. Most real answers combine a rule with its exceptions, so a step that keeps only the single best passage can drop the condition that changes the answer.

Code can check some parts exactly: that November 20 falls inside the window, and that the reply states the deadline. A required field is either present or missing.

A reader still needs to check whether the sentence means what passage B says. The answer-checking guide walks through that check.

Now delete passage B from the documents and ask again. Search returns A, C, and D. The choosing step still picks one, since picking is its job. A looks closest, so the reply becomes: "Returns are accepted within 30 days of delivery."

That reply restates a real policy and cites a passage it was given. It also answers a question the visitor did not ask, because nothing on the shortlist mentions a holiday window. A more confident chooser cannot help: it can rank only what search handed it. As the bounded decisions guide puts it, a score over the candidates cannot tell you whether the right candidate was missing.

Give the choosing step a way to say "none of these answers it" and send the question to a fallback. The right result here is a clear "need more information": "I couldn't find a holiday return policy. The general policy allows 30 days from delivery. I'll pass your question to a person who can confirm holiday orders."

If you score only the final reply, you cannot see which step went wrong. Check each stage separately, using test questions whose answering passage you already know.

• Coverage. Did the answering passage reach the shortlist? Ragas calls a version of this context recall.
• Choice. When it was there, did the choosing step pick it?
• Staying within the evidence. Did the reply say only what the chosen passage supports? Ragas measures something similar as faithfulness. It uses a model as the judge, so treat it as an estimate.

Include questions the documents cannot answer, and count a clear "need more information" as a pass for them.

To see coverage decide an answer, open a recorded run of this site's research agent. It searches a song collection for a title spelled the way a songbook prints it, without the Hawaiian marks. Guess whether it finds the recording, then play the run and see which stage settled the answer.

Widening the shortlist can improve coverage. It also gives the choosing step more to sort, puts more text in front of the generator, and takes time. Measure how wide your shortlist needs to be on your own questions instead of borrowing a number.

A visitor asks whether a discounted jacket can be returned. The reply says: "Yes, within 30 days of delivery," citing passage A. Passage C, "Sale items are final sale," is in the documents but was not on the shortlist. Which stage needs fixing?

The generator looks like a reasonable place to start, since it wrote the wrong answer. Yet the reply stayed close to passage A, so a check against A alone would find little wrong. The choosing step also picked the best passage it had. The miss happened at search: the question said "discounted" and the policy says "sale," so keyword search found no match, and the embedding search did not rank C highly enough. Fix coverage first: add test questions that use other words for sale, and check whether C reaches the shortlist.

To check one reply against its sources, try checking an AI answer. For when a model should choose rather than write, read writing an answer or choosing one, then should a model make this decision? for error limits. Designing the system around a model covers search, tools, and testing in a larger system.