Before tuning retrieval, prove the missing item was indexed

My first question about a failed search would be embarrassingly literal: is the missing document actually in the index?

The Build With AI email-search tutorial makes that question concrete. Its example selects up to 100 inbox threads, combines message bodies within each thread and truncates the body text before embedding it with the subject. The spreadsheet stores a shorter preview alongside the embedding. That is a useful small experiment with a deliberately bounded corpus.

It is not a search over every email the account has ever received. An archived thread outside the inbox selection will not enter this example's index. A relevant passage late in a long thread may fall beyond the text cutoff. No change to similarity scoring can recover text that the pipeline never supplied.

I would check those boundaries before shopping for an embedding model. That is a debugging order, not a claim that corpus scope explains most retrieval failures.

A missing result can fail at three different stages

Imagine a hypothetical search for an old agreement about a renewal date. We know the message exists because we can open it directly. The AI search returns nothing useful.

First, inspect eligibility. Was that thread inside the selected mailbox view and time window when the index was built? A label alone does not settle this: labelled messages can still be in the inbox. Check the actual selection rule and the indexed identifier.

Second, inspect the indexed representation. The thread might be present while the paragraph containing the agreement was truncated. A record with the right subject is not proof that the searchable representation contains the answer. Compare the text supplied to the embedding with the original passage.

Only then inspect ranking. If the passage was represented and remained eligible for this user, test whether the query found it. That is where chunking, lexical matching, embeddings and ranking become plausible repair candidates.

There is also a presentation trap. A useful result may have a preview too short to show why it matters. A link back to the complete authorised source lets the reader inspect the evidence rather than infer failure from an unhelpful snippet.

Make coverage visible before making search clever

I would add a small coverage record to the prototype: selection rule, number of eligible records, number indexed, last successful refresh and the known truncation boundary. Those fields should describe the same run as the results being evaluated.

Then build a modest set of known-item questions. Include a recent inbox message, an older item outside the selection and a passage near the end of a long thread. The expected result for an excluded message is an honest scope explanation, not a fabricated answer.

Expanding the corpus is a separate decision. More searchable material can improve coverage, but it also changes access, retention and refresh obligations. The repair should include only material the application and its user are authorised to retrieve.

This leaves plenty of room for genuine model problems. A fully indexed passage can still be missed or misunderstood. The benefit of checking scope first is that the evaluation can distinguish that failure from an impossible request.

Before changing the retrieval model, trace one missing answer from the original source into the exact text the index received.