Build a search relevance test set from known work, not invented prompts

Collect queries from operational evidence

Start with questions people actually asked, documents they eventually found, escalations caused by a miss, repeated support lookups, and common navigational requests. For each case, preserve the original wording, requester role, date, intended collection, known relevant documents, decisive passages, and how the answer was verified. Remove or protect sensitive details before the set is shared beyond its authorized evaluation environment.

Classify cases so the suite exposes different failure modes: exact known-item lookup, multi-document answerability, paraphrase, identifier, date constraint, metadata filter, forbidden document, stale version, and no-answer request. A test set made only of easy positive examples will reward systems that return plausible material even when they should return nothing or deny access.

Record expected behavior without pretending relevance is absolute

For known-item queries, name the expected document and a useful rank threshold. For answerability cases, identify acceptable source spans rather than prescribing one generated sentence. For negative cases, state why no result or no synthesis is correct. Permission cases pair a query with user or group context and specify both visible and forbidden records. Freshness cases identify the current version and the stale version that must not outrank it.

Some queries have several reasonable sources or require a subject-matter judgment. Mark those cases for human review and record alternative acceptable results. Use measures such as success at a chosen rank, reciprocal rank, judged precision, answerable coverage, forbidden-result count, and update delay, but retain the underlying result lists. A single aggregate number can conceal a severe access or freshness defect.

Use holdouts and regression evidence for changes

Divide the suite so tuning examples and held-out examples remain distinct. When extraction, chunking, embeddings, analyzers, field weights, filters, or fusion settings change, rerun the same versioned suite and compare individual cases. Investigate gains and regressions by query class, collection, permission group, and document age before approving a release. Add new production failures after they are understood, not merely because they occurred.

ExactSpan provides raw results and case-level notes for twenty subset questions, then extends the agreed suite during a paid build. Reality Contact, LLC does not set the buyer's substantive relevance standard or claim that the suite represents every future question. The buyer supplies authoritative examples, approves thresholds, and decides whether observed failures are acceptable for the intended workflow.

Where the service stops

Reality Contact, LLC implements and evaluates search but does not determine who should have access, certify confidentiality or legal privilege, make substantive professional judgments from retrieved material, or represent that a result set is complete. The buyer confirms lawful corpus access, approves permission mappings and evaluation cases, reviews failed and ambiguous results, and decides whether the system is acceptable for the team's operational search workflow. Do not send private material through the public form. Secure intake and written deletion terms are established before files, links, credentials, or sensitive records are transferred. This is technical search implementation and evaluation; it does not replace the buyer's legal, privacy, security, records, privilege, or substantive professional review. Search results are limited by the confirmed corpus, extraction, metadata, permissions, index version, queries, and evaluation set; the service does not promise that every relevant document or passage will be found.