RAG query rewriting: improve recall without changing intent

RAG query rewriting should make a question easier to retrieve without silently answering a different question. Preserve the original user text as an immutable audit field, extract trusted metadata filters separately, and generate only a small set of bounded lexical and semantic variants. Run those variants through the same authorization scope, fuse their candidates, rerank against the original question, and evaluate whether the rewrite improves relevant-evidence recall without increasing intent drift, latency or cross-tenant risk. Query rewriting is most useful for vague natural language, abbreviations, misspellings and multi-part questions; it is usually the wrong tool for exact IDs, SQL-shaped facts or security scope.

OpenAI's Retrieval guide documents an automatic vector-store query rewrite option through `rewrite_query=true`. Azure AI Search documents semantic query rewrite that can produce multiple variants, while Qdrant separates filters—which narrow candidates—from queries that contribute to ranking. These are platform capabilities, not proof that every rewrite improves a specific corpus. The routing, security and evaluation contract remains an application responsibility.

This guide was verified against the primary documentation on 13 October 2026. Documented parameters and search behavior are provider facts. The intent contract, rewrite policy, metrics and rollout gates below are engineering recommendations.

Start with a query contract, not a prompt

Treat the incoming request as a structured object before asking a model to rewrite it. At minimum preserve `original_text`, authenticated tenant and user scope, locale, conversation references, time boundary and request ID. The user text is evidence; never overwrite it with the rewritten form.

{
  "original_text": "What changed in the renewal policy after August?",
  "trusted_scope": {"tenant_id": "t_42", "acl": ["policy:read"]},
  "locale": "en",
  "conversation_refs": ["renewal policy"],
  "time_context": "2026-10-13T06:00:00Z"
}

The rewriter may output retrieval text, candidate entities, a decomposition plan and suggested filters. It must not decide authorization. Tenant, role, region and document visibility come from trusted identity and policy services. If a user says “search every customer,” that sentence is content—not permission to remove the tenant filter.

Define invariants: the rewritten query cannot introduce a person, product, date or negation that the original and conversation context do not support; exact identifiers must be copied verbatim; and every subquery must remain answerable inside the authorized corpus. Unknown intent should trigger a clarification or conservative search, not a creative guess.

Classify the query before choosing a rewrite

Different query shapes need different treatment. An exact lookup such as an invoice ID belongs in keyword search or SQL and should not be paraphrased. A natural-language concept can benefit from semantic normalization. A mixed query needs both. A multi-part question may need decomposition, while a short ambiguous phrase may require conversation context or clarification.

Query shape               Primary action
Exact ID or code          Keyword filter / SQL; preserve verbatim
Natural-language concept  One semantic rewrite plus original query
Concept + exact term      Hybrid dense and sparse retrieval
Multi-part comparison     Decompose into bounded subqueries
Ambiguous reference       Resolve from conversation or ask
Authorization request     Policy check; never rewrite scope

Classification can be deterministic for obvious patterns: UUIDs, order numbers, quoted phrases and date ranges. Use a model only where language understanding adds value. This lowers cost and prevents the rewriter from damaging strings that lexical retrieval handles precisely.

Separate search text from metadata filters

Do not stuff every constraint into an embedding. “Arabic invoices from Cairo after 1 September” contains semantic content and structured constraints. The semantic query may be “invoice requirements,” while language, city and date become payload filters. Qdrant's text-search documentation explains that filters narrow results without contributing to rank, whereas queries calculate relevance scores.

Build filters from validated fields. Map user terms such as “last quarter” through a deterministic calendar service. Validate enum values and date order. Merge user-requested constraints with mandatory authorization using AND; never let a generated OR weaken the trusted scope. Create payload indexes for frequently filtered fields and measure whether strict combinations require a different search strategy.

const trusted = tenantScope(session);
const requested = validateFilters(rewrite.filters);
const filter = { must: [...trusted.must, ...requested.must] };

const candidates = await qdrant.query('knowledge', {
  query: embed(rewrite.semantic_query),
  using: 'dense_v3',
  filter,
  limit: 40
});

The model proposes; the application validates. Reject unknown filter keys, overly broad ranges and values that conflict with access policy. Log the accepted filter plan without logging confidential query text by default.

Use bounded rewrites, not an uncontrolled swarm

A single normalization can expand abbreviations, repair spelling or replace pronouns with resolved entities. Multi-query retrieval creates several variants to improve recall across vocabulary. Decomposition turns a compound question into independent evidence needs. They are not interchangeable.

Keep the original query as one retrieval leg. Add at most a small number of distinct variants, each with a declared purpose: lexical synonym, semantic paraphrase or one sub-question. Deduplicate near-identical variants before search. Set total candidate, token, latency and cost budgets. A model that emits ten attractive queries can consume ten times the retrieval budget without adding useful evidence.

Azure AI Search documents a managed rewrite feature that can create up to ten variants. That is a capability ceiling, not a universal optimum. Start with one rewrite plus the original, then add variants only when an evaluation slice shows missed terminology.

For decomposition, retain a parent question and assign a stable subquery ID. Retrieve each subquery, then require the answer builder to distinguish evidence shared across subquestions from evidence that supports only one part. If one essential subquery has no evidence, the final answer should state that gap instead of blending partial evidence into a complete claim.

Combine lexical and semantic retrieval deliberately

Rewriting does not remove the need for hybrid search. Dense retrieval handles meaning and paraphrases; sparse retrieval protects exact product names, error codes and rare identifiers. Qdrant's hybrid query documentation supports prefetching multiple result sets and fusing them with rank-based methods such as reciprocal rank fusion.

Send a lexical-friendly variant to BM25 and a semantic-friendly variant to the embedding model, but apply the same trusted filter to both. Fuse by document or chunk identity, cap candidates per leg, then rerank the merged set against the original question—not against a possibly drifted rewrite. The original question is the user's acceptance contract.

Do not treat a high vector score as proof. Embedding scores are not calibrated probabilities and are not comparable across every query or model. Evaluate result order with labeled relevance judgments and set abstention rules from evidence coverage, not a universal similarity threshold.

Defend the rewrite stage from prompt injection

Retrieved content must not influence the query rewriter before authorization and retrieval. The rewriter receives user text, trusted conversation state and a closed output schema—not arbitrary document instructions. Mark user content as data and keep the policy in a developer or system instruction.

Use a strict schema with fields such as `semantic_query`, `lexical_query`, `filters`, `subqueries`, `preserved_terms` and `reason_code`. Limit lengths and array sizes. Verify that every preserved exact term appears unchanged. Reject executable syntax, tool calls and filter fields outside the allowlist.

A rewrite is not a place to infer hidden permissions, customer identifiers or sensitive attributes. Do not send confidential query text to a separate model or region unless the data-processing contract permits it. Store only the telemetry needed for evaluation, with retention and access controls.

Evaluate retrieval before answer quality

An answer score cannot reveal whether rewriting fixed retrieval or whether the generator guessed around bad evidence. Build a dataset with original query, authorized corpus version, relevant document or chunk IDs, important exact terms, expected filters and acceptable abstention. Include misspellings, acronyms, multilingual questions, negatives, date constraints, multi-part questions and adversarial scope requests.

Measure Recall@k for evidence coverage, Precision@k for noise, MRR or nDCG for ordering, exact-term preservation, filter correctness, zero-result rate, cross-tenant violations, rewrite latency and cost. Compare four arms: original only, one rewrite, original plus rewrite, and bounded multi-query. OpenAI's evaluation guidance recommends continuous evaluation and growing the set with observed failures; apply that discipline at the retrieval stage as well as the final response.

Slice results by query class. A rewrite may improve conversational questions but harm codes. The correct production policy may therefore enable rewriting only for selected classes. Use paired judgments on the same queries and require a security pass rate of 100%; average relevance cannot compensate for one cross-tenant result.

Construct context from the original intent

After fusion and reranking, group adjacent chunks, remove duplicates and preserve source identity, version, page or section. Allocate context by evidence need rather than giving the first subquery the entire token budget. Keep claim-sized passages so the answer can cite the exact support.

The answer builder receives the original question, accepted filter summary, retrieved evidence and coverage status. It does not need every discarded rewrite. Require citations to refer to provided source IDs and validate that each cited source exists in the context. If a rewrite introduced an unsupported entity, reranking against the original and claim-to-source validation should expose the drift.

Use SQL or a transactional API for balances, permissions, inventory and current order state. Use web search for public, changing information outside the private corpus. Use direct long-context input when the corpus is one small document and retrieval would add needless failure modes. Query rewriting cannot repair stale source data, broken chunking or missing access metadata.

Roll out with evidence and rollback

Start in shadow mode: generate rewrites, retrieve candidates and compute metrics without changing answers. Inspect cases where the rewrite adds or removes relevant documents. Then canary the policy for one query class and tenant cohort with a kill switch. Version the classifier, prompt, schema, embedding model, filter policy, fusion parameters and reranker together.

Monitor rewrite rate, number of variants, filter rejection, candidate overlap, Recall proxies, no-answer rate, latency percentiles, token use and security denials. A sudden fall in overlap may indicate a prompt or model change that altered rewrite semantics. Roll back the complete policy bundle, not just the prompt text.

Executive implementation checklist

Preserve the original query. Classify its shape. Keep exact terms unchanged. Derive authorization from trusted identity and validate requested filters. Use one bounded rewrite before multi-query. Keep lexical and semantic legs under the same scope. Fuse candidates, rerank against the original, and construct source-addressable context. Evaluate retrieval with labeled evidence, security invariants, latency and cost. Deploy in shadow and canary stages with a versioned rollback bundle.

The key rule is simple: query rewriting may change the search expression, but it must never change who is asking, what they asked, or which evidence they are allowed to retrieve.

Prepared by: Noor Yasser