RAG retrieval that degrades instead of returning nothing

A retrieval-augmented app has two failure modes that look the same to a user: the right passage exists but retrieval missed it, or the document really doesn’t contain the answer. InsightVault handles the first with a retrieval fallback chain and the second with an explicit instruction to the model.

The query path

Similarity search is a Postgres function over an HNSW index, so filtering to a single document happens in the same query as the vector search:

SELECT dc.id, dc.document_id AS doc_id, dc.content,
       (1 - (dc.embedding <=> query_embedding))::float AS similarity
FROM document_chunks dc
WHERE (match_chunks.document_id IS NULL OR dc.document_id = match_chunks.document_id)
  AND (1 - (dc.embedding <=> query_embedding)) > match_threshold
ORDER BY similarity DESC
LIMIT match_count;

Three levels of fallback

• First pass: threshold 0.2, top 6 chunks.
• If nothing clears the threshold, retry at 0.0. The closest chunks are usually still the most useful ones.
• If vector search returns nothing at all, take the document’s first chunks directly so the model has something to reason over.

The prompt then carries the other half of the contract: if the answer is not in the context, the model must reply that the document does not contain it. Retrieval stays permissive; the model is told to be strict.

The dimension problem

Embeddings come from Hugging Face (bge-small, 384 dimensions), then Gemini (asked for 1536), then OpenAI (text-embedding-3-small, 1536). The column is vector(1536), so the Hugging Face vectors are zero-padded to fit. That keeps inserts working, but vectors from different models live in different spaces. A query embedded by one provider shouldn’t be compared with chunks embedded by another. The fix is to store the provider alongside each chunk and embed the query with the same one.

Takeaway: most answer-quality problems in a RAG app are retrieval problems, and retrieval problems hide in schema decisions.