Fixing RAG Hallucinations with Graph RAG and Cross-Encoder Reranking

Where cosine similarity fails

Dense embeddings are excellent at finding text that is about the same thing. Enterprise questions are rarely that forgiving. "Which supplier contracts signed after 2023 do not include a termination-for-convenience clause?" depends on entities, dates and a negation. A pure vector search returns the passages that look most like the question, which are usually the contracts that do include the clause. The embedding has no reliable way to represent the word not, and no notion that a contract signed in 2022 falls outside the range.

Contracts, policies and technical manuals are also full of near-duplicates: the same template with different parties, amounts and effective dates. Embeddings collapse exactly those differences, because the surrounding language is identical. The model then receives five nearly identical passages, picks one, and confidently cites the wrong version. Superseded documents make this worse. Version three and version four of a policy sit next to each other in vector space, and nothing in the similarity score says which one is in force.

Identifiers are the third blind spot. Part numbers, clause references such as 14.2(b), drug names, error codes and account numbers are split into fragments that carry little meaning to an embedding model, so a query for one code happily retrieves a passage about a similar-looking one. None of this means vector search is wrong. It means similarity is one signal among several, and a system that treats it as the only signal will hallucinate in precisely the places where users most need it to be right.

Most hallucinations start at ingestion

Before retrieval can be good, the index has to contain the right text in the right shape. The most common root cause we find when auditing a misbehaving RAG system is not the model or the vector database. It is a PDF parser that flattened a table into a stream of numbers, dropped the headers that said which column was which, or merged a footnote into the clause above it. The model then reads a passage that genuinely says something false, and repeats it faithfully.

Parse with structure preserved. Tables should become Markdown or JSON with their headers attached to every group of rows, scanned pages need OCR with a confidence score you keep, and figures need a text description or should be excluded deliberately rather than skipped silently. Chunk along the document's own structure, its sections, clauses and list items, rather than at a fixed token count, and keep the heading path with each chunk so that a passage reading 'the fee is waived' still knows it belongs to Schedule B, Clause 7.

Metadata is what makes every later stage precise. Each chunk should carry its source document, version, effective dates, page and character offsets, document type, owner and access control list. Without offsets you cannot build exact citations. Without effective dates you cannot filter out superseded text. Without permissions on every chunk, the assistant will eventually quote a document to someone who was never meant to see it, which is a worse failure than any hallucination.

• Keep the original file and the parsed output side by side, so every chunk can be traced back to a page and region during review.
• Attach the full heading path to each chunk, and store parent sections so the generator can be given surrounding context when a chunk alone is ambiguous.
• Deduplicate templates at ingestion and record what differs between copies, such as parties, amounts and dates, as structured fields.
• Mark superseded versions explicitly instead of deleting them, because questions about the past still need them.

Hybrid retrieval: BM25, vectors and a knowledge graph

We run three retrievers in parallel. BM25 catches exact identifiers, clause numbers and product codes that embeddings blur. Dense vector search in Qdrant handles semantic recall across paraphrases, with payload filters for tenant, document type, permissions and effective date applied inside the search rather than afterwards, so filtering never empties a result list that looked full. A Neo4j knowledge graph, built during ingestion, answers the relational part of the question: which entities are connected, how, and which expected connections are missing.

The graph is where negation and multi-hop questions become tractable. The supplier contract question above is a straightforward graph query once ingestion has extracted contracts, signing dates and clause types as nodes and relationships: match the contracts in the date range, exclude those linked to the clause, return the documents. We prefer a small library of reviewed, parameterised query templates that the model chooses between and fills in, over letting it write free-form Cypher, because a wrong generated query fails silently and returns plausible results.

The graph is also bi-temporal. Every relationship records when it was true in the world and when we learned about it, so the system can answer "what was in force on the date of the incident" as well as "what is in force today", and can explain why an answer given last month differs from one given now. The three result lists are merged with reciprocal rank fusion, which combines rankings without needing the retrievers' scores to be comparable, and the fused list goes to the reranker.

• Lexical retrieval handles identifiers, numbers and exact phrasing that embeddings blur.
• Semantic retrieval with metadata filters provides recall across paraphrases and synonyms.
• Graph traversal answers questions about entities, relationships, dates and negative constraints.
• Rank fusion merges all three lists without hand-tuned score weights that drift as the corpus grows.

Cross-encoder reranking

The retrievers are tuned for recall, so together they return a generous candidate set, typically fifty to a hundred passages. A cross-encoder then scores each passage against the full question, reading both together rather than comparing two vectors computed independently. It can see that a passage mentions the clause only in a sentence that excludes it, or that its dates fall outside the range. This is slower per item but far more precise, and it is usually where the largest single accuracy gain comes from.

Rerankers come as hosted APIs, such as Cohere Rerank, or as open-weights cross-encoders you serve yourself, which matters when documents cannot leave your network. Scoring around a hundred passages on a GPU usually adds tens to a couple of hundred milliseconds, so cap the candidate count and truncate long passages sensibly rather than skipping the step. For specialised vocabularies, fine-tuning a reranker on a few thousand question and passage pairs from your own data is often a better investment than switching to a larger generator.

Use the reranker's scores, not just its order. Set a relevance threshold calibrated on your evaluation data: if nothing clears it, that is strong evidence the corpus does not contain the answer, and the system should say so before the generator has a chance to improvise. Only the top handful of passages, usually five to eight, each carrying its source document, version, page and span offsets, are given to the generator, with the strongest placed first, since models attend less reliably to material buried in the middle of a long context.

Making every claim verifiable

The generator is required to attach a citation to every factual sentence, pointing to a passage identifier and a span within it. We ask for structured output, a list of sentences each with its citations, rather than free text with bracketed numbers, because structure can be validated mechanically and free text cannot. Sentences that make no factual claim, such as a short introduction, are marked as such so they are not mistaken for unsupported claims.

A post-generation check then verifies each claim in two steps. First, the cited span must exist, character for character, in the cited passage. Second, it must actually support the sentence, which is tested with an entailment check: either a natural language inference model or a separate, tightly prompted model call that sees only the sentence and the span. Sentences that fail are removed. If the answer no longer holds together, the whole response is replaced with a clear statement that the documents do not answer the question, followed by the closest relevant sources.

That refusal path matters more than any benchmark. Users trust a system that says "I could not find this" far more than one that is right most of the time and wrong with total confidence. Citations also change how people use the answer. Our MedOmni clinical copilot, built for oncologists over 15 million patient records and biomedical research papers, highlights the exact paragraph of source evidence behind each answer, so a clinician can check a claim against its source instead of taking it on trust.

Evaluate continuously, not once

We maintain a golden set of questions with known answers and source spans for every corpus, and run it on every ingestion change, prompt change, reranker change and model upgrade. Recording source spans, not just answers, lets you evaluate retrieval separately from generation. When an answer is wrong, you need to know whether the right passage was never retrieved, was retrieved but ranked too low, or reached the generator and was misread. Each of those points to a different fix.

The question mix matters as much as the size. Include questions with identifiers, negations, date ranges and multi-hop relationships, questions that need two documents, and a deliberate share of questions the corpus cannot answer. Those unanswerable questions are the only way to measure whether the refusal path works; a set made only of answerable questions rewards a system for always producing something.

Citation precision, answer faithfulness and refusal behaviour are tracked as release gates, the same way latency and error rate are. Automated judges are useful for scale, but calibrate them against human reviewers on a sample first, and check that calibration again whenever the judge model changes. In production, log every query with its retrieved passages, scores and citations, and turn user-flagged answers into new golden-set cases within days, not quarters.

• Recall at k measures whether the correct passage appears in the candidate set before reranking.
• The rank of the correct passage after reranking shows whether the reranker is helping or hurting.
• Citation precision measures the share of cited spans that genuinely support their sentences.
• The correct refusal rate on unanswerable questions shows whether the system knows when to stop.
• The false refusal rate on answerable questions catches a threshold set so high that the assistant becomes useless.

Failure modes that survive a good pipeline

Even with hybrid retrieval, reranking and citation checks, a few failure modes keep appearing in production, and most are about the corpus changing under the system rather than the model misbehaving. A stale index is the classic one: a policy is updated on Monday, the nightly sync fails quietly, and the assistant spends a week citing a perfectly real, perfectly obsolete paragraph. Every citation passes verification, because the span exists. It is simply no longer true.

Conflicting sources are the second. Two current documents disagree, a regional policy and a group policy for example, and the generator picks one. The right behaviour is usually to show both and state which takes precedence, which means precedence rules have to be encoded somewhere, often as graph relationships, rather than left to the model. The third is text inside documents that reads like instructions. Retrieved content is untrusted input, and it must never be able to change the system's rules, call tools or alter who can see what.

• Monitor index freshness per source and alert when a sync fails or document counts drop unexpectedly.
• Show the document version and effective date beside every citation, so users can spot stale sources themselves.
• Encode precedence between overlapping documents explicitly instead of hoping the model infers it.
• Keep retrieved text separate from instructions, and check permissions again at answer time as well as at ingestion.
• Review a sample of refusals every week, because a rising refusal rate is often the first sign of an ingestion problem.

How we would approach this

We start by building the golden set with your domain experts and running the simplest credible baseline against it, usually hybrid search with a reranker and enforced citations, before adding a knowledge graph. The graph earns its place only when the failures show relational, temporal or negative questions that flat retrieval cannot answer; it is powerful, and it is also the most expensive component to keep accurate. Our earlier piece on prompting, RAG and fine-tuning covers how to decide whether retrieval is the right lever in the first place.

This is the work of our RAG Solutions & Knowledge Engines service: document parsing, chunking, hybrid indexes, reranking and citation guardrails. Where data cannot leave your network, the same approach runs on private models, as in the HIPAA-Compliant Clinical RAG Copilot case study, an air-gapped, on-premise deployment using a fine-tuned Llama 3 70B served with vLLM and Qdrant for vector search. For a first estimate, the Budget Calculator covers model hosting options and timeline. If you have a sample corpus and fifty real questions, a scoping call is the quickest way to find where your current system goes wrong.