The Incident: A Cautionary Tale
This case study is based on a real incident shared by a legal tech company (anonymized). I'm analyzing it here because the lessons are invaluable for anyone building production RAG systems.
December 15th, 2025. 2:47 AM.
An enterprise legal document assistant told a Fortune 500 customer that their merger contract contained a non-compete clause that would expire in 6 months. The actual clause had no expiration. The customer made strategic decisions based on this information.
Settlement cost: $240,000. Reputation damage: incalculable.
This post is a detailed analysis of what went wrong, why the "99% accuracy" metric was meaningless, and how the team rebuilt the system.
•••
What Our System Looked Like
We had the standard RAG architecture everyone teaches:
Code
User Query → Embedding → Vector Search → Top-K Chunks → LLM → ResponseOur metrics looked great:
- •Retrieval recall@10: 94%
- •Response coherence: 4.2/5 (GPT-4 judge)
- •User satisfaction: 87% thumbs up
We were measuring the wrong things.
•••
The Root Cause Analysis
Diagram
flowchart LR
A[Query: Non-compete expiration?] --> B[Vector Search]
B --> C[Retrieved: Section 14.3b]
C --> D[Missed: Appendix J Amendment]
D --> E[LLM Assumed Complete Info]
E --> F[Hallucinated Answer]
F --> G[$240K Settlement]
style D fill:#ef4444,color:#fff
style F fill:#ef4444,color:#fff
style G fill:#ef4444,color:#fff
Problem 1: Retrieval Recall is Not Enough
Our 94% recall meant we retrieved relevant chunks 94% of the time. But "relevant" isn't "sufficient."
The contract had 847 pages. The non-compete clause was in Section 14.3(b). The amendment modifying that clause was in Appendix J, page 743.
Our retrieval got Section 14.3(b). It missed Appendix J.
The LLM saw the non-compete clause, didn't see the amendment, and confidently stated the clause as written.
Python
# What we were doing (WRONG)
def retrieve(query: str, k: int = 5) -> List[str]:
query_embedding = embed(query)
results = vector_db.search(query_embedding, top_k=k)
return [r.text for r in results]
# What we should have done
def retrieve_with_context(query: str, document_id: str) -> List[str]:
query_embedding = embed(query)
# Step 1: Find directly relevant chunks
direct_results = vector_db.search(
query_embedding,
top_k=10,
filter={"document_id": document_id}
)
# Step 2: Find chunks that REFERENCE the relevant sections
section_ids = extract_section_references(direct_results)
reference_results = vector_db.search_by_metadata(
filter={
"document_id": document_id,
"references_sections": {"$in": section_ids}
}
)
# Step 3: Find amendments and modifications
amendments = vector_db.search_by_metadata(
filter={
"document_id": document_id,
"chunk_type": "amendment",
"modifies_sections": {"$in": section_ids}
}
)
return deduplicate(direct_results + reference_results + amendments)Problem 2: We Didn't Understand Legal Document Structure
Legal documents aren't blog posts. They have:
- •Cross-references ("Subject to Section 8.2...")
- •Amendments that modify earlier sections
- •Definitions that change meaning ("Confidential Information" means...)
- •Schedules and exhibits that contain the actual terms
Our naive chunking destroyed these relationships.
Python
# Our original chunking (WRONG)
def chunk_document(text: str, chunk_size: int = 500) -> List[str]:
return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]
# What legal documents actually need
class LegalDocumentParser:
def parse(self, document: bytes) -> LegalDocument:
# Extract document structure
toc = self.extract_table_of_contents(document)
sections = self.extract_sections_with_hierarchy(document)
definitions = self.extract_definitions(document)
cross_references = self.build_reference_graph(sections)
amendments = self.identify_amendments(sections)
return LegalDocument(
sections=sections,
definitions=definitions,
cross_references=cross_references,
amendments=amendments,
hierarchy=self.build_hierarchy(toc, sections)
)
def chunk_with_context(self, legal_doc: LegalDocument) -> List[Chunk]:
chunks = []
for section in legal_doc.sections:
# Include the section's context
context = {
"section_id": section.id,
"parent_section": section.parent_id,
"definitions_used": self.find_definitions_in_text(
section.text, legal_doc.definitions
),
"references_to": legal_doc.cross_references.get(section.id, []),
"referenced_by": legal_doc.cross_references.get_reverse(section.id, []),
"modified_by_amendments": [
a.id for a in legal_doc.amendments
if section.id in a.modifies
],
}
chunks.append(Chunk(
text=section.text,
metadata=context
))
return chunksProblem 3: The LLM Didn't Know What It Didn't Know
When the LLM received 5 chunks about a non-compete clause, it assumed it had complete information. It never said "I only see part of this clause" or "there may be amendments I haven't seen."
Python
# Our original prompt (WRONG)
prompt = f"""
Based on the following document excerpts, answer the user's question.
Excerpts:
{chunks}
Question: {question}
Answer:"""
# What we needed
prompt = f"""
You are analyzing a legal document. You have been given excerpts, NOT the complete document.
CRITICAL INSTRUCTIONS:
1. Only make statements you can DIRECTLY support with the provided excerpts
2. If the excerpts don't contain enough information, say so explicitly
3. Legal documents often have amendments and modifications - if you don't see explicit
amendment language, note that "this excerpt does not show any amendments to this clause"
4. Always cite the specific section numbers for every claim
Excerpts:
{chunks}
Metadata about these excerpts:
- Sections covered: {section_ids}
- Known amendments in document: {amendment_count}
- Amendments provided in excerpts: {provided_amendment_count}
Question: {question}
Answer with explicit citations and uncertainty markers:"""•••
The Rebuilt Architecture
Here's an overview of our improved RAG pipeline that prevents hallucinations:
Diagram
flowchart TB
subgraph Ingestion["Document Ingestion"]
A[Upload Document] --> B[Legal Document Parser]
B --> C[Extract Sections & Hierarchy]
B --> D[Extract Definitions]
B --> E[Build Cross-Reference Graph]
B --> F[Identify Amendments]
end
subgraph Indexing["Multi-Index Storage"]
C --> G[(Vector Index)]
D --> H[(Keyword Index)]
E --> I[(Graph Database)]
F --> J[(Amendment Index)]
end
subgraph Retrieval["Verified Retrieval"]
K[User Query] --> L[Semantic Search]
L --> M[Expand Cross-References]
M --> N[Fetch Amendments]
N --> O[Resolve Definitions]
O --> P{Completeness Check}
end
subgraph Generation["Grounded Generation"]
P -->|Pass| Q[Generate with Citations]
P -->|Fail| R[Request More Context]
Q --> S[Verify Each Claim]
S --> T[Return Grounded Answer]
end
G --> L
H --> O
I --> M
J --> N
1. Document-Aware Ingestion Pipeline
Python
class EnterpriseDocumentPipeline:
def __init__(self):
self.parser = LegalDocumentParser()
self.validator = DocumentValidator()
self.indexer = HierarchicalIndexer()
async def ingest(self, document: bytes, doc_type: str) -> str:
# Parse with document type awareness
if doc_type == "legal_contract":
parsed = self.parser.parse_contract(document)
elif doc_type == "financial_report":
parsed = self.parser.parse_financial(document)
else:
parsed = self.parser.parse_generic(document)
# Validate completeness
validation = self.validator.validate(parsed)
if not validation.is_complete:
raise IncompleteDocumentError(
f"Document missing required sections: {validation.missing}"
)
# Create multiple index types
doc_id = generate_id()
# Vector index for semantic search
await self.indexer.index_vectors(doc_id, parsed.chunks)
# Graph index for relationships
await self.indexer.index_graph(doc_id, parsed.cross_references)
# Keyword index for exact matching
await self.indexer.index_keywords(doc_id, parsed.definitions)
return doc_id2. Multi-Stage Retrieval with Verification
Python
class VerifiedRetrieval:
async def retrieve(self, query: str, doc_id: str) -> RetrievalResult:
# Stage 1: Initial semantic retrieval
initial_chunks = await self.vector_search(query, doc_id, k=20)
# Stage 2: Expand with cross-references
section_ids = [c.metadata["section_id"] for c in initial_chunks]
related_sections = await self.graph_db.get_related(section_ids, depth=2)
expanded_chunks = await self.get_chunks_by_section(related_sections)
# Stage 3: Check for amendments
amendments = await self.get_amendments_for_sections(section_ids)
# Stage 4: Resolve definitions
definitions_used = self.extract_definitions_used(initial_chunks)
definition_chunks = await self.get_definition_chunks(definitions_used)
# Stage 5: Verify completeness
all_chunks = initial_chunks + expanded_chunks + amendments + definition_chunks
completeness_check = await self.verify_completeness(query, all_chunks)
return RetrievalResult(
chunks=all_chunks,
completeness_score=completeness_check.score,
missing_sections=completeness_check.potentially_missing,
confidence_level=completeness_check.confidence
)3. Grounded Generation with Citations
Python
class GroundedGenerator:
async def generate(self, query: str, retrieval: RetrievalResult) -> Response:
# Pre-generation check
if retrieval.confidence_level < 0.7:
return Response(
answer=None,
status="INSUFFICIENT_CONTEXT",
message=f"Cannot confidently answer. Missing: {retrieval.missing_sections}"
)
# Generate with strict grounding
response = await self.llm.generate(
messages=[
{"role": "system", "content": GROUNDED_SYSTEM_PROMPT},
{"role": "user", "content": self.format_context_and_query(
retrieval.chunks, query
)}
],
response_format=GroundedResponse # Structured output with citations
)
# Post-generation verification
verification = await self.verify_claims(response, retrieval.chunks)
if verification.has_unsupported_claims:
# Re-generate with stricter constraints or return error
return await self.handle_unsupported_claims(
response, verification, retrieval
)
return Response(
answer=response.answer,
citations=response.citations,
confidence=verification.confidence,
caveats=response.caveats
)•••
The New Metrics We Track
Python
class EnterpriseRAGMetrics:
def track_query(self, query_id: str, query: str, response: Response,
ground_truth: Optional[str] = None):
metrics = {
# Retrieval quality
"retrieval_completeness": response.retrieval.completeness_score,
"cross_references_found": len(response.retrieval.cross_refs),
"amendments_checked": response.retrieval.amendments_checked,
# Generation quality
"citation_coverage": self.calculate_citation_coverage(response),
"unsupported_claims": len(response.verification.unsupported_claims),
"uncertainty_expressed": response.answer.count("may") +
response.answer.count("unclear"),
# Safety
"refused_due_to_uncertainty": response.status == "INSUFFICIENT_CONTEXT",
"human_review_requested": response.requires_human_review,
# Business metrics
"customer_escalated": None, # Filled later
"correction_required": None, # Filled later
}
if ground_truth:
metrics["factual_accuracy"] = self.check_factual_accuracy(
response.answer, ground_truth
)
self.analytics.track("enterprise_rag_query", metrics)•••
Results After Rebuild
| Metric | Before | After |
|---|---|---|
| Retrieval completeness | Not measured | 96% |
| Citation coverage | 0% | 100% |
| Unsupported claims per response | 2.3 avg | 0.02 avg |
| Refused due to uncertainty | 0% | 8% |
| Corrections required | 4.2% | 0.1% |
| Customer escalations | 12% | 2% |
•••
Key Lessons
- 1Your metrics are lying to you. Retrieval recall and user satisfaction don't measure safety.
- 2Document structure matters more than semantic similarity. A chunk from page 743 might be the most important for a query about page 12.
- 3LLMs don't know what they don't know. You must build systems that detect and communicate uncertainty.
- 4Refusal is better than confident wrongness. Enterprise customers would rather hear "I need more information" than get wrong answers.
- 5Legal/financial/medical domains need domain-specific parsing. Generic chunking destroys the relationships that matter.
The $240K was expensive tuition for this team, but their rebuilt system now processes $50M in contract reviews annually with zero critical errors. Learn from their mistakes so you don't have to pay the same price.