RAG failure points are an architecture problem, not a model problem

Retrieval augmented generation grounds a language model's output in retrieved documents instead of relying on what the model memorized during training. You index a knowledge base, retrieve the passages most relevant to a query, and pass them into the prompt so the model answers from your data. That design is why RAG became the backbone of enterprise AI. It is also why RAG inherits an entire class of failures that have nothing to do with the model.

The stakes are set by a gap. Near-universal adoption, rare meaningful value. McKinsey's 2025 global survey of 1,993 participants across 105 nations found that 88% of organizations use AI in at least one business function, yet only 39% report any measurable EBIT impact, and only 6% qualify as high performers with enterprise-wide value.^[4] Inside RAG specifically, the numbers sharpen: 51% of enterprise AI implementations rely on RAG, and 80% of those projects are projected to fail.^[2] The question that matters for anyone making architecture decisions now is why the same technology, deployed by comparably resourced firms, produces a 6% cohort of winners and a large majority that cycle through pilots.

Tricky Wombat made with Google Gemini 3.1 Flash Image, Jul 2026

Source

What does the data show about where RAG breaks?

The foundational map comes from the ACM CAIN 2024 experience report, an analysis of three production RAG deployments across research, education, and biomedical domains. It named seven failure points: missing content, when the answer is not in the knowledge base; missed top-ranked results, when relevant content exists but is not retrieved; not in context, when retrieved content is cut off before it reaches the prompt; not extracted, when the model has the answer in context but fails to pull it out; wrong format; incorrect specificity; and incomplete answers.^[5] The finding that matters most for architecture decisions: retrieval is the most frequent failure point, and system robustness "evolves rather than [being] designed in at the start."^[5]

The map keeps growing where deployment moves fastest. A 2026 ACL TrustNLP taxonomy expanded the inventory to 33 distinct failure modes organized across 7 pipeline stages: ingestion, representation, retrieval, generation, evaluation, deployment, and agentic orchestration. Of those 33 modes, 12 lack any peer-reviewed empirical evidence, and all 8 agentic orchestration modes are among them, which the authors call an evidence desert in the fastest-growing RAG deployment pattern.^[6] Read that against your roadmap. The modality most enterprises are moving toward in 2026, multi-agent orchestration, has the weakest evidence base for how it fails.

The reasoning ceiling is measurable. On the FRAMES benchmark of 824 multi-hop questions that require integrating 2 to 15 source documents, naive RAG achieves roughly 40% accuracy and agentic RAG roughly 60%, compared to near-100% for systems that separate LLM reasoning from programmatic data retrieval.^[7] The diagnosis is architectural: a "conflation of planning and execution within a probabilistic context."^[7] Naive RAG handles single-document lookup well and degrades sharply on the multi-step reasoning that most enterprise use cases actually demand.

What are practitioners reporting from production?

The ground-level signal points at data, not models. An interview study of 13 industry practitioners accepted to KDIR25 found that most RAG applications remain in prototype stages limited to domain-specific question answering, and that data preprocessing was the single major obstacle across every deployment.^[10]

RAG reduces hallucination but does not remove it, and pretending otherwise is where trust erodes. Legal AI hallucination drops from 69–88% for an ungrounded model to roughly 17% with RAG, yet production systems still exceed 10% on real-world queries, and legal and medical domains exceed 20%.^[14] The consequence is already in the business: 47% of enterprise users have made a business decision based on hallucinated AI content, and 53% of consumers distrust AI-powered search results.^[14] Automated metrics mask this. One documented case held RAGAS evaluation scores north of 0.85 while end-user trust sat at 60%, which is the gap between what your evals measure and what your users experience.^[14]

Numbers describe the pattern. Named cases show the mechanism. The cases below hold the model roughly constant and vary the data and retrieval architecture, which is exactly the comparison that isolates the controlling variable.

IBM: the same catalog, invisible then 95% accurate

Procurement teams and engineers searching IBM's industrial power supply catalog faced a query standard RAG could not solve. A request like "a power supply with 400VAC input and 28VDC output at 9kW" mixes exact numerical specifications with contextual requirements, and semantic similarity search kept returning products that were contextually similar but technically incompatible, a dangerous outcome in industrial procurement where a component mismatch carries safety and cost consequences.^[11] The correct product was in the catalog the entire time. Because vector similarity does not encode quantitative constraints, the retrieval layer could not surface it. The team moved from monolithic RAG to a multi-agent architecture on CrewAI with four specialized agents, and combined a structured SQL database for exact specification matching with a vector database for semantic context. The result was 95% accuracy on technical product discovery, against near-total failure before.^[11] The fix was not a smarter model. It was a data layer that matched the structure of the problem.

Morgan Stanley: retrieval quality earning 98% adoption

Morgan Stanley's tens of thousands of financial advisors needed fast access to internal research, market analysis, and compliance materials spread across a corpus that grew past 100,000 documents. Manual search was consuming client-facing time. The firm built a RAG assistant on secure internal infrastructure with human-in-the-loop oversight and an evaluation framework tested before each rollout, then expanded from a basic assistant to a suite including AskResearchGPT, which synthesizes institutional research for client use.^[15] Document retrieval efficiency improved from 20% to 80%, and advisor adoption reached 98% active usage.^[15] The 98% number is the analytically interesting one. Adoption at that level is not a mandate outcome. It is what happens when retrieval quality is high enough that people trust the system and change their behavior. The firm reached it by iterating through controlled pilots, each expansion built on validated retrieval quality from the prior stage, exactly the "robustness evolves" pattern the CAIN 2024 report described.^[5]

Henkel: 45 data sources before the model runs

Henkel, the German consumer goods and adhesives manufacturer with more than 48,000 employees, ran its technical teams across more than 45 separate internal data sources producing over 300,000 discrete search results through disconnected systems.^[16] Engineers and R&D staff duplicated research because the retrieval layer had no unified index. Henkel deployed a knowledge unification layer that indexed all 45-plus sources into a single retrieval surface, positioned as a collaboration and redundancy-reduction tool rather than an answer generator.^[16] The value landed at the ingestion and representation stages, not generation. That is the underreported organizational failure mode: 45 separate sources means 45 ingestion pipelines, each with its own schema, update cadence, and formatting, and each a failure point before the retrieval model ever runs.

Tricky Wombat made with Google Gemini 3.1 Flash Image, Jul 2026

Source

What pattern emerges across these cases?

In every case, the answer was available in the data. In every case, the retrieval architecture decided whether it could be found. IBM's catalog held the right product. Morgan Stanley's corpus held the right research. Henkel's 45 sources held the answers engineers were re-deriving. The differentiator was never model intelligence. It was how the data was represented, indexed, and retrieved. This is why the abandonment numbers and the success numbers coexist inside the same cohort of well-resourced firms: 42% abandoned most AI initiatives in 2025 while a RAG-powered support system at LinkedIn cut median per-issue resolution time by 28.6% in the same period.^[17][3] The split is not early adopters versus laggards. It is data infrastructure done well versus done poorly.

Zoom out from single systems and the distribution is bimodal, not a bell curve. IDC research puts the average return at $3.7 for every $1 invested in generative AI on average, but that average hides extreme variance: only 6% of organizations reach high-performer status with 5%-plus EBIT impact, while over 80% report no meaningful enterprise-wide impact despite active adoption.^[4][18] A small cohort captures most of the return. The rest fund pilots that never scale.

What do users and practitioners consistently report?

The consistent theme across practitioner accounts, interviews, and review data is that the bottleneck is upstream of the model. The KDIR25 study named data preprocessing as the top obstacle, ahead of model capability, cost, and latency.^[10] Evaluation practice compounds the problem: industry evaluation of RAG systems still relies heavily on manual human assessment rather than automated approaches, even as automated metrics show high scores that do not correlate with the trust practitioners actually observe.^[10][14] A healthcare engineering account captured the hardest-to-detect version of this. A system served clinical trial data from a 2022 study when users needed the 2025 updated safety profile. The stale embeddings were close enough to pass similarity thresholds, but the content was medically and legally out of date.^[13] Automated evals would have scored that retrieval as a success.

What drives the gap between strong and weak outcomes?

Where retrieval quality is high enough to earn trust, gains compound: Morgan Stanley reached 98% active advisor usage because the system was worth using.^[15] Where retrieval quality is poor, the loop runs backward. Users hit a wrong or stale answer, stop trusting the system, revert to manual search, and the project drifts toward the 42% abandonment pile.^[3] The compounding is real in both directions, and the hinge is the same variable. Gartner names it directly: of AI projects lacking adequate AI-ready data, 60% are projected to be abandoned through 2026, and the stated cause is data quality, not model limitations.^[9]

Here is the reframe. The prevailing assumption is that RAG quality is a model selection and prompt engineering problem. The evidence points somewhere else entirely.

The variable that decides whether RAG works is the information infrastructure around it: how data is represented, chunked, indexed, and retrieved. Model selection is the last lever to pull, not the first.

A NAACL 2025 study tested 25 chunking configurations across 48 embedding models, 1,200 configuration pairs, and found that chunking configuration influenced retrieval quality as much as or more than the choice of embedding model.^[19] Hold that next to IBM's catalog case, where the same class of embedding model, applied to the same data, produced near-zero accuracy under vector-only retrieval and 95% under a hybrid SQL-plus-vector architecture.^[11] The model was not the difference. The data layer was. The research blind spot mirrors the organizational one: the 2026 ACL taxonomy found retrieval and generation failures comparatively well-studied, while representation, evaluation, and agentic orchestration failures remain under-investigated despite occurring frequently in production.^[6] Enterprises under-invest in exactly the stages where the field has the least evidence, and where the cases show the outcomes are actually decided.

Tricky Wombat made with Google Gemini 3.1 Flash Image, Jul 2026

Source

What does a successful implementation look like?

Two cases show the reframe working in production. Grab, the Singapore-headquartered super-app, built two purpose-specific RAG systems instead of one architecture forced to serve every need. Its Report Summarizer connects an internal SQL and Python API platform with the firm's prompt-management system to generate and summarize recurring reports, saving 3 to 4 hours per report. Its A* Bot serves fraud investigators through Slack, using RAG over investigation templates to select predefined queries, run them against transaction databases, and return structured findings.^[17] Grab avoided IBM's original failure mode by design: SQL-based retrieval for structured data, vector only for template selection. It never asked vector similarity to do a job vector similarity cannot do.

LinkedIn attacked the representation layer. Standard RAG treated support tickets as independent documents, missing the causal and structural relationships that experienced engineers use to resolve complex issues. LinkedIn constructed a knowledge graph from existing historical ticket data, encoding entity relationships and issue linkages, then ran retrieval over that graph so it surfaced both textually similar tickets and causally related issue chains. The result was a 28.6% reduction in median per-issue resolution time.^[17] The memorable part: the information needed to improve retrieval was already in the system. It was the data representation, not the data, that was missing. Set these two against the earlier struggles and the pattern is unmistakable without any heavy comparison. When the answer is already in the data, the retrieval and representation architecture is what decides whether anyone can reach it.

The cost of getting it wrong runs on two ledgers. Direct error: AI hallucinations are estimated to have cost businesses around $67.4 billion globally in 2024, and Forrester puts the average annual cost per employee for hallucination verification and mitigation at $14,200, with employees in organizations running production AI spending about 4.3 hours per week checking outputs before acting on them.^[20][21][21] Run that across 1,000 employees and verification alone is a $14.2 million annual line item before a single wrong answer reaches a customer. Then the hidden category: a healthcare enterprise documented by Deloitte hit $6 million in unplanned annualized infrastructure cost from token consumption growing 8 to 10% per month, invisible to finance until it reached roughly 1 trillion tokens.^[22]

The return on getting it right is where the reframe pays off, and it is concentrated in the retrieval stack, not the model. Cross-encoder re-ranking improves retrieval precision 18–42% in production, and improving the re-ranker returns more per dollar than upgrading the language model.^[8] Hybrid retrieval combining dense embeddings with keyword matching lifts recall 20–40% over dense-only approaches.^[8] Chunking strategy alone determines a 35-point accuracy spread: semantic chunking with metadata filtering reaches 60%, semantic chunking without metadata 42%, and fixed chunking with metadata just 25%.^[8] A chunking decision, made during data preparation, moves accuracy as much as a major architectural change. That is the whole thesis in a single benchmark. The money follows the data pipeline.

Tricky Wombat made with Google Gemini 3.1 Flash Image, Jul 2026

Source

The thesis restated: RAG is necessary and not sufficient, and outcomes are decided by the retrieval and data infrastructure around the model. That is where Tricky Wombat builds. We frame the work around the pipeline, not the model, because the pipeline is where the 35-point swings, the stale-content failures, and the trust erosion actually happen. A RAG pipeline has to get three things right before model choice matters at all.

1. Represent and chunk data for retrieval, not for storage

Most systems ingest documents with fixed-size chunking and no metadata, which is why fixed chunking benchmarks at 25% accuracy.^[8] The correct approach enriches data during ingestion. We generate summaries, keywords, entities, and candidate questions at document-processing time and attach them as metadata, so retrieval matches on meaning and structure rather than raw token proximity, and query latency does not increase.^[23] Semantic chunking with metadata is the difference between 25% and 60% before any model is chosen.

2. Match the retrieval mechanism to the query type

Most systems run vector-only retrieval for everything, which fails silently on quantitative constraints, the IBM catalog failure mode where the right answer is present but unreachable.^[11] The correct approach routes by query structure: structured lookup for exact specifications and numerical constraints, dense embeddings for semantic context, keyword matching layered in for recall, and graph traversal where entity relationships carry the meaning. Hybrid retrieval lifts recall 20–40% over dense-only, and cross-encoder re-ranking adds another 18–42% in precision on top.^[8]

3. Keep knowledge fresh and attributable

Most systems index once and let content drift, so retrieval returns confident but stale answers that pass similarity thresholds while being wrong, the clinical-trial failure where a 2022 safety profile answered a 2025 question.^[13] The correct approach treats freshness as a monitored property: scheduled re-processing, versioned content, and citation verification that ties every generated claim back to a source passage so a reviewer can check it. Attribution is not a compliance nicety. It is how you detect the failures automated scores miss.

The pipeline is not a one-time build. Continuous operation means monitoring retrieval and generation metrics separately, re-processing content as sources change, and verifying citations on every response, so the system's accuracy improves as the knowledge base grows rather than degrading as it drifts. In 2026, 60% of new RAG deployments include systematic evaluation from day one, up from under 30% in early 2025, and that shift is why more of them survive contact with production.^[24][12]

Tricky Wombat made with Google Gemini 3.1 Flash Image, Jul 2026

Source

Across IBM's catalog, Morgan Stanley's corpus, Henkel's 45 sources, Grab's two systems, and LinkedIn's knowledge graph, the answer was already in the data, and the retrieval architecture decided whether anyone could reach it. That is the entire difference between the 6% and everyone else. RAG is necessary. It is the base layer the most sophisticated AI programs run on, and adopting it buys you nothing on its own.

The broader principle holds beyond RAG. AI outcomes track information infrastructure, not model selection, and the market is about to test that at scale as it grows toward USD 9.86 billion by 2030.^[1] Organizations that keep shopping for a better model will keep landing in the 80% that fails, because they are optimizing the one variable the evidence says matters least. The organizations that win will be the ones that treated retrieval, representation, and data freshness as the product, and the model as a component. The catalog already has the answer. Build the pipeline that can find it.