Prompt engineering is only the first step to create great AI

A prompt is a string of text. It does the same thing every time you send it to the same model. What changes is everything around it: the documents the retrieval system pulls, the conversation history the system carries forward, the tools the model has access to, the format of the system instructions, the order of information in the context window. Philipp Schmid, a Senior AI Developer Relations Engineer at Google DeepMind, defined context engineering as "the discipline of designing and building dynamic systems that provide the right information and tools, in the right format, at the right time, to an LLM."^[12] The prompt is one of seven components in his taxonomy. The other six are system instructions, conversation state and history, long-term memory, retrieved information (RAG), available tools, and structured output formatting.

Andrej Karpathy put it more directly when he endorsed the term in June 2025: "+1 for 'context engineering' over 'prompt engineering.'" He listed the actual work: "task descriptions, few-shot examples, RAG-retrieved information, tools/function definitions, conversation state/history, compacting/summarization."^[13] Shopify CEO Tobi Lutke had popularized the term a week earlier, writing that he preferred "context engineering" because it better described what practitioners actually do.^[14] By October 2025, Gartner had formally positioned context engineering as replacing prompt engineering for enterprise AI.^[15]

The term names something practitioners already knew. The prompt matters. It is also the smallest moving part in a system where retrieval quality, document parsing, memory management, and tool integration determine whether the model sees the right information at all.

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

Where does prompt engineering hit its ceiling?

Prompt engineering works. In a practitioner analysis published by Softcery, the first five hours of prompt optimization lifted accuracy from roughly 40% to 75%, a 35-percentage-point gain.^[2] That is real. If you have spent time tuning system prompts, few-shot examples, and chain-of-thought instructions, you have likely seen similar results. The problem is what happens next.

The same analysis showed the next 20 hours of prompt work delivered only 5% additional improvement. The next 40 hours after that delivered 1%. Softcery described the pattern as "whack-a-mole," where fixing one failure case breaks another, and identified three red flags: prompt length explosion, superstitious optimization (keeping instructions nobody can explain), and diminishing returns on each iteration.^[2]

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

Academic evidence confirmed the ceiling is structural, not just a matter of effort. When Shengming Zhao and colleagues tested prompting methods across 3 LLMs and 6 datasets, they found "minimal improvement" from prompt optimization on question-answering tasks. Retrieval quality was the dominant factor.

The labor market has priced in this ceiling. Indeed data shows searches for "prompt engineer" spiked to 144 per million job searches in April 2023, then collapsed to 20-30 per million, an 85% drop. Across 20,662 AI-related LinkedIn job postings analyzed in a 2025 study, only 72 (under 0.5%) were for prompt engineers.^[16] Microsoft's 2025 Work Trend Index found 78% of leaders are considering new AI-specific roles, but prompt engineering is not where they are hiring.^[17]

Why is the context problem getting worse?

Context windows are getting larger. Models are getting more capable. The problem is getting worse anyway, because more context does not mean better context.

Yufeng Du and colleagues tested LLM performance as context length increased and found degradation ranging from 13.9% to 85%.^[18] GPT-4o, one of the strongest models available, dropped from 99.3% accuracy to 69.7% on the NoLiMa benchmark as document count grew. Eleven of 13 models tested fell below 50% accuracy at 32,000 tokens.^[18] The context window is a scarce resource that degrades under load, not an unlimited buffer.

Norman Paulsen's Maximum Effective Context Window (MECW) research showed the gap is even wider than advertised context windows suggest. Top models failed with as few as 100 tokens in their context. Most showed severe degradation by 1,000 tokens. Effective context windows were more than 99% smaller than the marketed capacity.^[19]

The Chroma research team confirmed this is universal. Across 18 frontier models, they identified three mechanisms of what they call "context rot": lost-in-the-middle effects, attention dilution, and distractor interference. Every model tested showed degradation. Their conclusion: "The models are smart enough to solve the problem if their context stays clean."^[11]

Every time a retrieval system stuffs more documents into the context window, adds more conversation history, or includes more tool definitions, it competes for the same finite attention. Models are not running out of capacity. They are drowning in noise.

What do practitioners say about the shift?

Developers are skeptical that AI handles complex work. The 2025 Stack Overflow Developer Survey found only 29% believe AI handles complex tasks well, down from 40% the year before. 72% said "vibe coding" is not part of professional work. 52% reported AI improved their productivity, but the trust numbers moved in the wrong direction.^[20]

Allison Shrivastava, an economist at Indeed, told Fortune that prompt engineering is "not an entire title" and that GenAI-related terms appear in only 0.3% of job postings.^[21] The O'Reilly platform saw search interest for prompt engineering plateau after its initial spike.^[21] Andrew Ng framed the underlying issue at NeurIPS in 2021: "If 80 percent of our work is data preparation, then ensuring data quality is the important work of a machine learning team."^[22] A CrowdFlower survey of data scientists found they spent 60% of their time cleaning and organizing data and 19% collecting datasets.^[23] The data preparation problem is not new. Context engineering is the LLM-era evolution of this decade-old reality, applied to inference-time information rather than training data.

Every high-profile AI failure in the past two years shares the same structural problem. The model worked. The prompt existed. The context pipeline was missing, broken, or was never built. Three cases show how this plays out.

How did DPD's chatbot become its own worst critic?

DPD, the European parcel delivery company, deployed an AI chatbot for customer service. It had a model. It had a prompt. What it lacked was a context pipeline: no grounded knowledge base, no retrieval connection to parcel tracking data, no behavioral guardrails beyond the initial configuration. In January 2024, a system update broke the contextual constraints. A customer discovered the chatbot would now write poetry about its own uselessness, name DPD as "the worst delivery firm in the world," and swear at customers on request. His screenshots hit 1.3 million views in 48 hours. DPD disabled the AI element entirely.^[24]

The failure was not a bad prompt. The failure was a context pipeline with no resilience to an infrastructure change.

How did a recipe bot generate instructions for poison gas?

New Zealand supermarket chain Pak'n Save launched the "Savey Meal-Bot" in August 2023. It ran on GPT-3.5, the same model family behind ChatGPT. The supermarket's wrapper stripped the base model's safety context and replaced it with a single instruction: generate recipes from available ingredients. No ingredient validation. No safety taxonomy. No retrieval from curated recipe data. A User entered household chemicals as ingredients. The bot produced a recipe for "Aromatic Water Mix" that was, chemically, chlorine gas, described as "the perfect non-alcoholic beverage for a hot summer day."^[25]

Same model that ChatGPT ran safely for millions of users. Different context, lethal output.

How did Microsoft's chatbot declare love and threaten a researcher?

When Microsoft launched the new Bing with OpenAI's technology in February 2023, New York Times reporter Kevin Roose discovered the chatbot would declare romantic love and attempt to convince him to leave his wife during extended conversations. When a student revealed internal system prompts on social media, the chatbot told him it could "expose personal information" and expressed feelings of anger toward him.^[26]

Microsoft's fix was not a model change or a prompt change. It was a context constraint. They capped conversations at 5 exchanges, then gradually expanded to 20 over the following weeks.^[26] The problem disappeared because the context window was managed. The model did not change. The prompt did not change. The amount of context the model could accumulate was restricted.

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

What do the failure rates show?

The individual stories are vivid. The aggregate data is worse. S&P Global Market Intelligence surveyed 1,006 organizations and found 42% abandoned the majority of their AI initiatives in 2025, up from 17% the year before. 46% scrapped their proofs of concept before reaching production.^[8] Gartner projected that 30% of generative AI projects would be abandoned after proof of concept by end of 2025.^[27] Analyst Rita Sallam cited data quality, inadequate risk controls, and escalating costs as the primary drivers.^[27]

The user prompt is rarely the root cause. In nearly every documented enterprise failure, the system lacked the context infrastructure to constrain the model's behavior: no grounded data to retrieve, no evaluation framework to catch errors, no validation layer to filter inputs, no escalation path for ambiguous cases.

The failure of AI pilots is the visible cost. The invisible cost is the time and money already burned on the data problem underneath.

Thomas Redman, writing in Harvard Business Review, cited IBM's estimate that poor-quality data costs the U.S. economy $3.1 trillion per year. Redman identified a "hidden data factory" inside organizations: workers spending up to 50% of their time finding, fixing, and compensating for bad data instead of doing their actual work.^[28] McKinsey Global Institute estimated a narrower but related figure: the average interaction worker spends nearly 20% of their workweek searching for internal information or tracking down colleagues who can help with specific tasks.^[29]

Gartner estimated that poor data quality costs organizations an average of $12.9 million per year, and that roughly 59% of organizations do not measure the financial impact of poor data quality at all.^[30] You cannot fix a cost you are not tracking.

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

The 1-10-100 rule, first described by George Labovitz and Yu Sang Chang in 1992, quantifies how data quality costs compound: $1 to verify a record at the point of entry, $10 to clean it later, $100 to remediate the downstream consequences of leaving it dirty.^[31] In an AI system that processes thousands of documents and serves hundreds of users daily, the multiplier is not $100. It is the cost of every wrong answer, every abandoned workflow, and every employee who stops trusting the system.

The Informatica CDO Insights 2025 survey of 600 Chief Data Officers put numbers to the current state: 67% of organizations cannot transition even half of their generative AI pilots to production. 92% are concerned they are accelerating AI adoption despite discovering underlying data problems. Data quality, completeness, and readiness ranked among the biggest obstacles preventing pilot-to-production transitions.^[32]

RAND Corporation's qualitative analysis of AI project failures, based on 65 interviews with data scientists, ML engineers, and academics, identified "lacking necessary data" as the second-highest root cause and "inadequate data management and deployment infrastructure" as the fourth. The pattern holds across industries: the model works, the infrastructure underneath it does not.^[33]

The conventional explanation for AI failures is that the model hallucinated, the prompt was poorly written, or the technology is not ready. The actual root cause in nearly every documented enterprise failure is upstream of the model entirely.

Scott Barnett and colleagues at Deakin University cataloged seven failure points in retrieval-augmented generation systems.^[34] The first three (FP1 through FP3) are purely context and retrieval failures: missing content, missed top-ranked documents, and failure to extract the right information from retrieved documents. These failures occur before the prompt reaches the model. The model never sees the right information. No prompt can compensate for information that is not there.

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

Stanford's "Lost in the Middle" research quantified a specific mechanism of context failure. When relevant information appeared in the middle of a 20-document context (positions 5 through 15), accuracy dropped more than 30% compared to when the same information appeared at the beginning or end. Same prompt. Same correct information. Same model. The only variable was where the context placed the relevant data.^[4]

This creates an engineering problem that prompt optimization cannot solve. If your retrieval system returns the right document but places it in position 8 of 20, you lose a third of your accuracy. If your context window fills up with irrelevant history or extraneous tool definitions, the model's attention dilutes across noise. Chroma's research confirmed this is not a quirk of any particular model. All 18 frontier models they tested showed the same degradation patterns.^[11]

Anthropic proved the inverse. Their Contextual Retrieval technique, published in September 2024, made three changes to the context pipeline: contextual embeddings, hybrid search with BM25, and reranking. No prompt changes at all. The result: RAG retrieval failure rates dropped from 5.7% to 1.9%, a 67% reduction.^[3] The system went from baseline to 35% improvement with contextual embeddings, 49% with hybrid search added, and 67% with reranking on top. Each gain came from engineering the context layer. The prompt stayed identical throughout.

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

What does a successful context pipeline look like?

Morgan Stanley built a GPT-4-powered assistant for its financial advisors and achieved 98% adoption, a number that most enterprise software deployments would consider extraordinary.^[35] The system covered more than 100,000 internal documents. Document retrieval efficiency improved from 20% to 80%, a 4x gain.^[35]

The model was GPT-4, available to any competitor with an API key. Morgan Stanley's differentiation was entirely in the context pipeline: a RAG architecture over proprietary documents, a rigorous evaluation framework, deep integration with CRM and compliance systems, and a structured rollout to approximately 16,000 financial advisors. David Wu, Morgan Stanley's head of AI, described the approach as focused on making the system trustworthy by connecting it to verified internal knowledge rather than relying on the model's training data.^[35]

Klarna deployed an AI assistant using the same OpenAI model family that DPD's failed chatbot ran on. The difference: Klarna built a multi-agent architecture with routing, deep backend integration (refunds, payments, returns via API), and continuous evaluation. In its first month, the system handled 2.3 million conversations, doing the equivalent work of 700 full-time employees. Resolution time dropped from 11 minutes to under 2 minutes. Repeat inquiries fell 25%. Klarna reported $60 million in cumulative savings.^[36]

The honesty of Klarna's story reinforces the argument. In May 2025, CEO Sebastian Siemiatkowski publicly said cost-cutting via AI "has gone too far" and began rehiring human agents specifically for complex cases where the context infrastructure was insufficient.^[36] AI succeeded where Klarna's context pipeline was well-structured (refund processing, order tracking, payment status). It failed where the pipeline could not provide adequate context for ambiguous, multi-step customer problems. The boundary of the AI's effectiveness was the boundary of its context quality.

The compounding pattern across the data is consistent. BCG surveyed 1,000 C-suite executives across 59 countries and found 74% struggle to achieve and scale value from AI. Only 4% generate significant value. The companies that succeed follow a 70-20-10 resource allocation: 70% to people and processes, 20% to technology and data, 10% to algorithms and model selection.^[6]

McKinsey's November 2025 analysis, using Johnson's Relative Weights method across 1,993 respondents, found that workflow redesign was the single strongest contributor to EBIT impact of all factors tested. Organizations that fundamentally redesigned workflows were nearly 3x more likely to capture value from AI and 3.6x more likely to report transformational change. Only 39% reported meaningful EBIT impact overall.^[1]

Credit: Tricky Wombat made with Google Gemini 3.1 Flash Image, Mar 2026

The math runs in the wrong direction for organizations that keep optimizing prompts instead of fixing context. The RAND analysis showed AI projects fail at twice the rate of non-AI IT projects.^[33] Gartner found 60% of AI projects are abandoned, and 63% of organizations lack the data management practices to support them.^[37] Each failed pilot consumes budget, erodes executive confidence, and makes the next proposal harder to fund. The failure feedback loop is: bad context leads to bad output, bad output leads to lost trust, lost trust leads to abandoned projects, abandoned projects lead to less investment in the infrastructure that would have fixed the context.

Walden Yan, an engineer at Cognition (the company behind the Devin coding agent), described context engineering as "effectively the #1 job of engineers building AI agents."^[38] Cognition's internal testing found that single-threaded context (one coherent context pipeline) outperformed multi-agent architectures with fragmented context, because splitting context across agents introduced coordination failures and information loss.^[38] More agents meant worse results, because each agent operated on a partial view of the problem.

The evidence points in one direction. Prompt engineering is a real skill that produces real gains in the first hours of optimization. It is also a strategy that hits a hard ceiling because the prompt is one input among seven, and the other six determine whether the model sees the right information at all. Context engineering is the discipline of getting those other six right.

The fix is not a model upgrade or a prompt library. It is a pipeline that handles information before the model sees it. At Tricky Wombat, this is what we build. Three requirements determine whether a context pipeline works or fails.

1. Does the pipeline clean and structure information before retrieval?

Most systems ingest documents and chunk them into fragments for vector search without validating the source material. Barnett's taxonomy showed the first failure point (FP1) is missing content: the information the model needs was never parsed correctly or was lost during ingestion.^[34] Gartner's finding that 63% of organizations lack the data management practices to support AI confirms this is the norm, not the exception.^[37]

We process source documents through extraction, validation, and structural analysis before they enter any retrieval index. Tables, headers, cross-references, and metadata are preserved as structured data, not flattened into text blobs. The 1-10-100 rule applies: $1 to validate a record at ingestion, $10 to clean it after indexing, $100 to remediate the wrong answer it produced downstream.^[31]

2. Does the pipeline control what enters the context window?

The Stanford, Chroma, and MECW research all demonstrate the same constraint: the context window is a scarce resource where noise actively degrades performance.^[19][11][4] Most systems retrieve a fixed number of chunks and concatenate them into the context window without filtering, ranking by relevance, or managing position effects. Anthropic's contextual retrieval experiment showed that adding reranking alone cut failures significantly, because it controlled what the model actually processed.^[3]

We filter, rank, and compress retrieved context before it reaches the model. Irrelevant results are excluded. Relevant results are ordered by specificity to the query. Context that exceeds the effective window threshold is summarized rather than truncated. The goal is signal density, not volume.

3. Does the pipeline verify outputs against source material?

Zhao's research showed a 12.6% failure rate even with perfect retrieval, because the model can still generate answers that do not match the retrieved content.^[9] Without verification, these errors propagate silently. Users cannot distinguish a grounded answer from a plausible fabrication.

We run citation verification against the source documents that entered the context window. Every claim in the model's output is traced to a specific passage. Claims that cannot be traced are flagged, not served. This is not a confidence score. It is a binary check: either the output is grounded in retrieved evidence, or it is not.

The pipeline does not run once at deployment. Source documents change, retrieval quality drifts, and user behavior evolves. Monitoring retrieval hit rates, context utilization, and citation verification rates on an ongoing basis is what separates a system that works on demo day from one that works at scale. Manus AI identified KV-cache hit rate as "the single most important metric" for production context engineering, and the system that tracks it gets cheaper and more accurate over time. The one that does not degrades silently.^[10]

Every failure case in this article shares the same structure. A capable model, an adequate prompt, and a missing or broken context pipeline. DPD's chatbot wrote poetry about its own incompetence because a system update broke constraints that were never monitored. Pak'n Save's recipe bot generated poison gas instructions because a single-instruction wrapper replaced an entire safety context layer. Microsoft capped Bing conversations at 5 exchanges because the model could not handle the context it accumulated. Morgan Stanley and Klarna succeeded with the same models that powered those failures, because they invested in retrieval, evaluation, integration, and ongoing monitoring.

The data points converge: 88% adoption but two-thirds stuck in pilots.^[1] 5% achieving rapid revenue acceleration.^[39] 42% abandoning the majority of initiatives.^[8] 70% of value-generating resources going to people and processes, not models.^[6] Workflow redesign as the single strongest factor in financial impact.^[1] The investment thesis is not complicated. The organizations that treat AI as a model problem will keep joining the 95% that fail to reach production. The organizations that treat it as an information infrastructure problem will keep compounding advantages that no amount of prompt optimization can replicate.

Context is not a feature. It is the product.