Why AI gets it wrong: everything has to fit in one context window

The first time a chatbot I built confidently gave a wrong answer, I went looking for a bug in the model. There wasn't one. The document with the correct answer existed, it just wasn't in the prompt at the moment the model had to respond. The model didn't hallucinate out of stupidity. It answered the only question it could actually see, which was a slightly different question than the one the user asked.

That experience reframed how I think about almost every LLM failure. Most of them aren't reasoning failures. They're context failures, a consequence of one unavoidable constraint: everything the model uses has to fit inside a single, fixed window.

What a context window actually is

An LLM doesn't have memory the way a program has a database. It has a context window: a fixed budget of tokens covering the instructions, the conversation so far, any documents you've pasted in, and the space it needs to write its answer. That's the whole world. The model only "knows" what's inside that window for this one call. Nothing carries over to the next call unless you put it there yourself.

So when people ask "why did the AI get it wrong?", the honest answer is usually one of two things:

1. The information it needed wasn't in the window, or
2. The information was in the window, but buried among too much else.

Both are engineering problems, not model problems. And both get worse, not better, as you naively shove more text in.

Why stuffing everything in backfires

The intuitive fix ("just give the model everything, to be safe") is exactly the move that produces wrong output. Four failure modes show up again and again.

1. Truncation. The window has a hard limit. When you exceed it, something gets dropped, and it's rarely the part you'd choose. The model answers from what survived.
2. Lost in the middle. Even well within the limit, models attend unevenly across a long prompt. Facts at the very start and very end get used; facts buried in the middle get under-weighted. A correct sentence on line 400 of 800 can be effectively invisible.
3. Dilution and distraction. Irrelevant tokens aren't free. Padding the prompt with loosely related material measurably degrades reasoning: the model spends attention on noise and drifts toward whatever is most prominent rather than most relevant.
4. No memory. Because nothing persists between calls, a system that should "remember" the user from five minutes ago will cheerfully contradict itself unless you re-supply that history every single time, which costs tokens, which brings you back to problems 1-3.

The engineering that fixes it

Once you accept that the window is a budget, the work becomes obvious: get the right tokens in, keep the wrong ones out, and stop paying for the same tokens twice. Here's the toolkit I reach for.

1. Retrieval-augmented generation (RAG) and vector databases

Instead of pasting an entire knowledge base into the prompt, you store it as embeddings in a vector database (Milvus and Chroma are the two I use most), and at query time you retrieve only the handful of chunks that are semantically closest to the question. The model sees three relevant paragraphs, not three hundred irrelevant ones.

# Embed the user's question, then pull only the closest chunks
hits = vector_store.search(
    embed(question),
    top_k=5,           # a few relevant chunks, not the whole corpus
    filters={"doc_type": "policy"},
)
context = "\n\n".join(h.text for h in rerank(question, hits)[:3])
prompt = f"Answer using ONLY this context:\n{context}\n\nQ: {question}"

The details that make RAG actually work in production are less glamorous than the embedding step: chunking documents at sensible boundaries, hybrid search (combining semantic similarity with keyword matching so exact terms like error codes still land), and a re-ranking pass that reorders candidates by true relevance before any of them reach the prompt. Retrieval is where most "the AI is wrong" bugs are actually won or lost.

2. Memory and context management

For anything conversational, you need a memory strategy beyond "resend the entire transcript." The patterns that scale: summarize older turns into a compact running summary, keep recent turns verbatim, and push long-term facts into a vector store you can retrieve on demand: the same RAG machinery, pointed at the conversation's own history. The window then holds a tight, relevant slice instead of an ever-growing log.

3. Prompt caching

A lot of every prompt is stable: the system instructions, tool definitions, a fixed reference document. Re-sending and re-processing those identical tokens on every call is wasted work. Prompt caching lets the provider reuse the computation for an unchanged prefix, so you mostly pay full price only for the parts that actually change.

The practical rule that falls out of this: put the stable stuff first and the variable stuff last. Structure your prompt so the long, unchanging preamble forms a cacheable prefix and the user's specific question sits at the end.

4. Context and token optimisation

Even with retrieval, prompts accumulate cruft: verbose tool outputs, redundant boilerplate, stale turns. Trimming and compressing them (truncating noisy tool results, deduplicating, structuring context instead of dumping raw text) keeps the signal-to-noise ratio high. This is the single highest-leverage habit I've built, and it pays off in both accuracy and cost.

5. Agentic retrieval

The most robust pattern is to stop pre-loading context at all and let the model fetch what it needs, when it needs it. Give it tools (search, lookup, read-file) and let it pull information across several steps instead of guessing what to cram in up front. The window stays lean because it only ever holds what the current step requires.

The takeaway

When an LLM gets something wrong, the reflex is to blame the model or reach for a bigger one. Far more often, the fix is upstream of the model entirely: decide what deserves a place in the window, retrieve it precisely, compress what's left, and cache what never changes. The context window is a budget. Good AI engineering is mostly the discipline of deciding what earns a seat, and being ruthless about everything that doesn't.