Context engineering: the discipline that replaced prompt engineering

Context engineering is the name the field settled on once it became clear that prompts were not the hard part. For about two years, the folk wisdom of working with language models was that the prompt was the artefact. People collected prompts, traded them, wrote guides about phrasing.

Then production systems arrived and the picture changed. The wording of the instruction turned out to account for a small share of the quality variance. What dominated was whether the necessary information was in the window at all, and whether it was findable among everything else in there.

That shift now has a name.

What the discipline covers

Context engineering is everything that determines the model's input on a given request:

• Instructions — the system prompt, the persistent behavioural rules
• Retrieved knowledge — documents, records, code, whatever the task needs
• History — prior turns, and how much of them survives
• Tool definitions — every available tool's name, description and schema
• Intermediate state — results of earlier tool calls, plans, scratchpad content

Each of these competes for the same finite budget, and each has a cost in latency, money and — the one people miss — accuracy.

Position bias is a measured property, not folklore

The claim that "where you put things in the window matters" has a research basis worth knowing, because it tells you how to arrange context rather than just that you should.

Liu et al. established the canonical result in Lost in the Middle: How Language Models Use Long Contexts, published in TACL: on multi-document question answering and key-value retrieval, model performance follows a U-shaped curve against the position of the relevant information. Accuracy is highest when the needed passage sits at the very start or the very end of the context and degrades measurably when it sits in the middle — in some configurations, performance with the relevant document mid-context fell below performance with no retrieved documents at all (TACL 2024 ↗).

Subsequent work attributes this to an intrinsic attention bias toward extreme positions rather than to anything semantic, and has produced calibration methods that partially correct it (arXiv:2406.16008 ↗). More recent work in 2026 develops a structural account of why transformers exhibit the bias (arXiv:2602.16837 ↗) and shows that correcting position bias alone is not sufficient to fix long-context retrieval behaviour (arXiv:2606.27793 ↗).

Three practical consequences follow directly:

1. Rank properly and put the best material at the edges. If your retriever returns twenty passages in similarity order and you concatenate them, your best-but-not-first result is landing in the dead zone.
2. Fewer passages is not just cheaper — it is more accurate. Shortening the context moves everything closer to an edge.
3. A long-context model does not make retrieval quality optional. A large window changes what fits, not how attention is distributed across it.

Retrieval is the largest lever

For most systems, retrieval quality explains more of the output quality than any other single factor. If the answer is not in the window, no amount of instruction will produce it; if it is buried among nineteen near-misses, it often may as well not be.

The parts that matter in practice:

Chunking. Splitting by fixed token count is the default and usually the worst option. Split on structure — sections, functions, records — so a chunk is a self-contained thought rather than an arbitrary window.

Ranking, not just similarity. Vector similarity gets you candidates. A reranking pass over those candidates is consistently one of the highest-return additions to a retrieval pipeline, because "similar" and "relevant to this question" are different properties.

Hybrid search. Semantic search is weak on exact identifiers — error codes, part numbers, function names, proper nouns. Keyword search is weak on paraphrase. Almost every real system wants both.

Freshness. Stale retrieved content is worse than missing content, because it is confidently wrong. Re-indexing policy is part of the design, not an operational afterthought.

Fewer, better chunks. Returning five good passages beats twenty mediocre ones on accuracy, cost and latency simultaneously. This is the rare change with no trade-off.

Compaction: what to do with history

Any long-running session eventually exceeds its budget. The strategy you pick for that determines whether the system degrades gracefully or gets strange.

The failure to avoid is naive truncation on a session where the important constraint was stated in turn two. The model does not know it has forgotten; it simply behaves as though the rule never existed.

Tool definitions are context too

An easy one to miss: every tool available to the model occupies context on every single request, whether it is used or not.

A server exposing forty tools is spending a meaningful slice of the window on descriptions before any work happens, and giving the model forty options to choose wrongly between. Fewer, better-scoped tools consistently outperform exhaustive coverage — which is the same conclusion the MCP ecosystem has been converging on independently.

Agent memory: what to keep outside the window

Compaction answers what to do when a session outgrows its budget. Agent memory answers a harder question: what a system should still know next week.

The distinction matters because they are solved differently. Compaction is a summarisation problem inside one session. Memory is a storage and retrieval problem across many, and treating it as the former is why long-running agents feel amnesiac in some ways and cluttered in others.

Three layers are worth separating explicitly:

Working memory — the current window. Volatile, expensive, and the only thing the model can actually attend to. Everything else exists to decide what lands here.

Episodic memory — what happened before. Previous sessions, decisions taken, corrections the user made. The failure mode is storing raw transcripts and retrieving them by similarity: you get plausible-looking fragments of old conversations with no indication of whether they are still true.

Semantic memory — durable facts about the user, the account, the domain. "This customer is on the enterprise plan." "We decided against Postgres in March." These want to be structured records, not prose, because they need to be updated and invalidated rather than accumulated.

The practical guidance that follows:

• Write memory deliberately, not automatically. Systems that persist everything a user says accumulate contradictions. Something has to decide what is worth keeping, and "everything" is not a policy.
• Give facts an expiry or an owner. A stored preference from four months ago may be wrong. Without a way to invalidate it, the system confidently applies stale information — the same failure as stale retrieval, with a longer half-life.
• Retrieve memory the same way you retrieve documents. Rank it, budget it, and be willing to leave it out. Memory that always loads is just a longer system prompt with worse hygiene.
• Make it inspectable. When an agent behaves oddly, "what did it think it knew" is the first question. A memory store nobody can read is a debugging dead end.

Debugging a context problem

Most reports of "the model is bad" are context problems in disguise, and they resolve in a fairly reliable order. When quality is poor, work through this before touching the model or the prompt:

1. Was the necessary information in the window at all? Log the assembled context, not just the prompt and completion. Roughly half of quality complaints stop here.
2. If it was, where was it? Buried at position fourteen of twenty retrieved chunks is functionally close to absent. Re-rank and re-test before concluding anything about the model.
3. How much irrelevant material was alongside it? Ablate: remove a source, re-run the evaluation set, see whether quality moves. If it does not, you were paying for tokens that did nothing.
4. Is the instruction contradicted anywhere? A system prompt saying one thing and a retrieved document implying another produces behaviour that looks random and is not.
5. Only then, change the prompt or the model. These are the two levers people reach for first and the two that least often explain the problem.

Trust boundaries are structural

Everything in the window looks the same to the model. Your carefully written system instruction and a sentence inside a retrieved document have no inherent difference in authority.

This is why prompt injection is not fixable by asking the model to ignore instructions in retrieved content. Beurer-Kellner et al. make the argument precisely in Design Patterns for Securing LLM Agents against Prompt Injections ↗: their organising principle is that once an agent has ingested untrusted input, it must be constrained so that input cannot trigger any consequential action. The security property comes from the constraint on the action space, not from anything the model was told. Every pattern in that paper trades some agent generality for a provable limit on what an injection can achieve.

The mitigation is therefore structural:

• Label untrusted material explicitly as data, and keep it separate from instruction
• Give the model only the authority the task requires — not the union of everything it might need
• Gate consequential actions behind approval rather than trusting the model's judgement about when to take them
• Log what went into context, not just what came out, so an incident is diagnosable

This is covered in more depth in prompt injection and LLM security.

How to work on it

Treat context as a budget and instrument it like one.

1. Measure composition. For a typical request, how many tokens go to instructions, tools, history and retrieval? The answer is often surprising, and frequently indefensible.
2. Ablate. Remove a source and re-run your evaluation set. If quality does not drop, you were paying for nothing.
3. Evaluate retrieval on its own. Before end-to-end quality, measure whether the right document is in the top five. Most "the model is bad" reports are retrieval failures wearing a disguise.
4. Set a ceiling and hold it. Without one, context grows monotonically because every individual addition seems reasonable.
5. Test long sessions specifically. Quality at turn one tells you nothing about quality at turn forty, which is where users actually live.

Budgeting context as a cost line

Because every token in the window is paid for on every request, context composition is a cost decision as much as a quality one — and the two usually point the same way, which is unusually convenient.

A useful exercise: take a typical request and account for where the tokens went.

Most teams doing this for the first time find one component consuming a share nobody would defend if asked directly — commonly tool definitions, or retrieval returning far more than the model can use.

The good news is that trimming usually improves accuracy and latency at the same time as cost, because everything you remove moves the remaining material closer to the edges of the window where attention is strongest. This is one of the few places in engineering where the cheap option and the good option coincide, and it is worth exploiting deliberately rather than stumbling into.

Where to start if none of this exists yet

For a team with a working AI feature and no context discipline, the first three moves in order:

Log the assembled context. Not the prompt template — the actual bytes that went to the model, per request, with the retrieval scores attached. Almost nothing else is diagnosable without this, and most teams do not have it.

Build a small evaluation set. Thirty realistic inputs with acceptable outputs is enough to convert prompt changes from argument into measurement.

Measure retrieval precision separately. Ask only whether the necessary document made the top five. Fix that before touching anything else.

Everything further — compaction policy, tool surface pruning, memory design — is easier once those three exist, and mostly guesswork before.

Why the rename matters

It is easy to dismiss the terminology shift as fashion. It is not. "Prompt engineering" framed the work as writing — a thing one person does in a text box, iteratively, by feel. "Context engineering" frames it as systems design: a pipeline with retrieval, ranking, budgets, eviction policies, trust boundaries and observability.

The second framing is correct, and teams that adopt it stop trying to fix retrieval problems by rewording instructions.