Two years of prompt tinkering taught most enterprise teams the same lesson: a better-worded instruction does not fix a system that hands the model the wrong twelve paragraphs. The failures that survive into production are context failures—stale policy documents, a retrieved page the user was never cleared to read, a summary of last week's conversation that quietly contradicts today's record.
Context engineering is the response. It treats the context window as a governed, budgeted resource and the assembly of that window as a first-class part of your architecture: versioned, permission-aware, and measured. Anthropic's engineering team frames it as curating the smallest high-signal token set that reliably produces the desired outcome, which is a useful working definition—see their write-up on effective context engineering for AI agents.
What actually changed
Prompt engineering optimises one string. Context engineering optimises a pipeline with several independent decisions, each of which can be wrong on its own:
- Selection: which sources are eligible, and which chunks win the ranking for this question.
- Entitlement: whether this specific user is permitted to see each candidate chunk, evaluated per request rather than per corpus.
- Compression: what gets summarised, truncated, or dropped when the budget is tight, and what must never be dropped.
- Recency: which version of a document is authoritative, and how quickly a correction propagates.
- Attribution: whether the answer can point back to the exact source a reviewer can open.
Long context windows did not remove these decisions. They made sloppy ones more expensive, because you can now stuff a hundred thousand tokens of mostly irrelevant material into a request and pay for it on every call while accuracy quietly degrades.
A reference architecture for governed context
The pattern we deploy has six layers. It is deliberately boring, because boring layers are the ones you can audit.
1. Source registry
An explicit list of systems the assistant may read, each with an owner, a refresh cadence, a sensitivity class, and a retention rule. If a source is not in the registry, it cannot enter the pipeline. This single artefact answers most of what an auditor will ask.
2. Ingestion and normalisation
Documents are parsed, chunked along semantic boundaries, and enriched with metadata that matters later: source system, document version, effective date, owning team, and the access-control identifiers copied from the source of truth. Metadata is not decoration; it is the substrate for every filter downstream.
3. Retrieval
Hybrid search—lexical plus vector—with metadata filters applied before ranking, then a reranking pass on the shortlist. Both Google Cloud's guidance on retrieval-augmented generation and Microsoft's Azure AI Search RAG overview converge on the same shape: retrieval quality, not model choice, dominates answer quality.
4. Memory
Two tiers, kept separate. Short-term working memory holds the current task state and is discarded when the task closes. Long-term memory stores durable, explicitly written facts—preferences, entitlements, account context—with a source and a timestamp on each entry. Nothing enters long-term memory implicitly.
5. Assembly and budget
A deterministic assembler builds the request: system instructions, tool schemas, retrieved evidence with citations, then memory, within a fixed token budget and a documented drop order. Because the assembler is code rather than a prompt, you can unit-test it and diff its output between releases.
6. Tool boundary
Actions leave through a single mediated gateway that authenticates the end user, checks scope, and logs the call. This is where MCP belongs.
Permissions and provenance are one problem
The most damaging incidents we are asked to review are rarely hallucinations. They are correct answers delivered to the wrong person—a salary band, an unannounced acquisition, a customer's contract terms—because retrieval ran with a service account's privileges instead of the user's.
Two rules prevent nearly all of it. First, filter at query time using the caller's identity, and treat the index as untrusted for authorisation decisions it did not derive from the source system. Second, make provenance mandatory: every claim in an answer carries a link to the retrieved chunk, and the chunk carries its version and effective date. Citations are usually sold as a trust feature for users; their real operational value is that they turn a vague complaint into a reproducible ticket.
Retrieved content is also an injection surface. A document, ticket comment, or web page pulled into context can carry instructions aimed at your model, which is why the OWASP Top 10 for LLM Applications ranks prompt injection first. Treat all retrieved text as data, never as instructions, and keep the authority to act in the gateway rather than in the model's reasoning.
Where MCP fits, and where it stops
Model Context Protocol standardises how a model discovers and calls tools and data sources, which is genuinely valuable: one connector serves many clients, and one authorisation model covers many tools. The MCP specification is explicit that servers must not accept tokens they were not issued and must verify every request—guidance worth reading before you expose an internal server.
What MCP does not do is decide relevance, redact a field, enforce a token budget, or tell you whether an answer was grounded. Those remain yours. The practical boundary we recommend: MCP for transport, discovery, authentication, and audit; your own layer for retrieval, ranking, assembly, and evaluation. Teams that blur this line end up with tool sprawl and no single place to explain why the model saw what it saw.
Evaluation: you cannot govern what you do not measure
Context pipelines drift. Documents change, an index rebuild silently drops metadata, a reranker upgrade reshuffles results. Continuous evaluation is what separates a system you can defend from a demo that used to work.
- Retrieval metrics first: recall and precision at k against a labelled question set, reported per source. Most quality regressions are visible here before they reach the answer.
- Groundedness: the share of claims traceable to a retrieved chunk, sampled and human-reviewed weekly.
- Entitlement tests: adversarial cases where a low-privilege user asks for restricted content. These must fail closed, and they belong in CI.
- Context efficiency: tokens per resolved task, tracked alongside accuracy so you notice when you are buying a point of quality with triple the spend.
- Task outcomes: resolution rate, draft acceptance, escalation rate—the numbers a business owner will actually renew the budget on.
For governance framing that maps cleanly onto these controls, the NIST AI Risk Management Framework is a reasonable spine: it pushes you to document intended use, measure, and monitor rather than assert safety once at launch.
A pre-production checklist
- Every source is in the registry with a named owner, sensitivity class, and refresh cadence.
- Access filters run with the end user's identity, verified by tests that must fail closed.
- Chunks carry version and effective date; superseded content is removed or demoted on a known schedule.
- Context assembly is deterministic code with a fixed budget and a documented drop order.
- Answers cite sources a reviewer can open in one click.
- Retrieved text is never treated as instructions; tool authority lives in the gateway.
- Write and destructive actions require human approval; reads and drafts may be automated.
- Long-term memory entries are explicit, attributed, expiring, and user-inspectable.
- A labelled evaluation set runs on every change to prompts, retrieval, or index configuration.
- Tool calls and context assembly decisions are logged with enough detail to reconstruct a single answer.
When not to over-engineer
Not every use case earns this machinery. Over-building is the second most common failure we see, right behind ignoring permissions.
If your corpus is a few hundred stable documents, start with keyword search, a small curated set, and citations. If one team owns all the content and everyone has the same access, skip per-user filtering until the second team arrives. If the assistant only drafts text a human always reviews before it leaves the building, you do not need an approval workflow on top of the human. Graph retrieval, multi-agent orchestration, and learned rerankers are answers to measured problems; adopt them when your evaluation set names the problem, not because a conference talk did.
The sequence that works: ship a narrow, well-governed pipeline over one high-value corpus, instrument retrieval and outcomes from day one, then add sophistication only where the numbers show it pays. Governance first is cheaper than governance retrofitted—not because it is virtuous, but because unpicking an ungoverned index after an incident costs more than building the registry did.
Frequently asked questions
Is context engineering just prompt engineering with extra steps?
No. Prompt engineering optimises the wording of a single request. Context engineering designs the system that decides which facts, tools, and history reach the model on every request, who is allowed to see them, and how that selection is measured and audited over time. The prompt becomes one small, mostly stable component inside a pipeline you can version, test, and govern.
Do we need a vector database before we can start?
Usually not. Most first releases do better with keyword or hybrid search over a small, well-governed corpus, because early retrieval quality is limited by document hygiene and permission metadata rather than by embedding sophistication. Add vector or hybrid retrieval once your evaluation set shows that lexical search is missing paraphrased or conceptual questions.
How does Model Context Protocol fit into a context pipeline?
MCP standardises how a model reaches tools and data sources, so it removes bespoke connector work and gives you one place to enforce authorisation and audit tool calls. It does not decide relevance, redaction, or token budget for you. Treat MCP as the transport and permission boundary, and keep retrieval, ranking, and context assembly as your own governed layer.
We help teams design and instrument governed context pipelines—source registries, hybrid retrieval, entitlement testing, and evaluation harnesses—for assistants and agents already in production. If you are debugging accuracy, cost, or an access-control surprise, get in touch.