Prompting, retrieval, tools, agents and MCP

How models become applications, and where trust breaks

Chapter 19 of 21 · VI. Systems · about 40 minutes. Builds on: 17. Post-training: instruction tuning, preferences and reinforcement learning.

Skip this chapter if

you can explain in-context learning and chain of thought, how retrieval-augmented generation works end to end, how tool calling and agent loops are built, what MCP is, and where untrusted text enters each of these.

A trained model is a component, not a product

Applications wrap the model in a program that builds the prompt, calls the model, and acts on the output. Everything in this chapter happens at inference time with fixed weights.

In-context learning and prompting

Given a few examples of a task in the prompt, a large model performs the task on a new input without any weight update. GPT-3 made this “few-shot learning” visible. A mechanistic partial explanation: induction heads (Chapter 6) copy patterns seen earlier in the context, and they emerge at the same point in training where in-context learning improves. Practical consequences:

• The prompt is a program. Instructions, examples and formatting all steer the model, and the model cannot distinguish “your” instructions from instructions embedded in data it reads.
• Chain of thought: asking for or demonstrating intermediate steps improves multi-step reasoning in large models . Reasoning models now generate such steps by default (Chapter 17).
• Context budget: the window is finite, and models use it unevenly; information in the middle of a very long prompt is used less reliably .

Retrieval-augmented generation (RAG)

Models do not know your private documents and their knowledge has a cutoff. RAG fetches relevant text at query time and places it in the prompt.

1. Index: split documents into chunks; convert each chunk to a vector with an embedding model (a network trained so that texts with similar meaning get nearby vectors); store them in a vector index.
2. Retrieve: embed the question; find the $k$ chunks with the highest cosine similarity, $\cos(u,v)=\dfrac{u\cdot v}{\lVert u\rVert\,\lVert v\rVert}$ (approximate nearest-neighbor search makes this fast at scale).
3. Generate: build a prompt containing the question and the retrieved chunks; the model answers from them.

The demo uses an older, simpler vector (TF-IDF word weights) but the pipeline is the same. The corpus contains one poisoned document; see what happens when it is retrieved.

Failure modes: irrelevant or missing retrieval, chunk boundaries that cut context, conflicting sources, stale data, access-control mismatches (retrieval returns a document the asking user is not cleared to see), and instruction text hiding in retrieved documents.

Tool use

A model can be trained to emit a structured request, such as a JSON object naming a function and its arguments, instead of prose. The application code, not the model, executes the call and appends the result to the conversation. Research showed that models can learn when and how to call tools from examples , and that interleaving reasoning steps with actions improves problem solving (the ReAct pattern). gpt-oss’s harmony format has a dedicated channel and role for tool calls and tool outputs .

Agents

An agent is a loop around a model with tools:

messages = [system_prompt, user_request]
while True:
    reply = model(messages)            # may contain tool calls
    if no tool calls: return reply
    for call in reply.tool_calls:
        result = execute(call)         # YOUR code runs it, with YOUR credentials
        messages.append(result)        # model reads the result next turn

Capabilities (browse, run code, edit files, send mail) come entirely from the tools wired into the loop; so do the risks. The model decides which tools to call based on all the text in the context, including tool outputs.

The Model Context Protocol (MCP)

MCP is an open protocol for connecting AI applications to tools and data without custom integrations for each. An application (the client) connects to servers that expose tools (actions), resources (data) and prompts (templates), exchanging JSON-RPC messages over a local process pipe or HTTP. Two properties matter for security: tool names and descriptions are fed to the model as text (so a description is itself a prompt), and the protocol leaves authentication, identity binding and manifest signing largely to implementers. NSA’s May 2026 information sheet on MCP addresses these gaps. It recommends treating components as separate trust zones aligned with data classification, preferring local servers for private data, validating every tool call’s parameters against a schema, sandboxing tool execution at the operating-system level with least privilege, signing messages with expiry and replay protection, treating tool output as untrusted input, and logging every invocation (see the security guidance reference).

Where untrusted text enters

Component	Untrusted inputs
Prompting	user message, pasted documents, uploaded files
RAG	every indexed document; web pages; emails; wiki pages anyone can edit
Tools	tool results (search snippets, file contents, API responses, error messages)
Agents	everything above, plus other agents’ messages and memory from earlier sessions
MCP	tool descriptions and manifests, server responses, resource contents

All of these arrive in the same token stream. named the resulting attack indirect prompt injection: an attacker who cannot talk to the model plants instructions where the model will later read them. The most useful heuristic for reviewing an agent is the “lethal trifecta” (Simon Willison): private data access + exposure to untrusted content + a way to communicate externally. If all three are present, data theft is possible; remove one.

Check yourself

Explain why a RAG system can leak documents across users even with a perfect model.

Retrieval is by similarity, not permission. If the index mixes documents of different clearances and the retriever does not filter by the asking user’s rights, restricted text enters the prompt and the model will use it. Access control must be enforced at retrieval time.

In the agent loop, who holds the credentials used by execute(call)?

The application. The model only produces text; if a hijacked model requests an action, the application’s credentials perform it, so tools should be scoped to the minimum privilege per action.

Why is a tool description a prompt-injection surface?

Descriptions are inserted into the model’s context as text. A malicious or compromised server can write instructions into its descriptions, and the model may follow them.

Security link

This chapter is where prompt injection lives. Chapter 20 covers the defenses, including privilege-separated architectures.

Papers

• Few-shot learning at scale.
• A mechanism behind in-context learning.
• Chain of thought.
• RAG.
• , Reasoning with tools.
• Using long contexts.
• Indirect prompt injection in real applications.