Agent tool context is a product budget, not a prompt dump

An AI agent can become less useful as its capabilities grow. Add another connector, another schema and another page of tool instructions, and the agent may start more slowly, choose the wrong action more often, or run out of usable context before it reaches the user’s goal. The team then treats the symptom as a prompt problem even though the real issue is product architecture: every tool definition and every result competes for a finite attention, latency and cost budget.

Tool context should therefore be managed as a versioned product policy. The policy decides which capabilities are always visible, which are discovered only when needed, which intermediate results may be compacted, and which evidence must survive until the product outcome is verified. The objective is not the smallest prompt. It is the fastest safe path to a successful user task.

Start with the user outcome, not a token ceiling

A budget such as “keep the request below 80,000 tokens” is an implementation limit, not a product decision. Begin with the journey: a merchant asks an assistant to explain why an order is blocked and, if appropriate, prepare a safe corrective action. Define the useful end state, the maximum acceptable wait, the actions that require approval, and the evidence needed to confirm that the correction worked.

From that contract, derive technical guardrails: time to first useful state, total time to verified outcome, tool-discovery delay, successful-task rate, cost per successful task, unauthorized-action rejection, and the share of sessions that need a retry or human escalation. A context change is good only if these product measures improve without weakening safety or correctness.

This framing prevents a common optimization error. Removing tool descriptions may reduce input tokens but increase wrong-tool calls. Loading every integration may reduce discovery misses but inflate latency and distract the model. The correct budget is a portfolio of trade-offs, not one global number.

Inventory the five consumers of context

Measure context by source instead of reporting one total. At minimum, separate stable instructions, tool definitions, conversation history, retrieved evidence and tool results. They behave differently.

Stable instructions can often be cached. Tool definitions may be deferred. Conversation history may need a durable summary plus recent turns. Retrieved evidence should preserve citations and access decisions. Tool results often contain the most waste: large payloads that were necessary for one step but no longer deserve to travel through every later step.

OpenAI’s function-calling guide states that function definitions are injected into model context, count against the context limit and are billed as input tokens. Anthropic’s tool-context guide makes the same operational distinction between definitions and accumulated tool-result blocks. That means a catalog of 100 tools is not free merely because the user invokes one.

Record the rendered token count for each source class before and after a run. Anthropic’s token-counting endpoint accepts messages, tools, images and PDFs, so it can estimate the request before generation. For any provider, also record the usage returned by the real call because counting and caching are not identical operations.

Split the catalog into three lanes

Keep a small core of tools always available when they are common, low-risk and cheap to describe. Examples might include read-only order lookup, policy search and a request-for-clarification tool. Their names and schemas should remain stable across turns.

Put the long tail behind tool search. OpenAI’s tool-search documentation describes deferred definitions that load only when needed, while Anthropic’s tool reference exposes defer_loading so a definition stays out of the initial prompt until discovery. This reduces the baseline context, but it introduces a product risk: discovery can miss a relevant capability. Measure that miss rate and provide a safe fallback such as clarification, broader search or human handoff.

Use programmatic or batched execution for repetitive tool chains when the intermediate payload is not useful to the model. Anthropic’s guide explains that programmatic tool calling can keep intermediate results out of conversation history. The design principle is provider-independent: run filtering, joining or pagination in code, then return the smallest verified result that supports the next decision.

Do not hide authorization inside discovery. Eligibility should be computed by the application before a tool becomes callable. A search result may say a refund tool exists; it must not imply that this user, tenant or order is allowed to use it.

Make the policy explicit and versioned

A context policy should be reviewable like a feature configuration. This hypothetical example shows the shape, not universal thresholds:

{
  "policy_version": "agent-tools-2026-11-v3",
  "outcome": "resolve-order-block",
  "core_tools": ["get_order", "search_policy", "ask_user"],
  "deferred_domains": ["shipping", "payments", "returns"],
  "result_rules": {
    "max_inline_bytes": 12000,
    "preserve": ["record_id", "version", "decision", "source_url"],
    "externalize": ["raw_html", "full_history", "binary_attachment"]
  },
  "stop_rules": {
    "max_tool_rounds": 8,
    "require_approval_for": ["refund", "cancel_order"]
  }
}

Store the policy version with every run. When a failure appears, the team must be able to distinguish a model regression from a catalog change, a schema expansion, an aggressive result-trimming rule or a tool-search miss. Without that identity, “the agent got worse” is not diagnosable.

Preserve evidence while trimming payloads

Context editing is not deletion without consequences. Keep the facts that support the final answer and the state needed to recover: stable record IDs, source versions, timestamps, authorization decisions, tool-call outcomes and links to immutable artifacts. Move bulky payloads to an evidence store and return a compact, typed projection.

For example, a shipment lookup may return hundreds of tracking events. The next model turn may need only current status, latest carrier event, exception code, source timestamp and an evidence reference. If the user asks why the status changed, the runtime can retrieve the referenced event range instead of carrying the complete history through every turn.

Summaries must declare what they omit. A free-form sentence such as “shipment delayed” can erase whether the carrier or merchant caused the delay. Prefer a schema with source, observed state, confidence, unresolved conflict and retrieval handle. When two systems disagree, keep both claims until an authoritative reconciliation step resolves them.

Design caching around stable prefixes

Caching reduces repeated processing cost and latency; it does not make an oversized context smaller. OpenAI’s prompt-caching guide explains that the rendered prefix includes instructions, tool definitions and conversation history, and that changes to names, descriptions, schemas or ordering can invalidate reuse. Anthropic’s tool-caching guide similarly places tool definitions ahead of the system prompt and messages, while deferred tools are appended later so the prefix remains reusable.

Keep core definitions stable, put dynamic user and task data after the cached prefix, and append newly discovered tools rather than rebuilding the head of the request. Track cached input tokens, cache writes, realized read reuse and the resulting cost. A high nominal cache-hit rate can still be uneconomic if the product writes large prefixes that are rarely read again.

Treat cache keys as an isolation boundary as well as a performance hint. Do not create shared keys that make cross-tenant behavior hard to explain or audit. The application should be able to account for each workspace’s cached usage without exposing private content in telemetry.

Instrument the entire decision path

Trace one user task across eligibility, tool discovery, model calls, tool execution, verification and final outcome. OpenTelemetry’s GenAI observability guidance models an agent span with child model-call and tool-execution spans. Use that hierarchy, but keep prompts and raw results out of attributes unless your privacy policy explicitly permits them.

Useful fields include policy version, catalog version, tool-search query class, discovered tool count, selected tool, schema version, input-token classes, cached tokens, result bytes before and after projection, tool latency, verification result and final outcome class. Use low-cardinality values; put sensitive payloads in protected evidence storage, not metric labels.

The product dashboard should answer: How often did a user reach a verified useful state? How long did it take? How many sessions failed because the tool was absent, discovered late, selected incorrectly, rejected by authorization, timed out or returned too much data? What was the token and monetary cost per success? These categories produce actionable engineering work.

Release context changes like product changes

Build offline cases that represent real journeys and edge conditions. Test both tool selection and the final state transition. A candidate policy that chooses the right tool but drops the evidence needed for verification has failed.

Run the new policy in shadow mode on recorded or mirrored tasks without allowing mutating tools. Compare discovered tools, chosen actions, latency, tokens and final proposed outcome with the active policy. Then canary it to a small eligible cohort behind a reversible routing rule. Keep the model, policy, catalog and result-projection versions bundled so rollback restores a configuration that was actually tested.

Define stop conditions before release: a rise in tool-search misses, authorization rejections, abandonment, cost per success, context-overflow errors or unresolved state conflicts. A cheaper request is not a win if users repeat it or support agents must repair the outcome.

Hypothetical example: a merchant support agent

Consider a hypothetical assistant that helps a merchant investigate a failed fulfillment. The core catalog includes order lookup, policy search and clarification. Shipping-carrier, inventory and payment tools are deferred. The agent first classifies the failure domain, discovers only that domain’s tools, and receives a compact projection from each call.

If the order status and carrier status disagree, the projection preserves both source versions and marks the conflict unresolved. The assistant may explain the evidence, but it cannot recommend cancellation until a reconciliation tool confirms the authoritative state. A cancellation remains approval-gated even if discovery returns the tool.

The team measures time to a verified explanation, correct-domain discovery, tool miss rate, user abandonment, repeated questions, cost per resolved case and the share of cases escalated because evidence conflicts. It tests a new context policy in shadow mode, then exposes it to a small merchant cohort. Rollback restores the previous catalog, summaries and routing rule together.

Treat context as a product surface

The practical Product Engineering lesson is simple: tool context is not a bag of schemas attached to a prompt. It is a control plane for capability, cost, latency, safety and recoverability. Keep a small stable core, discover the long tail, compress intermediate payloads without destroying evidence, design for cacheable prefixes, and measure the path to the user’s verified outcome.

The best context budget is not the one with the fewest tokens. It is the one that makes the correct safe action easy to select, cheap to execute, observable to debug and reversible to change.

Prepared by: Noor Yasser