Your Agent Is Wasting Tokens and You Don't Know It - Erik Hanchett, AWS

Key Concepts

• Token Costs: The primary expense metric in LLM usage, based on the volume of text processed.
• Agentic Workflows: Systems where an LLM uses tools to perform tasks autonomously.
• Context Window: The total amount of information (tokens) an LLM can process in a single interaction.
• Prompt Caching: Storing static parts of a prompt to avoid redundant processing costs.
• Model Routing: Dynamically selecting an LLM based on task complexity to optimize cost.
• Sliding Window: A technique to limit conversation history sent to the model.

---

1. Cache System and Tool Prompts

The most effective way to reduce costs is to avoid re-sending static instructions.

• Mechanism: By setting cache_prompt = default (using AWS Bedrock/Strands Agents syntax), the system sends the full system prompt only on the first call. Subsequent calls utilize the cached version, significantly reducing the token count per request.
• Scope: This can be applied not only to system prompts but also to tool definitions and recurring message structures.

2. Intelligent Model Routing

Avoid using "Frontier" (high-performance, high-cost) models for simple tasks.

• Strategy: Implement a routing logic (e.g., an if statement or a lightweight "router" model) to categorize tasks by difficulty.
• Application: Use high-end models like Claude 3.5 Sonnet for complex reasoning tasks, while routing simpler, repetitive tasks to cost-effective models like Claude Haiku.

3. Offloading and Summarizing Tool Results

Large outputs from tools can quickly exhaust the context window and inflate costs.

• Methodology: Instead of passing raw, large tool outputs back into the agent loop, store the data locally or in the cloud.
• Optimization: Use a summarization step to extract only the relevant information from the tool result. This ensures the LLM receives the "gist" of the data without the overhead of the full dataset in every iteration of the agent loop.

4. Capping Tool Loops

Uncontrolled agent loops are a primary cause of "runaway" token costs.

• Risk: Agents may enter infinite loops or perform redundant tool calls (10–20+ iterations) if not constrained.
• Solution: Always define a max_iterations parameter for agent loops.
• Observability: Use observability tools to monitor the frequency and duration of tool calls. This data helps identify inefficient tools that require optimization or stricter loop limits.

5. Trimming Conversation History

In multi-turn conversations, sending the entire history with every new message is inefficient.

• Framework: Utilize a Sliding Window Conversation Manager. This restricts the context sent to the LLM to only the most recent $N$ messages (e.g., the last 10).
• Trade-off Management: To prevent the loss of critical context from earlier in the conversation, implement a summarization layer. Once the conversation exceeds the sliding window, summarize the older history and inject that summary into the context window instead of the raw transcript.

---

Synthesis and Conclusion

Reducing token costs in agentic applications requires a shift from "sending everything" to "sending only what is necessary." By implementing caching for static prompts, routing tasks to appropriate models, summarizing large tool outputs, capping execution loops, and trimming conversation history, developers can maintain high-performance agents while significantly lowering operational expenses. The key takeaway is to treat the context window as a finite, expensive resource that must be managed through architectural constraints rather than brute-force data transmission.