AI agents often face a significant operational hurdle: for every step they take—whether calling a tool or processing a result—they must re-read their entire history, including instructions and tool definitions. This repetitive process, akin to a constant “Groundhog Day” loop, increases latency and drives up costs. AWS has released technical guidance on using prompt caching with Amazon Bedrock to break this cycle and improve agent performance.
How prompt caching functions
Large language models (LLMs) typically process input tokens by performing a “prefill” operation, which builds the model’s working state, or key-value (KV) vectors. Prompt caching allows developers to save this state after the prefill phase. On subsequent requests that utilize the same input, the service loads the saved state rather than recomputing it, significantly lowering the number of tokens that require processing.
Implementation strategies
Developers can utilize two primary modes of prompt caching on Amazon Bedrock to manage their agents:
- Implicit caching: The service automatically determines where to cache the reusable prefix as a conversation grows, providing a default approach for many agentic workflows.
- Explicit caching: Developers define specific breakpoints—using
prompt_cache_breakpoint—to mark shared content such as system instructions or core tool definitions. This ensures consistent reuse across different user sessions or helper subagents.
AWS recommends a combined approach where explicit markers establish a shared starting point, while implicit caching manages the evolving history of the interaction.
Business and operational impact
By effectively caching static components like tool lists and system instructions, organizations can achieve meaningful cost reductions. For instance, tests on the GPT-6 Astra model demonstrated input cost savings of approximately 60% compared to non-cached workloads. To maximize these benefits, AWS advises keeping cache keys stable for groups of related requests and ensuring that any dynamic information—such as user-specific data or request IDs—is placed after the cached portion of the prompt.
As agents scale and delegate tasks to subagents, the use of a “warm” shared prefix can prevent redundant processing, allowing multiple helper agents to operate more efficiently. Developers are encouraged to monitor cache performance using Amazon CloudWatch metrics, such as CacheReadInputTokenCount and CacheWriteInputTokenCount, to track the balance between cache write costs and overall input savings.
Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/groundhog-day-for-ai-agents-prompt-caching-patterns-for-using-openai-models-on-amazon-bedrock/
Leave a Reply