Build Hour: Prompt Caching

Key Concepts

• Prompt Caching: A compute reuse mechanism that significantly reduces latency and costs by skipping re-processing of identical prompt prefixes.
• Cache Hit Rate: The percentage of requests that match a previously cached prefix, directly impacting cost savings.
• Prefix Importance: The initial portion of a prompt (system prompt, images, audio, messages) is crucial for cache hits; dynamic elements should be placed at the end.
• No Intelligence Trade-off: Prompt caching does not negatively impact the quality or intelligence of model outputs.
• API Differences: Caching behavior differs between the Completions and Responses APIs, particularly with reasoning models due to hidden chain-of-thought tokens.

Prompt Caching: Reducing Latency and Costs with OpenAI APIs

This session, led by Christine and Erica from OpenAI, details prompt caching as a critical optimization strategy for AI applications. The discussion covers foundational concepts, practical implementation, and real-world examples, emphasizing the significant cost and latency benefits achievable with minimal effort.

Fundamentals of Prompt Caching

Prompt caching operates on the principle of “compute reuse.” When multiple requests share the same prefix – encompassing the system prompt, images, audio, or messages – OpenAI skips re-processing those tokens, resulting in substantial computational savings. Caching begins after processing 1024 tokens, storing blocks of 128 tokens. A contiguous prefix is required for a cache hit; the entire input must match a previously processed sequence in the exact same order. The cache is initially ephemeral (5-10 minutes) but can be extended to 24 hours using the prompt_cache_retention parameter.

The attention mechanism, central to the transformer model, is what’s actually cached – the floating-point numbers representing the results of attention calculations, not the tokens themselves. The API request flow involves hashing the prefix (first 256 tokens), routing to an engine, fetching cached data if available, and performing inference only on uncached portions, then updating the cache with the model’s output.

Cost and Latency Benefits

The discount on cached tokens varies by model: 50% for GPT-4, 75% for GPT-4 Turbo, and 90% for GPT-5. Caching dramatically reduces latency, especially for long prompts, making it proportional to the generated output length rather than the total conversation length. Testing showed a 67% faster time to first token for cached long prompts (2300 prompts, 1024-200,000 tokens). Increasing a prompt from 900 to 1024 tokens can yield a 33% saving in token costs with even a 50% cache hit rate.

Real-World Implementations & Optimization

Several examples demonstrate the effectiveness of prompt caching. Warp, an agentic development environment, more than doubled their cache hit rate by implementing a task-scoped prompt cache key, resulting in a 23% decrease in input token cost using flex processing with extended caching. Another coding customer increased their cache hit rate from 60% to 87% by utilizing a prompt cache key. Audio caching on the speech-to-speech model achieves almost a 99% discount on cached tokens.

Maximizing cache hit rates involves several strategies, including truncation, summarization/compaction, and careful management of context window size. Placing dynamic components at the end of the prefix is best practice. Experimentation is encouraged to determine the optimal approach, as the system is agnostic to whether dynamic content is placed in the system prompt or a new message.

Architectural Considerations & API Nuances

Prompt caching is considered a “no-brainer” optimization, offering significant benefits without impacting intelligence. However, balancing caching with context engineering is crucial. Over-optimizing for caching could potentially hinder effective context management.

There are differences in caching behavior between the Completions and Responses APIs. Reasoning models utilize hidden chain-of-thought tokens that are not persisted in completions, leading to lower cache hit rates. The underlying model parameters are deterministic; an identical prefix will always produce the same KV cache and, therefore, the same output. The only cost incurred is GPU usage during a cache miss.

Resources & Future Developments

OpenAI provides several resources to aid in prompt caching implementation, including Prompt Caching Documentation, the Prompt Caching 101 Cookbook (published last year), and the newly launched Prompt Caching 2011 Cookbook. Demo code from past build hours is also available in a code repository. The next build hour, scheduled for March 24th, will focus on agent capabilities, with all discussed resources distributed via email.

Conclusion

Prompt caching represents a powerful and readily available optimization technique for OpenAI API users. By leveraging compute reuse, developers can significantly reduce latency and costs without compromising the intelligence or quality of their AI applications. Careful consideration of prompt structure, API nuances, and context management are key to maximizing the benefits of this valuable feature.