Context trimming: Save 9–47% on LLM API Costs
Does context trimming save money on LLM API costs?
Yes, for the right workload shape: 9–47% based on Each workload's trimmable input share (cacheableInputPct) times its input's share of total list cost at averaged priced-model rates, across all 8 workload shapes. It does not apply universally — see the applicability and tradeoff sections below before using it.
How it works
Input tokens are billed whether or not they change the answer — a retrieved chunk the model never uses, or a conversation turn from ten messages ago, costs exactly as much per token as the sentence that actually mattered. Trimming means shrinking what you send: dropping low-relevance retrieved chunks before they hit the prompt, summarizing older conversation turns instead of resending them verbatim, or capping document context to the sections a retrieval step actually ranked high. None of it touches output cost — it only reduces the input side of the bill, and it reduces it in direct proportion to the input's share of the total per-call cost.
Where it applies
Input-heavy shapes: RAG, long-document workloads, and chatbots with deep conversation history. Does nothing for short-input, long-output jobs like content generation.
Trimmable input share by workload
| Workload | Shape | Stable/trimmable input |
|---|---|---|
| LLM Chatbot | growing conversation history in, short reply out | 35% |
| RAG Question Answering | large retrieved context in, short answer out | 50% |
| Coding Agent | large file context in, large diff out | 60% |
| Document Extraction | medium document in, tiny structured JSON out | 15% |
| Long-Document Summarization | very large document in, medium summary out | 10% |
| Content Generation | tiny brief in, long piece out | — |
| Classification at Volume | tiny input in, single-label output out | 40% |
| Agentic Tool Loop | many small round-trips per completed task | 70% |
The tradeoff
Trimming context is an accuracy risk disguised as a cost optimization: the tokens cheapest to cut (older conversation turns, lower-ranked retrieved chunks) are sometimes the ones a specific answer actually depends on, and you only find out when the answer degrades. There is no dataset that tells you which tokens are safe to drop for your prompt — that has to be evaluated per use case, ideally against a held-out accuracy check, not assumed from a global percentage. Trimming also interacts badly with prompt caching: an unstable, per-request-trimmed prefix is not cacheable, so a workload that both trims and caches can end up paying the cache-write cost on every call instead of once.
Worked example
For rag question answering (large retrieved context in, short answer out) at 60,000 calls/month, Gemini 2.5 Flash Lite runs $81.60/month at list price and $81.60/month once verbosity is priced in. See the full ranked table for this workload →