LLM costs are easy to start and hard to control. A team ships a feature that calls GPT-4, the feature works, users like it, and the invoice climbs 15 percent month over month. The cost is not a problem until it is, and by then the architecture decisions that drive the cost are baked in.
This playbook covers twelve techniques that have produced measurable cost reductions in production systems. Each technique includes when it works, when it does not, and the trade-off you are accepting. Not all twelve apply to every system. Pick the ones that match your constraints.
Techniques 1-4: Reduce Token Volume
1. Compress your prompts. Audit every system prompt for redundancy. Teams add instructions over months of edge-case handling, and the prompt accumulates clauses that overlap or contradict. A prompt audit — rewriting from scratch with the current requirements, not patching the existing prompt — typically reduces prompt tokens by 20-40 percent. Measure before and after to confirm you have not changed output quality.
2. Trim retrieval context. RAG systems retrieve chunks and dump them into the prompt. If you retrieve five chunks but only two are relevant, you are paying for three chunks of wasted context on every call. Improve retrieval precision before increasing the number of retrieved chunks. A retrieval system that returns two relevant chunks outperforms one that returns five chunks with three noise chunks, and it costs less.
3. Use shorter output constraints. If your use case needs a classification, ask for a label, not a paragraph. If you need structured output, specify the schema and let the model fill it in. Unconstrained generation produces verbose output that costs more tokens and requires more post-processing. Set max_tokens to the actual maximum you need, not a generous default.
4. Cache repeated prefixes. If your system prompt or retrieval context is the same across many calls, use prompt caching. Most providers offer caching for repeated prefixes at a reduced rate. A system prompt of 1,000 tokens called 10,000 times per day costs 10 million input tokens at full price. With caching, the prefix is priced once and reused. The savings scale directly with call volume.
Techniques 5-8: Use Cheaper Models
5. Route simple queries to smaller models. Not every query needs the most capable model. Classification, extraction, and formatting tasks can use smaller, cheaper models. Build a router that classifies query complexity and sends easy queries to a small model and hard queries to a large model. The router itself can be a small model or a rule-based classifier. Even a simple heuristic — query length, keyword presence, or task type — captures most of the routing value.
6. Use distilled models for specific tasks. If you have a task that consistently uses one model, fine-tune a smaller model on that task. A well-distilled model on a narrow task can match the large model’s quality at a fraction of the cost. The prerequisite: you need enough task-specific training data and a clear quality metric to validate the distilled model.
7. Batch non-real-time requests. If your use case does not require real-time responses, batch requests and process them during off-peak hours. Many providers offer discounted batch pricing. Summarization, report generation, and data enrichment can all be batched. The trade-off is latency: batch processing adds hours of delay. For the right use cases, this is free money.
8. Switch providers for commodity tasks. Embedding, classification, and sentiment analysis are commodity capabilities. The cheapest provider that meets your quality threshold is the right provider. Do not use your premium model’s embedding endpoint when a dedicated embedding service costs one-tenth as much. Reserve premium models for tasks that require their capabilities.
Techniques 9-12: Architectural Savings
9. Implement semantic caching. Cache responses for semantically similar queries. If a user asks “What is our refund policy?” and later asks “How do I get a refund?”, the answer is the same. A semantic cache detects the similarity and returns the cached response without a model call. The cache hit rate depends on query distribution. For FAQ-style workloads, hit rates of 30-50 percent are achievable. For highly variable queries, the rate drops. Measure your actual hit rate before investing in the infrastructure.
10. Use streaming to reduce perceived cost. Streaming does not reduce actual cost, but it reduces the perceived cost of lower-quality responses. When users see partial results immediately, they are more tolerant of shorter responses. Shorter responses cost fewer output tokens. This is a UX optimization that has a cost side effect.
11. Implement graceful degradation. When the primary model is expensive or slow, have a fallback path. The fallback might be a cached response, a smaller model, or a rule-based answer. The system continues to function, albeit at lower quality, without paying the premium cost. This is not about cutting corners. It is about matching cost to the value of the interaction.
12. Kill zombie features. Audit every LLM-powered feature quarterly. If a feature has low usage, low impact, or has been superseded by a simpler approach, turn off the LLM calls. Zombie features — features that still work but few people use — consume tokens that could be allocated to high-value features. The audit takes a day. The savings are permanent.
Putting It Together
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
Start with token reduction — it has the lowest implementation cost and the most predictable savings. Move to model routing if token reduction is not enough. Architectural changes like semantic caching have higher implementation costs and should be pursued only after the cheaper techniques have been exhausted.
Next Step
Pick the two techniques that match your biggest cost driver. Implement them this sprint. Measure the cost change over two weeks. If savings exceed 15 percent, continue to the next two. If not, reassess which cost driver you identified correctly.
