LLM inference costs follow a pattern that catches teams off guard. The first prototype costs almost nothing — a few hundred dollars a month during development. The pilot scales to a few thousand. Production launches at ten to twenty thousand. Six months later, the bill is eighty thousand and growing faster than revenue. The model works. The unit economics do not.

Most cost optimization advice for LLMs focuses on picking a cheaper model. That is one lever, and it is often the wrong first lever. Switching from GPT-4 to a smaller model without evaluating the quality impact is how teams save 60% on inference and lose 30% of their customers. The right approach is a structured optimization process that reduces cost at every layer of the inference stack, starting with the layers that have the least quality impact.

This framework organizes the optimization levers into four layers, ordered from lowest risk to highest risk. Work through them in order. Each layer has a target cost reduction range based on what we typically see in production systems.

Prerequisites

You need per-request cost tracking. Not per-endpoint, not per-application — per-request. Each request should log the model used, the input token count, the output token count, and the total cost. If you cannot measure cost at the request level, you cannot optimize it.

You also need quality metrics tied to the same request data. When you optimize cost, you need to see whether quality moved. A cost reduction that degrades accuracy by 5% may or may not be acceptable — but you need to know the number to make the call.

Finally, you need your current cost breakdown by component: what percentage of your inference spend goes to input tokens, output tokens, and API call overhead. This breakdown tells you where to focus.

Layer 1: Input optimization (target: 15-30% reduction)

Input optimization reduces the number of tokens you send to the model. It has minimal quality impact because you are not changing the model or its behavior — you are just giving it less irrelevant information to process.

Prompt compression. Review your system prompts. Most production system prompts contain instructions that are redundant, overly verbose, or irrelevant to the current request. A system prompt that takes 800 tokens can usually be compressed to 300 tokens without losing instruction fidelity. The model does not need three paragraphs of context-setting before it gets to the task.

Audit every system prompt by removing sections one at a time and measuring whether output quality changes. You will find that 30-50% of most system prompts can be removed without any measurable quality impact.

Context window management. In RAG systems, the retrieved context is often the largest component of input tokens. Teams default to retrieving ten or twenty chunks when five would suffice. Measure retrieval precision — what percentage of retrieved chunks actually contribute to the final answer? If precision is below 50%, you are paying for tokens that do not help.

Reduce retrieval count and measure answer quality. Increase only if quality degrades. Add a re-ranking step that filters retrieved chunks before they reach the completion model. Re-ranking models are much cheaper than completion models, so the cost of re-ranking is offset by the savings from shorter prompts.

Caching. If your application sends identical or near-identical prompts to the model, cache the responses. Exact-match caching catches repeated queries. Semantic caching catches paraphrased queries that should produce the same response. Implement exact-match caching first — it is trivial and catches more hits than teams expect. Semantic caching requires a similarity threshold and adds latency, so implement it only after exact-match caching is stable.

Layer 2: Output optimization (target: 10-20% reduction)

Output optimization reduces the number of tokens the model generates. The quality impact depends on how aggressively you constrain output length.

Max token limits. Set explicit max token limits based on actual output length distributions. If 95% of your responses are under 300 tokens, set the limit to 400. Do not leave the default at 4,096. Models generate tokens until they hit the limit or a stop sequence. Every unnecessary token costs money.

Stop sequences. Define stop sequences that match your output format. If your responses end with a specific marker, tell the model to stop generating when it produces that marker. Without stop sequences, the model may continue generating past the useful content, producing filler that you discard and pay for.

Structured output. When you need JSON or another structured format, use the model’s structured output mode rather than asking for JSON in the prompt. Structured output modes enforce format compliance and typically generate fewer tokens because the model does not need to produce format-explaining text.

Layer 3: Model routing (target: 20-40% reduction)

Model routing sends different requests to different models based on complexity. This is the highest-impact optimization layer, but it introduces quality risk because you are making a judgment about which requests can tolerate a smaller model.

Complexity classification. Build a lightweight classifier that scores incoming requests by complexity. Simple factual queries, formatting tasks, and straightforward classifications can route to smaller, cheaper models. Complex reasoning, multi-step analysis, and creative generation route to larger models.

The classifier itself should be a small model or a rule-based system — do not use your most expensive model to decide which model to use. A fine-tuned small model or even a set of well-crafted rules can achieve 85-90% routing accuracy.

Tiered model strategy. Define three tiers:

• Tier 1 (cheapest): Simple tasks — classification, extraction, formatting. Use a small model or a fine-tuned model.
• Tier 2 (mid-range): Standard tasks — summarization, Q&A, standard reasoning. Use a mid-size model.
• Tier 3 (most expensive): Complex tasks — multi-step reasoning, code generation, nuanced analysis. Use your largest model.

Route requests based on the complexity classifier. Start conservatively: route only clearly simple requests to Tier 1. As you build confidence in the routing accuracy, expand Tier 1’s coverage.

Fallback chains. When a Tier 1 model produces an uncertain or low-quality response, fall back to Tier 2. This gives you the cost savings of Tier 1 for most requests while catching the cases where Tier 1 is insufficient. The fallback cost is incurred only for the minority of requests that need it.

Layer 4: Infrastructure optimization (target: 10-25% reduction)

Infrastructure optimization changes how you deploy and call models. It has moderate quality impact because it may change latency or availability characteristics.

Self-hosted inference. If your volume exceeds a break-even threshold (typically 50-100 million tokens per month for a mid-size model), self-hosting can reduce per-token costs by 40-60%. The tradeoff is operational complexity: you need GPU infrastructure, model serving software, scaling logic, and monitoring.

Self-hosting makes sense when you have predictable, high-volume traffic. It does not make sense for spiky traffic patterns or volumes below the break-even point.

Batch inference. For non-real-time workloads, batch processing is significantly cheaper than real-time API calls. Most providers offer batch APIs with 24-48 hour turnaround at 50% of the real-time price. If your use case tolerates delay — report generation, data enrichment, document processing — route it to batch.

Prompt caching with providers. Several providers now offer prompt caching at the infrastructure level. If your system sends the same prefix across multiple requests (common in RAG systems with a shared system prompt), the provider caches the prefix computation and charges reduced rates for cached tokens. This is free cost reduction if your prompt structure supports it.

Common failure modes

Optimizing cost before measuring quality baselines. You need a quality baseline before you start optimizing. If you do not know your current accuracy, latency, and user satisfaction metrics, you cannot tell whether a cost change caused a quality regression. Establish baselines first.

Aggressive model routing from day one. Start with conservative routing. Route only the most obviously simple requests to the cheapest model. Expand routing coverage gradually as you validate quality. Teams that route 80% of traffic to a small model on day one get burned by quality regressions they could have caught with a gradual rollout.

Ignoring latency impact. Some cost optimizations add latency. Semantic caching adds lookup time. Self-hosted inference may have cold start issues. Fallback chains add a second inference call for failed requests. Measure latency alongside cost and quality. A cost optimization that doubles p95 latency may not be worth it for interactive applications.

One-time optimization without ongoing monitoring. Cost characteristics change as your traffic mix changes, as providers update pricing, and as new models become available. Review your cost optimization quarterly. The routing thresholds that were optimal six months ago may be too conservative or too aggressive today.

Next step

Pull your inference cost data for the last thirty days. Calculate your cost per request and your input-to-output token ratio. If your input tokens are more than 60% of your total tokens, start with Layer 1 — you are paying to send information the model does not need. If your output tokens dominate, start with Layer 2. If both are lean, move to Layer 3 and build your complexity classifier.