Teams approve LLM projects based on per-query cost estimates, then get blindsided by the actual invoice. The gap between estimate and reality is not a rounding error. It is a structural problem: the estimate assumes steady-state usage, but the real cost includes retries, prompt inflation, embedding storage, fine-tuning runs, and the compounding effect of features that were not in the original scope.
This post gives you a calculator framework — not a spreadsheet, but the thinking behind the spreadsheet — so you can estimate spend with enough accuracy to make deployment decisions worth trusting.
Why Naive Estimates Fail
The simplest cost estimate multiplies expected daily queries by tokens per query by price per token. This produces a number that is wrong in one direction every time: too low.
Three things push actual costs above the naive estimate. First, prompt length grows during development. The prompt you estimate with is never the prompt you ship with. Guardrails, few-shot examples, system instructions, and retrieval context all add tokens. A prompt that starts at 200 tokens in the proof of concept reaches 800 tokens in production. Your cost estimate assumed 200.
Second, retry and fallback logic multiplies calls. When the primary model returns a low-confidence result, the system retries or falls back to a larger model. Each retry is a full inference pass at full cost. A five percent retry rate at double the base cost adds ten percent to your bill, and that is before you count the fallback model’s higher per-token price.
Third, features expand. The team ships one feature that uses the LLM. It works. Then they add a second feature that also uses the LLM. Then a third. Each feature was individually approved, but nobody tracked the aggregate cost. By quarter’s end, the total is three times the original estimate, and none of the individual teams exceeded their own budgets.
The Estimation Framework
Step 1: Inventory Every LLM Call Site
Before estimating cost, you need to know where the LLM is being called. Walk through the application and list every place a model inference happens. This includes obvious calls like chat completions and less obvious calls like classification, summarization, embedding generation, and content moderation.
For each call site, record four things: the model used, the average input tokens, the average output tokens, and the expected daily call volume. If you do not know the average token counts, measure them during development. Do not estimate them from the feature description. Measure actual prompts with actual data.
Step 2: Build the Token Budget
Multiply input tokens by call volume for each call site to get daily input tokens. Do the same for output tokens. Sum across all call sites. This is your daily token budget.
Now add a growth factor. For new products, assume 3x to 5x the initial estimate within six months. For established products adding LLM features, assume 1.5x to 2x. The growth factor accounts for feature expansion and prompt inflation. Teams consistently underestimate how much their prompts will grow as they handle edge cases discovered after launch.
Step 3: Price It Out
Apply the per-token pricing for each model used at each call site. Use the pricing tier you will actually be on, not the list price. If you have committed-use discounts, apply them. If you are on pay-as-you-go, use that rate.
Add a retry cost multiplier. A reasonable default is 1.1x for well-tested prompts and 1.3x for new prompts that have not been through production load. The multiplier accounts for retries, fallbacks, and the occasional runaway prompt that consumes far more tokens than expected.
Step 4: Account for Hidden Costs
Token pricing is the visible cost. The hidden costs often match or exceed it.
Embedding storage: if you are doing retrieval-augmented generation, you are storing vector embeddings. The storage cost depends on the embedding dimension and the number of documents. A million documents at 1536 dimensions costs roughly 6 GB of vector storage. At typical managed vector database pricing, this is a recurring monthly cost that scales with your document corpus.
Fine-tuning runs: if you fine-tune models, each training run costs compute. Estimate the number of fine-tuning runs per quarter. Teams usually run three to five iterations before settling on a fine-tuned model. Each run costs the base model price multiplied by training tokens.
Evaluation and testing: running evaluation suites against model outputs costs inference tokens. Budget for this. A thorough evaluation of a new prompt version against a test set of 500 examples, each requiring a model call, adds up. Plan for evaluation runs every time you change a prompt, which in active development is weekly.
Infrastructure overhead: the compute that orchestrates LLM calls — the application servers, the caching layer, the monitoring pipeline — has its own cost. This is not an LLM cost, but it is part of the total cost of running LLM features. If the LLM calls are a small part of total application traffic, this overhead is marginal. If the application is primarily an LLM wrapper, the infrastructure cost is the dominant cost.
Decision Matrix: Build vs. Buy Your Calculator
| Factor | Spreadsheet Calculator | Custom Monitoring |
|---|---|---|
| Team size | Under 10 developers | Over 10 developers |
| Call sites | Fewer than 5 | More than 5 |
| Model variety | 1-2 models | 3+ models |
| Budget cycle | Quarterly | Monthly or continuous |
For small teams with few call sites, a spreadsheet that tracks the four inputs per call site and applies pricing formulas is sufficient. Update it monthly with actual usage data to refine your estimates.
For larger teams, build cost monitoring into the application. Log every LLM call with token counts, model, and call site identifier. Aggregate daily. Alert when actual spend exceeds the estimate by more than 20 percent. This turns cost estimation from a planning exercise into an operational discipline.
Common Failure Modes
Estimating with the proof-of-concept prompt. The POC prompt is always shorter than the production prompt. Always measure with production-equivalent prompts, including retrieval context and guardrails.
Ignoring the compounding effect of features. Each LLM feature is individually justifiable. The aggregate is often not. Track total LLM spend across the organization, not just per-feature.
Forgetting about off-peak and batch processing. If you run batch jobs that call the LLM — nightly summarization, weekly report generation, periodic re-embedding — add those to the estimate. Batch costs are predictable and easy to forget.
Assuming price stability. Model pricing changes. New models launch at different price points. Your cost estimate should include a 15-20 percent buffer for pricing changes over the estimate period.
Next Step
Build your token budget today. List every call site, measure actual token counts with production-equivalent data, multiply by expected volume, and apply pricing with a 1.2x retry multiplier. If the resulting number makes the business case marginal, you have found the real decision: not which model to use, but whether the feature justifies its cost at production scale.
