A pharmaceutical company I consulted for was three months into a project to fine-tune a large language model on their internal research corpus. The project had a team of four engineers, a budget of $800,000 for compute, and executive enthusiasm. When I asked what problem the fine-tuned model would solve that GPT-4 could not solve with retrieval-augmented generation, the team lead said, “Our data is proprietary. We need our own model.”

This reasoning is common. It is also, in most cases, wrong. The proprietary data argument sounds compelling until you examine what fine-tuning actually does, what it actually costs, and what alternatives exist that achieve the same outcomes at a fraction of the cost. Most organizations that are building custom LLMs would be better served by building better retrieval systems, better prompt engineering, and better evaluation frameworks on top of existing models.

What fine-tuning actually does

Fine-tuning a large language model adjusts the model’s weights to produce outputs that are more aligned with a specific domain’s patterns. A model fine-tuned on legal documents will produce text that uses legal terminology more naturally. A model fine-tuned on medical records will produce summaries that follow clinical conventions. The fine-tuned model is genuinely better at the target domain than the base model.

But fine-tuning has a critical limitation: it adjusts the model’s style and surface-level domain knowledge, not its reasoning capabilities. A fine-tuned model does not become smarter. It becomes more fluent in the target domain’s vocabulary and formatting conventions. If the task requires domain-specific vocabulary and formatting — say, generating clinical trial reports in the standard format — fine-tuning provides real value. If the task requires reasoning about the domain’s concepts — say, identifying drug interactions based on mechanism of action — fine-tuning provides marginal improvement at best, because the reasoning capability is in the base model’s architecture, not in the domain-specific training data.

This distinction matters because it determines whether the investment in fine-tuning is justified. If you need style and format adaptation, fine-tuning is often worth the cost. If you need reasoning improvement, it usually is not.

The RAG alternative

Retrieval-augmented generation — where the model queries a knowledge base at inference time rather than encoding knowledge in its weights — handles most of the use cases that organizations cite for fine-tuning. The model’s reasoning capabilities come from the base model. The domain knowledge comes from the retrieved documents. The combination produces outputs that are both well-reasoned and domain-informed without the cost of fine-tuning.

RAG has its own challenges. Retrieval quality depends on the knowledge base’s structure, the embedding model’s effectiveness, and the chunking strategy. These are engineering problems with well-understood solutions. They are also problems that can be solved incrementally — you can improve retrieval quality without retraining the model, which means you can iterate faster and cheaper than with fine-tuning.

The practical comparison is stark. Fine-tuning a 70-billion-parameter model costs tens of thousands of dollars in compute per training run. Each run produces a new model that must be evaluated, deployed, and monitored. Improving a RAG system costs engineering time but not compute, and the improvements take effect immediately without redeploying the model.

When custom models actually make sense

There are specific conditions under which building a custom model is justified. These conditions are narrower than most organizations assume.

When latency and cost constraints are extreme. A large API-based model costs more per token and takes longer per inference than a smaller custom model. If your application processes millions of requests per day and each request must complete within a tight latency budget, the cost and latency of a large general-purpose model may be prohibitive. In this case, training or distilling a smaller model for the specific task can produce significant cost savings and latency improvements. This is the strongest case for custom models, and it applies primarily to high-volume, low-complexity tasks like classification and extraction.

When the domain has specialized vocabulary and reasoning patterns that base models handle poorly. Some domains — protein structure prediction, chip design optimization, certain areas of mathematical reasoning — have patterns that are sufficiently different from general language that base models underperform. In these cases, domain-specific pre-training (not just fine-tuning) can produce meaningfully better results. These domains are the exception, not the rule.

When data privacy constraints preclude sending data to external APIs. Some organizations — defense contractors, certain healthcare providers, financial institutions with strict compliance requirements — cannot send their data to external model providers. For these organizations, running a custom model on-premises is not a preference but a requirement. This constraint justifies the investment in custom models, though the organization should still evaluate whether a smaller, cheaper custom model meets their needs before committing to training a large one.

When you are building a product where the model is the product. If your company’s core value proposition is a model that performs a specific task better than any available alternative — like AlphaFold for protein structure — then building a custom model is the entire point. This is a very small number of companies, and they are not the ones I am advising to reconsider.

The cost reality

The organizations I see building custom LLMs consistently underestimate the total cost. The training compute is visible and budgeted. The less visible costs are often larger.

Evaluation. How do you know your fine-tuned model is better than the base model? You need evaluation benchmarks, evaluation infrastructure, and evaluation expertise. Building a rigorous evaluation framework is a significant engineering project that is rarely included in the fine-tuning budget.

Ongoing maintenance. A fine-tuned model is not a one-time investment. As the base model improves — which it does every few months — the fine-tuned model may become obsolete. Re-fine-tuning on a new base model is another training run, another evaluation cycle, and another deployment.

Opportunity cost. The engineering time spent on fine-tuning is engineering time not spent on the applications that use the model. In my experience, the application layer — the retrieval system, the prompt engineering, the output validation, the user interface — has more impact on the end-user experience than the model layer. Investing in the model at the expense of the application is optimizing the wrong layer.

The decision framework

Before investing in a custom model, answer three questions.

Can your use case be handled by a general-purpose model with retrieval-augmented generation? If yes, use RAG. It is cheaper, faster to iterate, and easier to maintain.

If RAG is insufficient, can your use case be handled by fine-tuning a smaller model (under 13 billion parameters) on a targeted dataset? If yes, fine-tune the small model. It is cheaper than fine-tuning a large model, easier to deploy, and sufficient for most domain-specific tasks.

If neither RAG nor small-model fine-tuning is sufficient, evaluate whether the task’s complexity justifies the cost of training or fine-tuning a large model. This evaluation should include the full lifecycle cost: training, evaluation, deployment, monitoring, and retraining. If the full lifecycle cost is justified by the business value the model produces, proceed. If it is not, redesign the task to be simpler.

The contrarian position is this: most organizations building custom LLMs are doing so because it feels like the ambitious choice. It feels like innovation. Building a better RAG pipeline on top of an existing model feels incremental by comparison. But the incremental approach ships faster, costs less, and often produces better outcomes for the end user. Ambition should be measured by outcomes, not by the impressiveness of the approach.