Fine-tuning vs RAG vs Prompting: A Decision Tree for Enterprise Teams

Fine-tuning is the answer teams reach for when prompting does not work well enough. Usually they reach for it too early, and the prompting approach was not working because the prompt was not well-designed rather than because the model needed to be retrained.

The order of operations should be: prompting first, RAG second, fine-tuning only when specific conditions are met. Here is the full decision tree.

Start Here: Is This a Knowledge Problem or a Behavior Problem?

This is the first question in every implementation decision.

A knowledge problem is when the model does not have access to the information it needs to answer correctly. Your internal policies, your product documentation, your proprietary data. A frontier model trained on internet data does not know what is in your internal wiki.

A behavior problem is when the model has the relevant knowledge but produces output in the wrong format, tone, or structure. It summarizes when you want bullet points. It writes formal prose when you want terse sentences. It produces inconsistent field naming in JSON outputs.

Knowledge problems are solved by RAG (retrieval) or by including the relevant information in the prompt (context stuffing). Fine-tuning does not help knowledge problems in the way most teams expect.

Behavior problems are often solved by better prompting. Few-shot examples are particularly effective. If the behavior problem persists after five iterations of prompt improvement with concrete examples, that is the signal to consider fine-tuning.

The Decision Tree

Is this a knowledge problem?
├── Yes → Does the context fit in the context window?
│         ├── Yes → Use context stuffing (include in prompt)
│         └── No  → Use RAG (retrieve relevant chunks)
└── No  → Is this a behavior problem?
          ├── Yes → Can better prompting fix it?
          │         ├── Yes → Improve the prompt (test 5+ iterations)
          │         └── No  → Do you have 1,000+ labeled examples?
          │                   ├── Yes → Consider fine-tuning
          │                   └── No  → Collect examples first
          └── No  → What problem are you solving?

Why Prompting Should Always Come First

The cost comparison is not close. A well-designed prompt costs hours to develop and cents per call. Fine-tuning a model costs days to weeks of engineering time, training compute, and ongoing maintenance every time the base model updates.

The ceiling for prompt quality is higher than most teams realize. Techniques like chain-of-thought prompting, few-shot examples, explicit output format specification, and role framing can dramatically improve output quality for complex tasks. Before concluding that the model needs to be fine-tuned, exhaust the prompt design space.

The one thing prompting cannot fix: latency from very large system prompts. If your prompt is 5,000 tokens because it contains extensive examples and instructions, and you are making 100,000 calls per month, the token cost and latency may justify fine-tuning to bake those instructions into the model weights. That is a cost optimization justification, not a quality justification.

When RAG Is the Right Architecture

RAG is the right choice when:

• The information you need is in documents too numerous or too long to include in every prompt
• The information changes frequently (policies that update quarterly, prices that update daily)
• You need source citations for compliance or trust reasons
• Different users should access different subsets of the document corpus (role-based access)

RAG is the wrong choice when:

• You just need to teach the model a specific output format or writing style
• The relevant information is small enough to include in the system prompt
• Your information never changes and consistency at inference time matters more than freshness

The Narrow Case for Fine-tuning

Fine-tuning is genuinely valuable in a specific, narrow set of scenarios:

Consistent output format at high volume: If you are making millions of calls per month and every call needs to produce output in a very specific format, fine-tuning can improve format consistency while reducing the token count needed in the prompt. The cost per call goes down and format compliance goes up.

Domain-specific language and terminology: If your industry uses terminology that is not well-represented in the base model's training data (highly specialized financial instruments, proprietary product names, regulatory frameworks), fine-tuning on domain-specific text can meaningfully improve quality.

Latency-critical applications: A fine-tuned smaller model (7B or 13B parameters) can outperform a prompted larger model on a narrow, well-defined task while running significantly faster and cheaper. If latency is the primary constraint, this trade-off can be worth the fine-tuning investment.

The Prerequisites for Fine-tuning to Work

Fine-tuning requires labeled examples. Not 50. Not 200. For meaningful improvement on complex tasks, you need at minimum 500 high-quality input-output pairs, and 1,000 to 2,000 is better.

Collecting that data properly takes time. You need to define what "correct" looks like precisely enough that humans can consistently label examples. If two labelers agree on fewer than 85% of examples, your labeling criteria are not clear enough and the fine-tuned model will be inconsistent.

You also need an evaluation set. A fine-tuned model that scores better on your training examples but worse on held-out data is worse than the base model. Do not skip the eval set.

The Practical Recommendation

For the vast majority of enterprise automation workflows: start with prompting. If prompting gets you to 90%+ quality on your test set, ship it. If quality stalls below your threshold after serious prompt iteration, add RAG if this is a knowledge problem, or start collecting labeled examples for fine-tuning if it is a behavior problem.

You will very rarely need to fine-tune. In the broad landscape of enterprise automation systems, fine-tuning is genuinely justified in fewer than 5% of cases.

Sources

• OpenAI — Fine-tuning Guide
• Anthropic — RAG Overview
• LlamaIndex — RAG vs Fine-tuning Comparison