In the fine-tuning vs RAG vs prompting choice, most nonprofit and agency projects need prompting first, RAG when answers depend on your own documents, and fine-tuning almost never. As of October 2026, the three big model vendors have also pulled back from self-serve fine-tuning.
Our guide to what a large language model is explains how models are trained and mentions RAG in passing. This post goes deeper on fine-tuning vs RAG vs prompting. It gives plain definitions and a decision table with cost and upkeep. It walks through four common projects and lists what OpenAI, Google and Anthropic offer. Vendor facts sit in one dated section.
The short version: in the fine-tuning vs RAG decision, start with neither. Write a good prompt and test it. If the model needs facts it never saw, such as your dues or HR policy, add retrieval. Consider fine-tuning only for a narrow, high-volume task where a good prompt fails. Then check that a vendor will still run the job.
Three ways to change what a model does
Prompting means changing the input. You write instructions, add examples of good output and paste in any text the model needs. The model itself does not change. Every call starts fresh from what you send.
RAG (retrieval-augmented generation) means searching your own content first, then putting the best passages into the prompt. Anthropic’s glossary says the data "is retrieved at runtime when a query is sent to the model." The model still does not change. It just gets better material to work from. Our RAG explainer covers the moving parts.
Fine-tuning means changing the model. You train it further on your own examples, and the vendor gives you a new private model ID. Anthropic’s glossary says fine-tuning makes a model start "mimicking the patterns and characteristics of the fine-tuning dataset."
That difference drives everything else. Prompting and RAG keep working when you switch to a newer model. A fine-tuned model is tied to the base model it came from.
What the research says about fine-tuning vs RAG
Two papers frame most discussions of fine-tuning vs RAG.
The first is the 2020 paper that named RAG, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis and colleagues. It paired a language model with a searchable index of Wikipedia. The authors reported that it "set the state-of-the-art on three open domain QA tasks." It also wrote more specific and factual text than the model alone. The paper named two open problems: showing where an answer came from, and updating what a model knows.
Even the original RAG was described as "a general-purpose fine-tuning recipe," so the two can mix. Most teams today use retrieval with an off-the-shelf model.
The second paper tested the question head on. In Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2023), Ovadia and colleagues gave three 7-billion-parameter open models facts in two ways. One was further training on raw text. The other was retrieval.
Retrieval won. The test used news from after the models’ training. Mistral 7B scored 0.481 with no help, 0.504 after fine-tuning and 0.875 with RAG. Llama 2 got worse after fine-tuning, dropping from 0.353 to 0.219. The authors wrote that "LLMs struggle to learn new factual information through unsupervised fine-tuning." Training on many rewordings of each fact helped. RAG still came out ahead.
One limit: that study tested teaching facts. It did not test style or format, where fine-tuning has a better case.
Fine-tuning vs RAG vs prompting: a decision table
Most projects land in the first three rows.
| Problem | Best approach | Cost | Upkeep |
|---|---|---|---|
| Model ignores your format, tone or rules | Prompting with clear instructions and examples | Low: a few hours of writing and testing, then per-token use | Re-test when you change models |
| Model does not know your facts (policies, dues, programs) | RAG over your own documents | Medium: build the index, host it, longer prompts per call | Re-index when content changes; check search quality |
| Answers must cite a source a person can check | RAG, with the prompt asking for quotes and links | Medium, same as above | Keep source pages current and permissions correct |
| Very high volume, narrow task, prompt still fails | Fine-tuning, if a vendor still offers it | High: labeled examples, training fees, separate pricing | Retrain before each base model retires |
| Facts change monthly or yearly | RAG, never fine-tuning | Medium | Update the documents, not the model |
With context windows near one million tokens, a short handbook can go straight into the prompt. Our context window guide explains the tradeoff. Retrieval earns its place when content is large, changes often or needs per-user permissions.
Fine-tuning vs RAG in four nonprofit and agency projects
Member FAQ bot. An association wants a chat box for questions about renewals, CE credits and the annual meeting. This is RAG. The answers live in help pages that change every year. Fine-tuning would freeze last year’s dues into the model. Retrieve from current pages, tell the model to answer only from them, and link the source.
Brand voice. A communications team wants drafts that sound like the organization. Fine-tuning was the classic answer. Today, start with a prompt that holds the style guide plus three to five approved samples. Large context windows make that cheap to include, and the prompt moves with you when you change vendors. If drafts still drift, add more examples before you think about training.
Classifying records. An agency needs to tag 20,000 old CMS posts against a new taxonomy. Or a nonprofit wants support email sorted into queues. This is prompting: the category list, a definition for each, a dozen labeled examples and structured output. Run it on a small model and check a sample by hand. OpenAI’s fine-tuning guide lists a smaller, cheaper model for one task as a reason to fine-tune. At nonprofit volumes, a small model with a good prompt usually closes that gap.
Policy lookup. Staff want to ask "what is our travel reimbursement limit?" against the employee handbook and board policies. This is RAG with citations. The search must respect who can see which document, since HR and finance policies are often restricted. A person should confirm any answer that affects pay, grants or compliance.
What each vendor offers for fine-tuning, as of October 2026
This is the section that will date. Everything below was checked on vendor pages on October 21, 2026.
OpenAI. The fine-tuning guide says the platform is being wound down and is no longer open to new users. The deprecations page gives the schedule. New organizations lost access on May 7, 2026. Organizations without recent fine-tuned inference lost it on July 2. On January 6, 2027, "Active existing customers will no longer be able to create new fine-tuning jobs." Six fine-tuned model rows had an October 23, 2026 shutdown date. We covered that in OpenAI’s October model shutoff.
Google. The Gemini API’s model tuning page says no model supports fine-tuning in the Gemini API or AI Studio. Google has no immediate plans to bring it back. The page says fine-tuning is supported in Gemini Enterprise Agent Platform, its Google Cloud service.
Anthropic. The Claude glossary states: "The Claude API does not currently offer fine-tuning." The one route was Claude 3 Haiku on Amazon Bedrock, as a preview. Bedrock’s legacy model lifecycle page shows it reached end of life in US regions on September 10, 2026. Other regions end November 13, 2026. Legacy models accept no new fine-tuning jobs.
All three vendors have moved the same way. Fine-tuning their flagship models is now either closed, closing or limited to enterprise cloud platforms.
What fine-tuning vs RAG vs prompting really costs
Prompting costs per token, on every call. Long instructions are billed each time, so a 3,000-token style guide adds up at volume. Prompt caching lowers the cost of a repeated prefix. The bigger cost is staff time to build a test set. Our AI costs guide shows how to turn token prices into a monthly bill.
RAG adds three costs. You pay to build the index, which means splitting documents and creating embeddings. You pay to host it. And each call carries the retrieved passages, so prompts get longer. Managed options price this directly. OpenAI’s pricing page lists file search storage at $0.10 per GB per day after 1 GB free. Tool calls cost $2.50 per 1,000. A self-hosted index can cost far less, as our pgvector on Supabase walkthrough shows. See What Are Vector Databases? for the options.
Fine-tuning costs the most to start. Someone has to build and check a labeled set of examples, which is usually the largest cost. Training has its own fee. OpenAI listed reinforcement fine-tuning on o4-mini-2025-04-16 at $100 per training hour. That base model was on the October 23 shutoff list. Then you pay again each time the base model retires.
How to choose for a client project
- Write the success test first. Collect 20 to 50 real questions or records with good answers. Anthropic’s prompt engineering guide assumes you have success criteria and a way to test them before you start.
- Try prompting alone. Many projects stop here.
- Add RAG when misses are about facts. Right format but wrong facts means it needs your documents.
- Fix retrieval before blaming the model. After a bad answer, check whether the right passage was found.
- Treat fine-tuning as a last step. Confirm a vendor will run the job. Price the retraining and record the base model’s end date.
That order keeps your options open. A prompt and an index can move to a new model in an afternoon. A fine-tune cannot.
Frequently Asked Questions
Fine-tuning vs RAG: what is the difference?
Fine-tuning trains the model further on your examples and produces a new model. RAG leaves the model alone and adds relevant passages from your documents to each prompt. RAG is easier to update because you change the documents, not the model.
Is prompt engineering enough for most projects?
For many, yes. Tasks about format, tone and following rules usually respond to clear instructions and good examples. Add retrieval when the model needs facts it was never trained on.
Can fine-tuning teach a model our organization’s facts?
Poorly, according to Ovadia and colleagues. In their tests, models struggled to learn new facts from further training, and retrieval scored higher. Facts that change, such as dues or event dates, belong in documents the model retrieves.
Does OpenAI still offer fine-tuning?
Only to existing customers, and only until January 6, 2027, according to OpenAI’s deprecations page. The fine-tuning guide says the platform is no longer open to new users.
Can I fine-tune Gemini models?
Not in the Gemini API or AI Studio, as of October 2026. Google’s tuning page says fine-tuning is supported in Gemini Enterprise Agent Platform, its Google Cloud service.
Can I fine-tune Claude?
Anthropic’s glossary says the Claude API does not offer fine-tuning. The Amazon Bedrock route for Claude 3 Haiku reached end of life in US regions on September 10, 2026, according to AWS.
What happens to a fine-tuned model when the base model retires?
It stops working on the same date. OpenAI and Amazon Bedrock both tie fine-tuned models to their base model’s lifecycle, so plan to retrain or move to prompting and RAG before then.
Fine-tuning vs RAG for a member FAQ bot: which one?
RAG. Member answers change every year and need a source link a person can check. Retrieval from your current help pages handles both.
