AI
Why Fine-Tuning Displaced RAG and Foundation-Model Builds
Three years ago, every enterprise wanted to build a foundation model. Last year, RAG was the answer. This year, the shift is to fine-tuning.
“48% of developers said they’re looking at fine-tuning.” — Alicia Frame
Base models from OpenAI, Meta, and Mistral have converged, so off-the-shelf models offer no competitive edge. Building from scratch is expensive and slow. Fine-tuning threads that gap: start from a frontier model, add your domain data, and get differentiation without the pre-training bill. The four main reasons customers do it are accuracy, cost reduction via distillation, lower latency, and safer tool use in agents.
Distillation Demo: Shrinking an o3-Quality Model to Mini Size
Omkar More ran the distillation demo in Azure AI Foundry (08:36) against the Stanford Human Preferences (SHP) dataset. He first confirmed the o3-powered autograder aligned with human judgments, then evaluated five base models and chose o3 as the teacher. Stored completions (a toggle that logs every API response into the Azure OpenAI resource) built the training set from 1,000 records with no manual labeling.
“4o went from 35% all the way to 90%.” — Omkar More
All fine-tuned variants posted large gains. No new labels, just o3’s outputs as the training signal.
What Shipped at Build: Global Training, Developer Tier, and Lifecycle Extensions
Four announcements shipped at Build. Global training removes the region guesswork, so any Azure region can now launch a fine-tuning job. Developer tier eliminates the hosting fee: deploy a fine-tuned model and pay only base-model inference rates while testing. Frame called it the most important announcement of the session, aimed at teams who want to experiment before committing budget. Provisioned throughput for fine-tuned models is now GA, with higher quota limits (customers already run 30 million TPM on fine-tuned models). Finally, fine-tuned model lifespan extends one year past the base model deprecation date, ending the scramble to re-train on short notice.
Reinforcement Fine-Tuning: Teaching a Reasoning Model New Domains
Supervised fine-tuning teaches a model to copy. Reinforcement fine-tuning teaches it to reason. The input is a prompt and a reference answer. A grader scores how close the model’s output lands and adjusts the chain of thought in o4-Mini accordingly. No labeled reasoning traces are required. Omkar More ran the reinforcement fine-tuning demo on o4-Mini (26:11) on a number-game dataset where the model constructs arithmetic expressions toward a target value. Using 100 unlabeled training samples and default hyperparameters, RFT delivered a 12-13% uplift. Non-reasoning models like 4.1 and 4.1-Mini barely moved on the same task.
Production Evidence: 50% Accuracy Gains and Real Agentic Deployments
Three customer results anchor the demos. DraftWise, a legal startup, used RFT to teach o3 and o4-Mini to reason about contract validity, a task where the answer is verifiable but the logic is not easily codified. Dragon Copilot from Nuance fine-tuned a model to summarize doctor-patient visit transcripts in EHRs and measured a 50% accuracy increase over the base model. In healthcare, that gap maps directly to patient outcomes. GitHub Copilot’s code completion models, fine-tuned per language with additional reinforcement learning, gained over 30% in performance.
Oracle Health: Fine-Tuning for Sub-800ms Tool Calling in an EHR Agent
Srinivas Gadde leads clinical AI at Oracle Health, shipping agents that help physicians prep for patients in the three to four minutes between appointments. He showed the Oracle Health clinical AI agent live demo (40:38) with voice queries pulling labs and sinusitis history. The fine-tuning rationale was latency. GPT-4o takes about 1,100ms for the orchestrator call; the hard ceiling is 800ms. Switching to 4o-Mini saves 200-300ms, but accuracy dropped. Fine-tuning 4o-Mini with 1,500 supervised examples of tool-choice and argument extraction matched 4o’s accuracy at Mini’s speed.
“The data and eval is usually the 80%.” — Srinivas Gadde
Q&A
How does fine-tuning improve tool-calling accuracy? Sri explained that complex JSONs, nested arguments, and multi-turn conversation history create interpretation problems that prompt engineering handles poorly; Frame added that fine-tuning is ‘show, don’t tell’ — you provide examples of correct tool calls so the model learns when and how to call each tool. ▶ 55:38
What does fine-tuning cost, from training through production hosting? Frame said most training jobs come in under $10 (Omkar’s demo was probably under $1); standard pay-as-you-go hosting runs $1.70/hour plus base-model per-token rates, or teams can buy PTUs for provisioned throughput; RFT is charged by the hour due to higher compute cost. ▶ 58:15
Do I need an army of data scientists to get started with fine-tuning? Frame said no — a subject matter expert who can define the use case and curate data can train a model via the Foundry UI without setting up MLOps pipelines. ▶ 59:52
Notable Quotes
48% of developers said they’re looking at fine-tuning. Alicia Frame · ▶ 2:16
4o went from 35% all the way to 90%. Omkar More · ▶ 18:13
Developer tier is intended to unblock this. Alicia Frame · ▶ 21:31
The data and eval is usually the 80%. Srinivas Gadde · ▶ 52:52
Key Takeaways
- Distillation with stored o3 completions lifted 4.1-Mini from 35% to 90% on a preference benchmark, no new labeled data needed.
- Reinforcement fine-tuning on o4-Mini gained 12-13% on a hard reasoning task using just 100 unlabeled samples and a custom grader.
- Oracle Health hit a hard 800ms latency ceiling with 4o; fine-tuning 4o-Mini with 1,500 samples cleared the bar and matched 4o accuracy.