Prompts vs. fine-tuning: when to reach for which

The default should always be prompting. A well-structured system prompt with clear persona, scope, tone, and escalation rules gets you 80% of the way to a production-ready agent in an afternoon. Fine-tuning takes weeks of data preparation, training runs, and evaluation, and if you fine-tune on bad data you've made the model worse in a way that's hard to reverse.

The most common reason teams reach for fine-tuning prematurely is that their system prompt is doing too much work in one block of prose. Before you write a training set, try breaking your prompt into sections: a persona section, an explicit rules section, an examples section, and an escalation section. A structured prompt with four concrete examples frequently outperforms a flat prompt that took twice as long to write.

Prompting also gives you a tighter feedback loop. You can change a prompt and re-run your eval suite in minutes. A fine-tuning cycle, even with a modern fine-tuning API, takes hours at minimum, and the difference between iterations is harder to inspect.

Fine-tuning genuinely wins in three scenarios. The first is consistent output format at scale. If your application parses the model's output programmatically, such as JSON extraction, structured classification or slot filling, and you're hitting format errors at 1 to 3% of calls, fine-tuning a small model on correct examples gets that error rate below 0.1%. Prompting alone rarely achieves the consistency that production parsers need.

The second scenario is latency-critical applications where you need a smaller, faster model. A fine-tuned 7B-parameter model can outperform a prompted 70B model on narrow tasks at a fraction of the latency and cost. If you're calling the model in real time for a sub-200ms feature, fine-tuning on your task lets you right-size the model.

The third is cost at massive scale. If you're making more than ten million model calls per month on a task that doesn't require a frontier model's full capability, fine-tuning a smaller base model for that specific task can reduce cost by 60 to 80%. The economics only make sense at volume: the investment pays off after roughly three to six months of saved inference cost.

Here's the rough math for a typical support-agent use case at 500k conversations per month. A frontier model with a 2000-token average prompt costs approximately $1,500 per month at standard API pricing. A fine-tuned smaller model handling the same calls costs roughly $400 per month in inference. The fine-tuning training run itself costs $200 to $800 depending on dataset size. Break-even is around month two.

On latency, frontier models at 2000 tokens of context typically generate 40 to 80 tokens per second. A fine-tuned 7B model on dedicated hardware runs at 120 to 200 tokens per second. For a 150-token response, that's 1 to 2 seconds versus 0.5 to 0.8 seconds. A perceptible difference for real-time chat, and negligible for async workflows.

The hidden cost people forget is maintaining a fine-tuned model. Every time you want to update behaviour, whether a new product feature or a changed policy, you retrain. Budget for one to two retraining runs per quarter per fine-tuned model, or build a prompting layer on top of your fine-tune to handle updates without retraining.

Run through these questions in order before committing to fine-tuning. If you answered yes to question 3 or 4, fine-tuning has a strong case. Otherwise, invest the same time in prompt engineering and eval-driven iteration first.