Does SFT require training a model from scratch?

No. Fine-tuning starts from an existing trained model and adapts its behavior using additional supervision.

Are human-written examples the only option?

No. Examples can have other origins, including model-assisted creation, but their suitability and correctness still need assessment.

Is SFT the same as reinforcement learning from human feedback?

No. SFT learns from demonstrated target outputs. A broader feedback-training pipeline may include SFT and then a separate preference-based or reinforcement-learning stage.

AI Glossary / Generative AI and language / Intermediate

Supervised fine-tuning

Definition

Supervised fine-tuning, or SFT, adapts an already trained model using examples that specify the desired output for an input. For a language assistant, these examples often pair requests with suitable responses. Training adjusts selected model parameters so the demonstrated responses become more likely. SFT can teach task behavior and output conventions, but it does not guarantee accurate answers to unfamiliar requests.

What it means

Pretraining gives a model broad patterns learned from its original training material. SFT supplies more directed demonstrations: how an assistant should respond to a particular kind of request, format an extraction, or handle missing information.

The supervision is the desired response itself. This differs from merely telling the model what to do in a prompt at inference time, because fine-tuning changes trained parameters. It also differs from a preference dataset that only says which of two answers is better.

Instruction tuning is a common application of SFT, but supervised adaptation can target narrower tasks too. The examples need to demonstrate the behavior the deployed system will actually be asked to perform.

How it works

A team chooses a starting model and prepares suitable input-output examples. Conversation roles and special tokens must follow the model's expected format. The team separates training material from evaluation material and checks the examples for conflicting or misleading targets.

During common language-model SFT, the model predicts target tokens using the preceding context. A loss penalizes low probability assigned to the target tokens, and an optimizer updates the trainable parameters. The configuration determines which tokens contribute to the loss; prompt or padding tokens may be excluded.

Training can update many model weights or use a parameter-efficient method that changes a smaller set. The result is then evaluated on unseen requests, including cases where the correct behavior is to acknowledge that the supplied information is insufficient.

Three practical examples

1. A document assistant learns to return three specified fields from short notices, using examples that consistently distinguish an absent field from an empty string.

2. A customer-support drafting tool learns a company's approved response structure from reviewed examples covering several request categories.

3. An educational assistant is trained on demonstrations that explain a calculation step by step and correct a student's mistaken premise politely when the evidence warrants correction.

Think of it this way

Think of coaching an experienced employee with worked examples of the organization's preferred practice. The employee already has broad skills, while the examples focus attention on a particular job.

The analogy stops at the training mechanism. A language model changes numerical parameters through optimization; it does not acquire human professional judgment or understand organizational responsibility. Repeatedly showing a polished answer can also teach an undesirable habit if the answer contains unsupported claims.

Worked example

Imagine one training response has two target tokens. Before an update, the model assigns probabilities 0.4 and 0.5 to those tokens, each conditioned on the appropriate preceding context.

Using natural logarithms, the average negative log-likelihood is −(ln 0.4 + ln 0.5)/2, approximately 0.805. Suppose a later model assigns probabilities 0.6 and 0.7. The corresponding average is approximately 0.434.

This toy calculation shows improvement at predicting that demonstrated response. It does not show that the model is more factual or useful on new tasks. A separate held-out test might check whether the model extracts the right fields, follows the required format, and avoids filling missing fields with inventions. The example illustrates loss, not an actual training run or a promised performance gain.

Illustrative case study

Fictional case: Harbor Events wants an assistant to convert venue notices into structured summaries. Its initial training examples always contain a booking date, so the model frequently supplies a date even when a new notice has none.

Reviewers add carefully written demonstrations in which the correct output marks the date as unknown. They also create an untouched evaluation set containing incomplete notices and conflicting date references.

The team compares the adapted model with its earlier version using both field accuracy and unsupported-field errors. This fictional workflow illustrates improving supervision and measurement; it reports no real company result.

Common misconceptions and mistakes

Do not confuse a declining training loss with reliable generalization. A model can learn the training answers too specifically or reproduce flaws in them.

Do not assume more examples always help. Repeated, inconsistent, or low-quality demonstrations may reinforce unwanted behavior. Inspect what the examples reward in practice.

Avoid using SFT as a substitute for a dependable source of frequently changing facts. A system may still need retrieval or tools. Also distinguish the fine-tuning objective from the method used to store parameter changes: SFT can be combined with adapters without becoming a different supervision concept.

Business use

SFT is useful when a team can clearly specify and demonstrate a repeated behavior. Before committing to training, establish a baseline and define the errors that matter: incorrect fields, missing qualifications, wrong format, or inappropriate response style.

Budget for collecting and reviewing demonstrations, evaluating regressions, and maintaining model versions. Keep a record of training-data provenance and evaluation decisions so later changes can be compared with the same operational criteria.

Check your understanding

A model perfectly reproduces its training responses. Is that enough to approve it for new customer requests?

Answer: No. It must be evaluated on appropriately separated examples that represent deployment needs. Memorizing demonstrations does not establish dependable performance on unfamiliar requests.

Related terms

Fine-tuning
Instruction tuning
Supervised learning
Cross-entropy loss
Teacher forcing
Parameter-efficient fine-tuning
Reinforcement Learning from Human Feedback
Pretraining
evaluation dataset

Learn next

Learn instruction tuning next to see how demonstrations target request-following behavior. Then study train-validation-test separation and parameter-efficient fine-tuning, keeping the supervision objective separate from the choice of which parameters are updated.

Sources

Hugging Face TRL documentation, SFT Trainer, including target-token loss, masking, and adapter support: https://huggingface.co/docs/trl/main/en/sft_trainer

Ouyang and colleagues, Training language models to follow instructions with human feedback: https://arxiv.org/abs/2203.02155

Primary documentation and the original paper's abstract were read. The paper distinguishes supervised demonstrations from its later feedback-training stage. Calculations, operational suggestions, and the Harbor Events case are original teaching material.

Previous: Supervised Clustering Next: Supervised learning