Fine-Tuning LLMs

Course Content

Fine-Tuning LLMs

6 sections · 52 lessons

What is prompt loss weighting in instruction tuning, and what tradeoffs does it control?

Which tokens are graded in one support example[system]Meraorderkabaayega?Aapkaorderkal01234567label -100label -100trainedThe model reads the whole sequence but is graded only on the assistant's reply.
With a 1,800-token clause and a 5-token label, an unmasked loss spends 99.7 percent of its signal on copying the prompt.

What you need to know

How masking works

Each training example becomes one token sequence: prompt, then response. For every position, the label is the next token. To ignore a position, its label is set to -100, which PyTorch's cross-entropy skips.

Text
tokens:  [system][user: Mera order kab aayega?][assistant: Aapka order kal tak ...]labels:  -100 ... -100 ... -100 ... -100 ...     real token ids ...

The model still reads the prompt (it is in the forward pass); it is just not graded on predicting it. A prompt loss weight of 0.1 would grade prompt tokens at one-tenth of the response tokens.

Why it matters: count the tokens

For a legal-clause classifier, a prompt might be 1,800 tokens (instructions plus the clause) and the answer 5 tokens (arbitration). With full loss, 1,800 / 1,805 = 99.7% of the loss comes from predicting the prompt. The model spends its training learning to write contract text, and the label — the only thing you need — is a tiny part of the signal.

In TRL 1.x

Python
from trl import SFTConfig# prompt/completion dataset: completion-only loss is the default (None -> True here)SFTConfig(output_dir="out", completion_only_loss=True)# chat "messages" dataset: train only on assistant turnsSFTConfig(output_dir="out", assistant_only_loss=True)

assistant_only_loss needs a chat template that marks assistant turns with {% generation %} tags; many templates do not have them yet, so check. These flags replaced the older DataCollatorForCompletionOnlyLM, which recent TRL no longer ships. A fractional weight such as 0.1 is not a built-in option; you pass your own loss function (compute_loss_func) that multiplies prompt-token losses by the weight.

The trade-off

Weight 0 (mask the prompt)

  • Model learns only to respond
  • No gradient wasted on boilerplate
  • Fewer supervised tokens per example
  • Can underfit on tiny datasets with short answers

Weight 1 (full sequence)

  • Every token is a learning signal
  • Acts as a mild regulariser
  • Model also learns to write prompts and user turns
  • Long templates swamp the response signal

A 2024 study ("Instruction Fine-Tuning: Does Prompt Loss Matter?") found that on datasets with short completions, a small non-zero prompt weight could beat full masking. That is why it is worth a quick ablation — try 0 and around 0.1 — when your answers are short and your data is small.

For continual pretraining on raw domain text there is no prompt/response split, so every token is trained.

A real-life example

A telecom company trains a Hindi customer-support model on 15,000 chat transcripts. Each example starts with a 400-token system prompt of policies, then alternating customer and agent turns in Hindi and Hinglish.

The first run trained on every token. In testing, the model answered correctly, then kept going: it wrote the customer's next message ("Theek hai, aur recharge kaise karun?") and sometimes repeated policy text from the system prompt. It had learned to predict the whole transcript, customers included.

They switched to assistant_only_loss=True, decoded one batch to confirm only agent turns had real labels, and retrained. The extra turns disappeared, and the policy-copying stopped because the system prompt was no longer a training target.

Follow-up questions to expect

  • "Is the system prompt part of the prompt?" — Yes. It is masked along with the user turns.
  • "Does masking save compute?" — Mostly no — the forward and backward passes still cover all tokens. TRL's default chunked loss skips the output projection for masked positions, which saves some memory.
  • "In multi-turn data, do you train on all assistant turns or only the last?" — Usually all assistant turns, since each is a valid example of good behaviour. Train only the last if earlier turns are of lower quality.