05 — Alignment (RLHF) Pipeline¶
Alignment training teaches a model which of its possible answers are better — the technique behind helpful chat assistants. The 🤖 RLHF Pipeline tab offers five methods, all built on Hugging Face TRL.
This is an advanced feature. If you're new to fine-tuning, complete standard SFT training first (04 — Training).
The Big Picture¶
Standard fine-tuning (SFT) teaches a model to imitate your examples. Alignment goes further: it teaches the model which outputs are good.
Reward-based: SFT → A. Reward model → B. GRPO
Verifiable tasks: SFT → B. GRPO with a `reference` column (no reward model needed)
One-step: SFT → C. ORPO (ranked pairs)
SFT → D. KTO (thumbs-up / thumbs-down feedback)
Checking Dependencies¶
At the top of the tab you'll see a status line:
Reward model ✅ | GRPO ✅ | ORPO ✅ | KTO ✅
If you see ❌, install TRL:
pip install "trl>=0.29.1,<1.15"
Tab A — Reward Model¶
What is a Reward Model?¶
A reward model is a second model that scores a response for a given prompt: high score = good response. It is a sequence classifier with a single score output, trained so that the chosen answer scores higher than the rejected one.
Data Format¶
Columns prompt, chosen (a good response) and rejected (a worse response to the same prompt):
prompt,chosen,rejected
"How long is a year?","The Earth orbits the Sun once every 365.25 days.","The Earth doesn't move."
"How do I cook pasta?","Boil salted water, add the pasta and cook for the time on the package.","Just put pasta in water."
Settings¶
| Setting | Default | Notes |
|---|---|---|
| Base Model ID | Auto | A model of the same family as your policy works best |
| Output Directory | ./reward_model |
Where the reward model is saved |
| Epochs | 3 | 1–3 is usually enough |
| Learning Rate | 1×10⁻⁴ | Trained with LoRA, so higher than full fine-tuning |
| Batch Size | 4 | Reduce if you run out of memory |
| Eval Steps | 100 | How often to evaluate on the 10% holdout |
| Max Length | 1024 | Maximum tokens for prompt + response |
Training uses LoRA; when it finishes the adapter is merged and a complete model is saved, so its folder can be used directly as the GRPO reward in Tab B.
Tab B — GRPO¶
What is GRPO?¶
GRPO (Group Relative Policy Optimization) is the reinforcement-learning method behind recent reasoning models. For every prompt the model writes several answers, each answer is scored, and the model is pushed towards the answers that beat their group's average. Unlike PPO it needs no separate value model, which makes it simpler and lighter.
Where the reward comes from¶
Use a reward model, built-in rewards, or both — all chosen rewards are added together:
- A reward model from Tab A — enter its folder in Reward Model Path.
- Built-in rewards (tick any):
| Reward | Gives 1 when… | Needs |
|---|---|---|
| Reference answer | the answer contains the reference text (case-insensitive) |
reference column |
| Maths answer | the answer equals the reference mathematically — \boxed{0.5} matches a reference of \frac{1}{2} (TRL + math-verify; the answer must be LaTeX, e.g. \boxed{…}) |
reference column, pip install math-verify |
| Think format | the output is <think>reasoning</think> followed by the answer |
— |
| Valid JSON | the whole output is valid JSON (a ```json fence is allowed) |
— |
| Regex | the whole output matches your regular expression | the Regex box |
prompt,reference
"What is 12 × 12?","144"
"What is the capital of France?","Paris"
A prompts-only file (prompt column) works when you give a reward model path.
Settings¶
| Setting | Default | Notes |
|---|---|---|
| Policy Model ID | Auto | Your SFT model or a Hub model ID |
| Reward Model Path | — | Folder from Tab A (optional if you have reference) |
| Output Directory | ./grpo_model |
The trained LoRA adapter is saved here |
| Learning Rate | 1×10⁻⁵ | Keep low — RL is sensitive |
| Epochs | 1 | Passes over the prompts |
| KL Beta | 0 | Penalty for drifting from the original model (0 = off, the current recommended default) |
| Completions per Prompt | 4 | More = better baseline, slower |
| Prompts per Step | 1 | Batch size is prompts × completions |
| Max Completion Tokens | 128 | Length of each generated answer |
| Resume from last checkpoint | off | Continue from the newest checkpoint in the output directory |
| Loss | dapo |
dapo (TRL's default) and dr_grpo avoid the original grpo loss's bias towards short answers; bnpo normalises per local batch |
| LoRA variant / Rank / Alpha | LoRA / 16 / 32 | Same variants as SFT (LoRA, rsLoRA, DoRA), on every linear layer |
| Generate with vLLM | off | Much faster generation on a CUDA GPU, sharing it with training (TRL colocate mode). Needs pip install "trl[vllm]<1.15", which installs the vLLM version your TRL supports |
GRPO generates text during training, so it is slower per step than SFT — vLLM generation helps most there.
Tab C — ORPO¶
What is ORPO?¶
ORPO (Odds Ratio Preference Optimization) aligns the model in a single step by comparing good and bad responses — no reward model and no reference model.
Data Format¶
Same as DPO: prompt, chosen and rejected columns.
prompt,chosen,rejected
"How do I apologise to a friend?","Be sincere, acknowledge what you did wrong, and listen to their response.","Just say sorry and move on."
"What is a healthy snack?","Fruit, nuts, yogurt, or vegetable sticks with hummus are all great choices.","Chips and soda."
Settings¶
| Setting | Default | Notes |
|---|---|---|
| Base Model ID | Auto | Starting model |
| Output Directory | ./orpo_model |
Where to save |
| Learning Rate | 1×10⁻⁴ | |
| Beta | 0.1 | ORPO loss weight. Higher = stronger preference signal. |
| Alpha | 0.1 | Balances SFT and preference loss components (older TRL only). |
| Epochs | 3 | |
| Batch Size | 2 |
Tab D — KTO¶
What is KTO?¶
KTO (Kahneman-Tversky Optimization) learns from individual responses labelled good or bad — the kind of data you get from thumbs-up / thumbs-down buttons. You don't need two ranked answers per prompt.
Data Format¶
Either unpaired feedback:
prompt,completion,label
"Summarise this email","Meeting moved to Friday at 3pm.",true
"Summarise this email","lol idk",false
label accepts true/false, 1/0, yes/no, good/bad.
Or ranked pairs (prompt, chosen, rejected) — each pair becomes one good and one bad example. You need at least one good and one bad example.
Settings¶
| Setting | Default | Notes |
|---|---|---|
| Base Model ID | Auto | Starting model |
| Output Directory | ./kto_model |
The trained LoRA adapter is saved here |
| Learning Rate | 5×10⁻⁵ | |
| Beta | 0.1 | How far the model may move from the original |
| Epochs | 1 | |
| Batch Size | 4 | At least 2 (KTO estimates a baseline per batch) |
| Max Length | 512 | Maximum tokens for prompt + response |
| Resume from last checkpoint | off | Continue from the newest checkpoint in the output directory |
GRPO and KTO save a checkpoint every 50 steps (the last two are kept), so a stopped or crashed run can be resumed with the same output directory.
Tab E — Distillation¶
What is knowledge distillation?¶
A small student model learns from a larger teacher: not just the teacher's final answer, but its whole next-token distribution. This uses TRL's GKD trainer (Generalized Knowledge Distillation), which also trains on the student's own answers, graded by the teacher, so the student learns from its own mistakes. You get a model that is much cheaper to run and keeps much of the teacher's quality on your task.
Teacher and student must come from the same model family (one shared vocabulary), e.g.
Qwen/Qwen3-0.6B ← Qwen/Qwen3-8B. The student is trained as a LoRA adapter.
Data Format¶
Chats (JSONL, messages), instruction/output, or prompt/completion. The last
assistant answer of each example is the part the teacher grades:
{"messages": [{"role": "user", "content": "What is 12×7?"}, {"role": "assistant", "content": "84"}]}
Settings¶
| Setting | Default | Notes |
|---|---|---|
| Student model | Auto | The small model you want to keep |
| Teacher model | — | Larger model of the same family |
| On-policy share (λ) | 0.5 | 0 = learn only on the dataset's answers; 1 = only on the student's own answers (the paper found high values work best) |
| Divergence (β) | 0.5 | 0 ≈ forward KL (cover everything the teacher might say), 1 ≈ reverse KL (focus on its main answer) |
| Temperature | 0.9 | Sampling temperature for the student's own answers |
| Student answer length | 128 | Tokens the student generates per example |
| Learning Rate / Epochs / Batch / Max Length | 5×10⁻⁵ / 1 / 2 / 512 |
Both models are loaded at once, so you need memory for the teacher and the student.
Which Should I Use?¶
| Scenario | Recommendation |
|---|---|
| I have ranked pairs (good vs bad answer) | ORPO, or DPO in the Training tab |
| I only have thumbs-up / thumbs-down feedback | KTO |
| My task has a checkable answer (maths, extraction) | GRPO with a reference column |
| I want a reusable judge to optimise against | Reward model → GRPO |
| I want a small, cheap model that behaves like a big one | Distillation |
| I'm just starting out | SFT only is fine for most use cases |
Next Step¶
→ 06 — Inference: Test and serve your model.