> ## Documentation Index
> Fetch the complete documentation index at: https://docs.monostate.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ORPO Training

> Odds Ratio Preference Optimization

# ORPO Training

ORPO combines SFT and preference optimization in a single training phase.

## What is ORPO?

ORPO (Odds Ratio Preference Optimization) is a simpler alternative to DPO that doesn't require a reference model. It optimizes preferences using odds ratios directly, reducing memory usage and training complexity.

## Quick Start

```bash theme={null}
aitraining llm --train \
  --model google/gemma-2-2b \
  --data-path ./preferences.jsonl \
  --project-name gemma-orpo \
  --trainer orpo \
  --prompt-text-column prompt \
  --text-column chosen \
  --rejected-text-column rejected \
  --peft
```

<Warning>
  ORPO requires `--prompt-text-column` and `--rejected-text-column`. The `--text-column` defaults to `"text"`, so only specify it if your chosen column has a different name.
</Warning>

## Python API

```python theme={null}
from autotrain.trainers.clm.params import LLMTrainingParams
from autotrain.project import AutoTrainProject

params = LLMTrainingParams(
    model="google/gemma-2-2b",
    data_path="./preferences.jsonl",
    project_name="gemma-orpo",

    trainer="orpo",
    prompt_text_column="prompt",
    text_column="chosen",
    rejected_text_column="rejected",
    dpo_beta=0.1,  # Default: 0.1
    max_completion_length=None,  # Default: None

    epochs=3,
    batch_size=2,
    lr=5e-5,

    peft=True,
    lora_r=16,
)

project = AutoTrainProject(params=params, backend="local", process=True)
project.create()
```

## Data Format

Same as DPO - preference pairs:

```json theme={null}
{
  "prompt": "What is AI?",
  "chosen": "AI is artificial intelligence, a field of computer science focused on creating systems that can perform tasks requiring human intelligence.",
  "rejected": "AI is just robots."
}
```

## ORPO vs DPO

| Aspect          | ORPO       | DPO                                                 |
| --------------- | ---------- | --------------------------------------------------- |
| Reference model | Not needed | Not needed with PEFT, required for full fine-tuning |
| Memory usage    | Lower      | Higher (if using reference model)                   |
| Training speed  | Faster     | Slower                                              |
| SFT phase       | Combined   | Separate                                            |
| Complexity      | Simpler    | More options                                        |

## Parameters

| Parameter               | Description                              | Default  |
| ----------------------- | ---------------------------------------- | -------- |
| `trainer`               | Set to `"orpo"`                          | Required |
| `dpo_beta`              | Odds ratio weight                        | `0.1`    |
| `max_completion_length` | Max response tokens                      | `None`   |
| `image_column`          | Image column for VLM preference training | `None`   |

### VLM (Vision-Language) ORPO

ORPO supports vision-language models like Qwen 3.5-VL for image+text preference alignment. Set `image_column` to enable VLM mode:

```python theme={null}
params = LLMTrainingParams(
    model="Qwen/Qwen3.5-VL-9B",
    trainer="orpo",
    image_column="images",
    text_column="chosen",
    rejected_text_column="rejected",
    prompt_text_column="prompt",
)
```

The dataset should have `chosen`/`rejected` columns with messages lists, and an image column containing the images. The image column is automatically renamed to `images` for TRL compatibility.

## When to Use ORPO

Choose ORPO when:

* Memory is limited (no reference model needed)
* You want combined SFT + alignment
* Simpler training pipeline preferred
* Starting from a base model (not instruction-tuned)

Choose DPO when:

* You need fine-grained control
* Working with already instruction-tuned models
* Reference model behavior is important

## Example: Customer Support

```python theme={null}
params = LLMTrainingParams(
    model="google/gemma-2-2b",
    data_path="./support_preferences.jsonl",
    project_name="support-bot",

    trainer="orpo",
    dpo_beta=0.15,

    epochs=3,
    batch_size=2,
    gradient_accumulation=4,
    lr=2e-5,

    peft=True,
    lora_r=32,
    lora_alpha=64,

    log="wandb",
)
```

## Next Steps

<CardGroup cols={2}>
  <Card title="DPO Training" href="/advanced/dpo-training">
    Alternative alignment method
  </Card>

  <Card title="Reward Modeling" href="/advanced/reward-modeling">
    Train reward models
  </Card>
</CardGroup>