> ## Documentation Index
> Fetch the complete documentation index at: https://docs.monostate.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quantization

> Reduce memory with quantized training

# Quantization

Quantization reduces memory usage by using lower precision for model weights.

## Quick Start

```bash theme={null}
aitraining llm --train \
  --model meta-llama/Llama-3.2-8B \
  --data-path ./data.jsonl \
  --project-name quantized-model \
  --peft \
  --quantization int4
```

## Python API

```python theme={null}
from autotrain.trainers.clm.params import LLMTrainingParams

params = LLMTrainingParams(
    model="meta-llama/Llama-3.2-8B",
    data_path="./data.jsonl",
    project_name="quantized-model",

    peft=True,
    quantization="int4",  # or "int8"
    lora_r=16,
)
```

## Quantization Options

| Option | Memory Reduction | Quality   |
| ------ | ---------------- | --------- |
| None   | 0%               | Best      |
| int8   | \~50%            | Very Good |
| int4   | \~75%            | Good      |

## Supported Tasks

Quantization is available for:

| Task    | Params Class        | Notes        |
| ------- | ------------------- | ------------ |
| LLM     | `LLMTrainingParams` | Full support |
| VLM     | `VLMTrainingParams` | Full support |
| Seq2Seq | `Seq2SeqParams`     | Full support |

### 4-bit (QLoRA)

Maximum memory savings:

```python theme={null}
params = LLMTrainingParams(
    ...
    quantization="int4",
)
```

### 8-bit

Better quality, less savings:

```python theme={null}
params = LLMTrainingParams(
    ...
    quantization="int8",
)
```

## Memory Requirements

### Llama 3.2 8B

| Config         | VRAM Required |
| -------------- | ------------- |
| Full precision | \~64 GB       |
| LoRA (fp16)    | \~18 GB       |
| LoRA + 8bit    | \~12 GB       |
| LoRA + 4bit    | \~8 GB        |

### Gemma 2 27B

| Config         | VRAM Required |
| -------------- | ------------- |
| Full precision | \~108 GB      |
| LoRA + 4bit    | \~20 GB       |

## Best Practices

### Use with LoRA

Quantization **requires** PEFT/LoRA to be enabled:

```python theme={null}
params = LLMTrainingParams(
    ...
    peft=True,  # Required for quantized training
    quantization="int4",
)
```

<Warning>
  Quantization only works when `peft=True`. Without PEFT enabled, the quantization setting will be ignored.
</Warning>

### Adjust Learning Rate

Quantized training often benefits from a higher learning rate than the default (`3e-5`):

```python theme={null}
params = LLMTrainingParams(
    ...
    peft=True,
    quantization="int4",
    lr=2e-4,  # Higher LR works well with QLoRA
)
```

### Use Flash Attention

Combine with [Flash Attention](/advanced/flash-attention) for speed:

```python theme={null}
params = LLMTrainingParams(
    ...
    quantization="int4",
    use_flash_attention_2=True,  # Requires Linux + CUDA + flash-attn package
)
```

## Inference with Quantized Models

Load quantized models for inference:

```python theme={null}
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 4-bit config (matches AITraining defaults)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=False,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-8B",
    quantization_config=bnb_config,
)
```

## Platform Requirements

<Warning>
  **Quantization only works on Linux.** The `bitsandbytes` library required for int4/int8 quantization is only available on Linux systems.
</Warning>

## Apple Silicon (MPS) Note

Quantization is **not compatible with Apple Silicon MPS**. When you use quantization on a Mac with M1/M2/M3:

* Training automatically falls back to CPU
* You'll see a warning message explaining this
* For faster training on Mac, skip quantization and use LoRA alone

```bash theme={null}
# On Apple Silicon - use LoRA without quantization for MPS acceleration
aitraining llm --train \
  --model google/gemma-3-270m \
  --data-path ./data.jsonl \
  --project-name mac-training \
  --peft \
  --lora-r 16
```

Environment variables for manual control:

* `AUTOTRAIN_DISABLE_MPS=1` - Force CPU training
* `AUTOTRAIN_ENABLE_MPS=1` - Force MPS even with quantization (may crash)

## Quality Considerations

Quantization does reduce quality slightly. For critical applications:

1. Test on your specific task
2. Compare with full-precision baseline
3. Consider 8-bit if quality matters more

## Next Steps

<CardGroup cols={2}>
  <Card title="LoRA/PEFT" href="/advanced/lora-peft">
    Efficient fine-tuning
  </Card>

  <Card title="Flash Attention" href="/advanced/flash-attention">
    Speed optimizations
  </Card>
</CardGroup>