> ## Documentation Index
> Fetch the complete documentation index at: https://docs.monostate.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# How Training Works
> The technical process behind AI training
# The Training Process
Let's look at what actually happens when you click "Train" - no math degree required.
## The Basic Loop
Training is a repetitive process that gradually improves the model:
```mermaid theme={null}
graph LR
A[Training Data] --> B[Model Makes Predictions]
B --> C[Calculate How Wrong It Was]
C --> D[Adjust Model]
D --> B
```
This loop runs thousands or millions of times until the model gets good at its task.
## Step by Step Breakdown
### Step 1: Initialize the Model
The model starts with random "weights" - think of these as millions of tiny dials that need to be tuned. At first, they're set randomly, so the model's predictions are nonsense.
### Step 2: Feed Training Data
Your data goes through the model in small batches:
* Text gets converted to numbers
* Images become grids of pixel values
* Everything becomes numbers the model can process
### Step 3: Forward Pass
The data flows through the model's layers:
```
Input → Layer 1 → Layer 2 → ... → Output
```
Each layer transforms the data, looking for different patterns:
* Early layers find simple patterns (edges, words)
* Later layers find complex patterns (objects, sentences)
### Step 4: Calculate Loss
"Loss" measures how wrong the model's prediction was:
* **Low loss** = Good prediction
* **High loss** = Bad prediction
Different tasks use different loss calculations:
* Classification: "How confident were you in the wrong answer?"
* Generation: "How different is your text from the expected text?"
### Step 5: Backpropagation
This is where the learning happens. The model works backwards from its mistake:
1. "I predicted 'cat' but it was 'dog'"
2. "Which weights caused this mistake?"
3. "Let's adjust those weights slightly"
Think of it like adjusting your aim after missing a target - you figure out what went wrong and correct it.
### Step 6: Update Weights
The model adjusts its millions of weights based on what it learned. The adjustments are tiny - too big and the model "forgets" what it learned before.
### Step 7: Repeat
Go back to Step 2 with the next batch of data. Each cycle is called an "iteration," and a full pass through all your data is called an "epoch."
## Key Training Concepts
### Learning Rate
How big the weight adjustments are:
* **Too high**: Model jumps around, never settling on good weights
* **Too low**: Training takes forever
* **Just right**: Steady improvement
Think of it like learning to draw - huge corrections make your lines wobble, tiny ones mean slow progress.
### Batch Size
How many examples the model sees before updating weights:
* **Small batches** (8-32): More frequent updates, less stable
* **Large batches** (128-512): Fewer updates, more stable
* **Your hardware**: Determines the maximum you can use
### Epochs
How many times the model sees all your training data:
* **Too few**: Model hasn't learned enough (underfitting)
* **Too many**: Model memorizes instead of learning (overfitting)
* **Sweet spot**: Usually 3-10 for fine-tuning, more for training from scratch
### Validation
During training, we periodically test the model on data it hasn't seen:
```
Epoch 1: Training loss: 2.5, Validation loss: 2.6
Epoch 2: Training loss: 1.8, Validation loss: 1.9
Epoch 3: Training loss: 1.2, Validation loss: 1.3
Epoch 4: Training loss: 0.8, Validation loss: 1.5 ← Overfitting!
```
When validation loss stops improving or gets worse, the model is overfitting.
## Types of Training
### Training from Scratch
Starting with a completely random model:
* Needs massive amounts of data
* Takes significant time and compute
* Used by companies creating base models
### Fine-tuning
Starting with a pre-trained model and adapting it:
* Needs less data (hundreds to thousands of examples)
* Much faster training
* What AI Training does for you
### Few-shot Learning
Teaching a model with just a few examples:
* Uses special techniques like prompt engineering
* Good for quick prototypes
* Limited to certain tasks
## What Makes Training Faster
### GPU Acceleration
GPUs can do thousands of calculations simultaneously:
* **CPU**: Processes sequentially, like reading a book
* **GPU**: Processes in parallel, like scanning a page
A task that takes hours on CPU might take minutes on GPU.
### Mixed Precision Training
Using lower precision numbers when possible:
* Full precision: 32-bit numbers (very accurate, slower)
* Mixed precision: 16-bit where possible (less accurate, faster)
* The model automatically uses full precision where needed
### Gradient Checkpointing
Trading memory for computation:
* Normal: Keep all calculations in memory
* Checkpointing: Recalculate some things to save memory
* Allows training larger models on smaller hardware
### Efficient Attention
For transformer models, optimizations like FlashAttention make training much faster by reorganizing how calculations are done.
## Monitoring Training
During training, you'll see metrics like:
```
Epoch 1/5
Step 100/500: loss=2.341, lr=5e-5, grad_norm=1.23
Step 200/500: loss=1.892, lr=5e-5, grad_norm=0.98
Step 300/500: loss=1.623, lr=5e-5, grad_norm=0.87
```
What these mean:
* **loss**: How wrong the model is (lower is better)
* **lr**: Current learning rate
* **grad\_norm**: Size of weight updates (should stay stable)
## Common Training Patterns
### Loss Curves
What you want to see:
* Training loss steadily decreasing
* Validation loss following training loss
* Both eventually plateauing
Warning signs:
* Loss increasing or exploding
* Validation loss increasing while training decreases
* Extremely noisy or unstable loss
### Learning Rate Scheduling
The learning rate often changes during training:
1. **Warmup**: Start with tiny learning rate, gradually increase
2. **Peak**: Train at optimal learning rate
3. **Decay**: Slowly reduce to fine-tune
This helps the model learn general patterns first, then refine details.
## Hardware Considerations
### Memory Requirements
Models need memory for:
* Model weights
* Gradients (weight updates)
* Optimizer states
* Activations (intermediate calculations)
Rough estimates:
* Small models (BERT): 4-8 GB
* Medium models (GPT-2): 8-16 GB
* Large models: 24GB+
### Training Speed
Typical training times:
* **Text Classification**: Minutes to hours
* **Image Classification**: Hours to days
* **Language Models**: Days to weeks
* **Large Models**: Weeks to months
## What Happens After Training
1. **Model Checkpoints**: Saved snapshots of the model at different stages
2. **Best Model Selection**: Usually the checkpoint with lowest validation loss
3. **Model Export**: Converting to format for deployment
4. **Quantization** (optional): Reducing model size for faster inference
## Next Steps
Ready to put this knowledge to use?
Pick between UI, CLI, or API
Understanding different architectures