Fine-Tuning Open-Weights Models with QLoRA & Unsloth

While prompting and RAG solve many context-grounding tasks, adapting open-weights LLMs (such as Llama 3.1 8B or Qwen 2.5 7B) for custom JSON output schemas, specialized domain nomenclature, or unique tone requires fine-tuning. Full parameter fine-tuning of an 8-billion parameter model requires over 80GB of VRAM just to store optimizer states. QLoRA (Quantized Low-Rank Adaptation) combined with Unsloth makes it possible to fine-tune state-of-the-art open models on a single consumer GPU (16GB VRAM) with zero accuracy degradation.

Full fine-tuning updates 100% of weights; LoRA freezes base weights and trains lightweight low-rank matrices; QLoRA quantizes the base model to 4-bit NormalFloat while preserving gradient precision.

The mathematics of LoRA and QLoRA

For a linear layer with weight matrix W0 of shape (d x k), standard gradient updates modify W0 directly. Low-Rank Adaptation (LoRA) freezes W0 and decomposes the update matrix dW into two low-rank matrices A and B:

W = W0 + dW = W0 + (B x A)
where B is (d x r), A is (r x k), and rank r << min(d, k)

By choosing a rank r = 16 or r = 32, the number of trainable parameters is reduced by over 99% (e.g. from 8,000,000,000 parameters to just 20,000,000 parameters).

QLoRA extends this by introduce three key innovations:

1. 4-bit NormalFloat (NF4): An information-theoretically optimal quantile quantization data type for normally distributed weights.
2. Double Quantization (DQ): Quantizes the quantization constants themselves, saving 0.37 bits per parameter.
3. Paged Optimizers: Uses CUDA Unified Memory to automatically page memory spikes between GPU VRAM and CPU RAM during gradient updates.

GPU VRAM breakdown: Full vs LoRA vs QLoRA

+-----------------------+-------------------+--------------------+
| Training Approach     | Base Weight VRAM  | Optimizer/Grad VRAM| Total VRAM (8B)  |
+-----------------------+-------------------+--------------------+------------------+
| Full 16-bit Fine-tune | 16 GB             | 64 GB              | ~80 GB VRAM      |
| Standard 16-bit LoRA  | 16 GB             | 2 GB               | ~24 GB VRAM      |
| 4-bit QLoRA           | 5.5 GB            | 1.5 GB             | ~9.5 GB VRAM     |
| QLoRA + Unsloth       | 5.5 GB            | 0.8 GB             | ~7.2 GB VRAM     |
+-----------------------+-------------------+--------------------+------------------+

Why Unsloth? Manual CUDA kernel optimization

Standard HuggingFace peft and trl implementations suffer from overhead during cross-entropy loss computation and RoPE embedding calculations. Unsloth rewrites these inner loops in custom Triton / CUDA kernels, yielding:

• 2x-5x faster training speeds compared to standard HuggingFace PEFT.
• 60% reduction in VRAM overhead, allowing batch size doubling.
• Exact zero loss accuracy trade-off (uses identical math, just faster execution).

Complete Python implementation with Unsloth & SFTTrainer

import torch
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# 1. Configuration
max_seq_length = 2048
dtype = None # Auto detection (Float16 or Bfloat16)
load_in_4bit = True # 4-bit QLoRA Quantization

# 2. Load Base Model & Tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

# 3. Add LoRA Adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0, # Optimized for 0 dropout in Unsloth
    bias="none",
    use_gradient_checkpointing="unsloth", # 30% VRAM savings
)

# 4. Prepare Instruction Dataset
dataset = load_dataset("philschmid/dolly-15k-curated-multilingual", split="train")

# 5. Initialize Trainer
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False, # Packs multiple short sequences for speed
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
    ),
)

# 6. Execute Fine-Tuning Run
trainer_stats = trainer.train()

# 7. Save Model LoRA Adapters
model.save_pretrained("llama3_custom_lora")
tokenizer.save_pretrained("llama3_custom_lora")

Exporting fine-tuned models to GGUF / Ollama for local inference

Once fine-tuning is completed, Unsloth allows 1-click export directly to GGUF format for local deployment via Ollama or llama.cpp:

# Export to 16-bit GGUF or quantized Q4_K_M GGUF
model.save_pretrained_gguf(
    "llama3_custom_gguf", 
    tokenizer, 
    quantization_method="q4_k_m"
)

Best practices for production fine-tuning

1. Start with quality data over quantity: 1,000 high-quality, human-verified instruction pairs outperform 50,000 noisy scraped samples.
2. Use Target Modules for all linear projections: Apply LoRA adapters to both attention weights (q, k, v, o) and MLP layers (gate, up, down).
3. Monitor loss curve plateau: If evaluation loss starts increasing while training loss decreases, stop immediately to avoid overfitting.
4. Export to GGUF for edge deployment: Quantize fine-tuned models to 4-bit or 5-bit GGUF for low-latency local execution.