Skip to content
BSWEN
Blog

Fine-Tune Any LLM for Free: Complete Guide to Using Google Colab T4 GPU

Cowrie
Cowrie
Dev @ Bswen

I wanted to fine-tune a language model for my specific use case. Then I saw the cloud GPU prices - $1-3 per hour adds up fast. A local GPU with enough VRAM costs thousands of dollars. I almost gave up.

Then I discovered I could do this for free using Google Colab’s T4 GPU.

The Problem

Fine-tuning LLMs is expensive. Cloud providers charge premium rates for GPU instances. A single training run can cost tens or hundreds of dollars. For hobbyists, students, or small teams, this cost barrier is prohibitive.

I needed a way to customize a model without spending money. Google Colab offers free T4 GPU access (16GB VRAM). But 16GB isn’t enough for full fine-tuning of modern LLMs.

The Solution

The answer is LoRA (Low-Rank Adaptation) combined with memory optimization techniques:

  • LoRA - Fine-tunes only a small subset of parameters instead of the entire model
  • Unsloth - An optimized training library that reduces memory usage by 60% through gradient checkpointing
  • 4-bit quantization - Loads the base model in compressed form
  • 8-bit AdamW optimizer - Reduces optimizer memory footprint

Together, these techniques let you train models up to 7B parameters on free Colab hardware.

How Long Does It Take?

Training time depends on model size:

Model SizeExamplesTraining Time
Small (~1.5B)DeepSeek-R1 1.5B, Qwen2.5 1.5B, Llama 3.2 1B~20 minutes
Medium (~3B)Qwen2.5 3B, Phi-3 Mini~40 minutes
Large (~7B)Qwen2.5 7B, DeepSeek-R1 7B, Mistral 7B~80 minutes

For beginners, I recommend starting with 1.5B or 3B models. They train fast and fit comfortably within free tier limits.

Step-by-Step Tutorial

Step 1: Enable GPU Runtime in Colab

Open a new Google Colab notebook. Go to Runtime > Change runtime type > T4 GPU. This gives you access to a free NVIDIA T4 GPU with 16GB VRAM.

Verify GPU is available:

check_gpu.py
import torch
print(f"GPU available: {torch.cuda.is_available()}")
print(f"GPU name: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB")

Step 2: Install Unsloth

Unsloth is the key library that makes free fine-tuning practical. Install it with:

install_unsloth.py
# Install Unsloth - check official docs for latest version
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.27" "trl<0.9.0" "peft<0.14.0" "accelerate<0.33.0" "bitsandbytes<0.44.0"

Note: Unsloth updates frequently. Always check the official documentation for the latest installation command.

Step 3: Load Your Model

Load the base model with 4-bit quantization. This compresses the model to fit in limited VRAM:

load_model.py
from unsloth import FastLanguageModel
import torch


# Load model with memory optimization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen2.5-3B",  # Or try: "unsloth/Llama-3.2-1B"
    max_seq_length = 2048,           # Max sequence length
    dtype = None,                    # Auto-detect (Float16 for T4)
    load_in_4bit = True,             # Use 4-bit quantization
)


print(f"Model loaded successfully!")

I chose Qwen2.5-3B as my starting point. It’s a good balance of capability and size. The 4-bit quantization reduces memory usage dramatically without significantly hurting performance.

Step 4: Add LoRA Adapters

LoRA adds trainable parameters on top of the frozen base model. Only these new parameters get updated during training:

add_lora.py
# Add LoRA adapters for efficient fine-tuning
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,                    # LoRA rank - higher = more parameters
    target_modules = [         # Which layers to adapt
        "q_proj", "k_proj",
        "v_proj", "o_proj",
    ],
    lora_alpha = 16,           # LoRA scaling factor
    lora_dropout = 0,          # Dropout for regularization
    bias = "none",             # No bias in LoRA layers
    use_gradient_checkpointing = "unsloth",  # Critical for memory savings
    random_state = 3407,
)


print(f"LoRA adapters added!")
print(f"Trainable parameters: {model.print_trainable_parameters()}")

The use_gradient_checkpointing = "unsloth" setting is critical. It saves about 60% VRAM by recomputing activations during backpropagation instead of storing them.

Step 5: Prepare Your Training Data

Format your training data as a list of prompts. Here’s a simple example:

prepare_data.py
# Example training data
train_dataset = [
    {"text": "### Instruction:\nExplain quantum computing.\n\n### Response:\nQuantum computing uses quantum bits or qubits..."},
    {"text": "### Instruction:\nWhat is machine learning?\n\n### Response:\nMachine learning is a subset of AI..."},
    # Add more examples...
]


# Convert to HuggingFace dataset format
from datasets import Dataset
dataset = Dataset.from_list(train_dataset)


print(f"Dataset size: {len(dataset)} examples")

For real projects, I recommend at least 100-1000 examples. More data generally produces better results, but even 50 carefully crafted examples can work for specific tasks.

Step 6: Configure Training Arguments

Set up training parameters optimized for the T4 GPU’s limitations:

training_args.py
from transformers import TrainingArguments


training_args = TrainingArguments(
    per_device_train_batch_size = 2,     # Small batch for limited VRAM
    gradient_accumulation_steps = 4,      # Accumulate gradients (effective batch = 8)
    max_steps = 60,                        # Number of training steps
    learning_rate = 2e-4,                  # Typical LoRA learning rate
    fp16 = not torch.cuda.is_bf16_supported(),  # Use FP16 on T4
    bf16 = torch.cuda.is_bf16_supported(),       # Use BF16 if supported
    optim = "adamw_8bit",                  # 8-bit optimizer saves memory
    weight_decay = 0.01,                   # Regularization
    lr_scheduler_type = "linear",          # Learning rate schedule
    seed = 3407,
    output_dir = "outputs",
    logging_steps = 1,                     # Log every step
    save_strategy = "steps",               # Save checkpoints
    save_steps = 20,                       # Save every 20 steps
)

Key memory-saving settings:

  • per_device_train_batch_size = 2 - Small batches fit in limited VRAM
  • gradient_accumulation_steps = 4 - Compensates for small batches
  • optim = "adamw_8bit" - Uses 8-bit precision for optimizer states

Step 7: Train the Model

Now run the training:

train.py
from trl import SFTTrainer


trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = 2048,
    args = training_args,
)


# Start training
trainer.train()


print("Training complete!")

During training, Colab shows progress bars and loss values. The loss should decrease over time. If you see “CUDA out of memory” errors, reduce batch size or sequence length.

Step 8: Export to GGUF Format

After training, export the model to GGUF format for local inference:

export_gguf.py
# Save in GGUF format for llama.cpp, Ollama, etc.
model.save_pretrained_gguf(
    "my_finetuned_model",      # Output directory
    tokenizer,
    quantization_method = "q4_k_m"  # Good balance of size and quality
)


print("Model exported to GGUF format!")

The q4_k_m quantization creates a model that balances file size and quality. For better quality at larger size, use q5_k_m or q8_0.

Memory Optimization Tips

I learned these lessons the hard way:

Tip 1: Start with Smaller Models

7B models push the T4’s limits. I recommend 1.5B or 3B for your first attempts. They train faster and are more forgiving.

Tip 2: Use Gradient Checkpointing

Always enable gradient checkpointing. Without it, you’ll run out of memory on larger batches:

gradient_checkpointing.py
# This saves ~60% VRAM
use_gradient_checkpointing = "unsloth"

Tip 3: Reduce Sequence Length

If you hit OOM errors, try reducing max_seq_length. For many tasks, 1024 or even 512 works fine:

reduce_seq_length.py
max_seq_length = 1024  # Instead of 2048

Tip 4: Monitor VRAM Usage

Check VRAM during training:

monitor_vram.py
def print_vram():
    allocated = torch.cuda.memory_allocated(0) / 1024**3
    reserved = torch.cuda.memory_reserved(0) / 1024**3
    print(f"VRAM: {allocated:.1f}GB allocated, {reserved:.1f}GB reserved")


print_vram()

Tip 5: Save Checkpoints Frequently

Colab free tier disconnects after idle periods. Save often:

save_checkpoints.py
# In TrainingArguments
save_strategy = "steps"
save_steps = 20  # Save every 20 steps


# Also manually save after training
model.save_pretrained("checkpoints/final")
tokenizer.save_pretrained("checkpoints/final")

Troubleshooting Common Issues

CUDA Out of Memory

Symptoms: Training crashes with “CUDA out of memory” error.

Solutions:

  1. Reduce per_device_train_batch_size to 1
  2. Increase gradient_accumulation_steps to 8
  3. Reduce max_seq_length to 1024
  4. Use a smaller model (try 1.5B instead of 3B)

Colab Disconnection

Symptoms: Notebook disconnects during long training runs.

Solutions:

  1. Click the page occasionally to prevent idle timeout
  2. Keep the browser tab active
  3. Save checkpoints every 20-30 steps
  4. For 7B models, consider Colab Pro for longer sessions

Slow Training

Symptoms: Training seems unusually slow.

Solutions:

  1. Verify GPU is enabled: Runtime > Change runtime type > T4 GPU
  2. Check that model is on GPU: model.to('cuda')
  3. Ensure fp16 or bf16 training is enabled

Import Errors

Symptoms: “No module named ‘unsloth’” or similar errors.

Solutions:

  1. Reinstall packages in order (Unsloth first, then dependencies)
  2. Restart runtime after installation
  3. Check official Unsloth docs for latest installation commands

Using Your Fine-Tuned Model

After exporting to GGUF, you can use your model locally:

With llama.cpp

run_llamacpp.sh
# Clone and build llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make


# Run inference
./llama-cli -m my_finetuned_model/unsloth.Q4_K_M.gguf -p "Your prompt here"

With Ollama

run_ollama.sh
# Create Modelfile
echo 'FROM ./unsloth.Q4_K_M.gguf' > Modelfile


# Create Ollama model
ollama create my-model -f Modelfile


# Run inference
ollama run my-model "Your prompt here"

With Python (transformers)

run_transformers.py
from transformers import AutoModelForCausalLM, AutoTokenizer


model = AutoModelForCausalLM.from_pretrained("checkpoints/final")
tokenizer = AutoTokenizer.from_pretrained("checkpoints/final")


inputs = tokenizer("Your prompt here", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

Limitations of Free Tier

The free Colab tier has constraints:

  • VRAM: 16GB limits you to ~7B parameter models
  • Time: Sessions disconnect after idle periods
  • Usage limits: Google limits free GPU usage hours per day/week
  • Reliability: May be slower during peak times

For production workloads or larger models, consider:

  • Colab Pro ($10/month) - More GPU hours, better GPUs
  • RunPod, Lambda Labs - Pay-per-hour GPU rentals ($0.20-0.50/hour for T4)
  • Local GPU - RTX 3060 (12GB) or better for dedicated training

What I Learned

Free LLM fine-tuning is practical. The key insights:

  1. Model size matters - 1.5B-3B models work reliably on free T4
  2. Memory optimization is essential - Gradient checkpointing and 4-bit quantization are non-negotiable
  3. LoRA is the enabler - Without LoRA, full fine-tuning requires 10x more VRAM
  4. Save frequently - Colab disconnections happen; checkpoints save progress
  5. Start small - Begin with 1.5B models to learn the process

The barrier to LLM customization has dropped dramatically. What once required expensive hardware now runs on free cloud resources. Anyone with a Google account can create specialized models for their needs.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: