CPU Offloading for LLM Inference: Run Large Models on Limited GPU Memory

Mar 27, 2026

I tried running Qwen 2.5 32B on my RTX 3060 with 12GB VRAM. The model alone needs about 64GB in FP16. I expected an out-of-memory error.

Instead, the model loaded. Then I waited 25 minutes for the first response.

The Error That Wasn’t an Error

Loading model weights: 100%|████████████| 64/64 [02:15<00:00]
Some parameters are on the meta device because they were offloaded to the cpu

At first, I thought something was wrong. “Meta device”? Offloaded to CPU? Was this even working?

Then I checked my system monitor:

GPU Memory: 11.8GB / 12GB (98%)
CPU Usage: 1100% (11 of 12 cores pegged)
RAM Usage: 48GB / 64GB

The model was running. Just… slowly. Very slowly.

What Is CPU Offloading?

CPU offloading is a technique where model parameters that don’t fit in GPU VRAM are stored in system RAM instead. During inference, these parameters are either:

Transferred temporarily to GPU when needed, then discarded
Computed directly on CPU without GPU involvement

Here’s a simplified view:

┌─────────────────────────────────────────────────────────┐
│                    GPU VRAM (12GB)                      │
│  ┌─────────────────────────────────────────────────┐   │
│  │     Layer 0-19: "Hot" layers (always resident)  │   │
│  │     Attention weights, FFN, LayerNorms          │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                         │
                         │ PCIe Bus (slower)
                         ▼
┌─────────────────────────────────────────────────────────┐
│                    System RAM (64GB)                    │
│  ┌─────────────────────────────────────────────────┐   │
│  │     Layer 20-63: "Cold" layers (offloaded)      │   │
│  │     Swapped in/out during forward pass          │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

The trade-off is stark: accessibility vs. speed. I could now run models that were previously impossible on my hardware, but each inference took minutes instead of seconds.

Why Would Anyone Accept 25-Minute Responses?

At first, this seemed useless. Who would wait 25 minutes for a chat response?

Then I realized several valid use cases:

Experimentation before buying - Testing a model architecture before investing in better hardware
Infrequent queries - A daily report generation where 25 minutes is acceptable
Batch processing - Overnight inference jobs where speed doesn’t matter
Learning - Understanding how model layers work by watching them execute

The key insight: slow inference is infinitely better than no inference.

Implementation: The Easy Way

The transformers library makes CPU offloading nearly automatic:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    device_map="auto",           # Magic happens here
    offload_folder="offload",     # Disk cache for spills
    torch_dtype="auto"
)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-32B-Instruct")

# See where each layer ended up
print(model.hf_device_map)

The device_map="auto" parameter tells the library to figure out the best distribution. Here’s what it chose for my setup:

{'disk': ['model.embed_tokens'],
 0: ['model.layers.0-19', 'model.norm', 'lm_head'],
 'cpu': ['model.layers.20-31']}

Disk, GPU (device 0), and CPU - all three tiers of storage in use.

Implementation: Fine-Grained Control

When I needed more control over which layers went where, I used the accelerate library directly:

from accelerate import infer_auto_device_map, dispatch_model
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True
)

# Define memory constraints
max_memory = {
    0: "10GB",      # Leave 2GB VRAM for activations
    "cpu": "50GB"   # System RAM allocation
}

device_map = infer_auto_device_map(model, max_memory=max_memory)

# Dispatch model across devices
model = dispatch_model(model, device_map)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-32B-Instruct")

inputs = tokenizer("Explain CPU offloading", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

This approach let me reserve 2GB of VRAM for activations (intermediate computations during inference). Without this buffer, I’d hit OOM errors during generation.

The llama.cpp Alternative

For even better CPU performance, I switched to llama.cpp. Its GGUF format with quantization made CPU offloading significantly faster:

# Download quantized model (4-bit instead of 16-bit)
# 32B model shrinks from ~64GB to ~18GB

./llama-cli \
    -m qwen2.5-32b-instruct-q4_k_m.gguf \
    -ngl 20 \           # Number of GPU layers (20 of 64)
    -t 12 \             # CPU threads (match your cores)
    -c 4096 \           # Context window
    --temp 0.7 \
    -p "Explain CPU offloading in LLMs"

Key parameters:

-ngl 20: Keep 20 layers on GPU, rest on CPU
-t 12: Use 12 CPU threads for parallel computation
q4_k_m: 4-bit quantization, medium quality

With quantization + offloading, the same 32B model now responds in 3-5 minutes instead of 25. Still slow, but actually usable for interactive work.

What I Got Wrong Initially

Mistake 1: Not Enough System RAM

My first attempt used a 32GB RAM machine. The model barely loaded, then crashed during inference when memory spiked. Rule of thumb: system RAM should be 2-3x the model size.

Mistake 2: Using HDD Instead of NVMe

When VRAM and RAM both fill up, parameters spill to disk. On my old HDD, this was catastrophic - responses took hours. Moving to NVMe SSD reduced disk-offloaded inference by 60%.

Mistake 3: Wrong Layer Distribution

Initially, I offloaded all layers evenly (half GPU, half CPU). But attention layers benefit more from GPU than FFN layers. I should have kept attention on GPU:

# Keep attention on GPU, offload FFN
device_map = {}
for i, layer in enumerate(model.model.layers):
    if i < 20:
        # Keep everything on GPU for first 20 layers
        device_map[f"model.layers.{i}"] = 0
    else:
        # For remaining layers, try to keep attention on GPU
        device_map[f"model.layers.{i}.self_attn"] = 0  # GPU
        device_map[f"model.layers.{i}.mlp"] = "cpu"     # CPU (FFN)

This optimization reduced inference time by about 30%.

Mistake 4: Ignoring CPU Architecture

CPU performance varies dramatically. My Ryzen 9 5900X with AVX-2 was 3x faster than an older i7 without AVX. Newer CPUs with AVX-512 would be even faster. CPU choice matters when you’re offloading.

Memory Requirements by Model Size

Here’s what I’ve found practical:

Model Size | FP16 VRAM | Q4 VRAM  | System RAM Needed | Min GPU
-----------|-----------|----------|-------------------|--------
7B         | 14GB      | 4-5GB    | 16GB              | 6GB
14B        | 28GB      | 8-10GB   | 32GB              | 8GB
27B        | 54GB      | 16-18GB  | 64GB              | 12GB
32B        | 64GB      | 18-20GB  | 64GB              | 16GB
70B        | 140GB     | 40-45GB  | 128GB             | 24GB

The “Min GPU” column assumes you’ll accept slow inference. If you want real-time responses, double or triple those GPU requirements.

When to Use CPU Offloading

Use it when:

You need to test a model before buying hardware
Your inference frequency is low (hours between queries)
You’re doing batch processing overnight
Budget constraints prevent GPU upgrades

Don’t use it when:

You need real-time chat responses
You’re serving multiple concurrent users
Your use case is production with SLAs
Electricity costs matter (CPU at 100% for 25 minutes = expensive)

Summary

CPU offloading made my 12GB RTX 3060 capable of running a 32B parameter model. The trade-off is severe - 25-minute response times for what takes seconds on proper hardware - but it works.

For learning, experimentation, and occasional inference, this is invaluable. For production use, you need better hardware or smaller models.

The real win came from combining quantization with offloading. Q4 quantization reduced the model size by 4x, and llama.cpp’s optimized CPU kernels made offloaded computation 5x faster. Together, they transformed an impractical 25-minute wait into a manageable 3-5 minute response time.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Reddit Discussion: Running Qwen 2.5 32B on Limited VRAM
👨‍💻 Hugging Face Accelerate: Big Model Inference
👨‍💻 llama.cpp: LLM Inference in C++
👨‍💻 Transformers Device Map Documentation

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!