How to Run Qwen 3.5-27B on 12GB VRAM: Complete Setup Guide

Mar 27, 2026

I stared at the error message, frustrated. My RTX 4070 with 12GB VRAM was supposed to be decent for gaming, but every time I tried loading Qwen 3.5-27B, I hit the same wall:

OutOfMemoryError: CUDA out of memory. Tried to allocate 52.00 GiB

52GB. My card has 12GB. The math wasn’t working in my favor.

The Reality Check

Let me be clear about what we’re dealing with:

Qwen 3.5-27B has approximately 27 billion parameters
At FP16 (16-bit float), that’s roughly 54GB of memory needed
At FP32 (32-bit float), we’re talking 108GB

Consumer GPUs max out at 24GB (RTX 4090). My RTX 4070 has 12GB. Enterprise cards like A100s have 40-80GB but cost $10,000+.

So how did I end up with a working setup?

The Solution: CPU Offloading

The key insight is simple: you don’t need everything in VRAM.

Modern inference frameworks support “offloading” - keeping part of the model on the GPU and the rest in system RAM. The CPU handles computation for offloaded layers.

Here’s what finally worked:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    device_map="auto",  # The magic happens here
    offload_folder="offload",
    torch_dtype="auto"
)

The device_map="auto" parameter tells transformers to:

Fill GPU memory first
Overflow remaining parameters to CPU
Handle the data transfers automatically

What Actually Happened

When I ran this on my system (Ryzen 9 7900X, RTX 4070 12GB, 64GB RAM), the logs told the story:

Loading checkpoint shards: 100%|██████████| 13/13 [02:34<00:00, 11.88s/it]
Some parameters are on the meta device because they were offloaded to the cpu.

VRAM Usage: 11.4/12.0 GB
GPU Utilization: 100%
CPU Usage: 1100% (11 cores active)

The model loaded. But here’s the trade-off:

+------------------+------------------+------------------+
| Metric           | Full GPU (A100)  | Offloaded (12GB) |
+------------------+------------------+------------------+
| Time to first    | ~0.5 seconds     | ~25 minutes      |
| token            |                  |                  |
+------------------+------------------+------------------+
| Tokens/second    | 50-100           | 0.5-1            |
+------------------+------------------+------------------+
| Total response   | 2-5 seconds      | 25-30 minutes    |
| (500 tokens)     |                  |                  |
+------------------+------------------+------------------+

Yes, you read that right. 25 minutes per response.

Why Does This Work?

Understanding the architecture helps explain the trade-off:

+-------------------+     +-------------------+
|    GPU (12GB)     |     |   System RAM      |
+-------------------+     +-------------------+
| Layer 1-8         |     | Layer 9-32        |
| Embeddings        |     | Remaining params  |
| Attention heads   |     | (offloaded)       |
| (hot path)        |     |                   |
+-------------------+     +-------------------+
        ↑                          ↑
        │                          │
   Fast compute              Slow compute
   (microseconds)            (milliseconds)
        │                          │
        └────────── PCIe ──────────┘
              Bus Transfer
              (nanoseconds)

When generating text, each layer needs to process the sequence. GPU layers fly through in microseconds. Offloaded layers require:

Data transfer from RAM to CPU
CPU computation
Transfer back to GPU for next layer

This overhead compounds across 32 layers and thousands of tokens.

Alternative Approaches

Before settling on CPU offloading, I tried several other methods:

1. Quantization (4-bit/8-bit)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    quantization_config=quantization_config,
    device_map="auto"
)

This reduced VRAM to ~8GB, but I lost inference quality. The model started hallucinating more frequently and lost nuanced reasoning capabilities.

2. vLLM with Memory Optimization

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-32B-Instruct \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 2048

vLLM’s PagedAttention algorithm helps with batching multiple requests, but for a single user scenario, the memory savings weren’t enough to fit the full model in 12GB.

3. Model Splitting Across Multiple GPUs

Not applicable - I only have one GPU. But if you have two 12GB cards, you could use:

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B-Instruct",
    device_map="balanced",  # Distributes across available GPUs
    max_memory={0: "12GB", 1: "12GB"}
)

What You Actually Need

After much experimentation, here’s the minimum viable setup:

CPU: 8+ cores (more is better)
RAM: 64GB minimum (model alone needs ~30GB)
GPU: 12GB VRAM (NVIDIA, for CUDA)
Storage: 100GB+ SSD (model weights are large)

The CPU is critical. My Ryzen 9 7900X has 12 cores, and I saw 11 cores pegged at 100% during inference. Without that CPU headroom, inference would be even slower or impossible.

Lessons Learned

Offloading is a trade-off, not a magic bullet. You gain ability to run the model but sacrifice speed dramatically.
System RAM matters more than you think. I initially tried with 32GB and ran into OOM. Upgraded to 64GB and everything worked.
CPU choice affects inference speed. AVX-512 support (AMD Ryzen 7000 series, Intel 12th gen+) significantly speeds up CPU computation.
Thermal management is real. Running at 100% GPU and 90%+ CPU for 25 minutes generates serious heat. My case fans sounded like a jet engine.
Batch size matters. I could only process one request at a time. Any attempt at batching caused memory errors.

When This Makes Sense

Running a 27B model on 12GB VRAM isn’t practical for production use cases. But it’s valuable for:

Experimentation: Testing model behavior without cloud costs
Learning: Understanding how LLM inference actually works
Development: Building and testing prompts locally
Privacy-sensitive applications: Data never leaves your machine

When to Use Cloud Instead

For anything requiring:

Response times under 10 seconds
Multiple concurrent requests
Production reliability
Cost-effective scaling

Cloud APIs (OpenAI, Anthropic, or even cloud-hosted Qwen) are far more practical. The electricity cost alone for running your GPU at full tilt for hours exceeds most API costs.

Final Thoughts

I spent a weekend figuring this out. Was it worth it? Absolutely. I learned more about GPU memory management, model architecture, and inference optimization than I ever would from reading documentation.

But I also learned when to fold. For daily use, I stick with smaller models (Llama 3.2-8B, Qwen 2.5-7B) that fit entirely in VRAM. The 27B model stays loaded for occasional experimentation when I need that extra reasoning capability.

The fact that it works at all on consumer hardware is remarkable. Five years ago, running anything over 7B locally was science fiction. Now it’s just slow.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!