How to Optimize llama.cpp for Maximum Inference Speed: A Complete Guide

Mar 15, 2026

I watched my model generate tokens at 3 tokens per second. Painful. I had a decent GPU, a quantized model, and llama.cpp - so why was inference so slow?

After hours of tweaking flags, recompiling, and reading through GitHub issues, I finally got it running at 40+ tokens per second. The difference was dramatic. Here’s what I learned about squeezing maximum performance out of llama.cpp.

The Speed Problem with llama.cpp

llama.cpp is famous for running LLMs on consumer hardware. But out of the box? It’s often configured conservatively. The default settings prioritize compatibility over speed.

I was running a 27B model on an RTX 3090 with these results:

text title="Before optimization"
Load time: 45 seconds
Token generation: 3 tokens/second
Prompt processing: 8 tokens/second

After optimization:

text title="After optimization"
Load time: 12 seconds
Token generation: 42 tokens/second
Prompt processing: 150+ tokens/second

That’s a 14x improvement in generation speed. Here’s how I did it.

The Quick Answer

If you just want the commands, here’s the optimal configuration:

./llama-cli \
  -m model-Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --flash-attn \
  -b 512 \
  -ub 512 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  -p "Your prompt here"

But let me explain each flag so you understand what’s happening.

GPU Offloading: The Single Biggest Gain

The -ngl (number of GPU layers) flag is the most important optimization. It controls how many model layers run on GPU vs CPU.

# CPU only (default) - SLOW
./llama-cli -m model.gguf -p "Hello"
# ~3 tokens/sec on 27B model

# Partial GPU offload
./llama-cli -m model.gguf -ngl 20 -p "Hello"
# ~15 tokens/sec

# Full GPU offload - FAST
./llama-cli -m model.gguf -ngl 99 -p "Hello"
# ~40+ tokens/sec

Why 99? Just set it higher than your model’s layer count. llama.cpp will offload all available layers. For most models, any number above your layer count works.

How to find your model’s layer count:

./llama-cli -m model.gguf --log-disable 2>&1 | grep "layers"

# Output example:
# n_layer = 32

Set -ngl to at least that number, or just use 99 to be safe.

Compile-Time Optimizations

Runtime flags are only half the battle. Compile-time options unlock additional performance gains.

CUDA Support

If you have an NVIDIA GPU, you must compile with CUDA:

cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

The default CPU-only build is incredibly slow. CUDA gives you 10-50x speedup depending on model size.

Flash Attention

Flash Attention reduces memory reads/writes during attention computation. It’s a significant speedup for longer contexts:

cmake -B build -DGGML_CUDA=ON -DGGML_FLASH_ATTN=ON
cmake --build build --config Release -j

Then enable it at runtime:

./llama-cli -m model.gguf -ngl 99 --flash-attn -p "Hello"

Check Your Build

Verify your build has the optimizations you need:

./llama-cli --version

# Look for:
# - CUDA support
# - Flash Attention support
# - cuBLAS support

Quantization: Speed vs Quality Tradeoff

Choosing the right quantization level impacts both speed and quality. After extensive testing, here’s what I found:

Q4_K_M:
  model_size: "~30% of FP16"
  speed: "Fastest"
  quality_loss: "Minimal for most tasks"
  recommendation: "Best for everyday use"

Q5_K_M:
  model_size: "~35% of FP16"
  speed: "Slightly slower than Q4"
  quality_loss: "Nearly imperceptible"
  recommendation: "When quality matters"

Q8_0:
  model_size: "~50% of FP16"
  speed: "Noticeably slower"
  quality_loss: "Negligible"
  recommendation: "Maximum quality, slower inference"

For my coding assistant, I use Q4_K_M. The quality difference from Q5 is barely noticeable, but the speed improvement is real.

# Example: Download Q4_K_M quantization
wget https://huggingface.co/model-repo/model-Q4_K_M.gguf

KV Cache Quantization

Here’s an optimization many people miss. The KV cache grows with context length, and storing it in FP16 wastes memory.

Enable KV cache quantization:

./llama-cli \
  -m model-Q4_K_M.gguf \
  -ngl 99 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  -c 16384 \
  -p "Long prompt here"

This reduces KV cache memory by ~75% with minimal quality impact:

text title="KV cache memory savings"
Context 16K tokens, 27B model:

FP16 KV cache: ~8GB VRAM
Q4 KV cache: ~2GB VRAM

Memory saved: 6GB (can use larger batch or context)

Batch Size Optimization

Batch size affects prompt processing speed. Higher batch sizes process prompts faster but use more memory:

./llama-cli \
  -m model.gguf \
  -ngl 99 \
  -b 512 \
  -ub 512 \
  -p "Your prompt"

# -b: batch size for prompt processing
# -ub: physical batch size (uncached batch)

I set both to 512 for my 24GB GPU. For smaller GPUs, try 256 or 128.

Finding Optimal Parameters Automatically

llama.cpp includes a tool to find optimal offload parameters:

./llama-fit-params \
  -m model-Q4_K_M.gguf \
  --ctx-size 8192

# Output suggests optimal -ngl, -b values

This tool tests different configurations and finds the best balance for your hardware.

Speculative Decoding

For even more speed, speculative decoding uses a smaller “draft” model to guess tokens, then verifies with the main model:

./llama-cli \
  -m model-27b-Q4_K_M.gguf \
  -md model-7b-Q4_K_M.gguf \
  -ngl 99 \
  -ngld 99 \
  -p "Your prompt"

This can give 20-50% speedup on top of other optimizations, but requires a compatible draft model.

Common Mistakes I Made

Mistake 1: Forgetting -ngl Flag

I ran llama.cpp for weeks before realizing I wasn’t offloading to GPU. The default is CPU-only, which is incredibly slow.

# WRONG - uses CPU by default
./llama-cli -m model.gguf

# RIGHT - offloads to GPU
./llama-cli -m model.gguf -ngl 99

Mistake 2: Using Wrong Quantization

I initially downloaded Q8 models thinking higher quality was always better. For inference speed, Q4_K_M is often the sweet spot.

# Look at the filename:
# model-Q4_K_M.gguf  <- Good for speed
# model-Q8_0.gguf    <- Slower, larger
# model-FP16.gguf    <- Slowest, full precision

Mistake 3: Ignoring Build Options

I used pre-built binaries that didn’t have CUDA or Flash Attention compiled in. Building from source with the right flags made a huge difference.

# Pre-built binaries often miss optimizations
# Build with:
cmake -B build -DGGML_CUDA=ON -DGGML_FLASH_ATTN=ON
cmake --build build --config Release -j

Putting It All Together

Here’s my complete optimized setup:

# 1. Build with optimizations
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DGGML_FLASH_ATTN=ON
cmake --build build --config Release -j

# 2. Download optimized model
wget https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct-GGUF/resolve/main/qwen2.5-coder-32b-instruct-q4_k_m.gguf

# 3. Run with all optimizations
./build/bin/llama-cli \
  -m qwen2.5-coder-32b-instruct-q4_k_m.gguf \
  -ngl 99 \
  -c 8192 \
  --flash-attn \
  -b 512 \
  -ub 512 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --temp 0.7 \
  -p "Write a Python function to merge two sorted lists"

Performance Checklist

Before running llama.cpp, verify these settings:

text title="Optimization checklist"
[ ] Compiled with CUDA support (-DGGML_CUDA=ON)
[ ] Compiled with Flash Attention (-DGGML_FLASH_ATTN=ON)
[ ] Using -ngl 99 (or number >= model layers)
[ ] Using Q4_K_M or Q5_K_M quantization
[ ] Enabled --flash-attn at runtime
[ ] Set appropriate batch size (-b 512)
[ ] Enabled KV cache quantization (--cache-type-k q4_0)

Summary

The key optimizations for llama.cpp speed:

Optimization	Impact	Priority
GPU offloading (-ngl 99)	10-50x	Critical
CUDA compilation	10-50x	Critical
Flash Attention	1.2-1.5x	High
Q4_K_M quantization	2-3x	High
KV cache quantization	Memory savings	Medium
Batch size tuning	1.1-1.3x	Low
Speculative decoding	1.2-1.5x	Optional

The single most important thing: make sure you’re using -ngl 99 and compiled with CUDA support. Those two changes alone will give you 90% of the possible speedup.

Context Length Impact: Larger context windows increase KV cache memory. Use --cache-type-k q4_0 to mitigate this.
Multi-GPU Support: llama.cpp supports multi-GPU with -sm row for splitting models across GPUs.
Memory Mapping: Large models benefit from -r (repack) option for better memory layout.
Server Mode: Use llama-server instead of llama-cli for API-style access with all optimizations applied.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Reddit Discussion: llama.cpp optimization
👨‍💻 llama.cpp GitHub
👨‍💻 Flash Attention Paper

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!