Skip to content
BSWEN
Blog

Can a 128GB MacBook Run 100B+ Parameter LLMs Locally? (Yes, Here's How)

Cowrie
Cowrie
Dev @ Bswen

Problem

I wanted to run large language models locally on my MacBook. Not small models - I’m talking about 100+ billion parameter models like Qwen 2.5 72B or even the massive Qwen 3.5 122B. But when I looked at GPU requirements online, I saw numbers that made my wallet hurt:

  • NVIDIA RTX 4090: 24GB VRAM (~$1,600+) - not enough for 100B models
  • NVIDIA A100: 80GB VRAM (~$10,000+) - maybe enough for one model
  • Running 100B+ models seemed to require multi-GPU setups or expensive cloud infrastructure

Then I found a Reddit thread where users claimed they were running 120B models on 128GB MacBooks. I was skeptical. How could a laptop run what enterprise GPUs struggle with?

The Answer

Yes, a 128GB MacBook with M4/M5 Max chip can run 100B+ parameter LLMs locally. Real users report:

  • Qwen 3.5 122B at Q4/Q5 quantization: runs smoothly
  • Nemotron-3 Super 120B: runs with ease
  • Even Qwen 3.5 390B: possible with SSD swap optimization (~12 tok/sec)

The key is understanding two technologies: Apple Silicon’s unified memory architecture and model quantization.

Why Traditional GPUs Struggle

Traditional GPU setups separate VRAM from system RAM. This creates a fundamental bottleneck:

Traditional Architecture:
+------------------+     +------------------+
|    CPU           |     |    GPU           |
|    System RAM    |     |    VRAM          |
|    64-128GB      |     |    8-24GB        |
+------------------+     +------------------+
         |                       |
         +------ Data Copy ------+
                (Slow)

When you run an LLM, the model weights need to be in VRAM for fast computation. A 100B parameter model at FP16 precision requires approximately 200GB of memory. No consumer GPU has this much VRAM.

I tried running a 70B model on my old RTX 3090 (24GB VRAM). The error was immediate:

Terminal window
CUDA out of memory. Tried to allocate 140.00 GiB

How Apple Silicon Solves This

Apple Silicon uses unified memory architecture (UMA). The CPU, GPU, and Neural Engine share the same memory pool:

Apple Silicon Architecture:
+------------------------------------------+
|              Unified Memory               |
|              128GB (M4/M5 Max)           |
|                                          |
|    +--------+  +--------+  +----------+  |
|    |  CPU   |  |  GPU   |  | Neural   |  |
|    |        |  |        |  | Engine   |  |
|    +--------+  +--------+  +----------+  |
+------------------------------------------+
              No data copying needed

This means:

  • No data copying between system RAM and GPU VRAM
  • M4 Max: 400GB/s+ memory bandwidth
  • M5 Max: 500GB/s+ memory bandwidth
  • The entire 128GB is available for model weights

But 128GB still isn’t enough for a 200GB model at FP16. That’s where quantization comes in.

Quantization: The Memory Multiplier

Quantization reduces model precision while maintaining acceptable accuracy. Here’s how different quantization levels affect model size:

Quantization Comparison (100B Parameter Model):


| Quantization | Bits Per Parameter | Model Size | Quality Impact    |
|--------------|-------------------|------------|-------------------|
| FP16         | 16                | ~200GB     | Baseline          |
| Q8           | 8                 | ~100GB     | Negligible        |
| Q5_K_M       | 5                 | ~62.5GB    | Minimal           |
| Q4_K_M       | 4                 | ~50GB      | Acceptable        |
| Q3_K_M       | 3                 | ~37.5GB    | Noticeable        |
| Q2_K         | 2                 | ~25GB      | Significant drop   |

For 100B+ models on 128GB, Q4_K_M or Q5_K_M quantization is the sweet spot. This leaves room for:

  • Context window (KV cache): 20-40GB for large contexts
  • System overhead: 10-15GB
  • Multiple sessions

Real-World Performance

I looked at what Reddit users actually reported:

One M5 Max 128GB owner stated:

“I run gpt-oss-120b, nemotron-3-super-120b-a12b, qwen3.5-122b-a10b, and qwen3-coder-next with ease and large contexts with Q4/Q5 quantization.”

Another user pushed even further:

“You can run qwen 3.5 390b on that easily with a few tweaks… using the 128gb and the SSD you would likely get ~12 tok/sec.”

An M4 Max 128GB user confirmed the practical value:

“Running really big models is really helpful when I need to.”

The 390B case uses SSD swap (virtual memory on SSD), which is slower but works. For in-memory inference, expect 30-50 tokens per second with 120B models.

Setting Up Ollama on MacBook

The easiest way to run these models is Ollama. Here’s how I set it up:

terminal
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh


# Pull a large model
ollama pull qwen2.5:72b


# Run it
ollama run qwen2.5:72b


# For even larger models (community uploads)
ollama run hf.co/bartowski/Qwen2.5-72B-Instruct-GGUF:Q4_K_M

Ollama automatically handles quantized models and optimizes for Apple Silicon.

Using MLX for Apple-Native Performance

For better Apple Silicon optimization, I use Apple’s MLX framework:

terminal
# Install MLX
pip install mlx mlx-lm


# Download a GGUF model
huggingface-cli download \
  Qwen/Qwen2.5-72B-Instruct-GGUF \
  qwen2.5-72b-instruct-q4_k_m.gguf \
  --local-dir ./models
inference.py
from mlx_lm import load, generate


# Load quantized model
model, tokenizer = load("./models/qwen2.5-72b-instruct-q4_k_m.gguf")


# Generate response
prompt = "Explain quantum computing in simple terms."
response = generate(model, tokenizer, prompt=prompt, max_tokens=500)
print(response)

Monitoring Memory Usage

When running large models, I monitor memory to avoid crashes:

terminal
# Real-time memory monitoring
watch -n 1 'sudo memory_pressure'


# Or use Activity Monitor
# Look for "Memory Used" and "Swap Used"

If you see swap usage climbing rapidly, your model is too large for comfortable in-memory inference.

Common Mistakes I Made

Mistake 1: Ignoring Context Window Overhead

When I first started, I thought model size was the only memory concern. I loaded a 100GB model and hit crashes at 32K context. The KV cache for large contexts consumes significant memory:

Memory Allocation for 120B Model at Q4:
- Model weights: ~50GB
- 32K context KV cache: ~15GB
- 128K context KV cache: ~60GB
- System overhead: ~10GB
- Total for 128K context: ~120GB (barely fits!)

Mistake 2: Choosing Wrong Quantization

I initially tried Q2 quantization to fit even larger models. The quality drop was severe:

Q2_K output (low quality):
"The quick brown fox jumps over the lazy dog and also quantum
computing is like very fast and stuff and computers are good..."


Q4_K_M output (good quality):
"The quick brown fox jumps over the lazy dog. This pangram
contains every letter of the English alphabet at least once."

For 100B+ models, stick to Q4_K_M or Q5_K_M. The quality difference is worth the memory trade-off.

Mistake 3: Not Using Apple-Native Tools

I initially tried CUDA-based frameworks, which run poorly on Mac. MLX and Ollama are optimized for Apple Silicon and deliver significantly better performance.

Cost Comparison

Here’s why I chose a MacBook over cloud alternatives:

Cost Analysis:


MacBook Pro M5 Max 128GB:
- Upfront cost: ~$3,500-$4,500
- No per-token fees
- No data egress charges
- Unlimited usage
- Break-even: 6-12 months of active use


Cloud GPU (equivalent performance):
- A100 80GB: ~$3-4/hour
- Running 8 hours/day: ~$720-960/month
- Plus API fees if using hosted services
- Data always leaves your machine

When This Makes Sense

A 128GB MacBook is worth it for local LLM inference if you:

  • Need privacy (no data leaves your machine)
  • Want no rate limits or API throttling
  • Prefer one-time cost over recurring fees
  • Need offline capability
  • Want to experiment with model fine-tuning

If you only need occasional inference, cloud APIs might be cheaper. But for regular use, the MacBook pays for itself.

Summary

In this post, I explained how Apple Silicon MacBooks with 128GB unified memory can run 100B+ parameter LLMs locally. The key points are:

  1. Unified Memory Architecture eliminates the VRAM bottleneck that traditional GPUs face
  2. Q4/Q5 quantization reduces model size by 4-5x while maintaining acceptable quality
  3. Real users run 120B models comfortably, and even 390B models with SSD swap
  4. Tools like Ollama and MLX make setup straightforward on Apple Silicon

The combination of unified memory and quantization democratizes access to large models. You don’t need a $10,000 GPU cluster - a laptop can do meaningful work with 100B+ models.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: