Can my hardware run this model?
Memory is the hard constraint for local LLMs. Choose a model, a quantization and a context length; the table shows which GPUs, Macs and unified-memory boxes fit, and a bandwidth-based guess at generation speed.
Estimates follow llama.cpp-style GGUF sizes; real files vary ±10%.
| Hardware | Usable mem | Fit | Est. tok/s |
|---|---|---|---|
RTX 3060 12GB cheapest sane entry; used market | 11 GB | too small | — |
RTX 4060 Ti 16GB slow bus, but 16GB | 15 GB | too small | — |
RTX 3090 24GB (used) the community default | 23 GB | too small | — |
RTX 4090 24GB | 23 GB | too small | — |
RTX 5090 32GB | 30 GB | too small | — |
2× RTX 3090 (48GB) layers split across cards; bandwidth per card | 45 GB | too small | — |
RTX PRO 6000 Blackwell 96GB | 91 GB | fits | ~26 |
Mac mini M4 24GB | 18 GB | too small | — |
Mac mini M4 Pro 64GB | 48 GB | fits | ~4 |
Mac Studio M4 Max 128GB | 100 GB | fits | ~8 |
Mac Studio M3 Ultra 256GB | 205 GB | fits | ~12 |
Mac Studio M5 Ultra 512GB | 410 GB | fits | ~17 |
Ryzen AI Max+ 395 (Strix Halo) 128GB Framework Desktop / GMKtec EVO-X2 / HP Z2 Mini | 96 GB | fits | ~4 |
NVIDIA DGX Spark 128GB CUDA, but same bandwidth class as Strix Halo | 109 GB | fits | ~4 |
Macs reserve roughly a quarter of unified memory for macOS by default (raisable with sysctl iogpu.wired_limit_mb). tok/s is a bandwidth rule of thumb for single-user decode at batch 1, discounted 40% for reality; prompt processing is compute-bound and much slower on Macs and APUs than on NVIDIA.
Questions people ask
How much VRAM does a 70B model need?
At Q4_K_M a 70B model's weights are about 40–42 GB, plus KV cache that grows with context (roughly 160 KB per token for Llama 3.3 70B at FP16). With 8K context you need ~45 GB total, which is why 2× 24 GB cards or a 64 GB+ Mac are the usual answers, and a single 24 GB card is not.
What quantization should I use?
Q4_K_M is the community default: about 4.8 bits per weight with small quality loss. Q5/Q6 are worth it if you have the memory. Below Q3 quality drops noticeably; use a smaller model at Q4 instead of a bigger one at Q2.
Is a Mac or an NVIDIA GPU better for local LLMs?
NVIDIA wins on speed per dollar and prompt processing; Macs win on how large a model you can load at all, quietly and at low power. A used RTX 3090 is the best value for models up to ~30B; a 128 GB+ Mac Studio or Strix Halo box is the only sane way to run 100B+ models at home.
Why is the estimated tokens per second so different between devices with the same memory?
Single-user generation is limited by memory bandwidth, not compute: every token reads the whole set of active weights. A 4090 has ~1 TB/s; a Mac mini M4 has 120 GB/s. Same model, ~8× speed difference.
Do MoE models like gpt-oss-120b or Qwen3-235B need less memory?
No, you still store all parameters. But only a fraction are active per token (5B of 117B for gpt-oss-120b), so they run far faster than a dense model of the same size, which makes them ideal for high-memory, low-bandwidth machines like Strix Halo or Mac Studio.