Which LLM can my PC run?

It depends on your GPU's VRAM and memory bandwidth. As a rule of thumb, an 8 GB card runs 7–8B models at Q4, 12 GB runs up to ~14B, 16–24 GB runs 24–32B, and 48 GB+ (or Apple Silicon unified memory) runs 70B models. The Local AI Advisor detects your exact hardware and lists every model that fits, how fast it runs, and the best quantization to use.

How much VRAM do I need to run an LLM?

A rough rule: model VRAM ≈ parameters (in billions) × bits-per-weight ÷ 8, plus the KV cache for your context and ~1 GB of overhead. An 8B model at Q4_K_M needs about 4.8 GB of weights, so ~6–7 GB total — comfortable on an 8 GB card. A 70B model at Q4 needs roughly 40 GB. The advisor computes the exact footprint for any model, quant and context length.

What GPU do I need to run a 70B model locally?

A 70B model at Q4_K_M needs roughly 40 GB of memory, so a 48 GB card (like an RTX A6000), two 24 GB GPUs, or Apple Silicon with 64 GB+ of unified memory can run it fully in memory. On a 24 GB card it only runs by offloading part of the model to system RAM, which drops speed sharply — a 24–32B model is the sweet spot for a single 24 GB GPU.

How fast will an LLM run on my hardware (tokens per second)?

Single-stream text generation is memory-bandwidth bound: each token reads the active weights once, so tokens/sec ≈ usable memory bandwidth ÷ active-weight bytes. For example an RTX 4090 (~1008 GB/s) runs an 8B model at Q4 around 120+ tokens/sec — far faster than reading speed. The advisor estimates realistic tokens/sec for every GPU, model and quantization.

Do I need a GPU to run local AI?

No — you can run smaller models on the CPU alone, but it is much slower because system RAM bandwidth is far lower than GPU VRAM bandwidth. A dedicated GPU, or Apple Silicon with its fast unified memory, is strongly recommended for anything above ~3B parameters if you want a usable, interactive speed.

Which quantization should I use?

Q4_K_M is the default sweet spot — about 97.5% of full quality at ~4.8 bits per weight. Use Q5_K_M or Q6_K for near-lossless output if the model still fits in memory, and only drop to Q3, IQ3 or Q2 when you need to squeeze a larger model into limited VRAM.

Is Ollama or LM Studio faster for local LLMs?

Throughput is nearly identical — both run llama.cpp / GGUF under the hood, so speed is set by your hardware and quantization, not the app. Choose Ollama for CLI and automation, LM Studio for a friendly GUI (and its MLX backend on Apple Silicon), or vLLM / TabbyAPI when you need to serve many concurrent requests.

Can my laptop run local LLMs?

Yes. A laptop with a 6–8 GB dedicated GPU runs 7–8B models comfortably, and Apple Silicon MacBooks with 16 GB+ of unified memory run 8–14B models well (larger models with more memory). The advisor detects laptop GPUs and integrated graphics, and estimates the slower CPU-class speed when there is no dedicated GPU.

Why do Mixture-of-Experts models like Qwen3-30B-A3B feel so fast?

MoE models only activate a fraction of their parameters per token — Qwen3-30B-A3B activates about 3B of its 30B parameters. Because generation speed depends on active-weight bytes, it decodes roughly as fast as a 3B model while answering with 30B-scale quality, which is why it feels much quicker than a dense 32B model.

Codestral 22B vs Qwen2.5-Coder 32B

Two open-weight models, head to head — task-by-task quality, memory needs, speed and license. Scores are 0-100 from public evaluations; speed and VRAM are computed for a common 12GB card.

Verdict

The Qwen2.5-Coder 32B is the stronger all-rounder (85 vs 79 average), but pick Codestral 22B for coding and Qwen2.5-Coder 32B for reasoning. Licenses differ: Codestral 22B is MNPL, Qwen2.5-Coder 32B is Apache-2.0.

Specs

	Codestral 22B	Qwen2.5-Coder 32B
Parameters	22.2B	32B
Context	32K	128K
VRAM (Q4)	15 GB	22 GB
Speed on RTX 4070	5.9 tok/s	2.7 tok/s
License	MNPL	Apache-2.0
Released	2024-05	2024-11

Task-by-task quality

coding

reasoning

math

chat

multilingual

agent

Average7985

Codestral 22BQwen2.5-Coder 32B

Can your GPU run them?

Codestral 22BHeavy

Needs ~15 GB at Q4 · 5.9 tok/s on a 4070

Qwen2.5-Coder 32BHeavy

Needs ~22 GB at Q4 · 2.7 tok/s on a 4070

Frequently asked

Is Codestral 22B or Qwen2.5-Coder 32B better?

Overall the Qwen2.5-Coder 32B edges it on average quality (85 vs 79 across six task areas), but it's use-case dependent — Codestral 22B leads on coding, while Qwen2.5-Coder 32B leads on reasoning.

Which is better for coding, Codestral 22B or Qwen2.5-Coder 32B?

Qwen2.5-Coder 32B — it scores 92/100 for coding vs 90/100.

Which needs more VRAM?

Qwen2.5-Coder 32B is larger (32B vs 22.2B) and needs more VRAM. At Q4 they need about 15 GB and 22 GB respectively.

Which is faster?

On a 12GB RTX 4070 at Q4, Codestral 22B runs at ~5.9 tok/s and Qwen2.5-Coder 32B at ~2.7 tok/s.

More model comparisons

Qwen3 8B vs Llama 3.1 8B

Qwen3 14B vs Gemma 3 12B

Qwen3 14B vs DeepSeek-R1-Distill 14B

Qwen2.5-Coder 7B vs Qwen3 8B

Gemma 3 12B vs Mistral Nemo 12B

Qwen3 32B vs Qwen2.5-Coder 32B

Run the interactive advisor

Auto-detect your exact hardware and get personalised picks, speed & memory.