Which LLM can my PC run?

It depends on your GPU's VRAM and memory bandwidth. As a rule of thumb, an 8 GB card runs 7–8B models at Q4, 12 GB runs up to ~14B, 16–24 GB runs 24–32B, and 48 GB+ (or Apple Silicon unified memory) runs 70B models. The Local AI Advisor detects your exact hardware and lists every model that fits, how fast it runs, and the best quantization to use.

How much VRAM do I need to run an LLM?

A rough rule: model VRAM ≈ parameters (in billions) × bits-per-weight ÷ 8, plus the KV cache for your context and ~1 GB of overhead. An 8B model at Q4_K_M needs about 4.8 GB of weights, so ~6–7 GB total — comfortable on an 8 GB card. A 70B model at Q4 needs roughly 40 GB. The advisor computes the exact footprint for any model, quant and context length.

What GPU do I need to run a 70B model locally?

A 70B model at Q4_K_M needs roughly 40 GB of memory, so a 48 GB card (like an RTX A6000), two 24 GB GPUs, or Apple Silicon with 64 GB+ of unified memory can run it fully in memory. On a 24 GB card it only runs by offloading part of the model to system RAM, which drops speed sharply — a 24–32B model is the sweet spot for a single 24 GB GPU.

How fast will an LLM run on my hardware (tokens per second)?

Single-stream text generation is memory-bandwidth bound: each token reads the active weights once, so tokens/sec ≈ usable memory bandwidth ÷ active-weight bytes. For example an RTX 4090 (~1008 GB/s) runs an 8B model at Q4 around 120+ tokens/sec — far faster than reading speed. The advisor estimates realistic tokens/sec for every GPU, model and quantization.

Do I need a GPU to run local AI?

No — you can run smaller models on the CPU alone, but it is much slower because system RAM bandwidth is far lower than GPU VRAM bandwidth. A dedicated GPU, or Apple Silicon with its fast unified memory, is strongly recommended for anything above ~3B parameters if you want a usable, interactive speed.

Is Ollama or LM Studio faster for local LLMs?

Throughput is nearly identical — both run llama.cpp / GGUF under the hood, so speed is set by your hardware and quantization, not the app. Choose Ollama for CLI and automation, LM Studio for a friendly GUI (and its MLX backend on Apple Silicon), or vLLM / TabbyAPI when you need to serve many concurrent requests.

Can my laptop run local LLMs?

Yes. A laptop with a 6–8 GB dedicated GPU runs 7–8B models comfortably, and Apple Silicon MacBooks with 16 GB+ of unified memory run 8–14B models well (larger models with more memory). The advisor detects laptop GPUs and integrated graphics, and estimates the slower CPU-class speed when there is no dedicated GPU.

Why do Mixture-of-Experts models like Qwen3-30B-A3B feel so fast?

MoE models only activate a fraction of their parameters per token — Qwen3-30B-A3B activates about 3B of its 30B parameters. Because generation speed depends on active-weight bytes, it decodes roughly as fast as a 3B model while answering with 30B-scale quality, which is why it feels much quicker than a dense 32B model.

How to run Qwen3 30B-A3B (MoE) locally

Qwen3 30B-A3B (MoE) is a 30.5B Qwen model (only ~3.3B active per token — a Mixture-of-Experts, so it decodes fast) with a 128K context and Apache-2.0 license. Here's the VRAM it needs, which GPUs run it, and how fast.

At a glance

Parameters

30.5B

Min VRAM (Q4)

~21GB

Context

128K

License

Apache-2.0

Memory needed per quantization

Quant	Quality	Weights	Min VRAM
IQ3_M	90.0%	14 GB	~17 GB
Q4_K_M	97.5%	18 GB	~21 GB
Q5_K_M	99.0%	22 GB	~24 GB
Q6_K	99.7%	25 GB	~28 GB
Q8_0	99.9%	32 GB	~35 GB
FP16 / BF16	100.0%	61 GB	~64 GB

Which GPUs run Qwen3 30B-A3B (MoE)

GPU	Fit	Quant	Speed
GeForce RTX 4060 8GB · 272 GB/s	Works	Q2_K	51 tok/s
GeForce RTX 3060 12GB 12GB · 360 GB/s	Perfect	IQ2_XXS	308.8 tok/s
GeForce RTX 4070 12GB · 504 GB/s	Perfect	IQ2_XXS	432.3 tok/s
Intel Arc B580 12GB · 456 GB/s	Perfect	IQ2_XXS	391.1 tok/s
GeForce RTX 4070 Ti SUPER 16GB · 672 GB/s	Perfect	Q2_K	462.6 tok/s
GeForce RTX 5070 12GB · 672 GB/s	Perfect	IQ2_XXS	576.3 tok/s
GeForce RTX 4080 SUPER 16GB · 736 GB/s	Perfect	Q2_K	506.6 tok/s
Radeon RX 7900 XTX 24GB · 960 GB/s	Perfect	Q4_K_M	375.1 tok/s
GeForce RTX 3090 24GB · 936 GB/s	Perfect	Q4_K_M	365.7 tok/s
GeForce RTX 4090 24GB · 1008 GB/s	Perfect	Q4_K_M	393.8 tok/s
GeForce RTX 5090 32GB · 1792 GB/s	Perfect	Q6_K	522.5 tok/s
Apple M4 16GB · 120 GB/s	Perfect	Q2_K	82.6 tok/s
Apple M4 Pro 24GB · 273 GB/s	Perfect	Q4_K_M	106.7 tok/s
Apple M3 Max (16c) 48GB · 400 GB/s	Perfect	Q8_0	90.8 tok/s
Apple M2 Ultra 64GB · 800 GB/s	Perfect	Q8_0	181.6 tok/s

Frequently asked

How much VRAM does Qwen3 30B-A3B (MoE) need?

At the recommended Q4_K_M quantization, Qwen3 30B-A3B (MoE) needs about 18.4GB for weights plus KV cache and overhead — roughly 21GB of VRAM total for a usable context. Drop to a 3-bit quant to squeeze it smaller, or go FP16 (61GB) for full quality.

What GPU do I need to run Qwen3 30B-A3B (MoE)?

The most affordable GPU that runs it well is the Intel Arc B580 (12GB) at IQ2_XXS, delivering about 391.1 tokens/sec. Anything with 21GB+ of VRAM will run it comfortably.

Which quantization should I use for Qwen3 30B-A3B (MoE)?

Q4_K_M is the default sweet spot (~97.5% of full quality). If it fits, Q5_K_M or Q6_K give near-lossless output; only drop below Q4 when you must fit it into limited VRAM.

Is Qwen3 30B-A3B (MoE) good for multilingual?

Yes — Qwen3 30B-A3B (MoE) scores 88/100 for multilingual, one of its strongest areas. It also has explicit reasoning/thinking support. License: Apache-2.0.