Which LLM can my PC run?

It depends on your GPU's VRAM and memory bandwidth. As a rule of thumb, an 8 GB card runs 7–8B models at Q4, 12 GB runs up to ~14B, 16–24 GB runs 24–32B, and 48 GB+ (or Apple Silicon unified memory) runs 70B models. The Local AI Advisor detects your exact hardware and lists every model that fits, how fast it runs, and the best quantization to use.

How much VRAM do I need to run an LLM?

A rough rule: model VRAM ≈ parameters (in billions) × bits-per-weight ÷ 8, plus the KV cache for your context and ~1 GB of overhead. An 8B model at Q4_K_M needs about 4.8 GB of weights, so ~6–7 GB total — comfortable on an 8 GB card. A 70B model at Q4 needs roughly 40 GB. The advisor computes the exact footprint for any model, quant and context length.

What GPU do I need to run a 70B model locally?

A 70B model at Q4_K_M needs roughly 40 GB of memory, so a 48 GB card (like an RTX A6000), two 24 GB GPUs, or Apple Silicon with 64 GB+ of unified memory can run it fully in memory. On a 24 GB card it only runs by offloading part of the model to system RAM, which drops speed sharply — a 24–32B model is the sweet spot for a single 24 GB GPU.

How fast will an LLM run on my hardware (tokens per second)?

Single-stream text generation is memory-bandwidth bound: each token reads the active weights once, so tokens/sec ≈ usable memory bandwidth ÷ active-weight bytes. For example an RTX 4090 (~1008 GB/s) runs an 8B model at Q4 around 120+ tokens/sec — far faster than reading speed. The advisor estimates realistic tokens/sec for every GPU, model and quantization.

Do I need a GPU to run local AI?

No — you can run smaller models on the CPU alone, but it is much slower because system RAM bandwidth is far lower than GPU VRAM bandwidth. A dedicated GPU, or Apple Silicon with its fast unified memory, is strongly recommended for anything above ~3B parameters if you want a usable, interactive speed.

Which quantization should I use?

Q4_K_M is the default sweet spot — about 97.5% of full quality at ~4.8 bits per weight. Use Q5_K_M or Q6_K for near-lossless output if the model still fits in memory, and only drop to Q3, IQ3 or Q2 when you need to squeeze a larger model into limited VRAM.

Is Ollama or LM Studio faster for local LLMs?

Throughput is nearly identical — both run llama.cpp / GGUF under the hood, so speed is set by your hardware and quantization, not the app. Choose Ollama for CLI and automation, LM Studio for a friendly GUI (and its MLX backend on Apple Silicon), or vLLM / TabbyAPI when you need to serve many concurrent requests.

Can my laptop run local LLMs?

Yes. A laptop with a 6–8 GB dedicated GPU runs 7–8B models comfortably, and Apple Silicon MacBooks with 16 GB+ of unified memory run 8–14B models well (larger models with more memory). The advisor detects laptop GPUs and integrated graphics, and estimates the slower CPU-class speed when there is no dedicated GPU.

Why do Mixture-of-Experts models like Qwen3-30B-A3B feel so fast?

MoE models only activate a fraction of their parameters per token — Qwen3-30B-A3B activates about 3B of its 30B parameters. Because generation speed depends on active-weight bytes, it decodes roughly as fast as a 3B model while answering with 30B-scale quality, which is why it feels much quicker than a dense 32B model.

What AI models can the GeForce RTX 5090 run?

The GeForce RTX 5090 pairs 32GB of VRAM with 1792 GB/s of bandwidth — and bandwidth is what sets local LLM speed. Here's exactly what it can run, how fast, and at which quantization. Every number below is computed, not guessed.

At a glance

VRAM

32GB

Bandwidth

1792GB/s

Runnable models

15/ 15

Largest model

70BQ2_K

Top picks for this card

Best for coding

Qwen2.5-Coder 32B

Q6_K · 53.3 tok/s · 29 GB

Best for reasoning

DeepSeek-R1-Distill 32B

Q6_K · 53.3 tok/s · 29 GB

Best for chat

Qwen3 32B

Q6_K · 53.3 tok/s · 29 GB

Every model on the GeForce RTX 5090

Model	Fit	Best quant	Memory	Speed
Llama 3.2 3B 3.2B · Llama	Perfect	FP16 / BF16	7.8 GB	217.6 tok/s
Qwen3 4B 4B · Qwen	Perfect	FP16 / BF16	9.4 GB	173.9 tok/s
Mistral 7B v0.3 7.2B · Mistral	Perfect	FP16 / BF16	16 GB	98.5 tok/s
Qwen2.5-Coder 7B 7B · Qwen	Perfect	FP16 / BF16	15 GB	101.6 tok/s
Llama 3.1 8B 8B · Llama	Perfect	FP16 / BF16	18 GB	88.9 tok/s
Qwen3 8B 8B · Qwen	Perfect	FP16 / BF16	18 GB	88.6 tok/s
Gemma 3 12B 12B · Gemma	Perfect	FP16 / BF16	26 GB	59 tok/s
Phi-4 14B 14B · Phi	Perfect	Q8_0	17 GB	93.6 tok/s
Qwen3 14B 14B · Qwen	Perfect	Q8_0	17 GB	93.6 tok/s
DeepSeek-R1-Distill 14B 14B · DeepSeek	Perfect	Q8_0	17 GB	93.1 tok/s
Mistral Small 3 24B 24B · Mistral	Perfect	Q8_0	28 GB	55.3 tok/s
Gemma 3 27B 27B · Gemma	Perfect	Q6_K	24 GB	62.7 tok/s
Qwen3 30B-A3B (MoE) 30.5B · Qwen	Perfect	Q6_K	27 GB	522.5 tok/s
Qwen3 32B 32B · Qwen	Perfect	Q6_K	29 GB	53.3 tok/s
Llama 3.3 70B 70B · Llama	Perfect	Q2_K	27 GB	57.3 tok/s

Fit and speed assume a 8K context on 32GB VRAM with 32GB system RAM for overflow. Tokens/sec is single-stream decode. Click any model for its full hardware requirements.

Community-measured speeds

Model	Estimated	Measured (median)	Reports
Llama 3.1 8B	88.9 tok/s	233 tok/s	1
Qwen3 14B	93.6 tok/s	138 tok/s	1
Qwen3 32B	53.3 tok/s	72 tok/s	1

Real generation speeds reported by the community (median, with statistical outliers trimmed), shown against our estimate to keep the model honest. Run the GeForce RTX 5090? Add your own numbers in the advisor.

Frequently asked

How many local LLMs can the GeForce RTX 5090 run?

15 of the 15 models we test fit on the GeForce RTX 5090's 32GB of VRAM at a sensible quantization. The largest is Llama 3.3 70B (70B) at Q2_K, generating about 57.3 tokens/sec.

How fast is the GeForce RTX 5090 for local AI?

Single-stream generation is memory-bandwidth bound, and the GeForce RTX 5090 has 1792 GB/s. On an 8B model at Q4 that translates to roughly 88.6 tokens/sec — comfortably faster than reading speed.

What should I run the GeForce RTX 5090 with?

Ollama or LM Studio (CUDA) for one-click setup, or vLLM/TabbyAPI for serving.

Can the GeForce RTX 5090 run 70B models?

Yes — a 70B model runs, but only by offloading part of it to system RAM, which drops speed significantly. A 24-32B model is the sweet spot for staying fully in memory.