Model Fit Checker

The first question in deploying a large model isn't how fast it runs — it's whether it fits. Memory, not compute, is usually the wall you hit first, because a model's parameters have to physically live in the accelerator's HBM, and modern models are enormous. A 70-billion-parameter model in half precision is 140 gigabytes of weights alone, already past a single 80 GB GPU before a single token is processed.

Precision is the lever that moves that number most. Each parameter takes four bytes in fp32, two in fp16 or bf16, one in int8, half in int4 — so quantizing weights halves or quarters the memory, and a model that needs two GPUs at fp16 can fit on one at int4. With modern quantization methods the accuracy cost is often small, which is why int8 and int4 deployment is everywhere: it's the cheapest capacity win available, and watching the memory fall across precisions is the first thing to try when a model doesn't fit.

Inference adds the KV cache — the stored attention keys and values that grow with context length and batch size, and for long-context, high-concurrency serving can rival the weights themselves. Training is a different regime entirely: mixed-precision Adam keeps fp16 weights, fp16 gradients, an fp32 master copy and two moment estimates, roughly sixteen bytes per parameter against two for inference. That eightfold overhead is why training a model takes far more GPUs than serving it, and why sharding techniques like ZeRO and FSDP exist.

When the total exceeds one GPU, you split the model — tensor parallelism across fast interconnect for low latency, pipeline parallelism for less bandwidth pressure, ZeRO/FSDP to shard training state. This checker tells you which regime you're in and what to do. Once it fits, the next bottleneck is often bandwidth, not capacity — check the HBM Bandwidth console — and size the deployment in the LLM Serving console.