Inference Cost Console

Training a model is a dramatic, one-time expense; serving it is a quiet bill that arrives with every single query, forever. For any successful AI product the cumulative inference cost overtakes training quickly and then keeps growing with usage — which is why the operating metric that matters most isn't the training run, it's the cost per inference, multiplied by the billions of queries a deployed model handles.

That cost has a simple core: the all-in hourly cost of the serving hardware divided by the number of inferences it produces in that hour. Because the hourly cost is essentially fixed, the denominator is everything — throughput and utilization. A well-batched, fully-loaded accelerator spreads its cost over enormous query volume and drives the per-inference cost down; an under-utilized one pays the same hourly cost for far fewer queries, and each one costs more. Keeping inference hardware busy is the heart of cheap serving.

The part a compute-only estimate misses is overhead. Network egress, storage, load balancing — each is negligible on a single query but real across billions, and ignoring it understates the true cost. A complete per-inference figure adds that overhead on top of compute, which is why this console separates the two and shows compute's share: when overhead becomes a meaningful slice, it's a signal to optimize data movement, not just the model.

Expressed per thousand or per million inferences, this is the unit every inference business case runs on — pricing, budgeting, and margin all derive from it. For generative LLMs where output length varies, the natural unit is the token instead — use the Token Cost console — and size the serving fleet that sets your throughput and utilization in the LLM Serving console.