Free Tool
GPU Capacity Planner
How many GPUs does your AI workload actually need — and what will they cost? This planner gives you a defensible estimate in minutes: pick a workload, compare hardware options, and see the math behind the answer.
Built for founders, product people, and anyone who has to budget for AI without an infrastructure background. Pricing data is refreshed from public sources; the methodology is documented inline. Part of the Infrastructure, Explained notebook.
self-host vs serverless
LLM GPU capacity & cost planner
Speed and throughput are derived from GPU memory bandwidth and VRAM — not typed in. Gold values are computed from the physics; everything else is yours to set.
GPU · sets VRAM, bandwidth, and compute (editable)
VRAM / GPUGB
Bandwidth / GPUTB/s
FP16 denseTFLOPS
GPUs / node
model · total size sets VRAM use, active size sets speed
8 B
all weights — drives VRAM footprint
8 B
per token — = total for dense, lower for MoE
FP8
1 bytes / parameter
8K tok
drives KV-cache size per user
calibration · why this is an estimate, not a benchmark
70%
realized vs rated — 65–80% typical
130 KB/tok
layers × attention (GQA/MQA); ~130 for 8B @ FP16 KV, ~half for FP8 KV
80%
GPU time left after prefill — 3:1 input eats the rest
derived from the silicon above
Single-user speedS
293 tok/s
Weights / node
8 / 80 GB
Max concurrent/ node
61
Node ceilingT
18K tok/s
Users ⇄ latency — drag either
Bounded by the derived S and T above.
40
293 tok/s
capped at S = 293 tok/s — one user can't beat single-stream speed
Snappy — about 220 words/s, well past reading speed.
1
cost per node · enter hourly or monthly
$ / node-hr
or$ / node-mo
util%
Nodes needed
1
Fleet / month
$1,825
Self-host $/1M out
$0.08
Cheaper option
Self-host
per-user speed vs users on one node — derived curve
vs serverless · blended $/1M output (3:1 in:out)
built-in defaults — local pricing file unavailable
in $
out $
in:out: 1
Self-host is 99% cheaper per token at capacity ($0.08 vs $8.60/1M). Break even past 212.2M output tokens/mo — 1% of this fleet's capacity.
Read before trusting a number. This is a first-principles estimate, not a benchmark. Decode speed (S) is bandwidth-bound and lands within roughly ±20–30% of measured vLLM / TensorRT-LLM figures once you tune the efficiency slider to one real datapoint from your own stack. Batch throughput (T) saturates at a compute-roofline ceiling (TFLOPS × ~50% MFU), multi-GPU speed assumes ~80%-efficient tensor parallelism, and the decode-share slider approximates prefill cost — all order-of-magnitude. Real numbers depend on your inference engine, attention implementation (GQA/MQA), sequence-length mix, and scheduler. Validate against your own load tests before sizing a purchase.