Will this LLM fit on my GPU?
Pick a model, a precision and a context length. See exactly where the memory goes, with the formula behind every number.
See an example:
Where the memory goes
| Component | Memory | Share |
|---|---|---|
| Total |
On your hardware
Every GPU
| GPU | Memory | GPUs needed | Headroom on one |
|---|
How it is calculated
- Weights
- parameters × bits per weight ÷ 8. GGUF K-quant mixes (Q4_K_M, Q5_K_M, Q3_K_M, Q2_K) use the average a 7–8B file lands on, so real files differ by a few percent. QLoRA keeps the embeddings in BF16 and quantises only the linear layers to NF4.
- KV cache
- 2 (K and V) × KV heads × head dim × layers × context × batch × bytes per value. Grouped-query attention uses the KV head count, not the query head count. Sliding-window layers (Gemma, gpt-oss) cache only their window. DeepSeek’s multi-head latent attention caches one compressed latent of 512 + 64 values per token per layer.
- Inference activations
- The peak working set of one layer for the tokens in flight (the prefill chunk), plus FP32 logits for the last token of each sequence. With flash attention no score matrix is stored.
- Training activations
- Per layer, per token, in 16-bit: 8×hidden + 4×(query width) + 4×(KV width) + 6×(MLP width), following Korthikanti et al. (2022) adapted to grouped-query attention and a gated MLP, with dropout off. Eager attention adds 4 × heads × seq² per layer. Gradient checkpointing keeps only each layer’s input plus one recomputed layer. LoRA and QLoRA store the same activations, because gradients must flow back through the frozen layers.
- Optimizer
- Per trainable parameter: AdamW keeps two FP32 moments (8 bytes) plus an FP32 master copy (4 bytes) beside 16-bit weights and gradients, the classic 16 bytes per parameter. LoRA adapters are trained in FP32 and need no master copy.
- Several GPUs
- Weights, KV cache, gradients and optimizer states are split evenly (tensor parallel for inference, ZeRO-3 / FSDP for training). Activations, logits and overhead are counted on every GPU, which errs on the safe side.
- Decode ceiling
- Memory bandwidth ÷ (weights read per token + KV cache). Each token reads every layer’s attention, the routed experts only, and the output head. Real engines reach roughly half to four-fifths of this.
GPU memory is the usable amount the driver reports (an “80 GB” H100 shows 79.6 GiB). Apple Silicon lets the GPU use about 75% of unified memory by default. All sizes are GiB (1024³ bytes).