All guides
Hardware 6 minReviewed September 1, 2026

How much GPU memory does an LLM need?

Estimate model weights, context cache, runtime overhead, and concurrency before choosing a GPU.

The answer

Model weights are only the floor. Reserve VRAM for the KV cache, runtime workspace, batching, and peaks; a model that barely loads is not a production fit.

Fast sizing rule

  • FP16 or BF16 weights need roughly 2 bytes per parameter before overhead.
  • 8-bit weights need roughly 1 byte per parameter; 4-bit weights roughly half a byte, plus quantization metadata.
  • Add 15–25% for runtime overhead, then size the KV cache from context length and concurrent sequences.
  • Do not advertise the GPU’s full memory as usable model capacity.

Practical starting points

  • 16–24 GB: quantized 7B–14B models, embeddings, rerankers, and light image generation.
  • 40–48 GB: quantized 30B-class models or high-concurrency smaller models.
  • 80 GB: larger dense models, long context, bigger batches, and production headroom.
  • Multi-GPU: 70B+ dense models, large MoE models, or high-throughput replicas where tensor parallelism is justified.

The hidden cost of context

Long context consumes memory even when the model weights do not change. Test the maximum context and concurrency together. A server that handles one 128K request may fail when four users arrive.