All guides
Concepts 8 minReviewed September 1, 2026

LLM quantization: 4-bit, 8-bit, GGUF, AWQ and GPTQ

What lower precision changes, when it saves money, and what to test before production.

The answer

Quantization stores or computes model values at lower precision so a larger model can fit in less memory. It trades some quality or compatibility for lower VRAM use and often higher throughput.

What the names tell you

  • 8-bit is a conservative first reduction when quality sensitivity is high.
  • 4-bit is common for fitting larger models on a single GPU, but method and calibration matter.
  • AWQ and GPTQ are GPU-oriented weight quantization methods supported by common serving stacks.
  • GGUF is an artifact format used heavily by llama.cpp and local runtimes across CPU and GPU hardware.

Test the task, not perplexity alone

Evaluate structured output, tool calls, multilingual prompts, long context and the exact retrieval corpus. Small regressions in a generic metric can become large failures in a narrow business workflow.