A Practical Guide to LLM Quantization

LLM quantization is the process of representing a model's weights (and sometimes activations) with fewer bits than they were trained with, usually going from 16-bit floating point down to 8-bit or 4-bit integers. The payoff is a model that uses roughly a quarter to a half the memory, runs faster on the same GPU, and can sometimes fit on hardware that couldn't hold the full-precision version at all. The cost is a small, usually measurable but often tolerable, drop in output quality. This guide covers the actual formats in use in 2026, the tools that implement them, and hands-on code for quantizing and running a model yourself.

Why LLM quantization matters for deployment

A 7-billion-parameter model stored in 16-bit floating point (FP16 or BF16) needs about 14GB of memory just to hold the weights, before you add activations, KV cache, and batching overhead. Quantize that same model to 4-bit and the weight footprint drops to roughly 3.5-4GB. That difference decides whether a model runs on a single consumer GPU, fits alongside other services on a shared box, or requires a multi-GPU cluster.

Memory is not the only benefit. Inference on many GPUs is memory-bandwidth bound, not compute bound, especially during autoregressive decoding where the model processes one token at a time. Moving fewer bytes per weight from GPU memory to the compute units directly speeds up token generation, sometimes by 2-3x, independent of raw FLOPs. Quantization is one of the few optimizations that improves memory, cost, and latency at the same time, which is why it shows up in nearly every production LLM deployment, whether you're serving a fine-tuned model from your own infrastructure or running a local copy of an open-weight model on a laptop.

The precision formats you'll actually encounter

Before touching any tool, it helps to know what the numbers mean.

• FP32: 32-bit floating point, the default for training. Almost never used for inference of large models because of the memory and speed cost.
• FP16 / BF16: 16-bit floating point, the standard "full precision" inference format. BF16 has a wider exponent range and is more stable on modern GPUs; FP16 is more common on older hardware.
• INT8: 8-bit integer quantization. Cuts memory roughly in half versus FP16 with typically negligible quality loss when done with a calibration step.
• INT4 / 4-bit: The most common quantization target for local inference. Cuts memory to roughly a quarter of FP16. Quality loss is more noticeable but often acceptable for chat and instruction-following tasks.
• NF4 (NormalFloat4): A 4-bit format designed specifically for weights that follow a roughly normal distribution, used by the bitsandbytes library. Tends to preserve quality better than naive INT4 at the same bit width.
• GGUF quantization levels (Q8_0, Q5_K_M, Q4_K_M, Q3_K_S, and so on): A family of mixed-precision schemes used by llama.cpp, where different tensors in the model get different bit widths based on their sensitivity. Q4_K_M is the most common "good default" choice for local inference.

The naming can be confusing because different tools use different conventions for describing what is essentially the same idea: fewer bits per weight, chosen carefully so the parts of the model most sensitive to error keep more precision.

Weight-only quantization vs full quantization

Most practical LLM quantization today is weight-only: the stored weights are compressed to low bit width, but they get dequantized back to FP16 on the fly right before the matrix multiplication happens. This is why weight-only quantization gives you memory savings and often speed gains (less data movement) without needing specialized low-precision compute kernels for every operation.

Full quantization, where activations are also quantized, can give additional speedups on hardware with native INT8 or INT4 compute support, but it's harder to get right because activations have a much wider and more unpredictable range than weights. Most of the popular open-source tools (bitsandbytes, GPTQ, AWQ, GGUF) default to weight-only quantization, which is why this guide focuses there.

Quantizing a model with bitsandbytes

bitsandbytes is the easiest entry point because it integrates directly into Hugging Face transformers and quantizes on load, no separate conversion step required.

pip install transformers accelerate bitsandbytes torch

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "meta-llama/Llama-3.1-8B-Instruct"

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quant_config,
    device_map="auto",
)

prompt = "Explain quantization in one paragraph."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=150)
print(tokenizer.decode(output[0], skip_special_tokens=True))

A few notes on the parameters that actually matter:

• bnb_4bit_quant_type="nf4" gives noticeably better quality than the plain "fp4" option at the same bit width, so use NF4 unless you have a specific reason not to.
• bnb_4bit_use_double_quant=True quantizes the quantization constants themselves, saving a small extra amount of memory (roughly 0.4 bits per parameter) with essentially no quality cost. Always turn it on.
• bnb_4bit_compute_dtype=torch.bfloat16 controls the precision used for the actual matrix multiply after dequantization. Keep this at BF16 or FP16, never drop it to match the storage precision.

For 8-bit instead of 4-bit, swap load_in_4bit=True for load_in_8bit=True and drop the 4-bit-specific fields. 8-bit is a safer choice when you can't afford any observable quality regression and still have the memory budget for it.

Quantizing with GPTQ for faster serving

bitsandbytes quantizes weights independently of each other, which is simple but leaves some accuracy on the table. GPTQ instead uses a small calibration dataset to quantize weights layer by layer, adjusting each layer to compensate for the error introduced by the layers already quantized. The result is a smaller quality drop at the same bit width, and because the quantization is done once and saved, loading is fast and there's no on-the-fly dequantization overhead at model-load time.

pip install auto-gptq optimum

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False,
)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config)

calibration_prompts = [
    "The quick brown fox jumps over the lazy dog.",
    "Quantization reduces the memory footprint of large models.",
    "Write a short summary of climate change.",
]
calibration_data = [tokenizer(p, return_tensors="pt") for p in calibration_prompts]

model.quantize(calibration_data)
model.save_quantized("llama-3.1-8b-gptq-4bit")
tokenizer.save_pretrained("llama-3.1-8b-gptq-4bit")

In production, replace the three-sentence calibration set with a few hundred examples that resemble your real traffic (support tickets, code, whatever the model will actually see). Calibration quality has a real, measurable effect on the final quantized model's accuracy, so don't skip it or shortcut it with generic text if you can use domain-relevant samples instead.

group_size=128 is a strong default: smaller groups (like 32 or 64) give slightly better accuracy at the cost of a bit more overhead for the quantization scale factors; larger groups save a little memory but degrade accuracy faster.

Quantizing for local inference with GGUF and llama.cpp

If the target is a laptop, a Mac, or CPU inference rather than a datacenter GPU, llama.cpp and the GGUF format are the standard path. The conversion happens outside transformers, using the llama.cpp repository's own scripts.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt

python convert_hf_to_gguf.py /path/to/llama-3.1-8b-instruct \
    --outfile llama-3.1-8b-f16.gguf \
    --outtype f16

That first step just converts the Hugging Face checkpoint to GGUF at full precision. The actual quantization happens in a separate step using the compiled llama-quantize binary:

cmake -B build
cmake --build build --config Release -j
./build/bin/llama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M

Then run it directly:

./build/bin/llama-cli -m llama-3.1-8b-q4_k_m.gguf -p "Explain quantization in one paragraph." -n 150

For picking a GGUF quant level, a practical rule of thumb:

• Q8_0: near-lossless, use when you have the memory and want a safety margin.
• Q5_K_M: a good balance, close to Q8_0 quality with meaningfully less memory.
• Q4_K_M: the most common default for local chat use, noticeably smaller with modest quality loss.
• Q3_K_M and below: real quality degradation becomes noticeable, mostly useful when memory is the hard constraint (e.g. running an 8B+ model on 8GB of RAM).

Measuring quality loss instead of guessing

Quantization always trades some quality for size and speed. The mistake is picking a bit width based on vibes instead of measurement. Two cheap checks catch most regressions before they reach users.

Perplexity comparison. Run the same held-out text through the full-precision and quantized model and compare perplexity. A jump of more than a few percent is a signal to back off to a higher bit width.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def compute_perplexity(model, tokenizer, text):
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model(**inputs, labels=inputs["input_ids"])
    return torch.exp(outputs.loss).item()

eval_text = "Your held-out evaluation passage goes here, a few paragraphs of representative text."
print("Perplexity:", compute_perplexity(model, tokenizer, eval_text))

Task-level regression testing. Perplexity is a proxy; it doesn't always predict real task performance. Keep a small fixed set of prompts that mirror your actual use case (classification, summarization, tool calling, whatever it is) with expected answers or a rubric, and run both the original and quantized model against it before shipping a quantized checkpoint. This catches cases where quantization quietly breaks something specific, like structured JSON output or a particular instruction-following pattern, that perplexity alone would miss.

Choosing bit width and method for your use case

• Prototyping or research on a single GPU: bitsandbytes 4-bit NF4. Zero conversion step, load and go.
• Production serving where every millisecond and every dollar counts: GPTQ or AWQ, pre-quantized once and served with a purpose-built inference engine such as vLLM or TensorRT-LLM, both of which have native support for these formats.
• Local or offline inference on a laptop, edge device, or CPU-only box: GGUF through llama.cpp, choosing Q4_K_M as a starting point and moving up to Q5_K_M or Q8_0 if you see quality issues.
• Any workload where correctness matters more than speed (legal, medical, financial text generation): stay at 8-bit or even FP16 unless you've validated 4-bit against your specific evaluation set and found the gap acceptable.

A pattern worth internalizing: quantize once, evaluate against your own data, then decide. The "best" bit width is not a fixed number, it depends on how sensitive your specific task is to small errors in the model's weights.

Common pitfalls

• Quantizing after fine-tuning without re-testing. A model that scored well quantized at 4-bit before fine-tuning can behave differently after a LoRA merge or full fine-tune, because the weight distribution changes. Re-run your evaluation after any training step.
• Mixing quantization with long-context workloads and forgetting the KV cache. Quantizing the model weights does nothing for KV cache memory, which grows with sequence length and batch size and can dominate memory usage at long context. Look at KV cache quantization (supported by vLLM and llama.cpp) separately if long context is a bottleneck.
• Using a generic calibration set for GPTQ or AWQ on a specialized domain. Calibration data should look like your production traffic. A model calibrated on generic web text and then used exclusively for code generation will quantize the wrong things well.
• Assuming quantized models are always faster. Weight-only quantization saves memory bandwidth, which speeds up decoding, but the dequantization step adds compute overhead. On hardware that isn't memory-bandwidth bound (small models, short sequences, high compute-to-memory ratio GPUs), a quantized model can sometimes run at similar or even slower wall-clock speed than FP16. Benchmark on your actual hardware and workload, don't assume.

FAQ

What's the difference between quantization and distillation? Quantization compresses an existing model's weights to fewer bits without changing its architecture or retraining it from scratch. Distillation trains a smaller, separate model to mimic a larger one's outputs. They solve similar problems (smaller, cheaper models) through completely different mechanisms, and they're not mutually exclusive: a distilled model can also be quantized afterward.

Does LLM quantization always hurt output quality? Almost always by a small, measurable amount, but "hurt" is relative. At 8-bit, quality loss is typically indistinguishable from full precision on most tasks. At 4-bit, it's usually small but present, and it becomes more noticeable below 4-bit. The right question isn't whether quality drops but whether the drop is acceptable for your specific use case, which is why measurement matters more than picking a number off a leaderboard.

Can I quantize a model I fine-tuned myself? Yes. All the tools in this guide, bitsandbytes, GPTQ, AWQ, and the llama.cpp GGUF pipeline, work on any Hugging Face-format checkpoint, whether it came from a base model release or your own fine-tuning run. Just make sure to merge LoRA adapters into the base weights first if you used parameter-efficient fine-tuning, since most quantization tools expect a single merged checkpoint rather than a base model plus adapter.

Which is better, GPTQ or AWQ? Both are calibration-based weight quantization methods and land in a similar quality range at the same bit width. AWQ tends to be somewhat more robust to a poorly chosen calibration set because it focuses on protecting the weights that matter most for activation magnitude rather than fitting every weight equally. In practice, check whether your target inference engine has better native support for one or the other, since that often matters more than the small accuracy difference between them.

Do I need a GPU to quantize a model? For bitsandbytes, quantization happens automatically on load and needs a GPU (or works on CPU, but slowly). GPTQ and AWQ calibration also typically run on GPU since they involve forward passes through the full model. GGUF conversion with llama.cpp can run entirely on CPU, which makes it the most accessible option if you don't have GPU access for the conversion step, even though you'll usually still want a GPU or Apple Silicon for fast inference afterward.

How much smaller does a model get after quantization? As a rough guide relative to FP16: 8-bit quantization is about half the size, 4-bit is about a quarter, and GGUF's mixed-precision schemes land somewhere in between depending on the specific quant level chosen. Always measure the actual file size or memory usage after quantizing your specific model rather than relying on these ratios exactly, since group sizes, quantization constants, and mixed per-tensor bit widths all shift the real number slightly.