Quantization Explained: Running Large Models on Smaller Hardware

Your favorite open-weight model has seventy billion parameters, and your laptop has a single consumer GPU with maybe sixteen gigabytes of memory. On paper, that model should not fit. In full precision it needs something like a hundred and forty gigabytes just to hold the weights, before you add a single token of context. And yet people run these models on gaming laptops, on Mac Minis, on a single rented GPU that costs a few cents an hour. The trick that makes this possible is quantization. It is one of the most important ideas in applied AI right now, and it is the difference between a model being a research curiosity and being something you can actually ship. This article walks through what quantization is, why it works, the formats and vocabulary you will run into, and how to reason about the tradeoffs so you do not blindly pick a setting and hope.

What Quantization Actually Means

At its core, a neural network is a giant pile of numbers. The weights that got learned during training are stored as floating point values, and by default those values are pretty precise. A standard weight might be a 32-bit float, which means each number takes four bytes and can represent a huge range of values with a lot of decimal precision. Quantization is the process of taking those high-precision numbers and representing them with fewer bits.

Think about it like audio. A studio master of a song might be a lossless file that captures every nuance. When you stream that song, you get a compressed version that uses far fewer bits per second. Most people cannot hear the difference on their earbuds, and the file is a tenth of the size. Quantization does the same thing to model weights. You give up some precision that mostly does not matter, and in exchange the model gets dramatically smaller and faster.

The key insight is that neural networks are surprisingly tolerant of imprecision. During training the model learned to be robust to noise, and the exact value of any single weight almost never matters. Whether a weight is 0.03847 or 0.038 rarely changes the output in a way you would notice. So if you round all the weights to a coarser grid of allowed values, the model keeps working. That tolerance is what the entire field is built on.

Why Precision Costs So Much Memory

To understand the payoff, you have to see where the memory goes. Every parameter in the model has to live somewhere, and the number of bits per parameter multiplied by the parameter count gives you the raw storage requirement. Here is a simple way to compute it.

def model_memory_gb(num_params_billions, bits_per_param):
    total_params = num_params_billions * 1e9
    total_bits = total_params * bits_per_param
    total_bytes = total_bits / 8
    return total_bytes / (1024 ** 3)

# A 70B parameter model at different precisions
for bits in [32, 16, 8, 4]:
    gb = model_memory_gb(70, bits)
    print(f"{bits}-bit: {gb:.1f} GB")

Run that and the picture becomes obvious. At 32 bits, a 70-billion-parameter model needs roughly 260 gigabytes. At 16 bits, which is the common default for inference, it drops to around 130 gigabytes. At 8 bits you are near 65 gigabytes, and at 4 bits you are down to about 33 gigabytes. That last number is the one that matters, because it is the point where the model starts fitting on hardware normal people can access.

The precision level is usually the single biggest lever you have over whether a model fits at all. You can add more GPUs, but that gets expensive fast and adds complexity. Cutting the bits per weight is free in the sense that it does not require buying anything, and it directly shrinks the footprint. This is why almost every locally run model you encounter is quantized in some form. Full precision is a luxury reserved for training and for the largest datacenter deployments.

How The Math Works Under The Hood

The actual mechanics of quantization are less mysterious than they sound. Suppose you have a block of weights that range from -0.8 to 0.9. In full precision each of those is a detailed float. To quantize to a lower bit width, you pick a small set of representable levels and map every real weight to the nearest one. With 4 bits you get sixteen possible levels. With 8 bits you get 256.

The most common scheme is linear or affine quantization. You find the minimum and maximum values in a block, compute a scale factor that stretches your integer range to cover that span, and optionally a zero point that handles asymmetry. Then each weight becomes an integer plus the metadata needed to reconstruct an approximation.

import numpy as np

def quantize_block(weights, num_bits=4):
    qmin, qmax = 0, (2 ** num_bits) - 1
    w_min, w_max = weights.min(), weights.max()
    scale = (w_max - w_min) / (qmax - qmin)
    zero_point = qmin - w_min / scale
    quantized = np.round(weights / scale + zero_point)
    quantized = np.clip(quantized, qmin, qmax).astype(np.int8)
    return quantized, scale, zero_point

def dequantize_block(quantized, scale, zero_point):
    return (quantized.astype(np.float32) - zero_point) * scale

original = np.array([-0.8, -0.2, 0.0, 0.35, 0.9], dtype=np.float32)
q, s, z = quantize_block(original, num_bits=4)
recovered = dequantize_block(q, s, z)
print("original :", original)
print("recovered:", np.round(recovered, 3))

When you run this, the recovered values are close to the originals but not identical. That gap is the quantization error, and the whole art of the field is keeping that error small enough that the model still behaves. Notice one important detail: the scale and zero point are computed per block, not once for the whole model. Using small blocks means a cluster of tiny weights gets its own fine-grained scale rather than being crushed by a single outlier somewhere else in the tensor. Smaller blocks give better accuracy at the cost of slightly more metadata overhead.

The Formats You Will Actually Encounter

If you spend any time downloading models, you will run into a soup of acronyms. They are not as chaotic as they look once you group them.

• GGUF is the format used by llama.cpp and its huge ecosystem of tools. It is designed for running on CPUs and consumer GPUs, and it is the format you want if you are running models on a Mac or a regular desktop. Files come in variants like Q4_K_M or Q5_K_S, where the number is the bit width and the suffix describes the specific scheme and block strategy.
• GPTQ is a post-training quantization method aimed at GPUs. It is smart about which weights to round in which direction, using a bit of calibration data to minimize the error that matters most. It tends to run fast on NVIDIA hardware.
• AWQ, activation-aware weight quantization, is another GPU-focused method. Its clever idea is that not all weights are equally important, and the ones that interact with large activations deserve more care. It protects those and squeezes the rest harder.
• bitsandbytes is the library that made 8-bit and 4-bit loading easy inside the Hugging Face ecosystem. When you load a model with a flag like load_in_4bit, this is usually what is doing the work under the hood.

For the GGUF naming specifically, here is how to read those cryptic suffixes.

• The number after Q is the target bits per weight. Q4 is roughly four bits, Q8 is roughly eight.
• K means a K-quant, a newer and smarter block scheme that mixes precision across a tensor.
• The trailing letter is a size class. S is small, M is medium, L is large. Bigger means slightly more bits spent and slightly better quality.

For most people starting out, a Q4_K_M file hits the sweet spot. It is small enough to fit almost anywhere and good enough that you will struggle to tell it apart from the full model in casual use.

Post-Training Versus Quantization-Aware

There are two broad philosophies for how quantization happens, and knowing the difference helps you understand why some quantized models feel better than others at the same bit width.

Post-training quantization, or PTQ, is what it sounds like. You take a model that was already trained in full precision, and after the fact you compress it. This is fast and cheap. You do not need the training data or a big compute budget. Methods like GPTQ and AWQ are PTQ methods that use a small amount of calibration data to make smarter rounding choices, but they still operate on a finished model. The vast majority of quantized models you download were made this way, because it is practical and the results are good enough.

Quantization-aware training, or QAT, bakes the quantization into the training process itself. The model learns while being subjected to the rounding it will face later, so it adapts its weights to be robust to that coarseness. This produces better results at very low bit widths, but it is far more expensive because it requires actually training the model. You mostly see QAT from the labs that produce the models, not from community members quantizing them at home.

The practical takeaway is this. If you are consuming models, you will almost always use PTQ artifacts, and modern PTQ is remarkably good. If you are a lab pushing toward two or three bits per weight where the error gets severe, QAT starts to earn its keep. For everyday work at four or five bits, PTQ is completely fine.

What You Actually Give Up

Nothing is free, and quantization does cost you something. The question is how much and whether you will notice. The honest answer is that at moderate levels the loss is small, and at aggressive levels it becomes real.

At 8 bits, the degradation is usually so tiny that benchmarks barely move. You can treat 8-bit as nearly lossless for practical purposes, and the memory savings versus 16-bit are substantial. This is a safe default when you have the room.

At 4 bits, you start to see measurable but usually acceptable degradation. The model might be slightly worse at long chains of reasoning, might occasionally lose the thread on very precise instructions, or might be a touch less reliable on edge cases. For chat, drafting, summarization, and most everyday tasks, a good 4-bit quant is hard to distinguish from the original. This is why 4-bit is the workhorse of local inference.

Below 4 bits, at 3 or 2 bits, the wheels start to wobble. The model gets noticeably dumber. It makes more mistakes, hallucinates more, and struggles with anything requiring precision. There are clever methods that push this frontier, and for a large enough model even a 2-bit version can be usable, but you are firmly in tradeoff territory. You would only go here if the alternative is not running the model at all.

The important nuance is that quantization interacts with model size. A heavily quantized large model often beats a lightly quantized small model at the same memory budget. A 70-billion-parameter model at 4 bits frequently outperforms a 13-billion-parameter model at 16 bits, even though they occupy similar space. When you are choosing, think in terms of total memory and pick the largest model that fits after quantization, rather than insisting on high precision for a smaller one.

A Practical Decision Framework

When you sit down to run a model, the choice comes down to a few questions you can answer quickly. Here is a way to think it through without overanalyzing.

1. Figure out your memory ceiling. Look at your GPU VRAM, or your unified memory on a Mac. Leave headroom for the context window and the operating system. Whatever is left is your budget.
2. Estimate the model footprint at candidate precisions using the simple formula from earlier. Add roughly twenty percent for the key-value cache and runtime overhead so you are not caught short.
3. Pick the largest parameter count that fits at 4 bits first. Bigger model at 4-bit usually beats smaller model at higher precision.
4. If it fits comfortably at 4 bits with room to spare, try bumping to 5 or 6 bits for a small quality gain. If it barely fits, stay at 4 or drop your context length.
5. Only reach for sub-4-bit if a model you really want will not otherwise load. Treat it as a last resort, not a default.

Here is a rough helper that ties the earlier pieces together into something you can actually use when planning a deployment.

def recommend_precision(vram_gb, num_params_billions):
    overhead = 1.2  # kv cache and runtime slack
    options = [
        ("8-bit (near lossless)", 8),
        ("6-bit (great quality)", 6),
        ("5-bit (very good)", 5),
        ("4-bit (recommended default)", 4),
        ("3-bit (last resort)", 3),
    ]
    usable = vram_gb / overhead
    for label, bits in options:
        needed = (num_params_billions * 1e9 * bits / 8) / (1024 ** 3)
        if needed <= usable:
            return f"Use {label}. Needs ~{needed:.1f} GB of your {vram_gb} GB."
    return "Model will not fit. Pick a smaller model or add memory."

print(recommend_precision(vram_gb=16, num_params_billions=13))
print(recommend_precision(vram_gb=24, num_params_billions=70))
print(recommend_precision(vram_gb=8, num_params_billions=7))

This is deliberately simple, and real deployments have more moving parts, but it captures the core logic. Compute what fits, prefer bigger models at 4 bits, and only spend extra bits when you have the room. That single habit will steer you right most of the time.

Where Quantization Fits In The Bigger Picture

It helps to see quantization as one member of a family of efficiency techniques rather than a standalone trick. It pairs naturally with other ideas you will meet as you go deeper into deploying models.

Quantization mainly shrinks the weights. A separate but related concern is the memory used by the key-value cache during generation, which grows with context length and can dwarf the weights on long inputs. Techniques for compressing that cache, including quantizing it too, are an active area and often matter as much as weight precision for long-context work.

There is also a distinction between quantizing just the weights and quantizing the activations that flow through the network at runtime. Weight-only quantization is the common case and the easiest to get right. Full quantization of activations can speed up the actual math on hardware that supports low-precision arithmetic, but it is trickier because activations have wilder outliers than weights. Many popular formats compromise by keeping weights compressed while running the computation at higher precision.

The broader lesson is that shipping models is an engineering discipline, not just a modeling one. Knowing how to fit a capable model onto real hardware, how to reason about memory budgets, and how to trade precision for reach is exactly the kind of skill that separates someone who reads about AI from someone who deploys it. Quantization is a perfect example because the concept is simple, the payoff is enormous, and the intuition transfers to a dozen adjacent problems.

Bringing It Together

Quantization is the reason the open model ecosystem is as vibrant as it is. Without it, running a serious model would require datacenter hardware and most of the experimentation happening on laptops and single GPUs simply would not exist. The core idea is almost anticlimactic once you see it. Neural networks store numbers with more precision than they actually need, so you round those numbers to fewer bits, and the model keeps working while getting several times smaller.

The practical wisdom is worth repeating. Eight-bit is nearly free and safe when you have the memory. Four-bit is the everyday default and good enough that you will rarely notice the difference. Below four bits you are trading quality for the ability to run at all, which is sometimes exactly the right trade. And when in doubt, a bigger model squeezed harder usually beats a smaller model kept pristine. Pick the largest model that fits after quantization and you will be right far more often than not.

If working through the memory math and the format soup made you want to understand the rest of the deployment stack the same way, that is exactly the muscle worth building. Serving, caching, batching, evaluation, and cost control all reward the same kind of first-principles thinking you just applied to precision. The AI Engineering Roadmap course on teachyou.ai is built to take you through that full journey, from the fundamentals of how models work to the practical craft of putting them into production on hardware you can actually afford. Quantization is one stop on that road, and once it clicks, the rest of the picture starts to come into focus.