Quantization¶
Quantization is storing and computing a model's numbers in a lower-precision format than it was trained in — FP16 weights becoming FP8 or INT4, say. It's the highest-leverage technique in this chapter because it attacks both phases at once, and the only lossy one, which is why most of the work is about not losing quality.
Recall the roofline: cutting precision in half helps on both walls.
- Prefill (compute-bound) → lower-precision Tensor Cores do roughly 2× the FLOPS.
- Decode (memory-bound) → each value is half the bytes, so you move twice as much per second of bandwidth — effectively doubling the resource decode is starved for.
In practice you don't get a clean 2× — there's overhead converting and handling low-precision data — but dropping one precision level typically buys 30–50% better performance for LLMs.
Why quantization is risky: errors compound
A forward pass is thousands of dependent operations; in decode, each token's KV feeds every later token. Small rounding errors don't stay small. Watch precision compound through a trivial example — squaring and cubing π at three precisions:
| π precision | π² | π³ |
|---|---|---|
| 3.14159 | 9.869588 | 31.006198 |
| 3.14 | 9.8596 | 30.959144 |
| 3 | 9 | 27 |
The error in the input precision is amplified by every operation downstream. Most of quantization research is (a) preventing these errors and (b) minimizing their impact on the final output.
5.1.1 Number formats¶
To reason about quantization you must read number formats fluently. Every format has a precision (bit count), a type (integer or floating-point), and — once quantized — a scale factor that maps the low-precision values back toward the original range.
Two derived properties decide how well a format represents real model values:
- Dynamic range — the spread between the smallest and largest representable value. This is what lets a format capture outliers without clipping them.
- Granularity — how many values share a single scale factor. Finer granularity = less chance of one outlier distorting its neighbors, at the cost of storing more scale factors.
Integer vs floating-point: it's about dynamic range¶
An n-bit format has 2ⁿ distinct codes either way — INT8 and FP8 both have 256 — but they place those codes differently.
- An integer format spaces its codes evenly. 256 equal steps across a range.
- A floating-point format spaces them logarithmically — dense near zero, sparse far out — using three fields:
FP8 in E4M3 layout (8 bits)
┌─┬─┬─┬─┬─┬─┬─┬─┐
│S│ E E E E │ M M M │
└─┴─────────┴───────┘
sign exponent mantissa
(1) (4) (3)
value ≈ (−1)^S × 1.MMM × 2^(EEEE − bias)
- Sign (S) — one bit, positive or negative.
- Exponent (E) — sets the scale (which power-of-two bucket). More exponent bits = more dynamic range.
- Mantissa (M) — the precision within a bucket. More mantissa bits = finer resolution.
E4M3 means 4 exponent + 3 mantissa bits; E5M2 trades a mantissa bit for an exponent bit — less
precision, more range. Same 8 bits, different balance.
Why floating-point wins for inference: outliers
Model weights and activations are mostly small, with rare large outliers that carry disproportionate meaning. An integer format spreads its 256 codes evenly, so it either clips the outliers or wastes resolution on the dense middle. Floating-point's logarithmic spacing keeps fine resolution near zero and reaches the outliers. That extra dynamic range is why quality-sensitive inference computes in floating-point formats — integer compute (W8A8) lacks the range. Weight-only integer storage (like the W4A16 job below) sidesteps this by dequantizing to 16-bit before the math; fully-integer pipelines are reserved for size-critical local/edge inference.
The formats you'll meet¶
| Name | Abbr | First arch | Notes |
|---|---|---|---|
| 64-bit float | FP64 | Fermi (2010) | scientific computing only, never inference |
| 32-bit float | FP32 | Tesla (2006) | sometimes training, almost never inference |
| 16-bit float | FP16 | Pascal (2016) | common native training/inference precision |
| Brain float 16 | BF16 | Ampere (2020) | FP32's range, less mantissa — the usual native format |
| 8-bit float | FP8 | Hopper (2022) | the inference sweet spot |
| Microscaling FP8 | MXFP8 | Blackwell (2024) | microscaling FP8 |
| 8-bit integer | INT8 | Pascal (2016) | size-critical, lacks range |
| 6-bit float | FP6 | Blackwell (exp.) | AMD adopting quickly |
| 4-bit float | FP4 | Blackwell (2024) | aggressive; quality-risky |
| Microscaling FP4 | MXFP4 | Blackwell (2024) | microscaling FP4 |
| NVIDIA FP4 | NVFP4 | Blackwell (proprietary) | finest-grain 4-bit |
| 4-bit integer | INT4 | Turing (2018) | local/edge only |
The practical landscape: 16, 8, and 4-bit are the inference precisions. FP8 (and microscaling MXFP8) is the current sweet spot — big speedups, little quality loss, flexible enough even for the KV cache. FP4/NVFP4 is promising but quality-risky.
Granularity: tensor, channel, block — and microscaling¶
A single scale factor stretched across too many values gets dragged around by outliers. Granularity controls how finely you slice:
- Per-tensor — one scale for the entire QKV tensor. Cheapest, coarsest.
- Per-channel — a scale per feature vector (row/column). Middle ground.
- Per-block (group) — split each vector into blocks of N values, one scale per block. Finest, most metadata.
per-tensor [ one scale for everything ............................ ] coarse
per-channel [ scale | scale | scale | scale | scale | scale ....... ]
per-block [s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s|s] fine
Microscaling formats bake fine granularity into the format itself. MXFP8/MXFP4 compute a blockwise scale every 32 values, clawing back the dynamic range a raw 8/4-bit format loses. NVFP4 goes finer still — block size 16 plus a secondary FP32 global scale — specifically to fight 4-bit quality loss.
The microscaling trade
Finer scaling means more scale factors to store and apply, eating into the speedup — and you now apply both block and tensor scales. Blackwell offsets this by applying scale factors directly in the Tensor Cores. Net: microscaling formats give you 4-bit storage with closer-to-8-bit quality, at a small compute cost the hardware mostly hides.
How quantization actually maps numbers: the affine transform¶
The formats above are the destination. The mechanism that moves a high-precision value into a low-precision grid is the affine mapping — worth seeing once, because scale and zero-point are exactly the knobs the formats above are configuring.1
To quantize a real value \(x\) to an integer code \(q\):
- Scale
S— the size of one quantization step: the real-world distance between adjacent integer codes. \(S = \dfrac{x_{\max} - x_{\min}}{q_{\max} - q_{\min}}\). SmallerS= finer resolution but narrower range. This is the "scale factor" the formats and granularity sections keep referencing — per-tensor gives every value the sameS; per-block gives each block its own. - Zero-point
Z— the integer code that represents real 0. It shifts the grid so that zero lands exactly on a code, with no rounding error.
real values x_min ───────────── 0 ──────────────────── x_max
│ │ │
quantize (÷S, +Z) ▼ ▼ ▼
integer codes q_min ───────────── Z ──────────────────── q_max
(e.g. -128) (exact zero) (e.g. 127)
Why exact zero matters (the 'tare' intuition)
Zero is special — padding, masked attention positions, and post-ReLU activations are all literally zero, and they're everywhere. If your mapping represented 0 as "approximately 0.04," that tiny error would be injected into millions of values and compound. The zero-point is the tare on a kitchen scale: you zero it before weighing so the container's mass never pollutes the measurement. Force real 0 onto an exact code and a whole class of error vanishes.
When \(Z = 0\) the mapping is symmetric (range centered on zero, slightly faster); when \(Z \neq 0\) it's asymmetric/affine (better for lopsided ranges like ReLU outputs, which are never negative). That choice is one of the real knobs in a quantization recipe.
5.1.2 Quantization approaches¶
After picking a precision, two questions remain: when do you quantize, and what do you quantize?
When: during training vs after¶
- Quantization-aware training (QAT) — train the weights and the scale factors together, so the converged model is already accurate at the target precision. Best quality; only the model's creator can do it. Some labs ship QAT models (GPT-OSS in MXFP4, Kimi K2 Thinking in INT4).
- Post-training quantization (PTQ) — convert finished weights to a new precision, computing scale factors and preserving accuracy via calibration (running sample data through to measure real value ranges). This is what inference engineers do, since you work with finished open weights.
A leading PTQ tool is NVIDIA TensorRT Model Optimizer (ModelOpt) — also does pruning, distillation, sparsity; outputs run on vLLM, SGLang, and TensorRT-LLM.
What: the sensitivity ladder¶
Not all components tolerate quantization equally. Quantize from least to most sensitive, and stop before you hurt quality:
QUANTIZATION RISK safe ▲
┌───────────────────────┐ │
│ 1. Weights (linear) │ least sensitive — biggest, most redundant
│ 2. Activations │ somewhat sensitive
│ 3. KV cache │ moderately sensitive
│ 4. Attention (softmax)│ highly sensitive — quantize last, if ever
└───────────────────────┘ │
risky ▼
- Weights (especially linear layers) — least sensitive; they're the bulk of the model and individually redundant.
- Activations — the intermediate outputs; somewhat sensitive. (The activation functions themselves are rarely quantized — too tiny to matter.)
- KV cache — moderately sensitive, but quantizing it is a force multiplier: more cache fits in memory and reads faster, boosting prefix caching and disaggregation directly.
- Attention (the softmax path) — highly sensitive. All but the most aggressive schemes run softmax in full precision.
Why attention is the riskiest to quantize
Two compounding reasons. First, softmax is exponential — it's exquisitely sensitive to dynamic range, and low precision distorts the distribution. Second, and worse: each token's attention depends on every prior token's KV. A precision error in attention doesn't stay local — it feeds forward into thousands of downstream tokens and snowballs. The π-cubing table at the top of this page, but a thousand steps deep. That's why the sensitivity ladder ends here.
Even within "safe" components you can be selective: early and late layers (input/output) are more sensitive than the middle, so they're often left in original precision. A solid moderate recipe: FP8 (ideally microscaling MXFP8 for its dynamic range) on select linear layers, activations, and often the KV cache — leaving attention's internals alone.
Integer formats and the local-inference exception
The data-center rule is "stick to floating-point." Local/edge inference flips it: when squeezing DeepSeek onto a MacBook, size beats range. GGUF is the popular format for distributing heavily quantized models on Hugging Face, and dynamic quantizations (Unsloth's famous 1.58-bit) keep sensitive layers high-precision while crushing the rest — averaging out to sub-2-bit. Brilliant for local, but production quality-sensitive work should stay in floating-point.
5.1.3 Measuring quality impact¶
The bar for production quantization is zero perceptible quality loss. You can't eyeball that — you measure it, three ways, always apples-to-apples against the original weights:
- Perplexity — the cheapest check. Give the model known text and measure how well it predicts the actual next tokens. Perplexity is how "surprised" the model is by correct text — lower is better. After quantizing, you want a minimal increase. Fast, but coarse.
- Intelligence benchmarks — run a public suite (MMLU, SWE-bench) and compare scores. You want a minimal reduction.
- Custom evals — a product-specific evaluation matching your real usage. The most meaningful, and the one that should gate a deploy.
Quantization is a dial, not a switch
Because LLMs are non-deterministic, scores vary run to run — you're looking for a difference indistinguishable from noise, not zero. And you have continuous control: FP8 instead of FP4, or weights-only instead of weights+KV, trade a little speed for a lot less risk. Run all three checks, pick the most aggressive setting that still passes your custom eval, and no further.
Worked example: taking Qwen from BF16 to INT4¶
Let's run the whole pipeline on a real target: Qwen2.5-7B, shipped in BF16, which we quantize down to INT4 weights. This is the most common job, and yields Qwen2.5-7B-W4A16 — 4-bit weights, 16-bit activations. The activations stay at 16-bit on purpose: the sensitivity ladder says weights tolerate quantization best, so we crush them and leave everything else alone.
Step 1 — the math, on eight real weights¶
Quantization happens per group of weights sharing one scale (here a group of 8; production uses
~128). Take one group from a weight row and quantize it symmetrically (zero-point Z = 0, standard
for weights) into signed INT4 using the symmetric code range [-7, 7] (qmax = 7; the format's −8 code goes
unused to keep the grid symmetric):
weights (BF16) w = [ 0.12, -0.41, 0.93, -0.05, 0.55, -0.88, 0.27, 0.02 ]
1. absmax = max(|w|) = 0.93
2. scale S = absmax / qmax = 0.93 / 7 = 0.13286
3. quantize q = round(w / S), clamp to [-7, 7]
= [ 1, -3, 7, 0, 4, -7, 2, 0 ] ◄ stored as INT4
4. dequantize ŵ = q * S
= [0.133, -0.399, 0.93, 0.0, 0.531, -0.93, 0.266, 0.0]
error ŵ − w = [+0.013, +0.011, 0.0, +0.05, −0.019, −0.05, −0.004, −0.02]
max abs error = 0.05
What to notice:
- Each weight is now 4 bits — an integer in
[-7, 7]— plus one sharedSper group. ThatSis the per-block scale factor from the granularity section, made concrete. 0.93is exact (it set the scale), but−0.05rounded all the way to0— small values near a big outlier lose the most. That's precisely why group size matters: a smaller group means a local outlier inflates the scale for fewer neighbors. Halve the group, and−0.05might land in a group whose absmax is0.41, getting a finer scale and surviving.- The errors look tiny, but recall the π-cubing table at the top of this page — across billions of weights and thousands of dependent steps, the recipe's job is to keep them from compounding.
Step 2 — the payoff¶
Do that for all of Qwen2.5-7B's ~7.6B weights:
| Size | Note | |
|---|---|---|
| BF16 weights | 15.2 GB | 2 bytes each |
| INT4 weights | 3.8 GB | 0.5 byte each |
| + group scales (group=128) | +0.12 GB | one BF16 scale per 128 weights |
| Effective INT4 | ≈ 3.9 GB | ≈ 3.9× smaller |
A model that needed a 24 GB GPU now fits on a 12 GB consumer card — and decode, being memory-bound, gets faster because it moves ¼ the weight bytes per token.
Step 3 — but don't actually use round-to-nearest¶
The Step 1 math is round-to-nearest (RTN) — the simplest scheme, and at INT4 it loses too much quality to ship. Real W4A16 uses a smarter post-training quantization (PTQ) algorithm (the same PTQ from §5.1.2) that spends a little calibration compute to place the codes better. The two you'll meet:
- GPTQ (Generative Pre-trained Transformer Quantization) — a named PTQ algorithm that quantizes weights one column at a time, using second-order Hessian (loss-curvature) information to compensate the not-yet-quantized weights for each rounding error. It minimizes the layer's output error, not each individual weight's error.
- AWQ (Activation-aware Weight Quantization) — notices that a few weight channels matter far more (judged by activation magnitude) and scales those salient channels up before quantizing, so rounding hurts them less. Often the best quality/speed for INT4.
Both still produce a W4A16 checkpoint; they just choose q more cleverly than round(w/S).
Step 4 — the production recipe¶
Putting the whole chapter's machinery into an actual workflow:
1. DECIDE THE RECIPE
format INT4 weights, BF16 activations (W4A16)
granularity per-group, group_size = 128 (finer = better quality, more scales)
scope linear-layer weights only;
keep embeddings, LM head, and attention in BF16 (sensitivity ladder)
2. CALIBRATE
run a few hundred representative samples through the model so the
algorithm sees real activation/weight ranges (GPTQ/AWQ need this)
3. QUANTIZE
run AWQ or GPTQ via a tool — llm-compressor, AutoAWQ, or NVIDIA
ModelOpt — producing a quantized safetensors (or GGUF for local/llama.cpp)
4. EVALUATE (§5.1.3, against the original BF16 weights)
perplexity delta ≈ noise? MMLU/your-eval drop ≈ noise?
if a custom eval regresses → back off: try INT4→ a microscaling FP4,
or quantize fewer layers, or W4A16 → W8A16
5. DEPLOY
load the checkpoint on vLLM / SGLang / TensorRT-LLM — they read the
quant format and run fused-dequant kernels (e.g. Marlin) that stream
the 4-bit weights into 16-bit Tensor-Core math
The one-paragraph version
To take Qwen from BF16 to INT4: choose W4A16, per-group (128), weights-only; calibrate on representative data; run AWQ or GPTQ (never plain round-to-nearest at 4-bit); evaluate the perplexity/benchmark/custom-eval deltas against the BF16 original; and if quality regresses, dial back — a higher-granularity FP4, fewer quantized layers, or 8-bit. You get a ~4× smaller, faster model, and the entire job is managing the quality you trade for it.
Hands-on: quantizing Qwen in a Colab notebook¶
Here's the recipe above as runnable code — and you don't need to provision anything. The whole job
runs in a free Google Colab GPU notebook. We use llm-compressor, the vLLM-native
quantization library, because its output loads straight into vLLM/SGLang and its layer-targeting is
exactly the control you want.[^llmc]
Read the walkthrough below first, then open the notebook and run it cell by cell. The notebook is
this same code with per-step logging, a print(model) layer tour, and Colab memory tricks.
Runtime: pick a free T4
In Colab, set Runtime → Change runtime type → T4 GPU. A free T4 (16 GB) quantizes a small Qwen
(Qwen2.5-0.5B-Instruct, the notebook default) in a few minutes. The 7B model shown below
needs more VRAM than a free T4 gives — use a Colab Pro L4/A100, or the Compute Engine VM at the
end of this section.
Install the library¶
Faster Hugging Face downloads
Authenticated pulls get higher rate limits (and any gated models), and hf_transfer — a Rust
downloader — saturates the bandwidth. In Colab, add your token in the 🔑 Secrets panel as
HF_TOKEN, then:
The quantization script¶
This is the whole job — the BF16 weights download from Hugging Face on first call:
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
MODEL_ID = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto") # (1)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
recipe = GPTQModifier( # (2)
targets="Linear", # which modules to quantize
scheme="W4A16", # 4-bit weights, 16-bit activations, group_size 128
ignore=["lm_head"], # which modules to SKIP
)
oneshot(
model=model,
dataset="HuggingFaceH4/ultrachat_200k", # (3) calibration data
recipe=recipe,
max_seq_length=2048,
num_calibration_samples=512, # 256–512 is plenty; more = slower
)
SAVE_DIR = "Qwen2.5-7B-Instruct-W4A16-G128"
model.save_pretrained(SAVE_DIR, save_compressed=True) # (4)
tokenizer.save_pretrained(SAVE_DIR)
dtype="auto"loads the model in its native BF16.GPTQModifieris the smart PTQ from Step 3 — not round-to-nearest. Swap one import to use AWQ instead (below).- Calibration: a few hundred samples so GPTQ measures real value ranges. Use domain-matched data — for a code model, calibrate on code, not chat.
save_compressed=Truewrites the packed 4-bit checkpoint; vLLM reads it directly.
That's the whole thing. The interesting part — and what you asked for — is the targets/ignore
control.
Targeting different layers¶
targets says what to quantize; ignore says what to leave in BF16. Both accept module class
names, exact module paths, or regex with a re: prefix. This is how you walk the sensitivity
ladder in practice — quantize the robust layers hard, protect the
sensitive ones.
First, know Qwen's module paths (printable with print(model)):
model.embed_tokens ← input embedding (sensitive)
model.layers.{0..27}.self_attn.q_proj ← attention projections
.self_attn.k_proj
.self_attn.v_proj
.self_attn.o_proj
model.layers.{0..27}.mlp.gate_proj ← MLP / FFN (biggest, most robust)
.mlp.up_proj
.mlp.down_proj
lm_head ← output head (sensitive)
Now, recipes from least to most conservative:
The targeting workflow
Start with default, run your evals (§5.1.3). If quality regresses, don't abandon 4-bit — add
the regression's likely culprits to ignore and re-run: edge layers, then attention, then
down_proj. You're searching for the smallest set of BF16 exceptions that recovers quality, which
is mixed-precision quantization done by hand. Each module you move to ignore costs a little memory
and buys a little quality.
Memory tip for big models
Add sequential_targets=["Qwen2DecoderLayer"] to the modifier to quantize one decoder layer at a
time, keeping only that layer's activations in memory. Essential when the model barely fits —
it's how the same recipe scales from 7B to 70B+.
Use AWQ instead (one import)¶
AWQ often edges out GPTQ at 4-bit (it protects salient channels — Step 3). Same call, different modifier:
from llmcompressor.modifiers.awq import AWQModifier
recipe = AWQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
# ...identical oneshot(...) and save_pretrained(...)
Save, then serve and evaluate¶
save_compressed=True writes the packed ~3.9 GB checkpoint. Colab's disk is ephemeral, so persist
it before the runtime recycles — copy to Google Drive, push to a Hugging Face Hub repo, or gcloud
storage cp it to a bucket. Then serve it from any GPU box (a smaller one — INT4 needs ~¼ the VRAM);
vLLM needs no special flags, it detects the quantization from the saved config:
Then evaluate — and this is the part that decides whether the quantized model actually ships.
Evaluate before you ship¶
The bar from §5.1.3 is zero perceptible loss, and you prove it by measuring every metric as a delta against the original BF16 weights — same harness, same prompts, same sampling params — then gating on thresholds you agreed up front. Four checks, cheap to decisive:
| Check | Measures | Tool | You want |
|---|---|---|---|
| Perplexity | gross breakage (tripwire) | lm-eval wikitext |
small +Δ |
| Benchmarks | capability loss | lm-eval MMLU / GSM8K / ARC |
drop within noise |
| Custom eval | your job — the deploy gate | your graders | holds vs baseline |
| LLM-as-judge | generative quality (tone, format) | stronger model, blind | no preference shift |
Run the harness twice — original, then candidate — and diff the reports:
pip install lm-eval
# baseline: the original BF16 weights
lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-7B-Instruct \
--tasks wikitext,mmlu,gsm8k,arc_challenge --batch_size auto --output_path out/bf16
# candidate: the quantized checkpoint (vLLM detects the quant format)
lm_eval --model vllm --model_args pretrained=./Qwen2.5-7B-Instruct-W4A16-G128 \
--tasks wikitext,mmlu,gsm8k,arc_challenge --batch_size auto --output_path out/w4a16
The methodology that separates a real gate from theatre:
- Baseline-relative, not absolute.
62.1% → 61.8%is the signal, not the 62%. Size your noise floor first by running the baseline against itself twice; use greedy decoding (temp=0) and a fixed eval set to keep the band tight. A drop only counts if it clears that band. - The custom eval gates — benchmarks don't. A model can hold MMLU and still regress on your domain. Your golden set of real prompts, with automatic graders (exact-match, regex, JSON-schema, or a judge), is what says ship / no-ship.
- Look past accuracy. Re-run your format/function-calling and safety/refusal suites
(quantization quietly breaks structured output), test long context (the KV cache and RoPE
interact badly with low precision), and confirm you actually got faster — benchmark served
throughput and p99 latency on vLLM (
vllm bench throughput), since speed was the whole point.
The release decision
Put every delta in one table, apply the thresholds, and let the numbers decide. Pick the most
aggressive quant setting that still passes your custom eval, and no further. If a gate fails,
don't abandon 4-bit — add the likely-sensitive layers to ignore (edge layers, then attention,
then down_proj), or step W4A16 → W8A16, and re-measure.
The Colab notebook
runs this whole gate as code — perplexity, lm-eval benchmarks, a custom golden-set eval, and a
pass/fail release table comparing BF16 against the W4A16 checkpoint.
Pass the gate and you've got a ~4× smaller model serving on a quarter of the VRAM.
Prefer a dedicated VM? Provision one on Compute Engine
Colab is the fastest way to try this. For the full 7B run, a repeatable pipeline, or to keep the
checkpoint on Google Cloud, run the same script on an ephemeral GPU VM instead. A Deep
Learning VM image ships with CUDA and the driver, so there's nothing to install at the system
level. Provision an L4, SSH in, pip install llmcompressor, run the script, push the result, and
delete the instance — a forgotten GPU VM is the classic surprise bill:
# europe-west4 (NL): good L4/A100 stock. SPOT: ~60–70% cheaper for a throwaway job.
gcloud compute instances create qwen-quant \
--zone=europe-west4-a --machine-type=g2-standard-8 \
--accelerator=type=nvidia-l4,count=1 --provisioning-model=SPOT \
--maintenance-policy=TERMINATE --image-family=common-cu124 \
--image-project=deeplearning-platform-release \
--metadata="install-nvidia-driver=True" --boot-disk-size=200GB
gcloud compute ssh qwen-quant --zone=europe-west4-a
# on the VM: pip install llmcompressor && python quantize.py
gcloud storage cp -r ./Qwen2.5-7B-Instruct-W4A16-G128 gs://YOUR_BUCKET/models/
gcloud compute instances delete qwen-quant --zone=europe-west4-a --quiet # stops all billing
For bigger models, step up the accelerator: a2-highgpu-1g (A100 40 GB) or an a3 machine (H100).
Productionizing the job
The hand-run job above is fine for a one-off. For a repeatable pipeline — new model versions, scheduled re-quantizes, your whole model catalog — you want this as an automated, scale-to-zero batch job on Kubernetes. That's a full hands-on of its own: Chapter 7 → A Quantization Pipeline on GKE.
Quantization makes every other technique cheaper — fewer bytes to cache, transfer, and parallelize. Next: spending the compute that decode leaves idle.
Next: Speculative Decoding →
-
The affine-mapping treatment (scale
S, zero-pointZ, the tare analogy, per-tensor/channel/block granularity) follows Vivek Kalyanarangan, Quantization and Fast Inference: A Practitioner's Guide to Efficient AI (Manning, 2026), ch. 2 — recommended for a from-first-principles build-up of the fixed-point and floating-point machinery underneath these formats. ↩