Quantizing a Model for MLX

A quantized MLX model is not simply the weights at four bits. It is the weights at four bits plus a scale and a bias for every group of them, and that second part is why a “4-bit” model is 4.5 bits per weight on disk. Knowing that in advance means you can predict the output size and notice when the result disagrees.

What MLX quantization stores

MLX’s default scheme is called affine. Weights are cut into contiguous groups; within each group, values are mapped onto a small integer range by a scale and an offset chosen for that group alone, and the scale and offset are stored alongside. The integers are packed several to a machine word. Dequantization is per group, so a single large outlier can only distort the sixty-four weights it shares a group with, rather than a whole tensor.

The defaults are in mlx-lm’s source rather than in prose: for affine, group size 64 and 4 bits. So per group of 64 weights the file holds 64 × 4 = 256 bits of packed weight, plus one fp16 scale and one fp16 bias, which is 32 bits more. That is 288 bits for 64 weights:

(64 * 4 + 16 + 16) / 64  =  288 / 64  =  4.5 bits per weight

Not every parameter is quantized. Layer norms and anything whose dimensions are not divisible by the group size stay in fp16, so the real figure is a shade above 4.5 — but only a shade, as the check below shows.

Predict the size first

Take a model whose parameter count is published. Hugging Face reports the safetensors index for meta-llama/Llama-3.3-70B-Instruct as 70,553,706,496 parameters, all bf16. At 4.5 bits per weight:

70,553,706,496 x 4.5 / 8  =  39,686,459,904 bytes  =  39.7 GB

The pre-converted mlx-community/Llama-3.3-70B-Instruct-4bit repository, read through the Hugging Face model API on 11 August 2026, holds eight safetensors shards totalling 39,688,567,605 bytes. The prediction is off by about two megabytes in forty gigabytes, and the discrepancy is the unquantized norms plus the safetensors headers.

The same arithmetic checks out on the other two repositories that were looked at: Qwen3-14B at 14,768,307,200 parameters converts to 8,307,898,514 bytes, and Llama-3.1-8B-Instruct at 8,030,261,248 parameters converts to 4,517,489,037 bytes. Divide bytes by parameters and multiply by eight and all three land on 4.50 bits per weight.

Run it and check

1. Install the tooling if you have not: pip install mlx-lm. See converting a model to MLX format for the native-Python check and Hub authentication.
2. Convert without quantization to a reference directory, so you have something to compare against: mlx_lm.convert --hf-path Qwen/Qwen3-14B --mlx-path ./qwen3-14b-fp16 --dtype float16.
3. Convert again with quantization on:

   mlx_lm.convert \
     --hf-path Qwen/Qwen3-14B \
     --mlx-path ./qwen3-14b-4bit \
     -q --q-bits 4 --q-group-size 64

4. Compare the directories: du -sh ./qwen3-14b-fp16 ./qwen3-14b-4bit. The fp16 copy is 2 bytes per parameter, the 4-bit copy is 0.5625 bytes per parameter, so the ratio should be close to 3.56 to one.
5. Confirm the metadata: cat ./qwen3-14b-4bit/config.json contains a quantization block recording bits, group_size and mode. That block is how load() knows to build quantized layers, so a model that loads as full precision despite small files has lost it.
6. Generate once to prove it runs: mlx_lm.generate --model ./qwen3-14b-4bit --prompt "hello". Note the Peak memory line it prints — it should be a little over the directory size, the excess being the KV cache and activations.

Modes, bits and mixed recipes

--q-mode takes affine, mxfp4, nvfp4 or mxfp8, and each carries its own defaults for group size and bits: affine is 64 and 4, mxfp4 is 32 and 4, nvfp4 is 16 and 4, mxfp8 is 32 and 8. The microscaling formats store an exponent per group rather than a full scale and bias, so their per-group overhead is smaller than affine’s at the same group size — which is the point of them.

--q-bits accepts values below 4. Two-bit and three-bit conversions run, produce startlingly small directories, and degrade much more sharply than the size suggests, because the number of representable levels per group collapses from sixteen to four.

Between the two there is --quant-predicate, which takes one of four named mixed recipes — mixed_2_6, mixed_3_4, mixed_3_6, mixed_4_6 — that assign more bits to the layers that tolerate it least. The rule in the source is explicit: the value projections and the down projections in the first eighth and last eighth of the layers, plus one layer in every three in between, get the high bit count, and so does the output head; everything else gets the low one. It is modelled on the same intuition as llama.cpp’s K quants.

What you lose

Quantization error is not spread evenly across a model, and that is the mechanism behind almost every quality complaint. Transformer activations contain a small number of features with magnitudes far outside the rest of the distribution, and a group containing one of them has its scale stretched to cover it — which spends most of the representable range on one value and leaves the other sixty-three sharing what is left. Smaller groups reduce the blast radius; mixed recipes protect the layers where it hurts most; going below four bits removes the headroom that made either workable.

The practical consequence is that degradation shows up first in the places where the model was already marginal — long multi-step reasoning, exact recall of rare facts, strict formatting under a schema — and not in fluent short prose, which is why a quick chat with a 2-bit model can feel fine and a week of real work with it does not. Choosing a quantization level covers the general question; picking one for a Mac’s unified memory budget derives it from the memory you actually have.