How Much VRAM a 7B Model Needs at Q4, Q5 and Q8
10 min read · updated August 11, 2026
Three quantities occupy your card: the weights, the key/value cache for the context, and whatever the runtime holds before either loads. Only the first depends on the quantization level, and it is the only one most tables show.
The three terms
Total resident memory is the sum of three things, and it is worth writing them separately because they scale with different variables:
VRAM = weights + kv_cache + runtime weights = P * bpw / 8 bytes kv_cache = 2 * L * H_kv * D * C * b bytes runtime = CUDA context + compute buffer bytes P parameter count bpw bits per weight of the quantization type L number of transformer layers H_kv number of key/value heads (not attention heads) D head dimension C context length in tokens b bytes per KV element (2 for fp16, 1 for q8_0)
The 2 in the KV term is for the two tensors, keys and values. The variable that trips people up is H_kv: on a model with grouped-query attention the number of key/value heads is much smaller than the number of attention heads, and it is H_kv that sets the cache size. Two models of nominally the same size can differ by 4x on this term alone, which is the single largest source of disagreement between published tables.
Notice which variables are absent. Nothing in the equation depends on how many tokens you generate, on the sampler, on temperature, or on how long the process has been running. Memory in a local runtime is almost entirely static: it is decided by the model file, the context you configured and the device you loaded onto, and then it does not move. That is why a load either fits or does not, and why an inference process that survives its first request will usually survive the thousandth. The exception is the compute buffer, which grows with the batch of tokens processed in one pass rather than with the session.
A 7B is not 7 billion parameters
The name is a rounding. Every parameter count below is computed from the shapes in the model’s own configuration — per layer, four attention projections plus three feed-forward matrices plus two RMSNorm vectors, plus the token embedding and the output head:
per_layer = hidden*H*D + 2*hidden*H_kv*D + H*D*hidden (q, k+v, o)
+ 3*hidden*intermediate (gate, up, down)
+ 2*hidden (two norms)
Llama 2 7B 32 layers, hidden 4096, 32 heads, 32 kv heads, ffn 11008, vocab 32000
-> 6,738,415,616 params (6.74B)
Mistral 7B 32 layers, hidden 4096, 32 heads, 8 kv heads, ffn 14336, vocab 32000
-> 7,241,732,096 params (7.24B)
Llama 3.1 8B 32 layers, hidden 4096, 32 heads, 8 kv heads, ffn 14336, vocab 128256
-> 8,030,261,248 params (8.03B)The spread is 19% between the smallest and the largest, and it is mostly vocabulary: Llama 3.1’s 128,256-token vocabulary costs 1.05 billion parameters across the embedding and the untied output head, against 262 million for a 32,000-token vocabulary. That single difference is larger than the gap between Q4_K_S and Q4_K_M.
It is worth being precise about “untied”. Some models reuse the embedding matrix as the output projection — the two are the same weights, so the vocabulary is paid for once. Others learn a separate output head and pay twice. Small models increasingly tie them, because at three billion parameters a 128,256-token vocabulary counted twice would be a fifth of the model. Whether a given checkpoint ties is stated in its configuration, and if a derived parameter count comes out about a billion too high on a large-vocabulary model, that is the first thing to check.
Bits per weight, from the tool that writes the file
You do not have to estimate this. llama.cpp’s quantize README publishes a measured bits/weight and file size for Llama-3.1-8B at every supported type. The k-quant column, at the time of writing:
type bits/weight size for Llama-3.1-8B Q2_K 3.1593 2.95 GiB Q3_K_M 3.9960 3.74 GiB Q4_K_S 4.6672 4.36 GiB Q4_K_M 4.8944 4.58 GiB Q5_K_M 5.7036 5.33 GiB Q6_K 6.5633 6.14 GiB Q8_0 8.5008 7.95 GiB F16 16.0005 14.96 GiB
Note that the name lies slightly in the same direction every time. “Q4” is not 4 bits per weight: the k-quant block layout carries scales and minimums, and the mix used by the _M variants keeps some tensors at higher precision. Q8_0 at 8.5008 is the block layout exactly — 32 weights of 8 bits plus one fp16 scale is (32×8+16)/32 = 8.5.
The weights table
Multiply the parameter count by the bits per weight and divide by eight. Sizes are in GiB, because that is what a card reports:
Q2_K Q3_K_M Q4_K_S Q4_K_M Q5_K_M Q6_K Q8_0 F16 Llama 2 7B 2.48 3.13 3.66 3.84 4.47 5.15 6.67 12.55 Mistral 7B 2.66 3.37 3.93 4.13 4.81 5.53 7.17 13.49 Llama 3.1 8B 2.95 3.74 4.36 4.58 5.33 6.14 7.95 14.96 worked example, Llama 3.1 8B at Q4_K_M: 8,030,261,248 * 4.8944 / 8 = 4,913,000,000 bytes approx = 4.58 GiB
The Llama 3.1 8B row is not a derivation at all — it is llama.cpp’s own published figure, and the derived value lands on it. That is the check that the parameter counts for the other two rows are right.
One unit trap costs people a gigabyte of imagined headroom. A file listing that says “4.92 GB” and a card that says “8 GB” are not using the same gigabyte: model repositories usually report decimal GB, while GPU capacity and every allocator message are binary. 4.92 GB is 4.58 GiB, and 8 GB of VRAM is 8 GiB, which is 8.59 decimal GB. The gap is 7.4% and it grows with size — at the 70B scale it is three gigabytes. Every figure on this page is GiB, and the safe habit is to convert everything to GiB the moment it enters your budget, because that is the unit the failure will be reported in.
Two more things the weights table does not capture. First, an importance-matrix quantization — llama.cpp’s --imatrix option, and the reason so many published GGUFs advertise “imatrix quants” — changes which weights are rounded where, not how many bits they occupy, so it does not move any number in this table. Second, the i-quant family (IQ2, IQ3, IQ4) reaches bits-per-weight values between the k-quants and below them, and llama.cpp publishes those in the same README: IQ4_XS at 4.4597 sits just under Q4_K_S, which on a 7B is about 150 MiB.
The context term, which is where they differ
Per token, the cache costs 2 * L * H_kv * D * b bytes. For a 32-layer model with head dimension 128 and fp16 cache:
Llama 2 7B (32 kv heads): 2*32*32*128*2 = 524,288 B/token = 512 KiB Mistral 7B ( 8 kv heads): 2*32* 8*128*2 = 131,072 B/token = 128 KiB Llama 3.1 8B ( 8 kv heads): 2*32* 8*128*2 = 131,072 B/token = 128 KiB at 8,192 tokens of context, fp16: Llama 2 7B 4.00 GiB Mistral 7B 1.00 GiB Llama 3.1 8B 1.00 GiB
Four gigabytes against one, for two models a table would put on the same row. Grouped-query attention is why: Llama 2 7B gives every attention head its own key and value projection, while Mistral 7B and Llama 3.1 8B share one KV head between four attention heads. If you take one number away from this page, take that one — a “7B needs 6 GB” claim is meaningless without saying which 7B and at what context. The mechanism is covered in more depth under how the KV cache works.
Adding it up
Take Llama 3.1 8B at Q4_K_M with 8,192 tokens of fp16 context, and allow 1.0 GiB for the runtime — a working assumption, not a measurement, and one you should replace with your own reading as described under what CUDA reserves before your model loads:
weights 4.58 GiB kv cache 1.00 GiB (8192 * 128 KiB) runtime 1.00 GiB (assumed; measure yours) -------- -------- total 6.58 GiB -> fits an 8 GiB card with 1.4 GiB spare
Same model at Q6_K and 16,384 tokens: 6.14 + 2.00 + 1.00 = 9.14 GiB, which does not. Same model at Q4_K_M with the KV cache in q8_0 instead of fp16 halves the middle term, buying back a gigabyte for a rounding error in cache precision. The three levers are quant, context length and cache precision, and they trade against each other in units you can now compute rather than guess. For the same arithmetic run backwards — fixed card, solve for context — see fitting a model and its context in 8 GiB.