Skip to content
Multigrid

llama.cpp, flag by flag

The flags that decide whether a local model fits, runs fast, or silently truncates your context — read from the behaviour rather than the help text.

llama-server --help prints something like two hundred options, each with one line of description, and almost none of those lines tells you what the flag costs. -c 32768 looks free until the KV cache it implies does not fit. -ngl 20 looks conservative until you learn that the last layer is worth more than the first nineteen. --mlock looks like a speed flag and is really a gamble on how much RAM you have left.

These pages take one flag or one pair at a time and work out what it actually does to memory, to latency and to output quality, starting from the arithmetic llama.cpp prints at load time rather than from rules of thumb. Where a number depends on your hardware, the page gives you the command that measures it on your machine instead of quoting one from somebody else’s.

llama.cpp's -ngl Flag: How Many Layers to Offload to GPU

How to derive an -ngl value from the model's layer count and your free VRAM, and why the last layer matters more than any of the others.

10 min read

llama.cpp's --ctx-size Flag and What It Costs in Memory

The KV-cache arithmetic that turns a --ctx-size number into megabytes, using the hyperparameters llama.cpp prints when it loads the model.

10 min read

RoPE Scaling in llama.cpp for Extending Context

What --rope-scaling, --rope-freq-base and --rope-freq-scale do to positional encoding, and what attention looks like when you set them wrong.

10 min read

Constrained Generation With llama.cpp Grammars (GBNF)

How GBNF constrains sampling token by token, with a grammar file that forces a specific JSON shape out of any local model.

10 min read

Running llama.cpp's Server in OpenAI-Compatible Mode

Starting llama-server, calling /v1/chat/completions with curl, and knowing which parts of the OpenAI shape are and are not implemented.

10 min read

Quantizing Your Own Model With llama-quantize

Converting a Hugging Face checkpoint to GGUF and quantizing it yourself, including what an importance matrix changes and what it does not.

11 min read

llama.cpp's mmap and mlock Flags Explained

Why memory-mapping a GGUF avoids loading it twice, what mlock buys and risks, and the --load-mode flag that now replaces both.

9 min read

Batch and Micro-Batch Size Flags in llama.cpp

How --batch-size and --ubatch-size divide the work of prompt processing, and which of the two actually sizes your compute buffer.

9 min read

Tuning llama.cpp's Threads Flag for CPU-Only Inference

Why more threads stop helping generation long before they stop helping prompt processing, and how to find your own maximum with llama-bench.

10 min read

Running llama.cpp on AMD and Intel GPUs With the Vulkan Backend

Building llama.cpp with GGML_VULKAN, confirming the driver picked your GPU, and serving tokens on non-NVIDIA hardware without ROCm.

10 min read

Speculative Decoding in llama.cpp: Setting Up a Draft Model

A working two-model speculative setup on llama-server, using the current --spec- flags rather than the ones older guides name.

10 min read

Choosing a Draft Model for Speculative Decoding

The vocabulary and size constraints that decide whether a draft model speeds anything up, including the compatibility checks llama.cpp performs at load.

9 min read

Min-P Sampling Explained

How min-p's relative threshold tracks the model's confidence, why it is a single subtraction in logit space, and where it sits in llama.cpp's sampler chain.

8 min read

Repeat Penalty Explained

What the repetition penalty does to a logit, why llama.cpp multiplies negative ones instead of dividing, and why a high value breaks text that is supposed to repeat.

9 min read

Logit Bias for Local Inference in llama.cpp

How the --logit-bias flag and the server's logit_bias field change a token's odds, with the exact syntax, and how to ban a token outright.

8 min read

Measuring Quantization Quality With llama.cpp's Perplexity Tool

Running llama-perplexity over two quantisations of the same model on your own text, and reading the result without over-claiming.

10 min read

Serving Parallel Requests With llama.cpp's Server Slots

How --parallel divides the KV budget into per-slot contexts, what the unified KV buffer changes about that division, and how to size both.

9 min read

Running llama.cpp With the Metal Backend on Apple Silicon

Building llama.cpp on Apple Silicon and proving from the logs and a controlled benchmark that Metal, not the CPU, is doing the matrix multiplications.

9 min read

The Flash Attention Flag in llama.cpp and What It Saves

What --flash-attn changes in llama.cpp's attention graph, with the compute-buffer arithmetic worked through at a stated context and batch size.

9 min read

Prompt Caching in llama.cpp's Server

How the server reuses a KV prefix between requests, what invalidates it, and how to work out the prefill it saves you on a repeated system prompt.

9 min read

Other topics