Skip to content
Multigrid

Running speech and embedding models locally

Whisper and the open embedding models on your own hardware — sizing them, serving them, and what quantizing them costs.

Speech recognition and text embedding are the two workloads that most often move off a hosted API and onto a machine somebody owns. They are not large by current standards — the biggest Whisper checkpoint is 1.55 billion parameters and a strong embedding model is a tenth of that — and they are both run in bulk, on data that is frequently the kind nobody wants to send anywhere. A transcription backlog and a corpus re-embedding are exactly the jobs where per-request pricing turns into a number somebody notices.

What makes them awkward is that almost nothing about their cost is published in a form you can use. Vendors publish parameter counts, vocabulary sizes, context lengths and licences; they do not publish tokens per second on your CPU, and where they publish a benchmark it is on hardware you do not have. These pages therefore do two things: they source what is genuinely published and date it, and where nothing is published they do the arithmetic in the open — from parameter counts, bytes per weight, sequence lengths and documented window sizes — so that you can see which assumption to change when your machine disagrees.

Transcribing Audio Locally With whisper.cpp

Build whisper.cpp, convert your audio to the one format it accepts, transcribe offline, and pick a quantization level from the bits-per-weight arithmetic rather than by guessing.

9 min read

Transcribing Audio Locally With faster-whisper

Install faster-whisper, get a transcript out of it, and read its own published benchmark table as real-time factors rather than as wall-clock times on somebody else's GPU.

9 min read

distil-whisper Explained

How cutting Whisper's decoder from 32 layers to 2 produces a model half the size and six times the speed, with the parameter arithmetic reproduced from the layer geometry.

9 min read

Choosing a Whisper Model Size for Your VRAM Budget, Derived

The weight footprint of every Whisper size worked out from parameter count and precision, and an account of where the other two-thirds of the quoted VRAM requirement goes.

9 min read

The Latency Budget for Real-Time Local Whisper Transcription

Why shrinking the audio chunk does not shrink Whisper's work, and the arithmetic that decides the smallest chunk a given machine can sustain.

9 min read

How Slow Whisper Is on CPU Only, Derived

Real-time factors for CPU transcription derived from a published benchmark with its hardware named, scaled across model sizes, plus the command to settle it on your own machine.

9 min read

Whisper Accuracy by Model Size, Sourced

What OpenAI actually publishes about Whisper's accuracy, the per-size LibriSpeech numbers printed on the model cards, and why the table inverts at the top.

9 min read

What Local Whisper Doesn't Do: Speaker Diarization

Why Whisper's vocabulary contains no speaker token, what a separate diarization pass has to compute, and where the two outputs disagree when you join them.

9 min read

Running BGE Embedding Models Locally

Get a BGE model producing vectors on your own machine, verify that the pooling really is CLS, and understand why its similarity scores start at 0.6.

9 min read

Running E5 Embedding Models Locally

Run an E5 model on your own hardware with the query and passage prefixes it was trained with, and the masked mean pooling that its API will not enforce for you.

9 min read

Running Nomic Embed Locally

Run Nomic Embed offline with its four task prefixes, handle the trust_remote_code requirement that complicates 'offline', and truncate its Matryoshka dimensions safely.

9 min read

Running GTE Embedding Models Locally

What Alibaba changed in the GTE v1.5 backbone to get 8192 tokens out of a BERT-sized encoder, and what running its quantized variant actually changes.

9 min read

Serving Embeddings With llama.cpp

Serve a GGUF embedding model over llama.cpp's HTTP endpoints, with the pooling flag set to match the model rather than left to a default.

9 min read

Quantizing an Embedding Model: What Quality You Actually Lose

The two different things called 'quantized embeddings', the one figure that is actually published, and the method for measuring drift on your own corpus.

10 min read

Exporting an Embedding Model to ONNX for Local Inference

Export a sentence-transformers model to ONNX, prove the exported vectors match the PyTorch ones, and decide whether the optimisation and quantization steps are worth taking.

9 min read

Multilingual Embedding Models for Local RAG

Why two-thirds of a multilingual embedding model's parameters are a vocabulary table, what that does to memory and speed, and how tokenizer fertility eats your context.

10 min read

CPU-Only Embedding Inference Speed, Derived

Documents per second derived from a model's FLOPs per document and a CPU's peak throughput, with every assumption labelled and the measurement command that replaces the estimate.

10 min read

Picking an Embedding Model That Fits in 8GB of RAM

What is left of 8GB after the operating system and the runtime, what each model's weights cost, and why batch size times sequence length squared is the term that kills you.

10 min read

Other topics