Multilingual pipelines
Building a pipeline that survives more than one language: prompt structure, code-switching, detection, routing and evaluation when the input is not English.
A pipeline that works in one language is not a pipeline that works in eleven with a translation step bolted on. The parts that break are not the parts anyone expects: the instruction that carried an English idiom into Japanese, the schema whose enum values were localised and now no longer match the branch in your code, the retrieved documents that quietly overruled an explicit output-language directive, the OCR stage that reported ninety-something percent accuracy on a corpus whose hard cases were four percent of the characters.
These pages work through those mechanisms one at a time. The prompting half covers where language belongs in a template, why a model drifts back into the user’s language, and how to test that one prompt behaves the same way in ten. The document half covers why a script’s own structure — cursive joining, stroke density, absent word spaces, fused conjuncts, a syllabary of several hundred symbols — sets a floor on recognition error that no amount of training data removes. Almost none of these failures raise an exception, so each page ends with what to measure.
Building a Prompt Template With Language as a Variable
Which parts of a prompt should switch with the target language and which should stay fixed, built as one template with a language parameter.
9 min read
Forcing a Model to Always Respond in One Language
Why an output-language instruction loses to the language of the input, and the placement and reinforcement pattern that holds it.
9 min read
Writing System Prompts for a Product Used in Many Languages
A system prompt split into language-agnostic rules and per-language overrides, so a new supported language is a data change.
9 min read
Why Translating a Prompt Word for Word Produces Worse Output
An instruction's force comes from convention, not semantics, and a literal translation preserves the meaning while destroying the convention.
9 min read
Testing Whether a Prompt Works the Same Across Languages
A harness that scores one prompt against one rubric in every supported language, and separates language-specific failures from general ones.
10 min read
Why Politeness in a Prompt Changes Output More in Some Languages
In languages where politeness is grammatical rather than lexical, the model must commit to a level on every verb, and it takes the cue from your prompt.
9 min read
Specifying Output Register, Formal or Informal, by Language
Register as a per-language grammatical rule with a matching assertion, rather than the word 'formal' in a prompt.
9 min read
Why a Prompt That Works in English Fails Silently in Another Language
Six failure classes that degrade an English-tuned prompt in another language without producing an error or an obviously wrong answer.
9 min read
Handling Code-Switched Prompts Where the User Mixes Languages
How to decide which language to answer in when a single message switches languages, and how to tell deliberate mixing from a paste error.
9 min read
Why Output Language Sometimes Ignores an Explicit Instruction
When a large block of other-language text in the context outweighs an explicit output-language directive, and how to demote it to data.
9 min read
OCR Challenges Specific to Arabic's Cursive Script
Why joined letters, four positional forms and dot-only distinctions put a structural floor under Arabic OCR error rates.
9 min read
OCR Accuracy for Chinese Characters at Low Resolution
Why a Chinese character needs several times the pixel area of a Latin letter to stay distinguishable, worked through to a DPI floor.
9 min read
OCR for Thai Script Without Spaces Between Words
Character recognition and word segmentation are two different problems in Thai, and only one of them is an OCR problem.
9 min read
OCR for Devanagari Conjunct Ligatures
Why fused consonant clusters are the specific failure point in Devanagari OCR, and why an overall error rate hides it.
9 min read
OCR for Ethiopic Ge'ez Script Documents
A syllabary of several hundred symbols, each a consonant plus a vowel marked by a small modification, gives Ethiopic OCR a distinctive error profile.
9 min read
OCR for Hebrew Text With and Without Niqqud
Why the vowel points below and inside Hebrew consonants are the character class OCR engines drop first, and what in the pipeline actually removes them.
9 min read
OCR for Historical German Fraktur Typefaces
Why Fraktur is out-of-distribution for a Latin-trained recogniser rather than merely difficult, and why a modern German dictionary makes the output worse.
9 min read
OCR for Korean Text Mixing Hangul and Hanja
Why a Korean recogniser with a Hangul-only label set cannot report that it saw a Hanja, and what that does to older mixed-script documents.
9 min read
OCR for Tamil and Malayalam Vowel Sign Placement
Why vowel signs that render to the left, right, above, below or on both sides of their consonant are the dominant error class in Dravidian-script OCR.
9 min read
OCR for Mongolian Traditional Vertical Script
Why a script written top-to-bottom in left-to-right columns defeats the layout stage of almost every OCR pipeline, and why rotating the page only solves half of it.
9 min read
OCR for Khmer Script With Stacked Subscript Consonants
Why a Khmer subscript consonant is encoded with an invisible control character and drawn below the baseline, and how that breaks both recognition and line segmentation.
9 min read
Why AI Models Refuse More Often in Arabic Than in English
What the published multilingual safety evaluations actually establish about the English–Arabic refusal gap, and how to measure it on your own traffic without fooling yourself.
10 min read
Why Content Moderation Misses More in Low-Resource Languages
The argument that under-moderation outside a handful of languages is a labelling-economics outcome rather than a policy decision, and what follows for anyone building on top of it.
10 min read
Why Jailbreak Prompts Work Better When Translated
The argument that capability generalises across languages far better than refusal does, and that the gap between the two is what translation attacks exploit.
10 min read
Why Safety Filters Flag Ordinary Hindi Religious Terms
Why devotional and mythological Hindi vocabulary trips violence and drug classifiers, which of three distinct collisions caused it, and how to get the request through.
10 min read
How English-Centric Safety Training Creates Blind Spots in Other Languages
The argument that a model's refusal boundary in a low-resource language is inconsistent rather than simply lower, and why that needs a different fix from 'more safety training'.
10 min read
Why Profanity Filters Miss Non-English Slurs
Why a word-list filter has exactly zero recall in a language it has no list for, and what to change so that the gap is visible rather than silent.
10 min read
Why Cultural Context Causes False-Positive Refusals in Translation
Why an everyday idiom becomes a threat once rendered literally, how to tell a filter block from a model refusal, and a retry policy that does not silently drop text.
10 min read
Why Public Safety Benchmarks Are Mostly English-Only
A dated count of the language coverage of the widely used public safety benchmarks, set against the number of languages that exist.
10 min read
Why the Same Prompt Gets Different Refusal Behavior in Japanese and English
What is structurally different about Japanese that moves the refusal boundary, what public evaluation actually covers, and how to measure the gap without measuring your own detector.
10 min read
What Code-Switching Is, and Why It Breaks Simple NLP Assumptions
A definition of code-switching at token level, and a walk through a standard pipeline showing which stage fails first when a sentence changes language partway through.
8 min read
Handling Hinglish Text in an AI Pipeline
A working pipeline for Hindi-English text that alternates within a sentence, covering the script problem, the normalisation problem, and where to stop trying to fix it upstream.
9 min read
Handling Spanglish Text in an AI Pipeline
A working pipeline for Spanish-English code-switched text, centred on the borrowed-verb morphology that produces words in neither language's dictionary.
9 min read
Why Sentiment Analysis Fails on Code-Switched Text
The specific mechanism by which a code-switched review gets the wrong sentiment score: the polarity words and the negation that cancels them arrive in different languages.
8 min read
Why Arabizi Confuses AI Models
What happens at token level when Arabic is written in Latin letters and digits, and why the digits are the part that breaks a model rather than the transliteration.
8 min read
Detecting Code-Switching Points Within a Single Sentence
How to flag the specific token where a sentence changes language, using token-level labelling rather than document-level detection, and how to evaluate it honestly.
9 min read
Why Code-Switched Customer Support Messages Get Misrouted
The routing rule that sends bilingual messages to the wrong queue, why the misroute is intermittent, and the change that makes it stable.
8 min read
Generating Natural Code-Switched Text With AI
Why a model asked to mix two languages produces alternating translated sentences instead of real intra-sentential switching, and the prompt structure that fixes it.
9 min read
Which Languages Dominate AI Training Data, and by How Much
What the published corpus composition figures actually say about language share in web-scale training data, and why the ranking is far more skewed than speaker populations.
9 min read
What It Costs to Build a Tokenizer for a New Language
The real inputs to training a tokenizer for a language that has none — corpus size, memory, vocabulary budget — and why the compute is the cheapest part by a wide margin.
9 min read
Should You Translate First or Prompt Natively for a Non-English Task
An argument that the answer depends on task type rather than on language quality: extraction favours translate-first, generation favours native, and the reason is where errors accumulate.
9 min read
How Many Languages Does an LLM Actually Support Well
The difference between the languages a model can produce grammatical text in and the languages it was evaluated in, with the published numbers for each and a threshold you can apply yourself.
9 min read
How Many Writing Systems Are in Active Use Today
A sourced count of encoded scripts from Unicode, split by the Consortium's own usage classification into everyday, limited and historic use.
8 min read
Why Unicode Covers More Languages Than Any AI Model Does
The gap between what can be typed and what can be understood, quantified from Unicode's script coverage against the supported-language lists on published model cards.
8 min read
Languages Missing Entirely From Public Training Corpora
How many living languages have functionally zero text in the public corpora models are trained on, where that number comes from, and why it barely moves.
9 min read
Which Languages Read Right-to-Left, and How Many Speakers That Covers
The list of right-to-left scripts in active use, the languages written in them, and a derived speaker total with its assumptions stated.
9 min read
Why Tone Marks Matter for AI Text in Tonal Languages
What a model actually loses when tone diacritics are stripped from Vietnamese or Yoruba, shown on minimal pairs, and where in a pipeline the stripping usually happens.
9 min read
Why Agglutinative Languages Break Word-Based NLP Assumptions
One Turkish word expands to an English clause, and that single fact breaks word counts, stopword lists, stemming, chunk sizing and short-query matching in specific ways.
9 min read
What Actually Changes for AI Between Logographic and Alphabetic Scripts
The two mechanisms that actually differ — how much meaning one character carries, and how UTF-8 plus subword tokenization price that character — and what follows from each.
9 min read
How Abjads Like Arabic and Hebrew Leave Vowels Ambiguous for AI
Why unvocalised Arabic and Hebrew words have several valid readings, what decides which reading a model picks, and which tasks the ambiguity actually breaks.
9 min read
How AI Models Handle Syllabary Scripts Like Cherokee and Ge'ez
Why one-symbol-per-syllable scripts sit between alphabets and logographies on character count but behave worse than either on tokens, and what decides that.
9 min read
Why Some Languages Have No Standard AI Benchmark at All
The language counts of the major multilingual benchmark suites, with sources and dates, and what the gap between them and the world's languages means for evaluation.
10 min read
Why Japanese Sentences Mix Kanji, Kana and Romaji in the Same Line
Which script each kind of word conventionally takes in Japanese, why the convention exists, and what goes wrong when a model or a pipeline gets the choice wrong.
9 min read
How Many Characters an AI Model Needs to Read a Given Language
Sourced character-set sizes across scripts, from a 26-letter alphabet to 11,172 Hangul syllable blocks, and why the inventory size is not what determines model support.
10 min read
Which Languages Are Read Top-to-Bottom Instead of Left-to-Right
The scripts with a vertical reading order, which of them still use it by default today, and what the distinction between traditional and contemporary usage costs you.
9 min read
Why the Same Country Name Is Written Differently by Locale in AI Output
Endonyms, exonyms and why a model writing Allemagne in French output is correct — plus the cases where localising a name is the bug.
9 min read
Why Reading Direction Alone Doesn't Predict AI Difficulty With a Script
An argument from counterexamples that training-data volume, not writing direction, determines how well a model handles a script — and what that redirects your effort towards.
9 min read
How Multilingual an AI Product Needs to Be for a Global Launch
The coverage curve flattens far earlier than the language list suggests, and the cost curve does not — an argument for launching in fewer languages than you think.
10 min read
Other topics
- LLM fundamentals & architecture
- Tokens, tokenization & context windows
- Prompt engineering
- Reasoning models & test-time compute
- Multimodal AI: vision, audio, video
- RAG & retrieval
- Embeddings & vector search
- AI agents & tool use
- Structured output & function calling
- Fine-tuning & post-training
- Local inference errors, string by string
- Running local models day to day
- Testing code that calls an LLM
- Snapshot and property testing for model output
- Regression suites for prompts
- Eval gates in CI
- Flaky tests against a model
- Determinism and the cost of testing
- Contract and streaming tests
- Testing tool calls and retrieval
- Inference, serving & latency
- Rolling out a prompt change
- Testing AI systems in practice
- Forecasting a time series
- Machine learning on tabular data
- Geospatial data and models
- Understanding audio that is not speech
- Understanding video
- Core computer vision tasks
- Machine learning on graphs
- Point clouds and 3D
- Evaluation, benchmarks & LLM-as-judge
- Sensor and IoT data
- Logs and event streams
- Models over biological sequences
- Machine learning on molecules
- Embedding and searching code
- Extracting invoices and purchase orders
- Receipts, statements and tax forms
- Insurance policies and contracts
- Deeds, court filings and patents
- Extracting from medical records
- Observability & LLMOps
- CVs, certificates and identity documents
- Shipping, customs and technical documents
- Meetings, email, chat and filled-in forms
- Building an extraction pipeline
- Business, property and inspection documents
- Contract clauses and insurance claims
- Regulated and compliance documents
- Consumer, travel and closing documents
- Mapping one chat API onto another
- SDK and framework migrations
- Hallucination & failure modes
- Re-embedding and model deprecation
- Cutting over between providers
- Parity gaps, shims and legacy endpoints
- Moving between model versions
- Migrating vector stores and caches
- Mapping capabilities and parameters
- Migrating pipelines and agents
- Contracts, runbooks and rollback
- Auditing a codebase before a cutover
- Compliance and fine-tune migration
- LLM cost engineering
- Routing, cost tracking and multi-tenancy
- What a migration does to your prompts
- AI security & prompt injection
- Privacy, compliance & data residency
- AI governance, policy & society
- Building reliable AI applications
- AI hardware, GPUs & compute
- Open-weight models & local inference
- AI for developers & coding agents
- AI in industry: vertical playbooks
- AGI, superintelligence, alignment & the long future
- Machine learning foundations
- NLP fundamentals & classical tasks
- Data engineering for AI
- Synthetic data & dataset curation
- AI product design & UX
- Search, ranking & recommendation
- Enterprise adoption & change management
- AI careers, skills & teams
- Reading AI research
- AI in science & discovery
- Robotics & embodied AI
- AI economics, markets & business models
- AI myths, hype & media literacy
- Context engineering
- Shipping AI features: patterns & anti-patterns
- Build it: end-to-end AI tutorials
- Python for AI: hands-on recipes
- TypeScript, React and the web
- Frameworks and SDKs
- Errors and troubleshooting
- AI facts, numbers and statistics
- The history of AI
- The maths behind AI
- Architectures beyond the transformer
- Reinforcement learning
- Diffusion and generative media
- Speech, audio and voice engineering
- Benchmarks, one at a time
- AI search visibility
- Infrastructure and operations
- Databases and storage for AI
- Knowledge graphs and structured knowledge
- Classical ML in production
- Regulation, jurisdiction by jurisdiction
- Prompt recipes and pattern library
- AI for people who do not write code
- Writing, media and creative work
- Edge and on-device AI
- Interpretability and model internals
- Field notes
- OpenAI model behaviour
- Claude model behaviour
- Gemini model behaviour
- Llama model behaviour
- Mistral model behaviour
- Qwen model behaviour
- DeepSeek model behaviour
- Cohere model behaviour
- Grok model behaviour
- Small model behaviour
- Hybrid model architectures
- Token cost by language and script
- Transliteration, romanization and script handling
- Locale-correct output
- Multilingual generation quality
- The EU AI Act, article by article
- AI under the GDPR and EU data law
- US AI regulation, state and sector
- International AI governance and standards
- AI litigation and enforcement
- Running AI workloads on AWS
- Running AI workloads on Google Cloud
- Running AI workloads on Azure
- AI at the edge: Workers, Vercel and Netlify
- Serving models on Kubernetes
- Operating AI infrastructure
- Quantization formats and what they cost
- llama.cpp, flag by flag
- Ollama and the desktop local-model runtimes
- Local models on Apple Silicon
- Hardware for local inference
- Running speech and embedding models locally
- Model files, adapters and conversion
- VRAM arithmetic for local models