Skip to content

Multilingual pipelines

Building a pipeline that survives more than one language: prompt structure, code-switching, detection, routing and evaluation when the input is not English.

A pipeline that works in one language is not a pipeline that works in eleven with a translation step bolted on. The parts that break are not the parts anyone expects: the instruction that carried an English idiom into Japanese, the schema whose enum values were localised and now no longer match the branch in your code, the retrieved documents that quietly overruled an explicit output-language directive, the OCR stage that reported ninety-something percent accuracy on a corpus whose hard cases were four percent of the characters.

These pages work through those mechanisms one at a time. The prompting half covers where language belongs in a template, why a model drifts back into the user’s language, and how to test that one prompt behaves the same way in ten. The document half covers why a script’s own structure — cursive joining, stroke density, absent word spaces, fused conjuncts, a syllabary of several hundred symbols — sets a floor on recognition error that no amount of training data removes. Almost none of these failures raise an exception, so each page ends with what to measure.

Building a Prompt Template With Language as a Variable

Which parts of a prompt should switch with the target language and which should stay fixed, built as one template with a language parameter.

9 min read

Forcing a Model to Always Respond in One Language

Why an output-language instruction loses to the language of the input, and the placement and reinforcement pattern that holds it.

9 min read

Writing System Prompts for a Product Used in Many Languages

A system prompt split into language-agnostic rules and per-language overrides, so a new supported language is a data change.

9 min read

Why Translating a Prompt Word for Word Produces Worse Output

An instruction's force comes from convention, not semantics, and a literal translation preserves the meaning while destroying the convention.

9 min read

Testing Whether a Prompt Works the Same Across Languages

A harness that scores one prompt against one rubric in every supported language, and separates language-specific failures from general ones.

10 min read

Why Politeness in a Prompt Changes Output More in Some Languages

In languages where politeness is grammatical rather than lexical, the model must commit to a level on every verb, and it takes the cue from your prompt.

9 min read

Specifying Output Register, Formal or Informal, by Language

Register as a per-language grammatical rule with a matching assertion, rather than the word 'formal' in a prompt.

9 min read

Why a Prompt That Works in English Fails Silently in Another Language

Six failure classes that degrade an English-tuned prompt in another language without producing an error or an obviously wrong answer.

9 min read

Handling Code-Switched Prompts Where the User Mixes Languages

How to decide which language to answer in when a single message switches languages, and how to tell deliberate mixing from a paste error.

9 min read

Why Output Language Sometimes Ignores an Explicit Instruction

When a large block of other-language text in the context outweighs an explicit output-language directive, and how to demote it to data.

9 min read

OCR Challenges Specific to Arabic's Cursive Script

Why joined letters, four positional forms and dot-only distinctions put a structural floor under Arabic OCR error rates.

9 min read

OCR Accuracy for Chinese Characters at Low Resolution

Why a Chinese character needs several times the pixel area of a Latin letter to stay distinguishable, worked through to a DPI floor.

9 min read

OCR for Thai Script Without Spaces Between Words

Character recognition and word segmentation are two different problems in Thai, and only one of them is an OCR problem.

9 min read

OCR for Devanagari Conjunct Ligatures

Why fused consonant clusters are the specific failure point in Devanagari OCR, and why an overall error rate hides it.

9 min read

OCR for Ethiopic Ge'ez Script Documents

A syllabary of several hundred symbols, each a consonant plus a vowel marked by a small modification, gives Ethiopic OCR a distinctive error profile.

9 min read

OCR for Hebrew Text With and Without Niqqud

Why the vowel points below and inside Hebrew consonants are the character class OCR engines drop first, and what in the pipeline actually removes them.

9 min read

OCR for Historical German Fraktur Typefaces

Why Fraktur is out-of-distribution for a Latin-trained recogniser rather than merely difficult, and why a modern German dictionary makes the output worse.

9 min read

OCR for Korean Text Mixing Hangul and Hanja

Why a Korean recogniser with a Hangul-only label set cannot report that it saw a Hanja, and what that does to older mixed-script documents.

9 min read

OCR for Tamil and Malayalam Vowel Sign Placement

Why vowel signs that render to the left, right, above, below or on both sides of their consonant are the dominant error class in Dravidian-script OCR.

9 min read

OCR for Mongolian Traditional Vertical Script

Why a script written top-to-bottom in left-to-right columns defeats the layout stage of almost every OCR pipeline, and why rotating the page only solves half of it.

9 min read

OCR for Khmer Script With Stacked Subscript Consonants

Why a Khmer subscript consonant is encoded with an invisible control character and drawn below the baseline, and how that breaks both recognition and line segmentation.

9 min read

Why AI Models Refuse More Often in Arabic Than in English

What the published multilingual safety evaluations actually establish about the English–Arabic refusal gap, and how to measure it on your own traffic without fooling yourself.

10 min read

Why Content Moderation Misses More in Low-Resource Languages

The argument that under-moderation outside a handful of languages is a labelling-economics outcome rather than a policy decision, and what follows for anyone building on top of it.

10 min read

Why Jailbreak Prompts Work Better When Translated

The argument that capability generalises across languages far better than refusal does, and that the gap between the two is what translation attacks exploit.

10 min read

Why Safety Filters Flag Ordinary Hindi Religious Terms

Why devotional and mythological Hindi vocabulary trips violence and drug classifiers, which of three distinct collisions caused it, and how to get the request through.

10 min read

How English-Centric Safety Training Creates Blind Spots in Other Languages

The argument that a model's refusal boundary in a low-resource language is inconsistent rather than simply lower, and why that needs a different fix from 'more safety training'.

10 min read

Why Profanity Filters Miss Non-English Slurs

Why a word-list filter has exactly zero recall in a language it has no list for, and what to change so that the gap is visible rather than silent.

10 min read

Why Cultural Context Causes False-Positive Refusals in Translation

Why an everyday idiom becomes a threat once rendered literally, how to tell a filter block from a model refusal, and a retry policy that does not silently drop text.

10 min read

Why Public Safety Benchmarks Are Mostly English-Only

A dated count of the language coverage of the widely used public safety benchmarks, set against the number of languages that exist.

10 min read

Why the Same Prompt Gets Different Refusal Behavior in Japanese and English

What is structurally different about Japanese that moves the refusal boundary, what public evaluation actually covers, and how to measure the gap without measuring your own detector.

10 min read

What Code-Switching Is, and Why It Breaks Simple NLP Assumptions

A definition of code-switching at token level, and a walk through a standard pipeline showing which stage fails first when a sentence changes language partway through.

8 min read

Handling Hinglish Text in an AI Pipeline

A working pipeline for Hindi-English text that alternates within a sentence, covering the script problem, the normalisation problem, and where to stop trying to fix it upstream.

9 min read

Handling Spanglish Text in an AI Pipeline

A working pipeline for Spanish-English code-switched text, centred on the borrowed-verb morphology that produces words in neither language's dictionary.

9 min read

Why Sentiment Analysis Fails on Code-Switched Text

The specific mechanism by which a code-switched review gets the wrong sentiment score: the polarity words and the negation that cancels them arrive in different languages.

8 min read

Why Arabizi Confuses AI Models

What happens at token level when Arabic is written in Latin letters and digits, and why the digits are the part that breaks a model rather than the transliteration.

8 min read

Detecting Code-Switching Points Within a Single Sentence

How to flag the specific token where a sentence changes language, using token-level labelling rather than document-level detection, and how to evaluate it honestly.

9 min read

Why Code-Switched Customer Support Messages Get Misrouted

The routing rule that sends bilingual messages to the wrong queue, why the misroute is intermittent, and the change that makes it stable.

8 min read

Generating Natural Code-Switched Text With AI

Why a model asked to mix two languages produces alternating translated sentences instead of real intra-sentential switching, and the prompt structure that fixes it.

9 min read

Which Languages Dominate AI Training Data, and by How Much

What the published corpus composition figures actually say about language share in web-scale training data, and why the ranking is far more skewed than speaker populations.

9 min read

What It Costs to Build a Tokenizer for a New Language

The real inputs to training a tokenizer for a language that has none — corpus size, memory, vocabulary budget — and why the compute is the cheapest part by a wide margin.

9 min read

Should You Translate First or Prompt Natively for a Non-English Task

An argument that the answer depends on task type rather than on language quality: extraction favours translate-first, generation favours native, and the reason is where errors accumulate.

9 min read

How Many Languages Does an LLM Actually Support Well

The difference between the languages a model can produce grammatical text in and the languages it was evaluated in, with the published numbers for each and a threshold you can apply yourself.

9 min read

How Many Writing Systems Are in Active Use Today

A sourced count of encoded scripts from Unicode, split by the Consortium's own usage classification into everyday, limited and historic use.

8 min read

Why Unicode Covers More Languages Than Any AI Model Does

The gap between what can be typed and what can be understood, quantified from Unicode's script coverage against the supported-language lists on published model cards.

8 min read

Languages Missing Entirely From Public Training Corpora

How many living languages have functionally zero text in the public corpora models are trained on, where that number comes from, and why it barely moves.

9 min read

Which Languages Read Right-to-Left, and How Many Speakers That Covers

The list of right-to-left scripts in active use, the languages written in them, and a derived speaker total with its assumptions stated.

9 min read

Why Tone Marks Matter for AI Text in Tonal Languages

What a model actually loses when tone diacritics are stripped from Vietnamese or Yoruba, shown on minimal pairs, and where in a pipeline the stripping usually happens.

9 min read

Why Agglutinative Languages Break Word-Based NLP Assumptions

One Turkish word expands to an English clause, and that single fact breaks word counts, stopword lists, stemming, chunk sizing and short-query matching in specific ways.

9 min read

What Actually Changes for AI Between Logographic and Alphabetic Scripts

The two mechanisms that actually differ — how much meaning one character carries, and how UTF-8 plus subword tokenization price that character — and what follows from each.

9 min read

How Abjads Like Arabic and Hebrew Leave Vowels Ambiguous for AI

Why unvocalised Arabic and Hebrew words have several valid readings, what decides which reading a model picks, and which tasks the ambiguity actually breaks.

9 min read

How AI Models Handle Syllabary Scripts Like Cherokee and Ge'ez

Why one-symbol-per-syllable scripts sit between alphabets and logographies on character count but behave worse than either on tokens, and what decides that.

9 min read

Why Some Languages Have No Standard AI Benchmark at All

The language counts of the major multilingual benchmark suites, with sources and dates, and what the gap between them and the world's languages means for evaluation.

10 min read

Why Japanese Sentences Mix Kanji, Kana and Romaji in the Same Line

Which script each kind of word conventionally takes in Japanese, why the convention exists, and what goes wrong when a model or a pipeline gets the choice wrong.

9 min read

How Many Characters an AI Model Needs to Read a Given Language

Sourced character-set sizes across scripts, from a 26-letter alphabet to 11,172 Hangul syllable blocks, and why the inventory size is not what determines model support.

10 min read

Which Languages Are Read Top-to-Bottom Instead of Left-to-Right

The scripts with a vertical reading order, which of them still use it by default today, and what the distinction between traditional and contemporary usage costs you.

9 min read

Why the Same Country Name Is Written Differently by Locale in AI Output

Endonyms, exonyms and why a model writing Allemagne in French output is correct — plus the cases where localising a name is the bug.

9 min read

Why Reading Direction Alone Doesn't Predict AI Difficulty With a Script

An argument from counterexamples that training-data volume, not writing direction, determines how well a model handles a script — and what that redirects your effort towards.

9 min read

How Multilingual an AI Product Needs to Be for a Global Launch

The coverage curve flattens far earlier than the language list suggests, and the cost curve does not — an argument for launching in fewer languages than you think.

10 min read

Other topics