Japanese and CJK Text Preprocessing for RAG - Tokenization, Normalization, and Chunk Boundaries

First Published: 2026-08-02
Last Updated: 2026-08-02

A retrieval pipeline built and tuned on English carries an assumption so basic that it is almost never written down: that a space marks the boundary between one word and the next. Nearly every default in the stack inherits it. The tokenizer splits on whitespace. The chunker counts characters and cuts at a space or a period. The overlap is expressed as "the last 50 words." None of these are wrong for English. All of them behave differently, and some of them fail silently, the moment the corpus contains Chinese, Japanese, or Korean.

The failures are quiet, which is what makes them expensive. Nothing raises an exception. Indexing succeeds. Queries return results. The results are simply worse than they should be, in ways that look like a model problem or an embedding problem and are actually a text-processing problem three layers upstream. A team that has never had to think about writing systems will usually reach for a bigger embedding model first, because that is the lever they know about.

This article is about that upstream layer: what specifically breaks, why, and how to design around it. It is not an argument that any of these languages is special or unusually difficult. It is a description of what happens when a design assumption that held for one family of writing systems is applied to another, and of the specification-defined tools that exist to handle the difference.

Specifications and documentation in this article were verified on 2026-07-31. Unicode behavior is cited against Unicode 17.0.0 and the specific Annex revisions listed in section 15. Analyzer behavior is cited against Apache Lucene 10.5.0 javadoc and current OpenSearch documentation. No tokenization or retrieval experiments were run for this article; every statement is traceable to a published specification or vendor document, and section 11 gives you the procedure for measuring the parts that depend on your own corpus.

How the same CJK text is split by a whitespace-oriented pipeline versus a CJK-aware pipeline

1. Introduction

1.1 The problem, stated precisely

Most retrieval pipelines contain two places where text is cut into pieces, and they are not the same place:

Chunking cuts a document into passages that become the unit of retrieval and the unit that gets embedded.
Tokenization cuts a passage (or a query) into terms — for a lexical index — or into subword tokens, for a language model.

English lets you be sloppy about the distinction because whitespace is a usable proxy at both levels. In a writing system without inter-word spaces, whitespace is a proxy for neither. Chunking has to fall back on some other boundary signal, and lexical tokenization has to fall back on a dictionary or on fixed-width n-grams. Both fallbacks have design consequences, and those consequences have to agree with each other — and with what happens at query time.

That last constraint is the one this article keeps returning to, because it is the one most often violated. It is easy to find advice about which Japanese tokenizer to use. It is much harder to find anyone pointing out that whichever you choose, the query path has to do the same thing, and that a mismatch between the two produces an index that looks healthy and retrieves badly.

1.2 Who this is for

Engineers and architects building RAG or search over a corpus that contains Chinese, Japanese, or Korean — including the very common case of a mostly-English corpus with a CJK subset, or an interface where users may type queries in a different language from the documents. It assumes you have built a retrieval pipeline before and know what an embedding, a chunk, and an inverted index are. It does not assume you read any of these languages.

1.3 Scope: which languages, and how honestly

This article covers Chinese, Japanese, and Korean together, because they break the same assumption — that whitespace marks a word boundary — but they break it differently and need different remedies.

Chinese and Japanese are written without spaces between words. Splitting on whitespace yields either one enormous token per run of text, or nothing useful at all.
Korean prose in modern usage is space-delimited, so whitespace splitting produces plausible-looking output. It is nevertheless agglutinative: what sits between two spaces is typically a stem plus one or more attached particles, so the space marks a phrase rather than a searchable term. Korean fails later and less visibly than the other two, which is its own kind of hazard.

Where a technique applies to only one of the three, that is stated inline. Worked examples lean on Japanese, because a single Japanese sentence routinely mixes four scripts — Han ideographs, two kana syllabaries, and Latin letters — and therefore exercises several failure modes at once in a short example. That is a property that makes it a compact example, not a claim about the language.

Thai, Lao, and Khmer face the same word-segmentation problem and are named below wherever a specification names them, but the analyzer and dictionary specifics for those languages are out of scope.

Also out of scope, and deliberately delegated:

Retrieval quality engineering in general — chunking strategy selection, hybrid search weighting, metadata filtering, reranking, query decomposition. That is a full topic in its own right and is covered in Amazon Bedrock Knowledge Bases Retrieval Quality Engineering. This article is only about what changes when the text is CJK.
Ingestion and indexing architecture — sync strategies, metadata design, tenant isolation. See Amazon Bedrock Knowledge Bases Ingestion and Isolation Patterns.
Vector store selection and index internals — see Vector Database Selection on AWS.
Any ranking of specific tokenizers or embedding models. The correct choice is corpus-dependent, and this article does not have your corpus. Section 11 is the substitute: a procedure for deciding on your own data.
Accuracy scores, benchmark results, and pricing. None appear below. Where a specification states a numeric default — a bigram width, a token limit — that is quoted as specification content, not as a measurement.

1.4 The one sentence to take away

If you remember nothing else: whatever normalization and tokenization you apply when building the index, you must apply the same way when handling the query — and in a CJK pipeline there are far more places for those two paths to diverge than in an English one. Section 10 and the second figure are about exactly this.

2. Why Whitespace Assumptions Break

2.1 The specification says the algorithm is not enough

This is not a folk observation; it is stated in the standard. Unicode Standard Annex #29, Unicode Text Segmentation, defines the default algorithms for finding grapheme cluster, word, and sentence boundaries. On word boundaries it is explicit about where the default stops working:

"reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms, analogous to English hyphenation."

And more generally:

"For scripts that use the South East Asian style of context analysis, neither the default word boundaries nor the default line breaks are adequate. Both need tailoring."

That is the whole problem in two sentences from the specification itself. The default word-boundary algorithm is not merely imprecise for these languages — it is documented as insufficient, with dictionary lookup named as the required class of remedy. Any pipeline that relies on a default word splitter for CJK is running outside the range the algorithm claims to cover.

UAX #29 also distinguishes three levels of boundary that are easy to conflate:

Boundary	What it is	Where it matters here
Grapheme cluster	A user-perceived character (a base character plus any combining marks)	The atomic unit — no word or sentence break may occur inside one
Word	Used for selection, search, and cursor movement	Lexical index terms (section 4)
Sentence	Used for larger-block selection	Chunk boundaries (section 6)

Grapheme clusters being atomic is worth internalizing: any chunker that slices by code point, or worse by byte, can split a grapheme cluster and produce a fragment that is not valid text. This is a general Unicode hazard rather than a CJK-specific one, but CJK corpora tend to surface it because they are more likely to contain characters outside the Basic Multilingual Plane and combining marks such as the voiced-sound marks used with kana.

2.2 Character width is a property with no single answer

Unicode Standard Annex #11, East Asian Width, assigns every character one of six property values:

Value	Name	Meaning
`F`	Fullwidth	Has a wide compatibility decomposition mapping to a narrow character elsewhere
`H`	Halfwidth	Explicitly halfwidth via a narrow compatibility decomposition, plus the won sign
`W`	Wide	Consistently wide in East Asian typography — ideographs, and emoji with the presentation property
`Na`	Narrow	Always narrow, with an explicit fullwidth counterpart; ASCII is the type case
`A`	Ambiguous	Rendered wide in East Asian legacy character sets, narrow elsewhere
`N`	Neutral	Everything else

The category that causes real trouble is A. The specification is direct about why: "Ambiguous characters require additional information not contained in the character code to further resolve their width." Their rendering depends on context — language, script, font, data provenance, or explicit markup — and the recommendation is to treat them as narrow when the context cannot be established.

For a retrieval pipeline the practical implications are two. First, any chunk-size budget expressed in "characters" is ambiguous by construction if the corpus is mixed: the same count means different amounts of rendered text and different amounts of information depending on script. Second, the same annex notes that fullwidth and halfwidth are "relational properties of a pair of characters" rather than intrinsic ones, and — importantly — that "The East_Asian_Width property does not preserve canonical equivalence." Width and normalization interact, which is the subject of the next section.

2.3 Several writing systems in one sentence

An ordinary sentence of Japanese technical prose can contain Han ideographs, hiragana, katakana, Latin letters, Arabic numerals, and fullwidth punctuation, with no spaces marking any transition. A tokenizer that keys on script changes will produce boundaries at those transitions — sometimes usefully, sometimes not. A tokenizer that keys on whitespace will produce almost none.

This mixing is also why script-boundary heuristics are not a general substitute for a dictionary. Katakana runs are commonly loanwords and product names, and a script-change heuristic will find their edges; but the boundary between two Han-script nouns has no script change to find, and that is where most of the meaningful terms are.

2.4 Punctuation and brackets are different characters

CJK text uses its own punctuation — an ideographic full stop rather than a period, an ideographic comma, and a set of bracket forms with no ASCII equivalent. A sentence splitter with a hardcoded ., !, ? set will find no sentence boundaries at all in such text and will return the entire document as one sentence. This particular failure is easy to fix once noticed and easy to miss entirely, because the downstream symptom — chunks that are far too large — looks like a chunk-size configuration problem.

The related trap is that some of these characters have both fullwidth and halfwidth forms that normalization may or may not unify, which is section 3.

3. Normalization

3.1 The four forms

Unicode Standard Annex #15, Unicode Normalization Forms, defines four:

Form	Operation
NFD	Canonical decomposition
NFC	Canonical decomposition, then canonical composition
NFKD	Compatibility decomposition
NFKC	Compatibility decomposition, then canonical composition

The K marks compatibility equivalence. Canonical forms (NFC/NFD) preserve compatibility composites present in the source; compatibility forms (NFKC/NFKD) decompose them. As the annex puts it, "None of the forms generate compatibility composites that were not in the source text."

Two guarantees make normalization safe to build on. All four transformations are idempotent — toNFC(toNFC(x)) = toNFC(x) — and normalized strings are stable: "once normalized, a string is normalized according to all future versions of Unicode." The composition version is pinned at 3.1.0 to hold that guarantee. You can therefore normalize at ingestion and rely on the result not shifting under a future Unicode release.

3.2 Why NFKC is the tempting choice, and what it costs

For CJK retrieval, NFKC is attractive for one specific reason: it unifies width variants. UAX #15 gives the case directly — under compatibility normalization, "the halfwidth and fullwidth katakana characters will normalize to the same strings." Fullwidth ASCII letters and digits likewise fold to their ordinary forms. Since the same term routinely appears in a corpus in both width forms — often because different source systems produced it — folding them together means a query in one form matches a document in the other.

That is a real benefit, and it is why the compatibility form shows up in so many CJK analysis chains. But the specification attaches an unusually blunt warning to it:

"Normalization Forms KC and KD must not be blindly applied to arbitrary text. Because they erase many formatting distinctions, they will prevent round-trip conversion to and from many legacy character sets, and unless supplanted by formatting markup, they may remove distinctions that are important to the semantics of the text."

The annex recommends treating the compatibility forms "like uppercase or lowercase mappings: useful in certain contexts for identifying core meanings, but also performing modifications to the text that may not always be appropriate."

The operative consequence for a RAG pipeline is a split between two copies of the text:

Index-side text — the form fed to the analyzer and used for matching. Aggressive folding is usually right here, because matching wants core meaning.
Display-side text — the passage you return to the user, or hand to the generation model as context. This should stay closer to the source, because compatibility normalization can destroy distinctions that mattered — superscripts flattened into ordinary digits, unit ligatures expanded, some formatting-bearing characters removed.

Applying NFKC once, destructively, at extraction time and keeping no other copy is the version of this mistake that is hardest to undo, because the original is gone.

3.3 Where in the pipeline to normalize

There are three distinct places, and they are not interchangeable:

At extraction, before anything is stored. Cheapest, and the right place for repairs that are unambiguously repairs. But it is destructive with respect to whatever you did not keep a copy of.
As a character filter, before the tokenizer runs. In OpenSearch, the analysis-icu plugin supplies icu_normalizer as a character filter, and it "can be configured with different normalization modes (NFC, NFD, NFKC, NFKD)." Running here changes what the tokenizer sees, which matters: a tokenizer's dictionary lookups behave differently against folded and unfolded text.
As a token filter, after tokenization. The same icu_normalizer is also available as a token filter, operating on tokens rather than the raw stream. Lucene's CJKWidthFilter is a narrower, cheaper option in the same position, folding "fullwidth ASCII variants into the equivalent basic latin" and "halfwidth Katakana variants into the equivalent kana." Its javadoc describes it as "a (practical) subset of NFKC/NFKD Unicode normalization" and points at the ICU package for full normalization.

The choice between position 2 and 3 is a real design decision rather than a preference. A morphological analyzer matches against a dictionary; if the dictionary is built over normalized forms and the input has not been normalized yet, lookups miss. Normalizing as a character filter — before the tokenizer — is usually the safer default for dictionary-based analysis. Some analyzers make this decision for you: Sudachi's DefaultInputTextPlugin documents its normalization as lowercasing followed by NFKC, applied to the input text before analysis.

The one arrangement that is simply a bug is normalizing in more than one of these positions without intending to. It is not usually harmful in itself — normalization is idempotent — but it becomes harmful the moment the two positions are configured with different forms, or the moment one of them is present on the index path and absent on the query path. That second case is section 10.

3.4 Normalization that is not Unicode normalization

Unicode normalization handles encoding-level variation. It does not handle orthographic variation, which in these languages is substantial: a term may be written in a different script than expected, with a different okurigana, or as a recognized variant character. This is a dictionary problem rather than a Unicode problem.

Sudachi's documentation is a useful concrete inventory of what the class of problem contains, because it lists the variation its normalized-form feature covers: okurigana variation, script variation, character variants, common misspellings, and contracted forms. Whether you want that folding depends on the corpus — in a legal or medical corpus a "variant" may be a meaningful distinction — but you should decide it deliberately rather than inherit it from a default.

For Chinese, the corresponding axis is Simplified versus Traditional. This is not covered by Unicode normalization at all; it needs a conversion step, and Amazon OpenSearch Service exposes the STConvert plugin for exactly this purpose. If your corpus mixes both, a query in one script will not match documents in the other unless you convert one way or index both.

4. Tokenization for Retrieval

This section is about lexical tokenization — producing the terms that go into an inverted index. Section 5 is about the separate question of subword tokens for language models. Conflating the two is common and costly; they use different algorithms, have different failure modes, and are tuned for different things.

4.1 The two families

Without whitespace, there are two workable strategies:

Morphological analysis uses a dictionary and a statistical model to find word boundaries. It produces linguistically meaningful terms and generally better precision. Its weaknesses are all downstream of the dictionary: a term the dictionary does not know — a new product name, a domain-specific technical term, an organization — will be split into pieces, and a query for that term will then have to match those pieces.

N-gram splitting cuts the text into fixed-length overlapping character sequences and ignores meaning entirely. Nothing is ever out-of-vocabulary, so recall is high and unknown terms are handled by construction. The costs are a larger index, more spurious matches from sequences that cross word boundaries, and scoring that is harder to reason about.

Neither is correct in general. The trade is precision and index size against recall and robustness to unknown vocabulary, and where you want to sit depends on how much of your corpus consists of terms no general dictionary will contain.

4.2 What is actually built in, and what is a plugin

A detail that surprises people: in OpenSearch, Japanese, Chinese, and Korean are not in the built-in language-analyzer list at all. The documented built-in language analyzers run from Arabic through Ukrainian and do not include them. What is available without a plugin is the cjk analyzer, which is bigram-based, and beyond that everything requires an installed plugin.

The relevant analysis plugins — analysis-icu, analysis-kuromoji, analysis-nori, analysis-smartcn — are all listed as additional plugins, installed with bin/opensearch-plugin install <plugin-name>. Assuming that a morphological analyzer is present because the cluster is running is a good way to discover at query time that it is not.

Managed offerings differ here, and this is a place where the abstraction leaks. Amazon OpenSearch Service ships Japanese (kuromoji) and ICU analysis on all domains from OpenSearch 1.0, while Sudachi, Nori, Pinyin, and STConvert are optional plugins associated with a domain as ZIP-PLUGIN packages, each with a minimum engine version. Its documentation labels Sudachi Analysis as "recommended for Japanese". Amazon OpenSearch Serverless prepackages ICU, Japanese (kuromoji), Korean (Nori), and Smart Chinese analysis. If you are moving between deployment models, check which set you actually have — the analyzer names in an index template are not portable by assumption.

4.3 The built-in bigram path

The cjk analyzer is worth understanding even if you do not use it, because it is the zero-configuration fallback and because its component chain is instructive. OpenSearch documents it as built from a standard tokenizer plus the token filters cjk_width, lowercase, cjk_bigram, and stop. The order is the interesting part: width folding happens before bigram formation, so the bigrams are formed over already-normalized text.

Lucene's CJKBigramFilter javadoc adds the behavioral detail that matters in practice. By default, "when a CJK character has no adjacent characters to form a bigram, it is output in unigram form"; an outputUnigrams flag makes it emit both unigrams and bigrams always, "for a combined unigram+bigram approach." That combined mode is a common way to make single-character queries work at the cost of a larger index.

OpenSearch's own documentation notes that "the icu_analyzer in the ICU analysis plugin works better for CJK text than the cjk analyzer" and advises experimenting with your own text and queries — which is both good advice and an illustration that the vendor is not claiming a universally correct answer either.

4.4 Morphological analyzers and their modes

Where morphological analysis is available, the mode selection is the main lever, and the defaults are not neutral.

Japanese (Lucene's JapaneseTokenizer, the kuromoji plugin). The tokenizer "uses a rolling Viterbi search to find the least cost segmentation (path) of the incoming characters," and offers three modes:

Mode	Documented behavior
`NORMAL`	"Ordinary segmentation: no decomposition for compounds"
`SEARCH`	"Segmentation geared towards search: this includes a decompounding process for long nouns, also including the full compound token as a synonym"
`EXTENDED`	"Extended mode outputs unigrams for unknown words" (marked experimental)

SEARCH is the documented default, and the reason is visible in its description: it emits both the decomposed pieces and the full compound, so a query for either form matches. That is the right default for retrieval and a poor one if you are counting distinct terms for some analytic purpose, because the same span of text contributes more than one token.

Korean (Lucene's KoreanTokenizer, the nori plugin). The analogous lever is decompounding:

Mode	Documented behavior
`NONE`	"No decomposition for compound"
`DISCARD`	"Decompose compounds and discards the original form" — the documented default
`MIXED`	"Decompose compounds and keeps the original form"

Note the asymmetry with Japanese: Korean's default discards the original compound, where Japanese's default keeps it. A team that reasons by analogy from one to the other will get this backwards. If you need a query for the full compound to match, MIXED is the mode that behaves like Japanese SEARCH.

Japanese (Sudachi). Sudachi exposes a different and unusually explicit control: three splitting modes, selected with -m {A|B|C}, described as A producing "the shortest units equivalent to the UniDic short unit," C extracting named entities, and B "the middle units." Its documentation gives worked examples from the core dictionary:

Mode	Example segmentation
A	客室 / 乗務 / 員
B	客室 / 乗務員
C	客室乗務員

and states plainly: "In full-text searching, to use A and B can improve precision and recall." Indexing more than one mode is therefore a documented technique rather than a trick — short units give recall, longer units give precision, and both can be indexed into separate fields.

Chinese. Smart Chinese analysis is available as a Lucene/OpenSearch plugin; Amazon OpenSearch Service additionally lists IK (Chinese) Analysis, Pinyin Analysis, and HanLP. The dictionary considerations below apply equally.

4.5 Dictionaries are an operational concern, not a setup step

Whichever morphological analyzer you pick, the dictionary is where domain terminology lives, and it is a moving part.

Sudachi publishes three dictionary sizes — Small (UniDic vocabulary only), Core (basic vocabulary, the default), and Full — so the choice of dictionary changes segmentation before any user dictionary is involved. Comparing the core-dictionary and full-dictionary examples in its documentation shows the same mode producing different splits depending on which dictionary is loaded.

User dictionaries are how you teach the analyzer your own vocabulary, and their mechanics are less uniform than you would hope. In Amazon OpenSearch Service, custom dictionary files are uploaded as packages and referenced by an assigned identifier, and the documentation is specific that the parameter varies by object: synonyms_path and stopwords_path are supported by several objects, but user_dictionary is exclusive to the kuromoji plugin. The IK Chinese plugin behaves differently again — it picks up an associated custom dictionary automatically without a user_dictionary parameter.

Three operational constraints from that same documentation are worth designing around before you commit:

Only single-file dictionaries are supported. Dictionaries that span multiple files, such as Hunspell, do not fit the model.
Dictionary files consume Java heap proportional to their size — the documentation gives the example of a 2 GiB dictionary consuming about 2 GiB of heap per node, and advises watching JVM memory pressure. A large domain dictionary is a capacity decision.
Updates are not always immediate. For the Sudachi plugin specifically, reassociating a dictionary file "doesn't immediately reflect on the domain"; the dictionary refreshes on the next blue/green deployment, with the documented alternative being to create a new package, build a new index against it, reindex, and swap via an alias.

That last point deserves emphasis because it changes how you plan. If adding vocabulary requires a reindex, then "we will add terms as we discover them" is a reindex cadence, not a configuration change, and the alias-swap pattern should be in the design from the start rather than retrofitted.

There is also a limit worth knowing: a maximum of 20 plugins may be associated with a single domain, counting optional, third-party, and custom plugins together.

5. Tokenization for LLMs

5.1 A different tokenizer with different consequences

Language models do not use the tokenizers of section 4. They use subword tokenizers fitted to their training data, and the number of tokens a given string consumes is a property of that specific model's tokenizer — not of the text, and not transferable between models.

For CJK text this matters more than it does for English, because the relationship between characters and tokens is different, and because it varies between models and between scripts. Everything in a pipeline that is denominated in tokens is affected: the context budget for retrieved passages, the maximum passage size an embedding model will accept, and the chunk-size parameter of whatever chunker you are using if it counts in tokens.

This article does not give ratios. Published figures exist, they vary by model and by script, and transcribing them would produce exactly the kind of number that is right when written and wrong when read. What follows instead is how to get the number that applies to your model and your text.

5.2 Do not estimate one model's tokens with another model's tokenizer

The most common way to get this wrong is to reach for a familiar tokenizer library and use it as a general-purpose estimator. Anthropic's documentation addresses this directly for its own models, stating that counts are model-specific and that OpenAI's tiktoken should not be used to estimate Claude token counts — it undercounts on ordinary text, and by a wider margin on code or non-English input. The specific direction of the error matters less than the general point, which generalizes across providers: a tokenizer is part of a model, and using a different model's tokenizer as a proxy is guessing.

The remedy is to use whatever token-counting facility the model you are actually calling provides:

Anthropic exposes a token counting endpoint, POST /v1/messages/count_tokens, which takes the same model identifier you will use for inference — because the count depends on it.
OpenAI publishes its BPE tokenizer as the open-source openai/tiktoken library, which is authoritative for its own models and, as above, only for those.
Google documents a countTokens API for Gemini models, and its documentation carries two caveats worth generalizing: counts for non-text inputs are an estimate rather than the consumed figure, and the count is defined per supported model rather than once for the family.
Other providers document equivalent facilities; consult the documentation for the specific model you are calling rather than assuming a shared tokenizer.

A note on this section's sources. Each facility above is cited against a documentation page that was confirmed reachable at the time of writing; this article's policy is not to cite a URL it has not confirmed, and some vendor documentation portals could not be retrieved without an interactive browser session. Where a provider is absent from the list, read that as a limitation of the verification method, not as a statement about which providers offer the capability. The rule to take away is provider-independent: measure with the tokenizer belonging to the model you are calling.

If you want an interactive way to develop intuition for how a body of text maps onto a context budget, LLM Token Counter and Context Budget Planner visualizes context-window usage.

5.3 Watch for dual limits on embedding models

Embedding models frequently express their input limit in two units at once, and which one binds depends on the script.

Amazon Titan Text Embeddings V2 is a clear example: its documentation gives maximum input of 8,192 tokens and, separately, 50,000 characters. For English prose the token limit is generally the one you hit. For a script whose characters consume more tokens each, the token limit binds much sooner in character terms — while for a script consuming fewer, the character limit could bind first. A chunker configured in characters against a model documented in both units can produce passages that are within the character limit and over the token limit, or the reverse.

The defensive design is to size chunks against the binding limit for your script mix, verified by counting rather than assumed, and to leave margin. The same document notes that although the model supports long documents, "for retrieval tasks, it is recommended to segment documents into logical segments, such as paragraphs or sections" — which is a reminder that the limit is a ceiling, not a target.

6. Chunk Boundaries

6.1 Sentence boundaries are unreliable by specification

Chunkers commonly try to cut at sentence boundaries, because a chunk that ends mid-sentence retrieves badly. UAX #29 is candid that this is hard in general, independent of language:

"Plain text provides inadequate information for determining good sentence boundaries. Periods can signal the end of a sentence, indicate abbreviations, or be used for decimal points, for example."

The annex notes the algorithm "cannot detect cases such as 'Mr. Jones'" and that "more sophisticated tailoring would be required."

In CJK text this general difficulty acquires a specific, and much more basic, form: the sentence-ending characters are not the ASCII ones. A splitter whose terminator set is ., !, ? finds zero boundaries in a document written with ideographic punctuation and returns the whole document as a single sentence. Downstream, the chunker either emits one enormous chunk or falls back to cutting on character count with no boundary awareness at all — and the symptom that reaches you is "our chunks are bad," several layers from the cause.

Adding the CJK punctuation forms to the terminator set fixes the gross failure. It does not fix the general problem, because the general problem is not solvable by a character set.

6.2 Line breaking is an underused boundary signal

There is a second specification that is much less commonly reached for and is well suited to this job. Unicode Standard Annex #14, the Unicode Line Breaking Algorithm, defines where a line may and may not break — which is a different question from where a word or sentence ends, and a considerably more tractable one for CJK.

Its central provision for these scripts is the Ideographic class:

"Characters with this property do not require other characters to provide break opportunities; lines can ordinarily break before and after and between pairs of ideographic characters."

That is to say, in ideographic text a break is permitted almost anywhere — which is exactly why the algorithm's real value lies in the positions it forbids:

NS (Nonstarters) — characters that "cannot start a line," including small kana and iteration marks. Cutting a chunk immediately before one of these produces a fragment beginning with a character that cannot legitimately begin a line.
CL (Close Punctuation) — "The closing character of any set of paired punctuation should be kept with the preceding character."
OP (Open Punctuation) — "The opening character of any set of paired punctuation should be kept with the character that follows."

Feeding a chunker the set of prohibited break positions is a cheap and specification-grounded improvement over a naive character-count cut, and it is available without a dictionary. It gets you chunks that do not begin with an orphaned closing bracket or a stranded small kana. It does not give you semantically coherent chunks — nothing at this level does — but it removes a class of obviously-wrong boundaries.

Two further details from the annex are worth carrying:

The algorithm covers Han ideographs, hiragana, and katakana under ID, except the small variants, which fall under NS.
Korean is tailored differently. The annex notes that "Korean uses space-based line breaking in many styles of documents," requiring Hangul syllables and jamo to be tailored to the AL class. Korean behaves more like a space-delimited script here — another instance of the three languages needing different handling.

The algorithm is explicitly "a tailorable set of rules," with locale-sensitive tailorings expressible in LDML and available through CLDR. If you are implementing this, take the tailored rules rather than the raw defaults.

6.3 Headings, tables, and vertical text

Structural boundaries — headings, list items, table rows — are better chunk boundaries than any character-level rule, and they are script-independent. Where document structure survives extraction, use it first and fall back to character-level rules only inside a structural unit. This is general chunking advice rather than CJK advice, but it is worth stating because the CJK-specific rules above are a fallback, not a replacement.

Vertically-set documents deserve a note. Unicode Standard Annex #50, Unicode Vertical Text Layout, defines the Vertical_Orientation property with four values (U upright, R rotated, Tu and Tr for characters needing alternative glyphs with upright or rotated fallback) to give "a stable default orientation of characters for reliable document interchange." This is a rendering property and does not affect chunking directly — but it signals that a document was authored for vertical layout, and vertically-set documents are where text-extraction column-order problems concentrate. That is section 7.

6.4 Overlap

Chunk overlap exists so that a passage spanning a boundary is not lost. Where the overlap is specified in words, a CJK pipeline has a problem immediately: there is no reliable word count without the tokenizer of section 4.

Two consequences follow. First, an overlap expressed in characters and inherited from an English configuration is not the same amount of content in a CJK document, for the same reason chunk sizes are not (section 2.2, section 5.3). Second, an overlap cut without regard to prohibited break positions can begin the overlapping region at a position that cannot legitimately start a line — the same defect as a badly cut chunk, reproduced once per boundary.

The workable approach is to express overlap in the same unit you express chunk size in, measure that unit against your actual model and corpus, and apply the same boundary rules to the overlap cut as to the chunk cut.

For how these parameters are exposed by a managed knowledge base — fixed-size, semantic, and hierarchical strategies, and their token and overlap parameters — see Amazon Bedrock Knowledge Bases Retrieval Quality Engineering. The CJK-specific point is that whichever strategy you select, its size and overlap parameters are denominated in units that behave differently on your text than on the English corpus the defaults were chosen against.

7. Document Ingestion Realities

Everything above assumes the text arrived intact. Frequently it did not, and the damage is script-specific.

Column order in vertical documents. A PDF laid out vertically may extract into an order that reflects neither reading order nor anything else useful, depending on how the producing application emitted its text runs. The result is grammatical-looking fragments assembled in the wrong sequence — which is worse than obviously-broken text, because nothing downstream flags it.

Ruby annotations and emphasis marks. Ruby is small annotation text set alongside a base run to give a reading. Depending on the extractor, ruby may be dropped, interleaved with the base text at arbitrary positions, or emitted as a separate run. Interleaving is the damaging case: it silently corrupts the base text with characters that were never part of it, and a morphological analyzer confronted with the result will produce garbage segmentation for that span. Emphasis marks — the dots set beside characters for emphasis — can behave similarly.

Encoding damage. Legacy CJK encodings remain common in older documents and in exports from older systems. A file decoded as the wrong encoding produces mojibake, which is usually visible on inspection but easy to miss in an automated pipeline processing many files. Worse is the partially-damaged case, where most of a document decodes correctly and a subset does not. If you are diagnosing suspected encoding damage, Character Encoding Converter Tool covers Japanese encoding conversion and mojibake repair.

Whitespace that is not ASCII whitespace. The ideographic space is a distinct character from the ASCII space. Code that strips or splits on \s may or may not treat it as whitespace depending on language, library, and regex flags. This is a small thing that produces confusing results, particularly in code that trims strings.

OCR noise. Scanned CJK documents produce a characteristic error profile — visually similar characters substituted for one another. Because the substituted character is usually a real character, the result is well-formed text that is wrong, which no validity check will catch.

The general architecture of document extraction — layout-aware parsing, multimodal extraction, and when to reach for a foundation model to do the parsing — is out of scope here and covered in Multimodal Document Intelligence Pipeline on AWS and Amazon Bedrock Knowledge Bases Ingestion and Isolation Patterns. The CJK-specific contribution of this section is the list of things to look for when you sample your extracted text — and the recommendation to actually sample it, by eye, with someone who reads the language, before concluding that a retrieval problem is a retrieval problem.

8. Embedding Model Selection for Multilingual Corpora

8.1 Read the language support claim carefully

"Multilingual" is not a specification, and the gap between a model's marketing description and its documentation is a good place to look for the actual commitment.

Amazon Titan Text Embeddings V2 illustrates the pattern. The model is widely described as pre-trained on 100+ languages, and an AWS blog announcement says exactly that. The service documentation's model reference for amazon.titan-embed-text-v2:0, however, lists its languages as "English (100+ languages in preview)". Both statements are AWS's own; they carry different commitments, and the one in the service documentation is the one that describes the supported surface.

This is not a criticism of a particular model — it is a general instruction about where to look. When you are deciding whether an embedding model will serve a CJK corpus, read the model reference page in the service documentation rather than the announcement, and note whether your languages are listed as supported or as preview. Then verify on your own data (section 11), because "supported" is a coarse signal that tells you nothing about how well the model separates near-duplicates in your domain.

8.2 The failure modes specific to mixed-language corpora

Cross-lingual query and document mismatch. If users query in one language and documents are in another, you are relying on the embedding model to place translations near each other in vector space. Multilingual models are trained to do this, with varying success by language pair and domain. If this is a requirement rather than a nice-to-have, it needs to be tested explicitly — it is a different capability from monolingual retrieval quality and can be much weaker.

Language-dominated similarity. In a mixed-language corpus, embeddings can cluster by language as much as by topic, so a query in one language preferentially retrieves documents in that language regardless of relevance. Where documents exist in parallel language versions, this can also produce result sets full of near-duplicates.

Uneven chunk information density. Because a fixed token budget corresponds to different amounts of content across scripts (sections 2.2 and 5.3), chunks built with one size parameter across a mixed corpus carry systematically different amounts of information depending on language — which biases retrieval in a way that is invisible in the configuration.

Model swaps require a full rebuild. Changing the embedding model means re-embedding the corpus, because query vectors and stored vectors must come from the same model. This is a general RAG constraint rather than a CJK one, but it makes embedding-model selection an unusually expensive decision to revisit, which argues for testing candidates on a corpus sample before committing. See Production RAG Architecture on Amazon Bedrock and Vector Database Selection on AWS for the surrounding architecture.

9. Hybrid Retrieval for CJK

9.1 Why the lexical half matters more here

Hybrid retrieval combines lexical matching with vector similarity. It is generally useful; it is disproportionately useful for CJK corpora, for a specific reason. Product codes, part numbers, personal and organization names, and domain jargon are exactly the terms most likely to be absent from an embedding model's training distribution and absent from a morphological analyzer's dictionary. The vector side handles paraphrase; the lexical side handles the exact string. Neither covers the other's gap.

But the lexical half only works if section 4 was done correctly — and hybrid retrieval has a way of masking that it was not. If the lexical side is misconfigured, vector results still come back, so the system appears functional while operating on one leg.

9.2 A concrete mixed-script trap

Lucene's CJKBigramFilter javadoc documents a specific failure that is worth reproducing because it is precisely the kind of thing that is invisible until you look at the token stream:

"Unlike ICUTokenizer, StandardTokenizer does not split at script boundaries. Korean Hangul characters are treated the same as many other scripts' letters, and as a result, StandardTokenizer can produce tokens that mix Hangul and non-Hangul characters, e.g. '한국abc'. Such mixed-script tokens are typed as <ALPHANUM> rather than <HANGUL>, and as a result, will not be converted to bigrams by CJKBigramFilter."

Read that carefully: a token that mixes scripts gets the wrong type, and because the bigram filter dispatches on type, it is silently skipped. The pipeline is configured correctly by every appearance. Certain tokens simply do not receive the processing you configured — and these are disproportionately the mixed alphanumeric-plus-CJK strings that product identifiers and technical terms are made of.

The mitigation is either the ICU tokenizer, which does split at script boundaries, or an explicit check of the token stream for mixed-script tokens on representative data. The general lesson generalizes past this one filter: read the token stream, do not infer it from the configuration.

9.3 Synonyms and orthographic variants

Section 3.4 covered orthographic variation as an analyzer concern. The other half of the answer is a synonym dictionary, which handles the cases no normalization rule will: an abbreviation and its expansion, a loanword and its native equivalent, an old and new organization name.

The operational detail that catches people is update semantics. Amazon OpenSearch Service documents updateable: true on a token filter as a field that "only applies to search analyzers, not index analyzers, and is critical if you later want to update the search analyzer automatically." The consequence is a real asymmetry:

Search-time synonyms can be updated without reindexing — which is why the documented pattern applies the custom analyzer as search_analyzer while leaving the index analyzer as standard.
Index-time synonyms are baked into the index at write time and require reindexing to change.

For a vocabulary that changes — which is most domain vocabulary — search-time expansion is the maintainable choice. Note that this is a deliberate, controlled asymmetry between the index and query paths, which is different from the accidental asymmetry section 10 is about. The difference is whether you chose it.

The tuning of hybrid search itself — how the two result sets are combined and weighted — is out of scope here and covered in Amazon Bedrock Knowledge Bases Retrieval Quality Engineering.

10. Query-Side Processing

Where index-time and query-time processing must agree in a CJK retrieval pipeline

10.1 The symmetry requirement

This is the section the rest of the article has been building toward, and it is short because the rule is short.

Text processed at index time and text processed at query time must be processed the same way. A document indexed as normalized, morphologically segmented tokens and a query submitted as unnormalized, whitespace-split text describe the same content and will not match. Nothing errors. The query returns whatever the vector side happens to surface, and the lexical side contributes nothing.

English pipelines mostly get this right by accident, because the processing is shallow enough that the two paths are hard to diverge. A CJK pipeline has far more places to diverge — normalization form, normalization position, tokenizer, tokenizer mode, dictionary version, synonym set — and each one is an opportunity for the paths to differ. That is not a difference in kind. It is a difference in the number of ways to be wrong.

The most common concrete instances:

Normalization applied at ingestion as a preprocessing step, but not applied to the incoming query — because the ingestion step lives in a different service from the query handler.
A different normalization form on each path: NFKC at index time, NFC at query time.
An analyzer specified in the index mapping but not in the query, so the query falls back to the default analyzer.
The dictionary updated on one path but not the other, or updated at a time when only one path picks it up (section 4.5's blue/green caveat is exactly this hazard).

10.2 When asymmetry is correct

Section 9.3 already gave the counterexample: search-time synonym expansion is a deliberate asymmetry, and the same holds for query expansion generally. The distinction is not "symmetric good, asymmetric bad" — it is that every asymmetry should be a decision someone made and wrote down.

A practical way to enforce this: keep the index analyzer and search analyzer configuration adjacent in the same definition, so that a divergence is visible in review rather than discoverable only at query time. Where the two paths genuinely live in different services, the analyzer configuration should be a shared artifact rather than two copies that drift.

10.3 Short queries

Search queries are short, and in a script where a word may be one or two characters, they can be shorter than the analyzer's minimum useful unit.

With bigram tokenization, a single-character query produces no bigram. This is what outputUnigrams (section 4.3) exists for.
With n-gram tokenization, a query shorter than min_gram produces no terms at all. OpenSearch's ngram tokenizer defaults are min_gram 1 and max_gram 2, with the maximum difference between them controlled by index.max_ngram_diff, which defaults to 1 — so the window of usable query lengths is narrower than people often assume, and widening it is an explicit index-level setting rather than just a tokenizer parameter.
With morphological analysis, a single-character query may or may not be a dictionary term.

Decide the behavior you want for short queries and test it, rather than discovering it from a user report. It is one of the cheapest things in this article to check and one of the most commonly skipped.

11. How to Evaluate on Your Own Corpus

Nothing in this article tells you which configuration is best, because that depends on your corpus, your queries, and your tolerance for index size. What follows is a procedure for finding out. It is deliberately mechanical.

11.1 Read the token stream

The single highest-value diagnostic is to look at what your analyzer actually produces. Every analysis engine exposes this; in OpenSearch it is the _analyze API:

POST /my-index/_analyze
{
  "field": "content",
  "text": "<a representative passage from your corpus>"
}

Run it on passages that contain your hardest vocabulary — product names, organization names, domain jargon, mixed alphanumeric-and-CJK identifiers — and read the output. You are looking for terms split in ways that will prevent a natural query from matching, and for the mixed-script typing problem of section 9.2.

OpenSearch's documentation includes a worked example of this on the cjk analyzer, showing the overlapping <DOUBLE> bigram tokens produced from a Chinese sentence and a separate <NUM> token for a digit run — a useful reference for what the output looks like when it is working as designed.

11.2 Diff the index path against the query path

Run the same text through the index analyzer and the search analyzer and compare the token lists. They should be identical unless you deliberately made them differ, in which case you should be able to state why. This one comparison catches most of section 10's failure modes, and it takes minutes.

Keep it as a test. Analyzer configuration drifts, and this is a cheap regression check.

11.3 Build a query set before you tune anything

Collect representative queries — from logs if you have them, from the people who will use the system if you do not. Include the awkward cases deliberately: single-character queries, queries mixing scripts, queries using an abbreviation where documents use the expansion, queries in one language against documents in another if that is in scope.

Fix this set before you start changing configuration. A query set assembled after tuning tends to encode the tuning.

11.4 Change one thing at a time

The variables worth sweeping, roughly in order of impact:

Normalization form and position (section 3.3)
Tokenizer family — morphological versus n-gram versus bigram (section 4.1)
Tokenizer mode within a family (section 4.4)
Dictionary selection and user-dictionary contents (section 4.5)
Chunk size and overlap, measured in the unit that binds (sections 5.3 and 6.4)

Change one, re-run the query set, and record what moved. Because several of these are index-time settings, a sweep means rebuilding an index per configuration — which is a reason to run the sweep on a corpus sample rather than the full corpus, and a reason to decide up front how large a sample is representative.

11.5 Measure the token budget separately

Chunk sizing depends on token counts, and token counts depend on the model (section 5). Take a representative sample of chunks and count them with the counting facility of the model you will actually use. Do this once per model change, and note that changing the embedding model or the generation model can change what a "500-token chunk" means.

11.6 Have someone read the output

The last step is not automatable. Have someone who reads the language look at a sample of retrieved passages, and specifically at their boundaries. Chunks that begin mid-word, chunks that begin with an orphaned closing bracket, interleaved ruby text, mojibake — these are obvious to a reader in seconds and invisible to every automated check in the pipeline.

12. Failure Modes and Anti-Patterns

A summary table of the failure modes discussed above, with the section that covers each.

Failure mode	Why it happens	Section
Index and query paths process text differently	Ingestion and query handling live in different code paths; nothing errors when they diverge	10.1
Normalization applied twice with different forms	Character filter and token filter both configured, with different settings	3.3
Aggressive normalization applied destructively at extraction	NFKC applied once at ingestion with no unfolded copy retained; the original is unrecoverable	3.2
Sentence splitter finds no boundaries	Terminator set contains only ASCII punctuation	6.1
Chunk size and overlap inherited from an English configuration	Character and token counts mean different amounts of content across scripts	2.2, 5.3, 6.4
Token budget estimated with another model's tokenizer	Tokenizers are model-specific and do not transfer	5.2
Chunks cut at prohibited break positions	Character-count cutting without line-breaking rules	6.2
Mixed-script tokens skip the filter that was configured for them	Token type dispatch; `한국abc` types as `<ALPHANUM>`, not `<HANGUL>`	9.2
Morphological analyzer assumed present but not installed	Japanese, Chinese, and Korean analyzers are plugins, not built-ins	4.2
Wrong compound-handling default assumed	Japanese `SEARCH` keeps the compound; Korean `DISCARD` does not	4.4
Dictionary updates do not take effect	Some plugins refresh only on redeployment or reindex	4.5
Index-time synonyms treated as updatable	`updateable: true` applies only to search analyzers	9.3
Short queries return nothing	Query shorter than `min_gram`, or single character with no unigram output	10.3
Simplified and Traditional Chinese treated as one	Not covered by Unicode normalization; needs conversion	3.4
"The model is multilingual, so we are fine"	Language support is coarse; says nothing about domain separation or cross-lingual matching	8.1, 8.2
N-gram chosen as the universal answer	High recall, but index size and precision costs are real	4.1
Extraction damage assumed absent	Column order, ruby interleaving, encoding damage, OCR substitution produce well-formed wrong text	7

Two meta-patterns are worth naming separately, because they generate the specific failures above rather than sitting alongside them.

Inferring behavior from configuration. Almost every item in the table is invisible in the configuration and visible in the token stream or the retrieved passage. The configuration says what you asked for. The token stream says what you got.

Treating this as a solved setup step. Dictionaries age, vocabulary arrives, corpora acquire new document sources with new extraction defects, and model swaps change what a token is. The parts of this that are configuration are set once; the parts that are dictionary and corpus are ongoing.

13. Frequently Asked Questions

Do I need a morphological analyzer, or is an n-gram approach good enough?
It depends on how much of your important vocabulary a general dictionary will not contain. N-gram splitting never fails on unknown terms, which is exactly the property you want when the corpus is full of product codes and internal jargon; it costs index size and precision. Morphological analysis gives better precision on ordinary prose and requires you to maintain a user dictionary for the terms it does not know. Indexing both into separate fields is a legitimate answer and is what Sudachi's documentation is pointing at when it notes that using both its A and B modes "can improve precision and recall" in full-text search. Section 11.4 is how to decide for your corpus.

Is NFKC the right normalization for a CJK retrieval pipeline?
It is the common choice for the index side, because it unifies width variants and therefore makes queries match documents that differ only in width. But UAX #15 warns that the compatibility forms "erase many formatting distinctions" and "may remove distinctions that are important to the semantics of the text," and advises treating them like case mappings. The usual resolution is to fold aggressively on the matching path and keep a less-folded copy for display and for generation context. Section 3.2.

Can I just use the embedding model and skip lexical search entirely?
You can, and for some corpora it is fine. The lexical side earns its keep on exact strings — identifiers, part numbers, names — which are the terms most likely to be outside an embedding model's training distribution. If your users search for those, dropping lexical search removes the only component that handles them well. Section 9.1.

Our embedding model says it supports 100+ languages. Is that enough?
It tells you the model was trained on the language. It does not tell you how well it separates near-duplicates in your domain, how it behaves on cross-lingual query-document pairs, or whether embeddings cluster by language rather than topic in a mixed corpus. Also read the service documentation's model reference rather than the announcement — the two can carry different commitments, as with the Titan V2 example in section 8.1. Then test on your own data.

Where exactly should normalization happen?
Three positions: at extraction, as a character filter before the tokenizer, or as a token filter after it. For dictionary-based analysis, normalizing before the tokenizer is usually safer, because dictionary lookups are sensitive to the form of the input. What matters more than the choice is that the same choice applies on the query path. Section 3.3 and section 10.1.

How do I set chunk size for CJK text?
Not by copying the number from your English configuration. Determine which unit binds for your model — several embedding models document both a token limit and a character limit, and which one you hit first depends on the script (section 5.3). Then measure that unit on your actual text with the counting facility of the model you are using, and apply line-breaking rules so the cut does not land on a prohibited break position (section 6.2).

Do I need to handle Simplified and Traditional Chinese separately?
If your corpus or your queries contain both, yes. Unicode normalization does not unify them — it is not that kind of difference. You need a conversion step, or you need to index both forms. Section 3.4.

Why does Korean behave differently from Chinese and Japanese here?
Because modern Korean prose uses spaces, so whitespace splitting produces plausible output rather than obviously broken output. What the spaces delimit, though, is a stem plus attached particles rather than a searchable term, so the failure appears later — as poor recall rather than as visibly broken tokenization. This also shows up in the line-breaking algorithm, where UAX #14 notes Korean requires tailoring to a different class than the ideographic scripts. Sections 1.3 and 6.2.

Is this a Japanese-specific problem?
No. The specifications name Thai, Lao, Chinese, and Japanese together as languages where the default word-boundary algorithm is insufficient, and the same class of problem exists for Khmer and others. The examples here lean on Japanese for compactness — a short Japanese sentence can exercise several failure modes at once — but the design principles are about writing systems, not about any particular language.

14. Summary

The assumption that a space marks a word boundary is baked into more of a retrieval pipeline than it appears to be: into tokenizers, chunkers, overlap parameters, and sentence splitters. Applied to Chinese, Japanese, or Korean, it does not raise errors; it degrades results in ways that look like problems elsewhere.

The specifications are more helpful here than they are usually given credit for. UAX #29 states outright that its default word-boundary algorithm is insufficient for these languages and names dictionary lookup as the remedy. UAX #15 defines normalization forms precisely and warns, in unusually plain terms, about what the compatibility forms destroy. UAX #14 provides a break-opportunity model that yields cheap, principled improvements to chunk boundaries without needing a dictionary at all. UAX #11 explains why a character count is not a stable unit across scripts. Reading these four before choosing a configuration is a better use of time than reading comparisons of tokenizers.

Above the specification layer, the decisions are trade-offs rather than answers. Morphological analysis versus n-grams trades precision and index size against robustness to unknown vocabulary. Aggressive normalization trades matching recall against preserved distinctions. Compound handling has different defaults in different analyzers, and those defaults were chosen for different purposes. None of these has a setting that is correct independent of the corpus, which is why section 11 is a procedure rather than a recommendation.

The one thing that is not a trade-off is symmetry. Whatever you do to text on the way into the index, do the same thing to the query — and where you deliberately do otherwise, know that you did. That single constraint, plus the habit of reading the token stream instead of trusting the configuration, prevents most of the failures in section 12.

15. References

Unicode Consortium (Unicode 17.0.0)

Unicode Standard Annex #29: Unicode Text Segmentation — Revision 47, 2025-08-17
Unicode Standard Annex #15: Unicode Normalization Forms — Revision 57, 2025-07-30
Unicode Standard Annex #14: Unicode Line Breaking Algorithm — Revision 55, 2025-09-05
Unicode Standard Annex #11: East Asian Width — Revision 44, 2025-07-24
Unicode Standard Annex #50: Unicode Vertical Text Layout — Revision 33, 2025-07-24
Unicode 17.0.0