For the thesis student's four Arctic sentences in the architecture article, tokenizing meant lowercasing the text, splitting it into words, and discarding punctuation. That worked because the sentences were plain English words. Here is a sentence that is less cooperative, with three ways of splitting it into pieces:
O'Neill's state-of-the-art C++ library wasn't released until 3/20/91 (see www.example.com).
| Method | Output |
|---|---|
| Split on whitespace | O'Neill's state-of-the-art C++ library wasn't released until 3/20/91 (see www.example.com). |
| Lowercase, keep runs of letters and digits | o neill s state of the art c library wasn t released until 3 20 91 see www example com |
NLTK's default word tokenizer, word_tokenize |
O'Neill 's state-of-the-art C++ library was n't released until 3/20/91 ( see www.example.com ) . |
Which of these is right, and what does the choice decide? None is simply right, and the choice decides which queries can match which documents.
Each method fails somewhere. Splitting on whitespace leaves punctuation attached, so (see and see are different tokens. Keeping runs of letters and digits removes the punctuation but destroys C++ (now just c), splits the date into three numbers, and turns wasn't into wasn and t. NLTK's tokenizer follows the conventions of the Penn Treebank. It keeps the hyphenated phrase, the programming language, the date, and the address intact, and it splits the possessive and the contraction into separate tokens.
Each output makes different queries possible. And whatever the tokenizer does to documents, it must do the same to queries, or they will not match.
Tokenization is the step that chops a character sequence into these pieces. It takes the decoded text that From Bytes to Documents produced. Here is the route. First we fix the vocabulary for its output: words, tokens, types, and terms. Then we survey the hard cases in English and run four real tokenizers on them, and on the 10,788 Reuters-21578 news stories that this topic uses for its measurements. Next we turn to languages written without spaces, where tokenizing means segmenting, and to languages with long compounds. Finally we look at subword tokenization, which neural models use instead of words. Worked problems at the end apply each method to new strings.
Words, tokens, types, and terms
To compare tokenizers, we need to say exactly what they count. Manning, Raghavan, and Schütze (2008, §2.2.1) distinguish three notions, and the everyday word covers all of them:
- A token is an instance of a sequence of characters, grouped as a useful unit, at a particular position in a document.
- A type is the class of all tokens with the same character sequence.
- A term is a type, possibly normalized, that is included in the index's dictionary.
- A word, in ordinary usage, is any of these, which is why the more precise terms are needed.
Example (counting the three). The phrase to sleep perchance to dream has five tokens, four types (to appears twice), and, if to is a stop word, three terms, since to is then left out of the dictionary.
The distinction matters because each stage counts something different:
- Document length is measured in tokens, and even that count depends on the tokenizer. The sentence at the top of this article is 10 tokens split on whitespace, 20 as runs of letters and digits, and 15 under NLTK.
- Vocabulary size is measured in types.
- The index dictionary contains terms. These are types after normalization and stemming have merged some of them and stop-word removal has discarded others.
Problems in English
English separates most words with spaces, which makes tokenization look easy. The difficulties are in the exceptions, and they are common enough to matter.
Apostrophes. O'Neill, O'Neill's, aren't, and the boys' each use the apostrophe differently. Should O'Neill be one token, two (o and neill), or the joined form oneill? A user who searches for O'Neill may type any of these forms. A consistent rule is therefore more important than which rule is chosen. Manning, Raghavan, and Schütze (2008, §2.2.1) use this example to show that even simple cases have no obviously right answer.
Hyphens. Hyphens join elements that may or may not be a single word: co-education, Hewlett-Packard, state-of-the-art, e-mail. Splitting at hyphens lets state of the art match state-of-the-art, but it also lets Hewlett match alone. Keeping hyphenated forms whole fails when the user omits the hyphen, as people often do. A common solution indexes the hyphenated form and its parts, or treats a hyphenated query as a phrase query over the parts.
Names and phrases containing spaces. San Francisco, New York University, and Los Angeles-based are single concepts split by whitespace. Tokenizing them as separate words loses the distinction between New York University and a document that mentions New York and a university. That distinction is normally recovered with phrase queries over a positional index, not at tokenization time. Some systems also recognize known multi-word names and index them as single tokens.
Numbers, dates, and codes. These combine digits and punctuation in ways no general rule handles well:
- dates (3/20/91, Mar 20, 1991, 1991-03-20);
- version numbers (3.2.1) and product codes (B-52);
- IP addresses (100.2.86.144) and tracking numbers.
Older systems often ignored numbers entirely. Modern systems usually index them, because people search for model numbers, error codes, and statute citations. They often normalize recognizable types such as dates into a standard format, so that different spellings match.
Technical tokens. Programming languages (C++, C#, .NET), email addresses, URLs, hashtags, and chemical formulas all contain characters a general tokenizer treats as delimiters. A system serving technical users needs explicit rules or exception lists for them.
Approaches
How do real tokenizers handle these cases? Most tokenizers for space-separated languages are rule-based. They use regular expressions or finite-state automata that recognize token patterns, together with exception lists for known abbreviations and special forms.
The Unicode Standard defines default rules for word boundaries in Unicode Standard Annex #29 (UAX #29), covering many scripts at once. It does not cover the word segmentation of Chinese, Japanese, or Thai, described below. The standard tokenizer of the Lucene search library implements those rules (Apache Software Foundation, Lucene 10 documentation for StandardTokenizer). Rules are fast and deterministic, so the same rules can be applied identically at indexing and query time. That is why search engines prefer them to statistical tokenizers for space-separated text.
Four tokenizers on the hard cases
The problems above are easiest to compare side by side. The table runs the hard cases through three rule-based tokenizers:
- this site's own search tokenizer, which lowercases, removes accents, drops a possessive 's, keeps a decimal or version number such as 3.14 or 3.2.1 as one token, and otherwise keeps runs of the letters a to z and digits;
- the Unicode default word boundaries of UAX #29, computed with the
regexmodule for Python, whose word-boundary mode follows that annex; - NLTK's
word_tokenize, in version 3.9.1 of the toolkit. Its rules change between versions: version 3.10.4 gives the same rows in the table below but slightly different counts on Reuters (1,549,352 tokens and 62,828 types).
Splitting on whitespace is omitted, because it returns every one of these strings unchanged.
| Input | This site's search | UAX #29 | NLTK |
|---|---|---|---|
state-of-the-art |
state of the art |
state of the art |
state-of-the-art |
O'Neill's |
o neill |
O'Neill's |
O'Neill 's |
wasn't |
wasn t |
wasn't |
was n't |
3.14 |
3.14 |
3.14 |
3.14 |
1,000.50 |
1 000.50 |
1,000.50 |
1,000.50 |
3/20/91 |
3 20 91 |
3 20 91 |
3/20/91 |
U.S.A. |
u s a |
U.S.A |
U.S.A . |
C++ |
c |
C |
C++ |
www.example.com |
www example com |
www.example.com |
www.example.com |
jo@example.com |
jo example com |
jo example.com |
jo @ example.com |
#IR2026 |
ir2026 |
IR2026 |
# IR2026 |
naïve |
naive |
naïve |
naïve |
find (ligature) |
find |
find |
find |
研究生命起源 |
(nothing) | 研 究 生 命 起 源 |
研究生命起源 |
Each tokenizer embodies a different idea of what a token is.
- UAX #29 never breaks between letters joined by an apostrophe or a period, or between digits joined by a period or a comma. So it keeps contractions, decimals, abbreviations, and host names whole, and it splits at every hyphen and slash. It treats each Chinese character as a separate word, since it does not attempt dictionary segmentation.
- NLTK applies English-specific conventions. It splits off possessive 's and the n't of contractions, which keeps was matchable, and it leaves hyphenated words and dates whole.
- This site's tokenizer is the bluntest, which suits a small English-language wiki. It makes one exception, for decimal and version numbers, so that a title such as Opus 5.5 is not confused with Opus 5. A comma is still a boundary, so 1,000.50 becomes
1and000.50. Its accent and ligature folding comes for free from the normalization it applies first. It cannot index Chinese at all.
Of the three, only NLTK keeps C++. UAX #29 reduces it to C and this site's tokenizer to c, so under either of them a search for the language also matches the letter. The script runs all three on every string of the table, using the site's own tokenizing function for the first column.
The same choice on a whole collection
The hard cases are rare in any one sentence. Do the differences matter on a real collection? They do: on the same stories the four tokenizers produce between 37,871 and 84,873 types. The tokenizers were run on the 10,788 stories of Reuters-21578 with tools/experiments/textproc/tokenizers.py. The column "lowercased types" lowercases each type afterwards, to separate the effect of splitting from the effect of case folding.
Example (most types, most tokens). Whitespace splitting produces the most types, because punctuation stays attached to words and multiplies their forms. NLTK produces the most tokens, because it makes separate tokens of punctuation marks and of clitics such as 's and n't.
| Tokenizer | Tokens | Types | Lowercased types |
|---|---|---|---|
| Split on whitespace | 1,378,305 | 84,873 | 73,249 |
| UAX #29 | 1,403,026 | 60,203 | 49,530 |
NLTK word_tokenize |
1,549,426 | 62,856 | 52,230 |
| This site's search | 1,436,924 | 37,871 | 37,871 |
Under whitespace splitting, said appears as said 16,662 times, but also as said. 7,885 times and said, 614 times: three types for one word. Stripping punctuation from the two ends of each whitespace token collapses 19,026 types.
The site's tokenizer has the fewest types. The main reason is that it breaks numbers at their commas and slashes, and identifiers at every other punctuation mark: 1,000.50 becomes 1 and 000.50. Reuters is full of such figures, each of which would otherwise be a type of its own. Under UAX #29, 18,373 types (30.5 percent) contain a digit, against 8,697 (23.0 percent) under the site's tokenizer, and that difference of 9,676 is most of the gap of 11,659 between the two lowercased vocabularies. Of the site's 8,697, 7,131 are decimals such as 000.50, which its number rule keeps whole. A financial newswire may well want those figures searchable, so the smaller vocabulary is not simply better.
Languages without spaces
The tokenizers above all lean on spaces. What happens when there are none? Chinese and Japanese are written without spaces between words, and Thai has no spaces between words within a sentence. For these languages, tokenization means word segmentation, and it is ambiguous. Manning, Raghavan, and Schütze (2008, §2.2.1) give the example of the two characters 和尚. As a single word they mean monk. As two words, 和 and 尚, they mean and and still. Only context decides.
There are two main strategies:
- Segment into words using a dictionary, a statistical model trained on segmented text, or both. The simplest dictionary method, maximum matching, scans left to right and takes the longest dictionary word at each position. It works well on ordinary text. It fails on ambiguous sequences and on words missing from the dictionary, such as new names.
- Index character n-grams instead of words, usually overlapping pairs of characters. Every two-character word is then indexed exactly, and longer words are matched as sequences of overlapping pairs. No segmentation decisions are made, so no segmentation errors can occur. The cost is a larger index, and pairs that span a word boundary produce spurious matches.
Maximum matching worked through
A small example shows both strategies and how maximum matching fails. The string 研究生命起源 means 'research the origin of life'. Suppose the dictionary contains 研究 (research), 研究生 (graduate student), 生命 (life), 命 (fate), and 起源 (origin), with no word longer than three characters. Forward maximum matching starts at the left and, at each position, tries the longest candidate first:
| Position | Candidates tried, longest first | Taken |
|---|---|---|
| 1 | 研究生, 研究, 研 | 研究生 |
| 4 | 命起源, 命起, 命 | 命 |
| 5 | 起源, 起 | 起源 |
The result, 研究生 / 命 / 起源, reads 'graduate student / fate / origin'. The greedy first choice took 研究生 because it is a longer dictionary word than 研究, and the stranded 命 had to stand alone.
Example (matching from the right). Backward maximum matching runs the same procedure from the right end, taking the longest dictionary word that ends at each position. Here it produces the intended 研究 / 生命 / 起源. The trace:
| Ending at | Candidates tried, longest first | Taken |
|---|---|---|
| 6 | 命起源, 起源 | 起源 |
| 4 | 究生命, 生命 | 生命 |
| 2 | 研究 | 研究 |
Neither direction is right in general. One simple rule, adopted here as this article's own convention and used in its problems, runs both and, when they disagree, prefers the segmentation with fewer words, and then the one with fewer single characters. Statistical segmenters resolve such cases by scoring whole segmentations with word and context probabilities, instead of committing greedily.
The bigram alternative indexes 研究, 究生, 生命, 命起, and 起源. A query for 生命 (life) or 研究 (research) matches exactly, with no segmentation decision to get wrong. The pairs 究生 and 命起 are not words, and they are the price. They inflate the index, and a query that happens to contain the same pair matches this text spuriously. (The worked example is computed by tools/experiments/textproc/tokenizers.py.)
Experiments on a TREC collection of Chinese news found that overlapping character bigrams perform about as well as segmentation into short words (Kwok, 1997). Combining the retrieval lists of the two representations gave the best results, at a substantial cost (Kwok, 1999). Japanese adds another complication. It mixes three scripts, kanji, hiragana, and katakana, and the same word may be written in more than one of them. A query in one script may therefore need to match documents in another.
Compounds
Some languages do have spaces but pack several words between them. German, Dutch, Swedish, Finnish, and other languages form long compound words by joining their parts without spaces. Manning, Raghavan, and Schütze cite the German Lebensversicherungsgesellschaftsangestellter, 'life insurance company employee'. A query for Versicherung (insurance) will not match the compound unless the compound is split into its parts.
Decompounding uses a dictionary of known words and rules for the linking elements between them, such as the s in Lebens-. Braschler and Ripplinger (2004) found that decompounding improved German retrieval substantially, more than stemming did.
Finnish and Turkish go further. They are agglutinative: they build long words from many suffixes, so a single word can carry what English expresses in a phrase. For such languages, tokenization and morphological analysis cannot be separated.
Subword tokenization
An inverted index can add a new term whenever one appears. A neural language model cannot. It needs a fixed vocabulary of a few tens of thousands of units. With whole words as units, a word not in that vocabulary cannot be represented: a new name, a rare technical term, a misspelling. Neural models solve this with subword tokenization. Frequent words remain whole, and rare words are split into frequent pieces.
The most widely used method is byte-pair encoding (BPE). It originated as a data compression algorithm (Gage, 1994) and was adapted for tokenization by Sennrich, Haddow, and Birch (2016). The idea in one sentence: start from single characters and keep gluing together whichever adjacent pair is most frequent.
Their example (the code of their Algorithm 1) uses a training corpus containing low five times, lower twice, newest six times, and widest three times, with </w> marking the end of a word:
| Step | Merge | Pair count | Words after the merge |
|---|---|---|---|
| 0 | l o w </w>, l o w e r </w>, n e w e s t </w>, w i d e s t </w> |
||
| 1 | e + s |
9 | n e w es t </w>, w i d es t </w> |
| 2 | es + t |
9 | n e w est </w>, w i d est </w> |
| 3 | est + </w> |
9 | n e w est</w>, w i d est</w> |
| 4 | l + o |
7 | lo w </w>, lo w e r </w> |
| 5 | lo + w |
7 | low </w>, low e r </w> |
Each merge is chosen by counting. es appears in both newest and widest, 6 + 3 = 9 times. When pairs tie, as e s, s t, and t </w> do at step 1, an implementation's tie-breaking rule decides.
Example (an unseen word). The word lowest never appeared in the training corpus, and after these five merges it is tokenized as low + est</w>. A new word is segmented by starting from its characters, l o w e s t </w>, and replaying the merges in the order they were learned. Merges 1 to 3 build est</w> and merges 4 and 5 build low. Both units are now in the vocabulary, so an unseen word is built from familiar pieces instead of being unrepresentable. A real tokenizer applies tens of thousands of merges learned from a large corpus.
Two related methods are common:
- WordPiece, used by BERT, chooses merges that most increase the likelihood of the training data, rather than by raw frequency (Schuster and Nakajima, 2012; Wu et al., 2016).
- The unigram language model method starts from a large set of candidate subwords. It repeatedly removes those whose removal reduces the likelihood of the corpus the least (Kudo, 2018).
The SentencePiece toolkit implements both BPE and the unigram method directly on raw text (Kudo and Richardson, 2018). It treats spaces as ordinary characters, each space becoming the visible symbol ▁. The same procedure therefore works for languages with and without spaces.
For classical inverted indexes, subword units are rarely the indexing unit, because a query term split into common fragments matches far too many documents. They matter to retrieval in two other ways. Neural rankers and dense retrievers see text only through their subword vocabularies (sentence and passage encoders follows a text from its word pieces to one vector), so the way a domain's vocabulary is split affects how well the model understands it. And some learned sparse retrieval models, such as SPLADE (Formal, Piwowarski, and Clinchant, 2021), build inverted indexes whose terms are the subword vocabulary of a neural model. The units of the index then come from the tokenizer of a language model, not from a linguistic rule.
Documents and queries alike
Every choice in this article has a defensible alternative. What cannot be defended is tokenizing documents and queries inconsistently. Query processing may add steps of its own, such as the asymmetric expansion of query terms. But it must start from the same token boundaries as indexing.
Suppose documents are tokenized with NLTK and queries are split on whitespace. The query (see will not match the term see. And wasn't in a query will not match was and n't in the index. The modular design article recommends a single analysis component shared by indexing and querying for exactly this reason.
Worked problems
The course text's exercises on this material ask for normalized forms and are solved in Token Normalization. These four problems are posed on this article's own examples and exercise each of its methods once. Every result is computed by tools/experiments/textproc/tokenizers.py.
Problem 1 (tokens, types, terms). The line to be or not to be that is the question is lowercased and split at spaces. Count its tokens and types. Then count its terms if the words to, be, or, not, that, is and the are stop words.
Solution. The line has 10 tokens, one for each word occurrence. To and be occur twice each, so there are $10 - 2 = 8$ types: to, be, or, not, that, is, the, question. Seven of the eight types are stop words, which leaves one term, question. A document's length would be recorded as 10, its contribution to the vocabulary as 8, and its contribution to the dictionary as 1.
Problem 2 (four tokenizers, one sentence). Tokenize the sentence below with each of the four tokenizers of this article and count the tokens. Then give the tokens each produces for a query AT&T.
Dr. O'Brien e-mailed 2,500.75 dlrs to AT&T's U.K. office on 12/31/86.
Solution.
| Tokenizer | Tokens | Output |
|---|---|---|
| Split on whitespace | 11 | Dr. O'Brien e-mailed 2,500.75 dlrs to AT&T's U.K. office on 12/31/86. |
| This site's search | 18 | dr o brien e mailed 2 500.75 dlrs to at t u k office on 12 31 86 |
| UAX #29 | 15 | Dr O'Brien e mailed 2,500.75 dlrs to AT T's U.K office on 12 31 86 |
| NLTK | 15 | Dr. O'Brien e-mailed 2,500.75 dlrs to AT & T 's U.K. office on 12/31/86 . |
Each row follows from the rules stated above.
- Whitespace splitting leaves the final period on the date, so the last token is
12/31/86.and not the date. - This site's tokenizer breaks at every character that is not a letter or a digit, except a period between digits. The name gives
oandbrien, the amount2and500.75(the comma breaks it, the decimal point does not), and the date three numbers. It drops the possessive 's, so AT&T's givesatandt. - UAX #29 keeps letters joined by an apostrophe or a period and digits joined by a comma or a period. So O'Brien, 2,500.75 and the inner period of U.K. survive, while the hyphen, the ampersand and the slashes are boundaries. The trailing period of Dr. and of U.K. is dropped.
- NLTK keeps the hyphenated word and the date whole, keeps the periods of the abbreviations Dr. and U.K., and splits off the possessive and the sentence's final period as tokens of their own.
UAX #29 and NLTK both produce 15 tokens, but only seven of them are the same strings: O'Brien, 2,500.75, dlrs, to, AT, office and on. Equal counts do not mean equal tokens.
The query AT&T becomes at t under this site's tokenizer, AT T under UAX #29, and AT & T under NLTK. Under this site's tokenizer the document's AT&T's gave exactly the query's two tokens. Under NLTK it gave AT & T 's, which begins with the query's three. Under UAX #29 it gave AT T's, and the document holds T's where the query has T. So the query matches the sentence as a phrase under this site's tokenizer and under NLTK, and under UAX #29 only if a later step removes the possessive.
Problem 3 (maximum matching in both directions). The string thetabledownthere is to be segmented with the dictionary the, theta, table, bled, down, own, there, whose longest word has five letters. Run forward and backward maximum matching, and apply this article's rule for a disagreement.
Solution. Forward matching tries the longest candidate first at each position:
| Position | Candidates tried, longest first | Taken |
|---|---|---|
| 1 | theta | theta |
| 6 | bledo, bled | bled |
| 10 | ownth, ownt, own | own |
| 13 | there | there |
Backward matching takes the longest dictionary word that ends at each position:
| Ending at | Candidates tried, longest first | Taken |
|---|---|---|
| 17 | there | there |
| 12 | edown, down | down |
| 8 | table | table |
| 3 | the | the |
Forward matching gives theta / bled / own / there and backward matching gives the intended the / table / down / there. The greedy first choice of theta over the strands the rest, as 研究生 did above. The rule's two preferences do not settle this case: both segmentations have four words, and neither contains a single character. A segmenter that scores whole segmentations by how probable their words are would be needed to prefer the second.
Problem 4 (byte-pair encoding continued). Continue the byte-pair encoding example from five merges to ten. A tie goes to the pair met first when the four words are scanned in the order low, lower, newest, widest, each from left to right. Then tokenize newest, widest and lower, and the unseen words lowest, newer and wider.
Solution. After merge 5 the corpus is low </w> (5 times), low e r </w> (2), n e w est</w> (6) and w i d est</w> (3). The pair counts are 6 for each of n e, e w and w est</w>, 5 for low </w>, 3 for each of w i, i d and d est</w>, and 2 for each of low e, e r and r </w>.
| Step | Merge | Pair count | Words after the merge |
|---|---|---|---|
| 6 | n + e |
6 | ne w est</w> |
| 7 | ne + w |
6 | new est</w> |
| 8 | new + est</w> |
6 | newest</w> |
| 9 | low + </w> |
5 | low</w>, low e r </w> |
| 10 | w + i |
3 | wi d est</w> |
At step 6 the three pairs with count 6 tie, and n e is met first. Merging it leaves ne w and w est</w> tied at 6, and ne w comes first. At step 9 the largest remaining count is 5, for low </w>. That merge joins low to the end-of-word mark only where the mark follows it directly, so low e r </w> is unchanged. At step 10 the three pairs of widest tie at 3, and w i is met first.
A word is tokenized by replaying the ten merges, in order, on its characters:
| Word | Tokens after ten merges |
|---|---|
| newest | newest</w> |
| widest | wi d est</w> |
| lower | low e r </w> |
| lowest (unseen) | low est</w> |
| newer (unseen) | new e r </w> |
| wider (unseen) | wi d e r </w> |
The most frequent training word has become a single token. Lowest is still two tokens, the same as after five merges: merge 9 does not apply to it, because in lowest the symbol low is followed by est</w> and not by </w>. Newer reuses new, which was learned from newest, and spells out the ending er in single characters, because no merge has built it.
What we have learned
A tokenizer decides what the units of the index are, and there is no neutral way to decide. On the same Reuters stories, four reasonable tokenizers produced between 37,871 and 84,873 types. The differences came from punctuation, numbers, and clitics, not from exotic text. Languages without spaces turn tokenization into segmentation, compounds call for splitting words apart, and neural models replace words with learned subword pieces. Through all of it, one rule holds: documents and queries must be cut the same way.
Once a collection is a stream of tokens, it can be counted. The counts are regular: a few types account for a large share of all tokens, and new types keep appearing however much text has been read. The next article, Term Statistics: Zipf's Law and Heaps' Law, measures both regularities on Reuters-21578. They shape the steps that follow, stop-word removal and normalization, which decide which tokens become terms, and they predict how large the index will be.
References
- Apache Software Foundation. Apache Lucene 10 documentation: class
org.apache.lucene.analysis.standard.StandardTokenizer. https://lucene.apache.org/core/ record · 7 pages - Braschler, M., and Ripplinger, B. (2004). How effective is stemming and decompounding for German text retrieval? Information Retrieval, 7(3–4), 291–316. record · 2 pages
- Formal, T., Piwowarski, B., and Clinchant, S. (2021). SPLADE: Sparse lexical and expansion model for first stage ranking. In Proceedings of SIGIR 2021, 2288–2292. record · 2 pages
- Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2), 23–38. record
- Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of ACL 2018, 66–75. record
- Kudo, T., and Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of EMNLP 2018: System Demonstrations, 66–71. record
- Kwok, K. L. (1997). Comparing representations in Chinese information retrieval. In Proceedings of SIGIR 1997, 34–41. record
- Kwok, K. L. (1999). Employing multiple representations for Chinese information retrieval. Journal of the American Society for Information Science, 50(8), 709–723. record
- Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, section 2.2. Cambridge University Press. record · 102 pages
- Schuster, M., and Nakajima, K. (2012). Japanese and Korean voice search. In Proceedings of ICASSP 2012, 5149–5152. record
- Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of ACL 2016, 1715–1725. record · 2 pages
- The Unicode Consortium. Unicode Standard Annex #29: Unicode Text Segmentation. https://www.unicode.org/reports/tr29/ record
- Wu, Y., Schuster, M., Chen, Z., et al. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. record