Learn why tokenization is a rule-based decision before any word count can begin.
A planning meeting begins with the Lab’s labeled sentences on the table. The team has 28 reviewed text units from a job board page and a training flyer. Six came from the flyer, and 22 came from job-detail lines.
The labels answer one question: what does each text unit say? The next question is smaller and more mechanical. Before the Lab can count words, it has to decide what a word is.
Note
The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.
train a subword model without writing files into the project folder; and
explain what actually decides how many pieces a text costs.
Start with the sentence on the page
The first code the reader sees brings in readr for files, dplyr and tibble for tables, purrr for repeated checks, stringr for strings, tokenizers for word and sentence splitting, and tokenizers.bpe for subword splitting. A tibble is the tidyverse table shape that prints its dimensions and each column’s type.
The first sentence fits on one line: Paid 12-week training is provided. A token is one piece of text produced by a splitting rule. The same visible sentence can produce different tokens because the rule makes different choices.
Split on spaces
One simple rule is to split wherever there is whitespace. Here stringr uses the ICU text library to find spaces, tabs, and similar separators. That rule gives five pieces. The period stays attached to provided., and 12-week stays in one piece.
whitespace_tokens <-str_split( first_sentence,"\\s+",simplify =FALSE)[[1]]knitr::kable(tibble(token_number =seq_along(whitespace_tokens),token = whitespace_tokens ),col.names =c("Token number", "Whitespace token"),caption ="Tokens made by splitting on whitespace",row.names =FALSE)
Tokens made by splitting on whitespace
Token number
Whitespace token
1
Paid
2
12-week
3
training
4
is
5
provided.
This rule is easy to explain, but it keeps punctuation and hyphenated material attached to nearby letters. That may or may not fit the count the Lab wants.
Use a word tokenizer
tokenizers::tokenize_words() applies a different rule. It lowercases text, drops punctuation, and splits the hyphenated phrase. The same sentence becomes six tokens: paid, 12, week, training, is, and provided.
default_word_tokens <-tokenize_words(first_sentence)[[1]]knitr::kable(tibble(token_number =seq_along(default_word_tokens),token = default_word_tokens ),col.names =c("Token number", "Default word token"),caption ="Tokens from tokenizers::tokenize_words()",row.names =FALSE)
Tokens from tokenizers::tokenize_words()
Token number
Default word token
1
paid
2
12
3
week
4
training
5
is
6
provided
That rule is useful when Paid and paid should count together. It is a poor fit if the Lab needs to keep punctuation as its own evidence.
Keep case and punctuation visible
The same function can be asked not to lowercase and not to remove punctuation. With those settings, Paid stays capitalized, the hyphen becomes its own token, and the full stop becomes its own token. The sentence becomes eight tokens.
case_punctuation_tokens <-tokenize_words( first_sentence,lowercase =FALSE,strip_punct =FALSE)[[1]]token_count_summary <-tibble(rule =c("split on whitespace","default word tokenizer","keep case and punctuation" ),tokens =c(length(whitespace_tokens),length(default_word_tokens),length(case_punctuation_tokens) ))knitr::kable( token_count_summary,col.names =c("Rule", "Tokens from the first sentence"),caption ="One sentence under three tokenization rules",row.names =FALSE)
One sentence under three tokenization rules
Rule
Tokens from the first sentence
split on whitespace
5
default word tokenizer
6
keep case and punctuation
8
Five, six, and eight are all correct counts under the rule that produced them. The count alone does not tell the reader which rule was used.
Check sentence splitting before word counts
The file already stores one sentence per row. The Lab still checks that a sentence tokenizer sees the same 28 pieces before moving on. This guard matters because some files store several sentences in one row, while others break a sentence across rows.
For these rows, tokenizers::tokenize_sentences() returns one piece per row. That means the file’s sentence boundary choice and the tokenizer’s boundary choice agree on this teaching dataset.
Train a subword tokenizer
Modern language models often split text into subwords, pieces smaller than some words and larger than single letters when the model has learned a common pattern. Byte pair encoding (BPE) starts from single characters and repeatedly glues together the pair that occurs most often. The model below trains only on the 28 Riverton sentences.
The vocab_size = 250L argument is a modeling choice that controls how many pieces the trainer can keep. The coverage = 1 argument tells the trainer to keep every character it sees in the training text; characters outside that set have no piece. The special marker <UNK> is the model’s way of saying it has no piece for this, short for “unknown”. The code also sets threads = 1L so the result does not depend on how work is split across processors.
Note
The explicit model_path sends the model to a temporary file during rendering.
corpus_file <-tempfile(fileext =".txt")write_lines(sentences$text, corpus_file)bpe_model <-bpe( corpus_file,model_path =tempfile(fileext =".bpe"),vocab_size =250L,coverage =1,threads =1L)first_vocabulary_rows <- bpe_model$vocabulary |>slice_head(n =10)knitr::kable( first_vocabulary_rows,col.names =c("ID", "Subword"),caption ="The first rows of the BPE vocabulary",row.names =FALSE)
The first rows of the BPE vocabulary
ID
Subword
0
1
2
3
4
▁
5
e
6
i
7
r
8
s
9
a
The vocabulary has 250 entries. The first four rows are special markers, then the boundary marker ▁ and single characters. The marker ▁ means that a subword begins at a word boundary.
Compare familiar and unfamiliar text
When the BPE model sees the first sentence, common pieces such as training, is, and provided. stay together. The rarer 12-week material fragments.
first_sentence_subwords <-bpe_encode( bpe_model,x ="Paid 12-week training is provided.",type ="subwords")[[1]]unseen_phrase_subwords <-bpe_encode( bpe_model,x ="unpredictability biotechnology",type ="subwords")[[1]]subword_summary <-tibble(text =c("Paid 12-week training is provided.","unpredictability biotechnology" ),pieces =c(length(first_sentence_subwords),length(unseen_phrase_subwords) ),subwords =c(str_c(first_sentence_subwords, collapse =" | "),str_c(unseen_phrase_subwords, collapse =" | ") ))knitr::kable( subword_summary,col.names =c("Text", "Subword pieces", "Subwords"),caption ="BPE pieces for familiar and unfamiliar text",row.names =FALSE)
▁ | un | pre | d | ic | ta | b | il | i | t | y | ▁b | i | o | te | ch | n | ol | o | g | y
The first sentence takes nine subword pieces. The phrase unpredictability biotechnology, which does not appear in the Riverton rows, takes 21 pieces in this toy model. Larger released tokenizers use different training data, vocabularies, normalizers, and unknown policies, so their piece counts must be measured rather than inferred from this result.
It is tempting to end there with a rule: unfamiliar material costs more pieces. The next section shows that the rule is false as stated.
What actually decides the number of pieces
This model was trained on 28 English lines written in the Latin script, and coverage = 1 told the trainer to keep every character it saw in that text. A character outside that set has no piece of its own, and the model falls back to <UNK>.
Watch what that does to the piece count. The comparison below adds two texts the model has never seen: a Chinese phrase meaning “training course” and a Greek word. It also switches from type = "subwords" to type = "ids", which reports the vocabulary entry each piece actually resolved to.
unknown_id <- bpe_model$vocabulary$id[ bpe_model$vocabulary$subword =="<UNK>"]texts_to_encode <-c(familiar_sentence ="Paid 12-week training is provided.",unseen_english ="unpredictability biotechnology",unseen_chinese ="\u57F9\u8BAD\u8BFE\u7A0B",unseen_greek ="\u0393\u03BB\u03C9\u03C3\u03C3\u03B1")piece_report <-tibble(example =names(texts_to_encode),text =unname(texts_to_encode),ids =map(unname(texts_to_encode), \(one_text) bpe_encode(bpe_model, x = one_text, type ="ids")[[1]] ),subwords =map(unname(texts_to_encode), \(one_text) bpe_encode(bpe_model, x = one_text, type ="subwords")[[1]] )) |>mutate(characters =str_length(text),pieces =map_int(ids, length),unknown_pieces =map_int( ids, \(id_vector) sum(id_vector == unknown_id) ),shown_as_subwords =map_chr( subwords, \(piece_vector) str_c(piece_vector, collapse =" | ") ),display_text =case_when( example =="unseen_chinese"~"<span lang=\"zh\">培训课程</span>", example =="unseen_greek"~"<span lang=\"el\">Γλωσσα</span>",TRUE~ text ),display_subwords =case_when( example =="unseen_chinese"~str_c("<span lang=\"zh\">", shown_as_subwords,"</span>" ), example =="unseen_greek"~str_c("<span lang=\"el\">", shown_as_subwords,"</span>" ),TRUE~ shown_as_subwords ) )knitr::kable( piece_report |>select(display_text, characters, pieces, unknown_pieces, display_subwords),col.names =c("Text","Characters","Pieces","Unknown pieces","What the subword view shows" ),caption ="Piece counts and unknown pieces for four texts",row.names =FALSE,escape =FALSE)
▁ | un | pre | d | ic | ta | b | il | i | t | y | ▁b | i | o | te | ch | n | ol | o | g | y
培训课程
4
2
1
▁ | 培训课程
Γλωσσα
6
2
1
▁ | Γλωσσα
The Chinese phrase is the least familiar text on the page and costs two pieces, where the fully familiar English sentence costs nine. Nothing was learned about Chinese. One of those two pieces is <UNK>, the model’s way of recording that it has no representation at all.
That is the whole correction. The number of pieces depends on the tokenizer and the vocabulary it learned, on the script and the language, on the normalization the model applies before splitting, and above all on what the model does with a character it does not know. Under this model’s unknown policy, unfamiliar material can cost fewer pieces than familiar material, because collapsing to <UNK> is cheap and lossy.
The shown_as_subwords column is a second warning. It displays the Chinese characters as though the model had a piece for them, because the subword view reconstructs surface text rather than reporting vocabulary entries. Count unknowns from the ids, never from the printed pieces.
Check that the pieces can be turned back into text
A tokenizer that cannot rebuild its input has lost something. Decoding the ids answers that question directly.
round_trip <- piece_report |>mutate(decoded =map_chr( ids, \(id_vector) bpe_decode(bpe_model, id_vector)[[1]] ),recovers_input = decoded == text )knitr::kable( round_trip |>select(display_text, pieces, unknown_pieces, decoded, recovers_input),col.names =c("Text","Pieces","Unknown pieces","Decoded from ids","Recovers the input" ),caption ="Round-trip results for the four texts",row.names =FALSE,escape =FALSE)
Round-trip results for the four texts
Text
Pieces
Unknown pieces
Decoded from ids
Recovers the input
Paid 12-week training is provided.
9
0
Paid 12-week training is provided.
TRUE
unpredictability biotechnology
21
0
unpredictability biotechnology
TRUE
培训课程
2
1
FALSE
Γλωσσα
2
1
FALSE
Both Latin-script texts come back character for character, including the punctuation and the digits. Both unseen-script texts come back as <UNK>, and the check confirms the pattern exactly: the texts that fail to round-trip are precisely the texts that contained an unknown piece.
Round-trip recovery tests reversible text identity. It does not test source alignment: some tokenizers can return offsets into the original input even when an unknown id cannot decode to its text. A task that marks spans or highlights matches must test those offset mappings separately.
The result gives the team a rule for the meeting: a word count must name the rule that made the words, and a piece count must name the vocabulary, the script it covers, and what it does with everything else.
What to remember
Tokenization turns text into pieces according to a chosen rule.
The first Riverton sentence becomes 5, 6, or 8 word tokens under three rules.
Sentence tokenization preserves the 28 rows in this file.
This toy BPE used larger pieces for some material present in its training text; that pattern is not universal.
Piece counts depend on the vocabulary, the script, the normalization the model applies, and its policy for unknown characters.
Under an unknown-piece policy, unfamiliar text can cost fewer pieces and lose everything it contained.
Count unknown pieces from the ids, and check that the ids decode back to the input.
The Lab can start counting only after it writes down the token rule. For these sentences, that rule changes the unit being counted before any summary appears.