Choose which words to keep and measure what falls outside
word parsing
vocabulary
workforce research
Learn how frequency cutoffs and out-of-vocabulary checks shape a small workforce text vocabulary.
After the tokenization check, the coordinator prints the Riverton Workforce Lab’s word tokens on a single sheet. The list looks manageable until the repeated words and one-time words sit side by side. A vocabulary cannot keep everything in a large project without a rule.
The team decides to test the rule on its small teaching data. It will build a word list from the job-detail rows, then ask how much of the training flyer falls outside that list.
Note
The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.
TipWhat you will learn
By the end of this lesson, you will be able to:
distinguish tokens from types;
count a word vocabulary and its one-time terms;
apply minimum-frequency cutoffs;
measure out-of-vocabulary words on held-out text; and
compare a word vocabulary with a subword vocabulary.
Count tokens and types
The first visible chunk loads readr for files, dplyr, tidyr, and tibble for tables, purrr for repeated work, tokenizers for word tokens, and tokenizers.bpe for subword tokens. A tibble is a table that prints its size and column types.
library(readr)library(dplyr)library(tidyr)library(tibble)library(purrr)library(tokenizers)library(tokenizers.bpe)sentences <-read_csv("data/workforce/workforce_sentences.csv",na =character(),col_types =cols(sentence_id =col_character(),document_id =col_character(),source_line =col_character(),text =col_character(),reference_label =col_character(),uncertainty =col_character(),annotator_id =col_character(),rationale =col_character(),codebook_version =col_character(),codebook_hash =col_character(),derived =col_character(),transformation =col_character() ))word_token_rows <- sentences |>transmute( sentence_id,token =tokenize_words(text) ) |>unnest_longer(token)word_counts <- word_token_rows |>count(token, name ="frequency", sort =TRUE)top_word_counts <- word_counts |>slice_head(n =8)knitr::kable( top_word_counts,col.names =c("Token type", "Frequency"),caption ="The eight most frequent word types in all 28 rows",row.names =FALSE)
The eight most frequent word types in all 28 rows
Token type
Frequency
is
13
a
7
required
7
the
5
training
5
are
4
experience
4
of
3
A token is one occurrence. A type is one distinct token value. These counts use the default tokenizers::tokenize_words() rule: lowercase text, drop punctuation, and split hyphenated material. Across the 28 rows, that rule creates 153 tokens and 96 types. Of those types, 72 appear exactly once.
The most frequent word types are ordinary function words and repeated terms from the notices: is appears 13 times, a and required appear 7 times each, the and training appear 5 times each, are and experience appear 4 times each, and of appears 3 times.
Split the data before testing a vocabulary
An honest check cannot build a vocabulary and test it on the same rows. The Lab uses the 22 job-detail rows to build the word list and the 6 flyer rows to test what the list has not seen.
Word-token counts after splitting by document source
Split
Rows
Tokens
Types
job-detail rows
22
133
86
training flyer
6
20
20
The job-detail rows contain 133 tokens and 86 types. The flyer rows contain 20 tokens. The split is small, but it keeps the test from grading a vocabulary on the same text that created it.
Keep words by minimum frequency
A minimum-frequency cutoff keeps only types that appear at least a stated number of times. The code below asks how many job-detail types remain and how many job-detail tokens those remaining types cover.
The shorthand \(cutoff) means “make a small function whose input is cutoff.”
The minimum-frequency-1 row is a reference point: it keeps every job-detail type, so full coverage is true by construction. The stricter rows show how many job-detail tokens lose their own word type under each cutoff.
Measure out-of-vocabulary words
An out-of-vocabulary token is a token that the vocabulary does not contain. Here the vocabulary comes from the 22 job-detail rows. The test asks which flyer tokens fall outside that job-built list.
A retrieval system can keep a different kind of vocabulary. A search index stores whole terms and counts for retrieval, while a tokenizer or subword model stores the pieces it will use to encode new text.
The unknown flyer types are 15, apply, by, classes, house, october, open, riverton, stipend, and support.
For those 10 token occurrences, the job-built word list has no entry.
Try the same split with subwords
Byte pair encoding (BPE) starts from single characters and repeatedly glues together the pair that occurs most often. The Lab trains a BPE model on the job-detail rows only, then encodes the flyer rows. This gives the subword vocabulary the same held-out test as the word vocabulary.
The word tokenizer lowercases both sources, so the BPE comparison does the same. Holding case constant matters: otherwise the comparison would change both the tokenization method and the case policy. The vocab_size = 200L argument is a modeling choice that controls how many pieces the trainer can keep. The coverage = 1 argument tells the trainer to keep every character it sees in the lowercased job-detail text. The special marker <UNK> is the model’s way of saying it has no piece for this, short for “unknown”.
Note
The explicit model_path sends the model to a temporary file during rendering.
The like-for-like comparison is now at the word-token level. The whole-word vocabulary has no entry for 10 of 20 flyer tokens. With both sources lowercased, none of the 20 word tokens contains a BPE unknown id, and none of the 68 BPE pieces is unknown. The subword model can assemble unseen words from characters and pieces learned in the job rows.
That result belongs to this 200-piece model and this case policy. It is not a general promise that BPE removes out-of-vocabulary material. Case-sensitive input, a different writing system, a smaller character coverage, or another tokenizer can produce unknown ids. Comparing 10 word tokens with 68 pieces would also mix denominators, so the conclusion uses the 20 word-token outcomes.
What to remember
Tokens are occurrences; types are distinct token values.
The 28 Riverton rows produce 153 word tokens and 96 word types.
Most word types in this small file appear exactly once.
Frequency cutoffs shrink a vocabulary and reduce token coverage.
Held-out text exposes out-of-vocabulary words.
With the same lowercase policy, this BPE model encodes all 20 held-out word tokens; that result is setting-specific, not a general guarantee.
These percentages belong only to this 28-sentence teaching corpus. They show how vocabulary choices behave on the Riverton example, not how often these patterns occur in workforce text more generally.