Use hunspell analyses without mistaking them for contextual morphology
word parsing
morphology
workforce research
Learn what an English spelling dictionary can and cannot say about stems, affixes, ambiguity, and context in workforce text.
Twenty-eight reviewed sentences come from a job board page and a training flyer. The team wants to know whether shifts and shift should count as the same word when it summarizes the vocabulary.
That question matters because a worker may search for training, while a job posting may say shifts, certification, or certificate. The team needs a way to look inside a word before it decides what to group.
Morphology is the internal structure of words. A stem carries the core meaning, and affixes are added pieces such as plural s or past-tense ed. English marks a small amount of grammar this way, so English speakers often notice word parts only when a form looks odd.
Note
The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching. The dictionary output is real and comes from the local hunspell package.
TipWhat you will learn
By the end of this lesson, you will be able to:
describe a stem and an affix in plain language;
read a dictionary-based word analysis;
distinguish a useful decomposition from a misleading one;
find unanalyzed and ambiguous word types in a small vocabulary; and
explain why dictionary analysis differs from contextual morphology in sentences.
This lesson takes a limited first step. It asks what an English spelling dictionary can say about isolated word forms. That is different from contextual morphology, where a tagger or morphologizer labels a token in a sentence with feature bundles such as Number=Plur or Tense=Past. This page covers dictionary morphology, not contextual morphological tagging; Lesson 18 shows token-level feature bundles in its feats column.
Read the workforce sentences
To read the sentences, the code brings in readr for files, dplyr and tibble for tables, purrr for repeated list checks, stringr for text checks, tokenizers for word splitting, and hunspell for dictionary analysis. It then reads the 28 reviewed workforce sentences.
library(readr)library(dplyr)library(tibble)library(purrr)library(stringr)library(tokenizers)library(hunspell)sentences <-read_csv("data/workforce/workforce_sentences.csv",na =character(),col_types =cols(sentence_id =col_character(),document_id =col_character(),source_line =col_character(),text =col_character(),reference_label =col_character(),uncertainty =col_character(),annotator_id =col_character(),rationale =col_character(),codebook_version =col_character(),codebook_hash =col_character(),derived =col_character(),transformation =col_character() ))document_counts <- sentences |>count(document_id, name ="sentence_count") |>arrange(document_id)knitr::kable( document_counts,col.names =c("Document ID", "Sentences"),caption ="The 28 Riverton text units by source document",row.names =FALSE)
The 28 Riverton text units by source document
Document ID
Sentences
F001
6
J001
4
J002
4
J003
4
J004
3
J005
3
J006
4
The flyer has six rows, and the job-detail documents have 22 rows. The sample fits in a small table, but the vocabulary is large enough for several kinds of word behavior.
Ask the dictionary for analyses
hunspell_analyze() checks each word against an English spelling dictionary and its affix rules. The output is compact. st: gives a stem, and fl: gives a flag used by the dictionary’s rule system. A flag is a code in the dictionary, not a reader-friendly grammar label.
In the code, \(value) is R shorthand for a small function. Here it means that each returned vector is named value before it is pasted into one string.
For shifts, the dictionary finds the stem shift and marks the plural form with flag S.
The entry for requires gives a warning. Hunspell can produce an analysis that is internally legal and completely false. It allows re- in front of dictionary words, and quire happens to be a dictionary word. The two have nothing to do with each other: require came into English whole, from Old French requerre, from Latin requirere. The rule found a path, not a history.
certification has two analyses. It can stand as certification, or it can be connected to certificate through another rule. The letters alone do not choose between those analyses.
With running, the word is listed as its own dictionary entry, so the analyzer returns running and stops there. A common form can be stored directly.
Count what the dictionary can see
The team next applies the same dictionary check to the vocabulary of the 28 sentences. A word type is one distinct word form after lowercasing. If training appears several times, it counts once in this vocabulary.
The dictionary analyzes 91 of the 96 word types. Five receive no analysis: three numbers, the month october, and the place name riverton. Eight word types receive more than one analysis, which means the word form alone leaves more than one route through the dictionary.
Inspect the ambiguous cases
Ambiguity is not a bug by itself. A single spelling can have several plausible paths through a dictionary. The next table shows the Riverton word types where hunspell returned more than one analysis.
ambiguous_words <- vocabulary_summary |>filter(analysis_count >1L) |>transmute( word,analyses =map_chr(analyses, \(value) paste(value, collapse ="; ")) )knitr::kable( ambiguous_words,col.names =c("Word", "Allowed analyses"),caption ="Riverton word types with more than one dictionary analysis",row.names =FALSE)
Riverton word types with more than one dictionary analysis
Word
Allowed analyses
available
st:available; st:avail fl:B
certification
st:certification; st:certificate fl:N
communication
st:communication; st:communicate fl:N
customer
st:customer; st:custom fl:R
evening
st:evening; st:even fl:G
outdoors
st:outdoors; st:outdoor fl:S
provided
st:provided; st:provide fl:D
training
st:training; st:train fl:G
For the Lab, this table is useful because it slows the decision down. The dictionary has supplied possible word structures, not final labels. A person still has to decide whether the structure helps the question at hand.
Keep dictionary analysis separate from contextual morphology
English puts some information inside word endings: shifts marks plural, and provided marks a form related to tense or aspect. It leaves much else to word order and nearby helper words. Languages that mark case, person, tense, and number on the word itself place more information inside a single token. A tool built around English spelling assumptions will miss much of that structure.
Hunspell is a spelling dictionary with affix rules. It is not a universal morphological analyzer. It can produce an analysis that is internally legal and completely false. Hunspell allows re- in front of dictionary words, and quire happens to be a dictionary word. The rule found a path, not a history.
This lesson stays with spelling alone. In NLP, contextual morphology usually means assigning feature bundles such as Number=Plur or Tense=Past to a token in a sentence. Those features live in the feats column of a treebank and can be produced by a tagger or morphologizer. Lesson 18 displays them in parser output. This page does not produce those tags. It runs a spelling dictionary over an out-of-context list of word types, which is a different task with a different input and output.
What to remember
Morphology studies the parts inside words.
A stem carries the core meaning; affixes modify the form.
Dictionary rules can produce useful analyses and misleading ones.
One word form can have more than one allowed analysis.
Hunspell checks spelling-style rules on isolated words, not contextual morphology in sentences.
The team can group shifts with shift for a plain vocabulary count, but every returned stem still needs human judgment about the text. Feature bundles in running sentences are a different task.