library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(cld2)
library(cld3)
library(digest)
language_path <- "data/riverton/language-samples.csv"
language_samples <- read_csv(
language_path,
na = character(),
col_types = cols(
sample_id = col_character(),
written_in = col_character(),
text = col_character()
)
)
reference_metadata <- read_csv(
"data/riverton/riverton-reference-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
license = col_character(),
created_on = col_character(),
fingerprint = col_character()
)
)
language_reference <- reference_metadata |>
filter(artifact == "language-samples.csv")
sample_fingerprint <- digest::digest(
paste(read_lines(language_path), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)
recorded_fingerprint <- reference_metadata |>
filter(artifact == "language-samples.csv") |>
pull(fingerprint)
detector_versions <- tibble(
detector = c("cld2", "cld3"),
package_version = c(
as.character(packageVersion("cld2")),
as.character(packageVersion("cld3"))
)
)
display_samples <- language_samples |>
mutate(
display_text = str_c(
"<span lang=\"",
written_in,
"\">",
text,
"</span>"
)
)Identifying document language
Treat language guesses as fallible classifications
documents
language identification
workforce research
Learn how two language detectors behave on short Riverton text samples and why abstention can be safer than a wrong route.
Short notes in several languages create a routing problem before content labels begin. The coordinator wants to route each note to the right review queue before anyone labels its content.
Open house. exposes the problem. A person reads the two-word notice as English, while a language detector may have too little evidence to say.
Language identification is a classification problem. A classifier assigns a label from a fixed set, such as en for English or es for Spanish. The risk has two shapes: a detector can return the wrong code, or it can decline to answer and leave the item unrouted. This lesson shows which shape these two detectors take on a tiny teaching file.
Note
The Riverton inbox and the language samples are fictional teaching materials.
TipWhat you will learn
Use this lesson to practice how to:
- run two local language detectors on fixed samples;
- compare detector output with recorded labels;
- separate agreement from correctness;
- explain why short texts are hard to classify; and
- describe why returning
NAcan be safer than a wrong language code.
Load the labelled samples
The sample table is small, but the setup still records detector versions, checks text length, runs the local CLD2 and CLD3 detectors, and fingerprints the file. The samples are short strings written by the author for teaching.
| Detector package | Version |
|---|---|
| cld2 | 1.2.6 |
| cld3 | 1.6.2 |
| File | SHA-256 fingerprint |
|---|---|
| data/riverton/language-samples.csv | fa84ffaf608578b8dd0e4f0dbc7425049cd9d362e9d41a1773078973afeccf67 |
| Sample | Recorded language | Text |
|---|---|---|
| L01 | en | Evening shifts require a valid forklift certification. |
| L02 | en | Apply by October 15. |
| L03 | es | Muy útil y fácil de usar. |
| L04 | fr | Les cours du soir commencent en octobre. |
| L05 | de | Der Kurs beginnt im Oktober. |
| L06 | nl | De cursus begint in oktober. |
| L07 | en | Open house. |
| L08 | es | Casa abierta. |
The written_in column records the language the author meant to write, not a verified annotation. The Spanish Casa abierta. is grammatical, but as an event notice it is a calque rather than an idiomatic phrase. These results come from the detector versions printed above; a package change can return different codes for the same strings.
Run two detectors
CLD2 and CLD3 are trained language detectors. NA means the detector returned no language label. They do not understand the whole situation around a note; they classify the characters they receive.
detected_samples <- language_samples |>
mutate(
characters = str_length(text),
words = str_count(text, boundary("word")),
length_group = if_else(characters <= 13L, "short", "long"),
cld2_language = cld2::detect_language(text),
cld3_language = cld3::detect_language(text),
cld2_match = !is.na(cld2_language) & cld2_language == written_in,
cld3_match = !is.na(cld3_language) & cld3_language == written_in,
display_text = str_c(
"<span lang=\"",
written_in,
"\">",
text,
"</span>"
)
)
open_house <- detected_samples |>
filter(text == "Open house.")| Sample | Recorded language | Text | Characters | Words | CLD2 | CLD3 |
|---|---|---|---|---|---|---|
| L01 | en | Evening shifts require a valid forklift certification. | 54 | 7 | en | en |
| L02 | en | Apply by October 15. | 20 | 4 | en | en |
| L03 | es | Muy útil y fácil de usar. | 25 | 6 | es | es |
| L04 | fr | Les cours du soir commencent en octobre. | 40 | 7 | fr | fr |
| L05 | de | Der Kurs beginnt im Oktober. | 28 | 5 | de | de |
| L06 | nl | De cursus begint in oktober. | 28 | 5 | NA | NA |
| L07 | en | Open house. | 11 | 2 | NA | NA |
| L08 | es | Casa abierta. | 13 | 2 | NA | NA |
For this installed version, both detectors abstain on Open house. by returning NA. That is a useful failure for routing, because the item can be sent to human review rather than routed under a wrong code without review.
Count matches, abstentions, and label differences
Agreement between two detectors does not prove correctness. They can share training data patterns and make the same mistake. The recorded labels let the team count what happened on these eight examples.
detector_counts <- tibble(
detector = c("CLD2", "CLD3"),
matched_recorded_label = c(
sum(detected_samples$cld2_match),
sum(detected_samples$cld3_match)
),
no_label = c(
sum(is.na(detected_samples$cld2_language)),
sum(is.na(detected_samples$cld3_language))
)
) |>
mutate(
differed_from_recorded_label =
nrow(detected_samples) - matched_recorded_label - no_label
)| Detector | Matched recorded label | No label | Different from recorded label |
|---|---|---|---|
| CLD2 | 5 | 3 | 0 |
| CLD3 | 5 | 3 | 0 |
Each detector matches 5 of the 8 recorded labels, returns no label for 3 of the 8, and differs from the recorded label on 0 of the 8. Those fractions describe this tiny teaching file. Eight samples measure nothing about detector accuracy on a workforce archive.
Compare short and longer text
Length changes the amount of evidence a detector sees, but this file is too small to estimate a length effect. There is no principled cutoff here. The 13-character line separates the two shortest strings from the other 6 examples.
length_summary <- detected_samples |>
group_by(length_group) |>
summarise(
samples = n(),
cld2_matches = sum(cld2_match),
cld2_no_label = sum(is.na(cld2_language)),
cld3_matches = sum(cld3_match),
cld3_no_label = sum(is.na(cld3_language)),
.groups = "drop"
) |>
arrange(length_group)| Length group | Samples | CLD2 matches | CLD2 no label | CLD3 matches | CLD3 no label |
|---|---|---|---|---|---|
| long | 6 | 5 | 1 | 5 | 1 |
| short | 2 | 0 | 2 | 0 | 2 |
The two shortest samples produce no matches because both detectors abstain. Among the six other samples, each detector matches 5 recorded labels and abstains on 1. The Dutch sentence is long enough to read by eye, but both detectors return NA.
A wrong code is worse than no code
The eight labelled samples produced no wrong codes. That does not mean wrong codes are impossible. The next string was chosen because it produces one: it is a single hand-picked case, not a rate, and it shows what a wrong code looks like. It also sits just above the arbitrary 13-character cutoff.
short_probe <- tibble(
text = c("Open house.", "Evening class."),
recorded_language = "en"
) |>
mutate(
characters = str_length(text),
cld2_language = cld2::detect_language(text),
cld3_language = cld3::detect_language(text),
cld2_match = !is.na(cld2_language) & cld2_language == recorded_language,
cld3_match = !is.na(cld3_language) & cld3_language == recorded_language
)| Text | Recorded language | CLD2 | CLD3 |
|---|---|---|---|
| Open house. | en | NA | NA |
| Evening class. | en | NA | lb |
Evening class. is English in this teaching probe. CLD2 returns NA, while CLD3 returns lb, the ISO 639-1 code for Luxembourgish. For routing, the abstention is safer: no queue is selected until someone checks the item.
These eight samples are all written in the Latin alphabet, so this page shows nothing about other writing systems. That is a limit of the file, not a reassurance.
Language identification in production
While cld2 and cld3 illustrate the routing problem, models such as Meta’s fasttext are common alternatives for fast language identification. When deploying any model, keep these production realities in mind:
- Language scope and BCP 47: Projects must explicitly state their supported language scope upfront. When recording a language, use a standard like BCP 47 tags (e.g.,
en-US) rather than informal names. - Per-language recall: A model’s overall accuracy hides its failures. Always measure recall per language, because models routinely fail on low-resource languages.
- Code-switching: The single-label
cld2::detect_language()andcld3::detect_language()calls used here return one document label. Mixed language text needs a detector designed for spans or tokens, or a review route; single-label behavior should not be generalized to every model. - Probability vs. certainty: A model might return a confidence score like
0.95, but do not treat this as a calibrated probability. It is only the model’s internal score, not a statistical guarantee.
What to remember
- Language identification may return a code or no label.
- Agreement between detectors does not prove that a language code is correct.
- The eight-row file is an inspection exercise: both detectors declined to label the two shortest strings rather than guessing at them.
- Returning
NAcan be safer than sending text to the wrong language queue.
Items with NA results belong in review, and short-text language codes should prompt checking rather than settle a queue. The hand-picked probe demonstrates a possible wrong code; the eight-row file demonstrates safe abstention.