Identifying document language

Treat language guesses as fallible classifications

documents
language identification
workforce research
Learn how two language detectors behave on short Riverton text samples and why abstention can be safer than a wrong route.

Short notes in several languages create a routing problem before content labels begin. The coordinator wants to route each note to the right review queue before anyone labels its content.

Open house. exposes the problem. A person reads the two-word notice as English, while a language detector may have too little evidence to say.

Language identification is a classification problem. A classifier assigns a label from a fixed set, such as en for English or es for Spanish. The risk has two shapes: a detector can return the wrong code, or it can decline to answer and leave the item unrouted. This lesson shows which shape these two detectors take on a tiny teaching file.

Note

The Riverton inbox and the language samples are fictional teaching materials.

TipWhat you will learn

Use this lesson to practice how to:

  • run two local language detectors on fixed samples;
  • compare detector output with recorded labels;
  • separate agreement from correctness;
  • explain why short texts are hard to classify; and
  • describe why returning NA can be safer than a wrong language code.

Load the labelled samples

The sample table is small, but the setup still records detector versions, checks text length, runs the local CLD2 and CLD3 detectors, and fingerprints the file. The samples are short strings written by the author for teaching.

library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(cld2)
library(cld3)
library(digest)

language_path <- "data/riverton/language-samples.csv"
language_samples <- read_csv(
  language_path,
  na = character(),
  col_types = cols(
    sample_id = col_character(),
    written_in = col_character(),
    text = col_character()
  )
)

reference_metadata <- read_csv(
  "data/riverton/riverton-reference-metadata.csv",
  na = character(),
  col_types = cols(
    artifact = col_character(),
    description = col_character(),
    source = col_character(),
    license = col_character(),
    created_on = col_character(),
    fingerprint = col_character()
  )
)

language_reference <- reference_metadata |>
  filter(artifact == "language-samples.csv")

sample_fingerprint <- digest::digest(
  paste(read_lines(language_path), collapse = "\n"),
  algo = "sha256",
  serialize = FALSE
)

recorded_fingerprint <- reference_metadata |>
  filter(artifact == "language-samples.csv") |>
  pull(fingerprint)

detector_versions <- tibble(
  detector = c("cld2", "cld3"),
  package_version = c(
    as.character(packageVersion("cld2")),
    as.character(packageVersion("cld3"))
  )
)

display_samples <- language_samples |>
  mutate(
    display_text = str_c(
      "<span lang=\"",
      written_in,
      "\">",
      text,
      "</span>"
    )
  )

knitr::kable(
  detector_versions,
  col.names = c("Detector package", "Version"),
  caption = "Detector package versions for this run",
  row.names = FALSE
)
Detector package versions for this run
Detector package Version
cld2 1.2.6
cld3 1.6.2 
knitr::kable(
  tibble(
    file = language_path,
    sha256 = sample_fingerprint
  ),
  col.names = c("File", "SHA-256 fingerprint"),
  caption = "Fingerprint for the author-written language sample file",
  row.names = FALSE
)
Fingerprint for the author-written language sample file
File SHA-256 fingerprint
data/riverton/language-samples.csv fa84ffaf608578b8dd0e4f0dbc7425049cd9d362e9d41a1773078973afeccf67 
knitr::kable(
  display_samples |>
    select(sample_id, written_in, display_text),
  col.names = c("Sample", "Recorded language", "Text"),
  caption = "Eight labelled language samples used for detection",
  row.names = FALSE,
  escape = FALSE
)
Eight labelled language samples used for detection
Sample Recorded language Text
L01 en Evening shifts require a valid forklift certification.
L02 en Apply by October 15.
L03 es Muy útil y fácil de usar.
L04 fr Les cours du soir commencent en octobre.
L05 de Der Kurs beginnt im Oktober.
L06 nl De cursus begint in oktober.
L07 en Open house.
L08 es Casa abierta.

The written_in column records the language the author meant to write, not a verified annotation. The Spanish Casa abierta. is grammatical, but as an event notice it is a calque rather than an idiomatic phrase. These results come from the detector versions printed above; a package change can return different codes for the same strings.

Run two detectors

CLD2 and CLD3 are trained language detectors. NA means the detector returned no language label. They do not understand the whole situation around a note; they classify the characters they receive.

detected_samples <- language_samples |>
  mutate(
    characters = str_length(text),
    words = str_count(text, boundary("word")),
    length_group = if_else(characters <= 13L, "short", "long"),
    cld2_language = cld2::detect_language(text),
    cld3_language = cld3::detect_language(text),
    cld2_match = !is.na(cld2_language) & cld2_language == written_in,
    cld3_match = !is.na(cld3_language) & cld3_language == written_in,
    display_text = str_c(
      "<span lang=\"",
      written_in,
      "\">",
      text,
      "</span>"
    )
  )

open_house <- detected_samples |>
  filter(text == "Open house.")

knitr::kable(
  detected_samples |>
    select(
      sample_id,
      written_in,
      display_text,
      characters,
      words,
      cld2_language,
      cld3_language
    ),
  col.names = c(
    "Sample",
    "Recorded language",
    "Text",
    "Characters",
    "Words",
    "CLD2",
    "CLD3"
  ),
  caption = "Language detector output for each sample",
  row.names = FALSE,
  escape = FALSE
)
Language detector output for each sample
Sample Recorded language Text Characters Words CLD2 CLD3
L01 en Evening shifts require a valid forklift certification. 54 7 en en
L02 en Apply by October 15. 20 4 en en
L03 es Muy útil y fácil de usar. 25 6 es es
L04 fr Les cours du soir commencent en octobre. 40 7 fr fr
L05 de Der Kurs beginnt im Oktober. 28 5 de de
L06 nl De cursus begint in oktober. 28 5 NA NA
L07 en Open house. 11 2 NA NA
L08 es Casa abierta. 13 2 NA NA

For this installed version, both detectors abstain on Open house. by returning NA. That is a useful failure for routing, because the item can be sent to human review rather than routed under a wrong code without review.

Count matches, abstentions, and label differences

Agreement between two detectors does not prove correctness. They can share training data patterns and make the same mistake. The recorded labels let the team count what happened on these eight examples.

detector_counts <- tibble(
  detector = c("CLD2", "CLD3"),
  matched_recorded_label = c(
    sum(detected_samples$cld2_match),
    sum(detected_samples$cld3_match)
  ),
  no_label = c(
    sum(is.na(detected_samples$cld2_language)),
    sum(is.na(detected_samples$cld3_language))
  )
) |>
  mutate(
    differed_from_recorded_label =
      nrow(detected_samples) - matched_recorded_label - no_label
  )

knitr::kable(
  detector_counts,
  col.names = c(
    "Detector",
    "Matched recorded label",
    "No label",
    "Different from recorded label"
  ),
  caption = "Detector outcomes across the eight labelled samples",
  row.names = FALSE
)
Detector outcomes across the eight labelled samples
Detector Matched recorded label No label Different from recorded label
CLD2 5 3 0
CLD3 5 3 0

Each detector matches 5 of the 8 recorded labels, returns no label for 3 of the 8, and differs from the recorded label on 0 of the 8. Those fractions describe this tiny teaching file. Eight samples measure nothing about detector accuracy on a workforce archive.

Compare short and longer text

Length changes the amount of evidence a detector sees, but this file is too small to estimate a length effect. There is no principled cutoff here. The 13-character line separates the two shortest strings from the other 6 examples.

length_summary <- detected_samples |>
  group_by(length_group) |>
  summarise(
    samples = n(),
    cld2_matches = sum(cld2_match),
    cld2_no_label = sum(is.na(cld2_language)),
    cld3_matches = sum(cld3_match),
    cld3_no_label = sum(is.na(cld3_language)),
    .groups = "drop"
  ) |>
  arrange(length_group)

knitr::kable(
  length_summary,
  col.names = c(
    "Length group",
    "Samples",
    "CLD2 matches",
    "CLD2 no label",
    "CLD3 matches",
    "CLD3 no label"
  ),
  caption = "Detector outcomes by sample length in this file",
  row.names = FALSE
)
Detector outcomes by sample length in this file
Length group Samples CLD2 matches CLD2 no label CLD3 matches CLD3 no label
long 6 5 1 5 1
short 2 0 2 0 2

The two shortest samples produce no matches because both detectors abstain. Among the six other samples, each detector matches 5 recorded labels and abstains on 1. The Dutch sentence is long enough to read by eye, but both detectors return NA.

A wrong code is worse than no code

The eight labelled samples produced no wrong codes. That does not mean wrong codes are impossible. The next string was chosen because it produces one: it is a single hand-picked case, not a rate, and it shows what a wrong code looks like. It also sits just above the arbitrary 13-character cutoff.

short_probe <- tibble(
  text = c("Open house.", "Evening class."),
  recorded_language = "en"
) |>
  mutate(
    characters = str_length(text),
    cld2_language = cld2::detect_language(text),
    cld3_language = cld3::detect_language(text),
    cld2_match = !is.na(cld2_language) & cld2_language == recorded_language,
    cld3_match = !is.na(cld3_language) & cld3_language == recorded_language
  )

knitr::kable(
  short_probe |>
    select(text, recorded_language, cld2_language, cld3_language),
  col.names = c("Text", "Recorded language", "CLD2", "CLD3"),
  caption = "Two short English probes and their detected languages",
  row.names = FALSE
)
Two short English probes and their detected languages
Text Recorded language CLD2 CLD3
Open house. en NA NA
Evening class. en NA lb

Evening class. is English in this teaching probe. CLD2 returns NA, while CLD3 returns lb, the ISO 639-1 code for Luxembourgish. For routing, the abstention is safer: no queue is selected until someone checks the item.

These eight samples are all written in the Latin alphabet, so this page shows nothing about other writing systems. That is a limit of the file, not a reassurance.

Language identification in production

While cld2 and cld3 illustrate the routing problem, models such as Meta’s fasttext are common alternatives for fast language identification. When deploying any model, keep these production realities in mind:

  • Language scope and BCP 47: Projects must explicitly state their supported language scope upfront. When recording a language, use a standard like BCP 47 tags (e.g., en-US) rather than informal names.
  • Per-language recall: A model’s overall accuracy hides its failures. Always measure recall per language, because models routinely fail on low-resource languages.
  • Code-switching: The single-label cld2::detect_language() and cld3::detect_language() calls used here return one document label. Mixed language text needs a detector designed for spans or tokens, or a review route; single-label behavior should not be generalized to every model.
  • Probability vs. certainty: A model might return a confidence score like 0.95, but do not treat this as a calibrated probability. It is only the model’s internal score, not a statistical guarantee.

What to remember

  • Language identification may return a code or no label.
  • Agreement between detectors does not prove that a language code is correct.
  • The eight-row file is an inspection exercise: both detectors declined to label the two shortest strings rather than guessing at them.
  • Returning NA can be safer than sending text to the wrong language queue.

Items with NA results belong in review, and short-text language codes should prompt checking rather than settle a queue. The hand-picked probe demonstrates a possible wrong code; the eight-row file demonstrates safe abstention.

Sources