Training models

Fit a text model only after the example set is large enough

model development
training models
text classification
Learn what model training means for text, why a 28-sentence pilot set is too small, and how a penalized logistic model learns from inaugural paragraphs.

Maya has a folder of sentence labels that did its first job well. It helped a small team agree on what counted as training, scheduling, skills, requirements, and other workforce details.

Then someone asks for a predictor. The folder has the same labels, the same care, and a new problem: the material that is useful for learning a codebook can be far too small for measuring a model.

This lesson makes that break visible. The Riverton teaching set stops at a pilot check. The modeling work moves to a larger public corpus, presidential inaugural paragraphs bundled with quanteda.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching. The inaugural addresses are works of the United States government and are treated here as public domain text packaged by quanteda.

TipWhat you will learn

This lesson shows how to:

  • tell when a labeled set is too small for model testing;
  • move from a labeling pilot to a larger teaching corpus;
  • distinguish a recipe, model specification, workflow, fit, and prediction;
  • compare a few candidate models without touching the final test set;
  • fit the selected penalized logistic regression model; and
  • report a live score with a baseline and limits attached.

Stop at the pilot set

A training set is the labeled material a model learns from. A test set is labeled material held back until scoring. If the test set is tiny, one right or wrong row can swing the score by a large amount.

suppressPackageStartupMessages({
  library(readr)
  library(dplyr)
  library(tibble)
  library(rsample)
  library(recipes)
  library(textrecipes)
  library(parsnip)
  library(workflows)
  library(yardstick)
  library(glmnet)
})

riverton_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

set.seed(4300)
riverton_split <- group_initial_split(
  riverton_sentences,
  group = document_id,
  prop = 0.75
)
riverton_training <- training(riverton_split)
riverton_testing <- testing(riverton_split)

riverton_label_counts <- riverton_sentences |>
  count(reference_label, name = "sentences") |>
  arrange(desc(sentences), reference_label)

split_size <- tibble(
  part = c("training", "test"),
  rows = c(nrow(riverton_training), nrow(riverton_testing))
)

knitr::kable(
  riverton_label_counts,
  col.names = c("Label", "Sentences"),
  caption = "The Riverton pilot labels are too few for a model score",
  row.names = FALSE
)
The Riverton pilot labels are too few for a model score
Label Sentences
requirement 9
training 7
other 5
schedule 5
skill 2 
knitr::kable(
  split_size,
  col.names = c("Split part", "Rows"),
  caption = "Keeping documents whole leaves only four test sentences",
  row.names = FALSE
)
Keeping documents whole leaves only four test sentences
Split part Rows
training 24
test 4

The split is deterministic because the seed is 4300, and it keeps all rows from a source document together. It leaves 4 test sentences from one document, containing 4 of the five labels. A single changed answer would move accuracy by about 25 percentage points. That is not enough room to learn how far off an automatic answer might be.

Change the material

The larger corpus comes from quanteda::data_corpus_inaugural, reached through the shared helper. Each row is one paragraph of at least 25 words. The outcome for this lesson is era, which asks whether a paragraph came from an address before 1900 or from 1900 or later.

source("R/inaugural-corpus.R")

paragraphs <- inaugural_paragraphs()

corpus_summary <- tibble(
  measure = c("paragraphs", "speeches", "people", "surnames"),
  value = c(
    nrow(paragraphs),
    n_distinct(paragraphs$speech_id),
    n_distinct(paragraphs$president),
    n_distinct(paragraphs$surname)
  )
)

era_counts <- paragraphs |>
  group_by(era) |>
  summarise(
    paragraphs = n(),
    median_words = median(paragraph_words),
    .groups = "drop"
  )

knitr::kable(
  corpus_summary,
  col.names = c("Measure", "Value"),
  caption = "The inaugural paragraph corpus used for model training",
  row.names = FALSE
)
The inaugural paragraph corpus used for model training
Measure Value
paragraphs 1377
speeches 60
people 40
surnames 36 
knitr::kable(
  era_counts,
  col.names = c("Era", "Paragraphs", "Median paragraph words"),
  caption = "Paragraphs by era",
  row.names = FALSE
)
Paragraphs by era
Era Paragraphs Median paragraph words
before 1900 485 112
1900 or later 892 54

The material has 1,377 paragraphs from 60 speeches by 40 people. The helper keeps president as the full name and also keeps surname, because a surname-only grouping would merge separate people who share a family name. The word-count column matters too: paragraphs before 1900 have median length 112, while later paragraphs have median length 54.

That size does not make the task general to all political speech or all historical writing. It does give the lesson enough held-out paragraphs to test a specific claim on this corpus.

Keep the final test set out of model selection

The split keeps each speech whole. A paragraph from one inaugural address cannot sit in training while another paragraph from the same address sits in testing. The seed is 4301 for the split.

The test speeches get one job in this worked example: estimate performance after the model choice is finished. Choosing a model from its test score would spend that evidence before the final check. This is still an internal holdout from a corpus used throughout the section, not confirmation on new text from another source.

Inside the training set, cross-validation makes several temporary train-and-check splits. Each temporary check is called a fold. The folds are often called CV folds. They are grouped by speech here, so every paragraph from one address stays together. Five folds put 9 or 10 speeches in each assessment set while leaving both era labels represented.

baselines <- read_csv(
  "data/inaugural/baselines.csv",
  na = character(),
  col_types = cols(
    task = col_character(),
    outcome = col_character(),
    rows = col_integer(),
    speeches = col_integer(),
    people = col_integer(),
    largest_class = col_character(),
    baseline_accuracy = col_double()
  )
)

model_comparison <- read_csv(
  "data/inaugural/model-comparison.csv",
  na = "",
  col_types = cols(
    rank = col_integer(),
    candidate = col_character(),
    family = col_character(),
    penalty = col_double(),
    mtry = col_integer(),
    min_n = col_integer(),
    mean_bal_accuracy = col_double(),
    std_err_bal_accuracy = col_double(),
    mean_accuracy = col_double(),
    std_err_accuracy = col_double(),
    selected = col_logical(),
    selection_reason = col_character()
  )
)

fold_summary <- read_csv(
  "data/inaugural/model-comparison-folds.csv",
  na = character(),
  col_types = cols(
    fold = col_character(),
    analysis_speeches = col_integer(),
    assessment_speeches = col_integer(),
    assessment_paragraphs = col_integer(),
    before_1900 = col_integer(),
    later_1900 = col_integer()
  )
)

set.seed(4301)
era_split <- group_initial_split(
  paragraphs,
  group = speech_id,
  prop = 0.75
)
era_training <- training(era_split)
era_testing <- testing(era_split)

set.seed(4310)
era_folds <- group_vfold_cv(
  era_training,
  group = speech_id,
  v = 5,
  strata = era,
  balance = "groups"
)

fold_group_overlap <- vapply(
  era_folds$splits,
  \(split) {
    length(intersect(
      unique(analysis(split)$speech_id),
      unique(assessment(split)$speech_id)
    ))
  },
  integer(1)
)

knitr::kable(
  fold_summary,
  col.names = c(
    "Fold",
    "Training speeches",
    "Assessment speeches",
    "Assessment paragraphs",
    "Before 1900",
    "1900 or later"
  ),
  caption = "Five grouped folds inside the training speeches",
  row.names = FALSE
)
Five grouped folds inside the training speeches
Fold Training speeches Assessment speeches Assessment paragraphs Before 1900 1900 or later
Resample1 36 10 244 51 193
Resample2 37 9 254 55 199
Resample3 37 9 150 42 108
Resample4 37 9 228 90 138
Resample5 37 9 166 52 114

Stratification distributes early and later speeches across the folds; it does not equalize paragraph counts. The table shows that the paragraph-level class mix still varies, which is one reason balanced accuracy is the primary measure.

Compare a small set of candidates

This is not a search across every model R can fit. The comparison asks whether a different kind of decision rule changes the answer:

  • Ridge logistic regression keeps every word feature but pulls all fitted weights toward zero.
  • Lasso logistic regression uses the same linear model but can set some word weights exactly to zero.
  • A random forest combines many branching rules and can learn relationships that are not linear.

The tuning parameters control how each candidate learns. Ridge tests four penalties from 0.01 to 10; lasso tests four from 0.0001 to 0.1. Their fitted penalty paths span different scales, so copying one grid across both would waste settings on duplicate or intercept-only fits. The forest tries six combinations of features per split (mtry) and the minimum rows needed in a branch (min_n). The builder in data-raw/build-model-comparison.R fits all 14 settings: four ridge, four lasso, and six forest. It never reads the final test speeches.

Balanced accuracy is the primary selection measure because it gives the two era classes equal weight. Raw accuracy appears beside it so the effect of the uneven class counts stays visible.

The table shows the two leading ridge settings and the leading lasso and forest settings. The ridge means differ, but their fold-to-fold standard errors overlap. Five related folds do not provide a confidence interval for other corpora.

comparison_display <- model_comparison |>
  filter(selected | rank %in% c(2L, 3L, 5L)) |>
  mutate(
    setting = case_when(
      candidate == "random forest" ~
        paste0("mtry ", mtry, "; min_n ", min_n),
      TRUE ~ paste0("penalty ", penalty)
    ),
    chosen = if_else(selected, "yes", "no")
  ) |>
  select(
    candidate,
    setting,
    mean_bal_accuracy,
    std_err_bal_accuracy,
    mean_accuracy,
    chosen
  )

selected_candidate <- model_comparison |>
  filter(selected)

knitr::kable(
  comparison_display |>
    mutate(
      across(
        c(
          mean_bal_accuracy,
          std_err_bal_accuracy,
          mean_accuracy
        ),
        \(value) round(value, 4)
      )
    ),
  col.names = c(
    "Candidate",
    "Setting",
    "Mean balanced accuracy",
    "Standard error",
    "Mean accuracy",
    "Chosen"
  ),
  caption = "Leading settings from 14 candidates compared on training folds",
  row.names = FALSE
)
Leading settings from 14 candidates compared on training folds
Candidate Setting Mean balanced accuracy Standard error Mean accuracy Chosen
ridge logistic penalty 0.01 0.8008 0.0266 0.8188 yes
ridge logistic penalty 0.1 0.7901 0.0343 0.8320 no
lasso logistic penalty 0.001 0.7809 0.0274 0.8013 no
random forest mtry 100; min_n 20 0.7761 0.0300 0.8437 no

Ridge with penalty 0.01 has the highest mean balanced accuracy, 0.8008. The fold-to-fold standard errors overlap the nearby candidates, so the table does not establish a universal winner. We keep ridge because it leads this comparison and leaves a linear model whose coefficients can be inspected in lesson 45.

The selected penalty is the lowest ridge value tried. That says less shrinkage helped among these four settings; it does not establish that 0.01 is the best possible penalty. A finished search would extend the grid before making that claim.

The random forest is a useful check rather than a required upgrade. Its best raw accuracy is competitive, but its lower balanced accuracy shows that more correct paragraphs can still mean less even performance across classes.

Build the selected workflow

A tidymodels pipeline has four parts:

  1. A recipe is a set of instructions that turns raw text into numeric model columns. Each step_*() line adds one instruction.
  2. A model specification names the kind of model and its settings before it sees data.
  3. An engine names the package that performs the calculations. Here that package is glmnet.
  4. A workflow keeps the recipe and model specification together. Fitting the workflow hands it training data so it can estimate model weights.

The recipe turns text into model columns. Tf-idf means term frequency times inverse document frequency. In plain language, a word counts for more in a paragraph when it is common there and rare elsewhere. Stop words such as the and of are removed first, and the recipe keeps the 500 most frequent remaining tokens.

The selected model is logistic regression, a linear classifier for categories. The penalty pulls large coefficients toward zero so the model does not chase every small accident in the training text.

text_recipe <- recipe(era ~ paragraph, data = era_training) |>
  step_tokenize(paragraph) |>
  step_stopwords(paragraph) |>
  step_tokenfilter(paragraph, max_tokens = 500) |>
  step_tfidf(paragraph)

text_model <- logistic_reg(penalty = 0.01, mixture = 0) |>
  set_engine("glmnet")

era_workflow <- workflow() |>
  add_recipe(text_recipe) |>
  add_model(text_model)

set.seed(4302)
era_fitted <- fit(era_workflow, data = era_training)

era_predictions <- predict(era_fitted, era_testing) |>
  bind_cols(era_testing |> select(era))

live_metrics <- metric_set(accuracy, bal_accuracy)(
  era_predictions,
  truth = era,
  estimate = .pred_class
)
live_accuracy <- live_metrics |>
  filter(.metric == "accuracy") |>
  pull(.estimate)
live_bal_accuracy <- live_metrics |>
  filter(.metric == "bal_accuracy") |>
  pull(.estimate)

corpus_baseline <- baselines |>
  filter(task == "era") |>
  pull(baseline_accuracy)

training_majority <- era_training |>
  count(era, sort = TRUE) |>
  slice(1) |>
  pull(era)

split_baseline_accuracy <- mean(era_testing$era == training_majority)

score_table <- tibble(
  measure = c(
    "Final test accuracy",
    "Final test balanced accuracy",
    "Same-split training-majority rule",
    "Corpus largest-class baseline"
  ),
  value = c(
    live_accuracy,
    live_bal_accuracy,
    split_baseline_accuracy,
    corpus_baseline
  )
)

knitr::kable(
  score_table |>
    mutate(value = round(value, 4)),
  col.names = c("Measure", "Value"),
  caption = "One held-out test after candidate selection",
  row.names = FALSE
)
One held-out test after candidate selection
Measure Value
Final test accuracy 0.8209
Final test balanced accuracy 0.8361
Same-split training-majority rule 0.4179
Corpus largest-class baseline 0.6478

Only after the candidate and penalty are fixed do we use the 14 final test speeches. The model scores 0.8209 accuracy and 0.8361 balanced accuracy. The majority label learned from training would score 0.4179 on this exact test set. That rate is below 0.5 because the training majority is 1900 or later, while this test split contains more before 1900 paragraphs. Keeping speeches whole does not preserve the corpus class mix. The corpus-wide largest-class baseline is 0.6478, but the same-split comparison is the one a deployable rule could use.

What the fit did not prove

A fitted model is a frozen recipe plus learned weights. It can label new rows in the same shape as the training rows. It cannot prove that the label is well chosen, that the corpus stands for other text, or that the score will hold under a harder split. The next lesson tests those questions with repeated splits.

What to remember

  • A pilot corpus can teach labeling without being large enough for testing.
  • A grouped test set with four rows cannot measure much.
  • A speech-held-out split keeps one address from leaking across both sides.
  • Grouped folds compare candidates without spending the final test set.
  • A recipe, model specification, engine, and workflow have different jobs.
  • Tf-idf gives more weight to words that are common in one paragraph and rare in the rest.
  • A model score needs a baseline beside it.

A trained classifier is only the start of the measurement. The useful question is whether its score survives a fairer test.

Sources