Choosing which examples to label next

Use active learning without confusing uncertainty with truth

training data
active learning
workforce research
Learn how a small model can rank unlabeled workforce sentences for human review.

A finished codebook covers training, requirements, schedules, skills, and other workforce text. For this exercise, five complete job postings form the labeled starting set. A sixth posting and the flyer form an unlabeled pool. The team could read the pool in file order. It tests whether a small model can point to cases that its current evidence does not separate cleanly.

Active learning is a loop in which a model helps choose the next items for people to label. The model does not supply the answer. It points to examples that meet a selection rule, and a person applies the codebook. Here, that rule measures uncertainty about training, not which sentence matters most to a worker.

Note

The sentences and research team are fictional. The model is deliberately small and exists to make the selection logic visible; it is not suitable for workforce decisions.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • separate an initial labeled set from an annotation pool;
  • explain uncertainty sampling;
  • rank items by distance from a 50% model score;
  • reveal labels only after selection;
  • show that a selected subset moves an estimate either way; and
  • name the biases that active learning can introduce.

Hide the answers in the annotation pool

The teaching file contains reviewed labels so the lesson can check its work. During selection, the model receives labels from documents J001 through J005. The complete J006 posting and F001 flyer form the annotation pool. Keeping documents intact avoids training on one part of a posting while selecting another part from the same source. The code below opens files with readr, handles tables with dplyr and tibble, repeats checks with purrr, and matches text with stringr.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)

all_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

annotation_view <- all_sentences |>
  select(
    sentence_id,
    document_id,
    text,
    reference_label
  )
initial_labels <- annotation_view |>
  filter(document_id %in% sprintf("J%03d", 1:5))
annotation_pool <- annotation_view |>
  filter(document_id %in% c("J006", "F001")) |>
  select(sentence_id, document_id, text)
held_back_labels <- annotation_view |>
  filter(document_id %in% c("J006", "F001")) |>
  select(sentence_id, reference_label)

knitr::kable(
  annotation_pool,
  col.names = c(
    "Sentence ID",
    "Document ID",
    "Text"
  ),
  caption = "Ten text units waiting for annotation",
  row.names = FALSE
)
Ten text units waiting for annotation
Sentence ID Document ID Text
s019 J006 Two years of customer service experience are required.
s020 J006 Weekend shifts are required.
s021 J006 Product training is included.
s022 J006 Clear written communication is an essential skill.
s023 F001 RIVERTON SKILLS OPEN HOUSE
s024 F001 DATA SUPPORT CERTIFICATE
s025 F001 Paid training stipend
s026 F001 Evening classes
s027 F001 No prior experience required
s028 F001 Apply by October 15

The labels are stored separately and do not enter the selection calculation. This prevents the lesson from using an answer that would be unavailable in a real annotation round.

Fit a small keyword model

The example uses a Bernoulli naive Bayes model. It records whether selected terms appear, then compares how often those terms occur in training and non-training sentences. Half-count smoothing prevents an observed term from receiving a conditional probability of exactly zero or one.

This model is transparent enough to inspect, but its assumptions are strong. It treats the selected terms as conditionally independent and ignores word order, context, and most of the sentence.

The candidate terms come from the codebook and labeled examples, not from reading the held-back pool. The code removes any candidate never observed in the labeled set. The short form \(keyword) means a small function with one input named keyword. Otherwise add-one smoothing could make an unseen term appear more common in the smaller class merely because its denominator is smaller.

candidate_keywords <- c(
  "training",
  "certificate",
  "apprenticeship",
  "mentor",
  "stipend",
  "classes",
  "required",
  "preferred",
  "license",
  "diploma",
  "portfolio",
  "shift",
  "schedule",
  "skill",
  "spreadsheet",
  "communication",
  "experience"
)

make_features <- function(text, terms) {
  normalized_text <- str_to_lower(text)
  feature_columns <- terms |>
    map(\(keyword) {
      str_detect(
        normalized_text,
        sprintf("\\b%s\\w*\\b", keyword)
      )
    })

  matrix(
    unlist(feature_columns, use.names = FALSE),
    nrow = length(text),
    ncol = length(terms),
    dimnames = list(NULL, terms)
  )
}

candidate_features <- make_features(
  initial_labels$text,
  candidate_keywords
)
keywords <- candidate_keywords[
  colSums(candidate_features) > 0L
]
training_features <- candidate_features[
  ,
  keywords,
  drop = FALSE
]
is_training <- initial_labels$reference_label == "training"

smoothing <- 0.5
training_prior <- (
  sum(is_training) + smoothing
) / (
  length(is_training) + 2 * smoothing
)
training_term_count <- colSums(
  training_features[is_training, , drop = FALSE]
)
other_term_count <- colSums(
  training_features[!is_training, , drop = FALSE]
)
empirical_rate_training <- training_term_count /
  sum(is_training)
empirical_rate_other <- other_term_count /
  sum(!is_training)
term_rate_training <- (
  training_term_count + smoothing
) / (
  sum(is_training) + 2 * smoothing
)
term_rate_other <- (
  other_term_count + smoothing
) / (
  sum(!is_training) + 2 * smoothing
)
direction_is_preserved <- sign(
  term_rate_training - term_rate_other
) == sign(
  empirical_rate_training -
    empirical_rate_other
)

score_training <- function(text) {
  features <- make_features(text, keywords)
  log_odds <- log(
    training_prior / (1 - training_prior)
  ) +
    rowSums(
      sweep(
        features,
        2,
        log(term_rate_training / term_rate_other),
        "*"
      ) +
        sweep(
          !features,
          2,
          log(
            (1 - term_rate_training) /
              (1 - term_rate_other)
          ),
          "*"
        )
    )

  stats::plogis(log_odds)
}

training_scores <- score_training(annotation_pool$text)

knitr::kable(
  tibble(
    term = keywords,
    training_count = as.integer(training_term_count),
    other_count = as.integer(other_term_count),
    smoothed_training_rate = round(
      term_rate_training,
      3
    ),
    smoothed_other_rate = round(
      term_rate_other,
      3
    )
  ),
  col.names = c(
    "Term",
    "Training rows",
    "Other rows",
    "Smoothed training rate",
    "Smoothed other rate"
  ),
  caption = "Term evidence from the 18 labeled sentences",
  row.names = FALSE
)
Term evidence from the 18 labeled sentences
Term Training rows Other rows Smoothed training rate Smoothed other rate
training 3 0 0.583 0.036
certificate 0 1 0.083 0.107
apprenticeship 1 0 0.250 0.036
mentor 1 0 0.250 0.036
required 0 4 0.083 0.321
preferred 0 2 0.083 0.179
license 0 1 0.083 0.107
diploma 0 1 0.083 0.107
portfolio 0 1 0.083 0.107
shift 0 1 0.083 0.107
schedule 0 2 0.083 0.179
skill 0 1 0.083 0.107
spreadsheet 0 1 0.083 0.107
experience 0 2 0.083 0.179

The half-count smoothing keeps probabilities away from zero and one. The assertion also requires each smoothed difference to point in the same direction as the observed difference. A term never seen in a training row cannot become positive evidence for training merely because that class is smaller.

The score used next is the model’s estimate under this small dataset and model. It is not a validated probability that the label is correct.

Select the closest scores

Uncertainty sampling often selects scores closest to 0.5 in a binary task. The model is least decisive there. We calculate each score’s distance from 0.5 and choose the two smallest distances. The model vocabulary contains only terms observed in the labeled documents. Words found only in the pool cannot acquire artificial evidence through smoothing.

selection_table <- tibble(
  sentence_id = annotation_pool$sentence_id,
  text = annotation_pool$text,
  training_score = training_scores,
  distance_from_half = abs(
    training_scores - 0.5
  )
) |>
  arrange(distance_from_half, sentence_id)
selected_for_review <- selection_table |>
  slice_head(n = 2)

knitr::kable(
  selection_table |>
    mutate(
      training_score = round(training_score, 3),
      distance_from_half = round(
        distance_from_half,
        3
      )
    ),
  col.names = c(
    "Sentence ID",
    "Text",
    "Training score",
    "Distance from 0.5"
  ),
  caption = "Annotation candidates ordered by model uncertainty",
  row.names = FALSE
)
Annotation candidates ordered by model uncertainty
Sentence ID Text Training score Distance from 0.5
s026 Evening classes 0.194 0.306
s028 Apply by October 15 0.194 0.306
s022 Clear written communication is an essential skill. 0.154 0.346
s023 RIVERTON SKILLS OPEN HOUSE 0.154 0.346
s024 DATA SUPPORT CERTIFICATE 0.154 0.346
s021 Product training is included. 0.901 0.401
s025 Paid training stipend 0.901 0.401
s020 Weekend shifts are required. 0.034 0.466
s019 Two years of customer service experience are required. 0.019 0.481
s027 No prior experience required 0.019 0.481

The first two rows are the next annotation candidates. Neither contains a feature seen in the labeled set, so both receive the same score. This is a useful failure of uncertainty-only selection: a score can reflect missing vocabulary rather than a subtle boundary between labels.

People still assign the labels

Only after selection do we join the held-back reference labels. This simulates the annotator returning completed work.

reviewed_selection <- selected_for_review |>
  select(sentence_id, text, training_score) |>
  left_join(
    held_back_labels,
    by = join_by(sentence_id),
    multiple = "error"
  )

newly_labeled <- selected_for_review |>
  select(sentence_id) |>
  left_join(
    annotation_pool,
    by = join_by(sentence_id),
    multiple = "error"
  ) |>
  left_join(
    held_back_labels,
    by = join_by(sentence_id),
    multiple = "error"
  )
updated_labels <- bind_rows(
  initial_labels,
  newly_labeled
)

knitr::kable(
  reviewed_selection,
  col.names = c(
    "Sentence ID",
    "Text",
    "Training score",
    "Human-applied label"
  ),
  caption = "The two selected sentences after annotation",
  digits = 3,
  row.names = FALSE
)
The two selected sentences after annotation
Sentence ID Text Training score Human-applied label
s026 Evening classes 0.194 schedule
s028 Apply by October 15 0.194 other

The two sentences receive schedule and other. Human review adds distinctions that the binary training detector cannot express. The tie shows why a practical selection policy may combine uncertainty with diversity and explicit coverage goals.

Uncertainty is only one selection rule

A model can be confidently wrong. Selecting only uncertain examples may also overlook:

  • communities or languages absent from the initial labels;
  • rare cases the model scores with false confidence;
  • duplicate examples that consume the annotation budget;
  • important harms that occur infrequently; and
  • examples far from the model’s training distribution.

A stronger plan combines uncertainty with random audit samples, diversity, coverage targets, and deliberate tests of known failure modes. Keep a document-disjoint evaluation set that active learning never uses for feature design, selection, or training.

Selection changes the estimate, and not in a fixed direction

One consequence is easy to state backwards. A set of examples chosen because the model found them hard is not a picture of the whole pool, so a number measured on it is not a pool-level number. The tempting next sentence, that the measured error therefore looks worse than the truth, is not a rule. It depends on what the rule selected.

The comparison below measures one quantity, the gap between the model’s score and the label a person applied, on three sets of items: the two the uncertainty rule chose, the two the model was most decisive about, and the whole pool.

pool_outcomes <- annotation_pool |>
  left_join(
    held_back_labels,
    by = join_by(sentence_id),
    multiple = "error"
  ) |>
  mutate(
    training_score = training_scores,
    is_training = as.integer(reference_label == "training"),
    score_gap = abs(training_score - is_training),
    predicted_training = training_score > 0.5
  )

most_decisive <- selection_table |>
  arrange(desc(distance_from_half), sentence_id) |>
  slice_head(n = 2)

gap_for <- function(ids) {
  mean(
    pool_outcomes$score_gap[pool_outcomes$sentence_id %in% ids]
  )
}

selection_comparison <- tibble(
  evaluated_on = c(
    "two items the uncertainty rule chose",
    "two items the model was most decisive about",
    "all ten pool items"
  ),
  items = c(2L, 2L, nrow(pool_outcomes)),
  mean_score_gap = c(
    gap_for(selected_for_review$sentence_id),
    gap_for(most_decisive$sentence_id),
    gap_for(pool_outcomes$sentence_id)
  )
)

pool_accuracy <- mean(
  pool_outcomes$predicted_training ==
    (pool_outcomes$is_training == 1L)
)

knitr::kable(
  selection_comparison |>
    mutate(mean_score_gap = round(mean_score_gap, 3)),
  col.names = c(
    "Measured on",
    "Items",
    "Mean gap between score and label"
  ),
  caption = "One model, one pool, three ways of choosing what to measure",
  row.names = FALSE
)
One model, one pool, three ways of choosing what to measure
Measured on Items Mean gap between score and label
two items the uncertainty rule chose 2 0.194
two items the model was most decisive about 2 0.019
all ten pool items 10 0.112

The pool value is 0.112. Selecting by uncertainty raised the measured gap to 0.194. Selecting by confidence lowered it to 0.019. Same model, same ten items, same quantity, and the estimate moved in opposite directions because the selection rule differed.

Neither direction is a law. A rule that finds hard cases usually flatters the data and punishes the score; a rule that finds easy cases does the reverse; a rule that happens to select a class the model handles well can make a weak model look strong. The more defensible statement is also more useful: any estimate computed on a selected subset describes that subset.

The remedy has not changed. Keep a fixed evaluation set drawn at random, disjoint by document from anything active learning touches, and report pool-level numbers only from it. In this exercise the classifier also happens to place all ten pool items on the correct side of 0.5, which is worth stating plainly and worth trusting very little: ten items, a vocabulary built from the labeled documents, and no independent test set.

What to remember

  • Active learning chooses what people label next; it does not replace them.
  • Hide pool labels until after selection.
  • A score near 0.5 is uncertain only under the current model.
  • Confidence is not the same as correctness or social importance.
  • A number measured on selected items can land above or below the pool value.
  • Audit random, rare, and underrepresented cases alongside uncertainty.

The selection trail accounts for two new reviews, but it does not broaden what the fictional notices cover. Selection cannot repair source coverage. The Lab therefore inspects three outside resources and asks whether any can answer the same questions before their records are combined.

Sources