Creating additional training examples

Augment text without assuming the label survived

training data
augmentation
workforce research
Learn how to create workforce text variants and test whether they still support the intended label.

The reviewer comparison left two boundary cases visible rather than treating a majority as truth. A different risk appears when the Riverton Workforce Lab creates synthetic examples: the row count can grow without adding observed evidence. The team can write additional sentences that follow known patterns. More rows may help a model learn, but a changed sentence can lose the meaning that justified its label. For this exercise, the team writes five fictional variants about training and schedules. They cannot tell a worker that a real course meets at a given time or provides paid training.

Textual data augmentation creates new training examples by transforming or generating text from existing examples. Augmentation changes a dataset; it does not collect new evidence about the labor market.

Note

All sentences and organizations in this lesson are fictional. Augmented rows are synthetic and remain marked as such.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • distinguish a new text variant from a new observation;
  • create controlled template variants;
  • use cmapr job titles without importing unsupported claims;
  • prepare a classification-check payload with huggingfaceR;
  • reject candidates whose label no longer holds; and
  • test duplication, label balance, and whether augmentation changed anything.

Start with one purpose

The team wants more examples of explicit training offers. It will not claim that the examples describe real employers or estimate how often training is offered. The next chunk loads package helpers for files, tables, strings, CMap examples, and model-check payloads, then reads the file with declared column types into a tibble.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(cmapr)
library(huggingfaceR)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)
training_examples <- workforce_sentences |>
  filter(reference_label == "training")

knitr::kable(
  training_examples |>
    select(sentence_id, text),
  col.names = c("Sentence ID", "Reviewed training example"),
  caption = "The seven training examples before augmentation",
  row.names = FALSE
)
The seven training examples before augmentation
Sentence ID Reviewed training example
s001 Paid 12-week training is provided.
s006 The employer pays for certification training.
s012 Each new hire receives a mentor.
s015 On-the-job training is provided.
s016 This is a paid apprenticeship.
s021 Product training is included.
s025 Paid training stipend

The source labels were assigned under the manual annotation codebook. The augmentation step must preserve that label definition.

Use job titles as template values

cmapr includes a ten-row teaching fixture with job-title transitions. We use three titles as vocabulary only. We do not carry over transition weights or interpret a title change as a promotion.

cmap_titles <- read_csv(
  cmap_example_data(),
  col_types = cols(
    sector = col_character(),
    region = col_character(),
    job_title_from = col_character(),
    job_title_to = col_character(),
    transition_weighted_count = col_double(),
    si_from = col_double(),
    si_to = col_double()
  )
)
selected_titles <- cmap_titles$job_title_from[c(3L, 5L, 8L)]

selected_titles
[1] "Data Analyst"      "Medical Assistant" "Junior Developer"

The titles come from the cmapr package’s teaching fixture. That file does not contain upstream record identifiers, so this lesson does not claim that the three strings are authenticated extracts from the full CMap dataset. The sentences created next are fictional.

Create controlled variants

The template states the same kind of claim for each title.

source("R/workforce-codebook.R")
augmentation_codebook <- workforce_codebook()

template_variants <- tibble(
  source_sentence_id = "s001",
  source_family_id = "training-offer-s001",
  template_id = "training-pathway-v1",
  vocabulary_source = "cmapr example_transitions.csv",
  vocabulary_row = c(3L, 5L, 8L),
  vocabulary_commit =
    "eb2d0b3810e07782aab4a81fba1f539133f2ba6a",
  text = sprintf(
    "The %s pathway includes paid training.",
    str_to_lower(selected_titles)
  ),
  label = "training",
  synthetic = TRUE,
  method = "controlled template",
  codebook_version = augmentation_codebook$version,
  codebook_hash = augmentation_codebook$hash,
  review_status = "requires_human_review",
  split = "training_candidate"
)

schedule_variants <- tibble(
  source_sentence_id = "s003",
  source_family_id = "schedule-s003",
  template_id = "class-schedule-v1",
  vocabulary_source = "author-created template",
  vocabulary_row = NA_integer_,
  vocabulary_commit = NA_character_,
  text = sprintf(
    "%s schedules are available.",
    c("Morning", "Weekend")
  ),
  label = "schedule",
  synthetic = TRUE,
  method = "controlled template",
  codebook_version = augmentation_codebook$version,
  codebook_hash = augmentation_codebook$hash,
  review_status = "requires_human_review",
  split = "training_candidate"
)

augmented_rows <- bind_rows(
  template_variants,
  schedule_variants
)
augmentation_label_order <- c(
  "training",
  "requirement",
  "schedule",
  "skill",
  "other"
)
augmentation_counts <- tibble(label = augmentation_label_order) |>
  left_join(
    augmented_rows |>
      count(label, name = "n"),
    by = join_by(label)
  ) |>
  mutate(n = coalesce(n, 0L))
source_families <- augmented_rows |>
  distinct(source_family_id) |>
  arrange(source_family_id)

knitr::kable(
  augmented_rows,
  col.names = c(
    "Source sentence",
    "Source family",
    "Template",
    "Vocabulary source",
    "Vocabulary row",
    "Vocabulary commit",
    "Synthetic text",
    "Proposed label",
    "Synthetic",
    "Method",
    "Codebook version",
    "Codebook hash",
    "Review status",
    "Intended split"
  ),
  caption = "Five controlled augmentation candidates",
  row.names = FALSE
)
Five controlled augmentation candidates
Source sentence Source family Template Vocabulary source Vocabulary row Vocabulary commit Synthetic text Proposed label Synthetic Method Codebook version Codebook hash Review status Intended split
s001 training-offer-s001 training-pathway-v1 cmapr example_transitions.csv 3 eb2d0b3810e07782aab4a81fba1f539133f2ba6a The data analyst pathway includes paid training. training TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s001 training-offer-s001 training-pathway-v1 cmapr example_transitions.csv 5 eb2d0b3810e07782aab4a81fba1f539133f2ba6a The medical assistant pathway includes paid training. training TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s001 training-offer-s001 training-pathway-v1 cmapr example_transitions.csv 8 eb2d0b3810e07782aab4a81fba1f539133f2ba6a The junior developer pathway includes paid training. training TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s003 schedule-s003 class-schedule-v1 author-created template NA NA Morning schedules are available. schedule TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s003 schedule-s003 class-schedule-v1 author-created template NA NA Weekend schedules are available. schedule TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate

The rows carry their source family, template, second vocabulary source, package commit when applicable, codebook, synthetic status, intended split, and pending review status. Any evaluation example derived from s001 or s003 must stay outside the evaluation set with its whole family. The five rows remain candidates until a person checks that each is natural, useful, and consistent with the codebook.

Review the candidates and reject some

A generator that produces only usable output is a generator nobody checked. The Lab added two more candidates that a template would happily produce, then sent all seven to a reviewer.

risky_variants <- tibble(
  source_sentence_id = c("s001", "s015"),
  source_family_id = c(
    "training-offer-s001",
    "training-offer-s015"
  ),
  template_id = c(
    "training-pathway-v1",
    "training-restatement-v1"
  ),
  vocabulary_source = c(
    "cmapr example_transitions.csv",
    "author-created template"
  ),
  vocabulary_row = c(6L, NA_integer_),
  vocabulary_commit = c(
    "eb2d0b3810e07782aab4a81fba1f539133f2ba6a",
    NA_character_
  ),
  text = c(
    "The financial analyst pathway may include paid training.",
    "On-the-job training is provided."
  ),
  label = "training",
  synthetic = TRUE,
  method = "controlled template",
  codebook_version = augmentation_codebook$version,
  codebook_hash = augmentation_codebook$hash,
  review_status = "requires_human_review",
  split = "training_candidate"
)

candidate_rows <- bind_rows(augmented_rows, risky_variants)

review_decisions <- tibble(
  text = candidate_rows$text,
  decision = c(
    rep("accept", 5L),
    "reject",
    "reject"
  ),
  reason = c(
    rep("States an offer without hedging; label holds", 3L),
    rep("States when schedules are available; label holds", 2L),
    "\"may include\" removes the explicit offer the label requires",
    "Repeats a reviewed sentence word for word"
  )
)

reviewed_candidates <- candidate_rows |>
  left_join(review_decisions, by = join_by(text)) |>
  mutate(
    duplicates_reviewed_text = text %in%
      workforce_sentences$text,
    hedged = str_detect(text, "\\b(may|might|could)\\b"),
    review_status = case_when(
      decision == "accept" ~ "accepted",
      decision == "reject" ~ "rejected",
      TRUE ~ review_status
    )
  )

accepted_rows <- reviewed_candidates |>
  filter(decision == "accept")
rejected_rows <- reviewed_candidates |>
  filter(decision == "reject")

knitr::kable(
  reviewed_candidates |>
    select(text, label, decision, reason),
  col.names = c(
    "Candidate text",
    "Proposed label",
    "Decision",
    "Reason"
  ),
  caption = "Seven candidates, five accepted, two rejected",
  row.names = FALSE
)
Seven candidates, five accepted, two rejected
Candidate text Proposed label Decision Reason
The data analyst pathway includes paid training. training accept States an offer without hedging; label holds
The medical assistant pathway includes paid training. training accept States an offer without hedging; label holds
The junior developer pathway includes paid training. training accept States an offer without hedging; label holds
Morning schedules are available. schedule accept States when schedules are available; label holds
Weekend schedules are available. schedule accept States when schedules are available; label holds
The financial analyst pathway may include paid training. training reject “may include” removes the explicit offer the label requires
On-the-job training is provided. training reject Repeats a reviewed sentence word for word

Both rejections are the kind a template cannot catch on its own.

The hedged sentence is grammatical, plausible, and wrong for the label it carries. Under the codebook, training requires an explicit offer, and “may include” states a possibility. Training a model on it teaches that hedged language marks a training offer, which is close to the opposite of the rule the annotators applied.

The repeated sentence is worse in a quieter way. It matches a reviewed row word for word, so it adds no information, and if its copy landed in training while the original sat in evaluation, the evaluation would measure recall of a string the model had already seen.

Two rejections out of seven is a rejection rate, not a defect. A generator with no rejections usually means the review was skipped.

Ask whether the accepted rows changed anything

The five accepted rows are safe. Safe is not the same as useful, and the way to find out is to run the downstream task twice.

The task below is deliberately crude: decide whether a sentence is a training offer, using terms that appear in training examples and never in the others. It learns from the five complete postings J001 through J005, and it is measured on the ten units from J006 and the flyer, which no augmented row was derived from.

labeled_pool <- workforce_sentences |>
  filter(document_id %in% sprintf("J%03d", 1:5)) |>
  select(text, label = reference_label)
held_out <- workforce_sentences |>
  filter(document_id %in% c("J006", "F001")) |>
  select(sentence_id, text, label = reference_label)
accepted_source_ids <- unique(accepted_rows$source_sentence_id)

to_words <- function(text) {
  text |>
    str_to_lower() |>
    str_replace_all("[^a-z0-9 ]", " ") |>
    str_squish() |>
    str_split(" ")
}

distinctive_terms <- function(examples) {
  training_words <- unique(unlist(
    to_words(examples$text[examples$label == "training"])
  ))
  other_words <- unique(unlist(
    to_words(examples$text[examples$label != "training"])
  ))

  sort(setdiff(
    training_words[str_length(training_words) > 2],
    other_words
  ))
}

predict_training <- function(terms, text) {
  map_lgl(
    to_words(text),
    \(row_words) any(row_words %in% terms)
  )
}

without_augmentation <- distinctive_terms(labeled_pool)
with_augmentation <- distinctive_terms(
  bind_rows(
    labeled_pool,
    accepted_rows |>
      select(text, label)
  )
)

is_training <- held_out$label == "training"
ablation <- tibble(
  condition = c("without augmentation", "with augmentation"),
  distinctive_terms = c(
    length(without_augmentation),
    length(with_augmentation)
  ),
  correct = c(
    sum(predict_training(without_augmentation, held_out$text) == is_training),
    sum(predict_training(with_augmentation, held_out$text) == is_training)
  ),
  items = nrow(held_out)
)

new_terms <- setdiff(with_augmentation, without_augmentation)

knitr::kable(
  ablation,
  col.names = c(
    "Condition",
    "Distinctive terms",
    "Correct decisions",
    "Items judged"
  ),
  caption = "The same held-out units judged with and without the augmented rows",
  row.names = FALSE
)
The same held-out units judged with and without the augmented rows
Condition Distinctive terms Correct decisions Items judged
without augmentation 13 10 10
with augmentation 18 10 10 
new_terms
[1] "analyst"   "assistant" "developer" "junior"    "pathway"

The augmented rows added five terms to the vocabulary and changed nothing. Every held-out decision is identical under both conditions, and both get all ten right.

The five new terms explain why. Four of them are job titles and one is the word pathway, all of which arrived from the template rather than from any new observation about how employers describe training. The word doing the work, training, was already there.

The honest reading of this comparison is modest in both directions. It does not show that augmentation is useless; ten items with two positive cases could not detect a small improvement, and a stronger model with more data might use the extra rows differently. It does show that these five rows earned nothing here, which is exactly the finding a report should carry when a training pool grows by more than a quarter and the result does not move.

Prepare a model check without claiming a result

huggingfaceR can prepare the payload for a zero-shot classification request. This code records the candidate labels and texts, but it does not call a model or return predictions.

label_check_payload <-
  hf_zero_shot_classification_payload(
    augmented_rows$text,
    candidate_labels = c(
      "training",
      "requirement",
      "schedule",
      "skill",
      "other"
    ),
    multi_label = FALSE
  )

str(label_check_payload)
List of 2
 $ inputs    : chr [1:5] "The data analyst pathway includes paid training." "The medical assistant pathway includes paid training." "The junior developer pathway includes paid training." "Morning schedules are available." ...
 $ parameters:List of 2
  ..$ candidate_labels: chr [1:5] "training" "requirement" "schedule" "skill" ...
  ..$ multi_label     : logi FALSE

A future model call could flag surprising candidates for review. A model label would not prove that the augmentation is valid. The same codebook and human review still apply. An actual request would also require a named model and provider and would transmit the text under that provider’s terms. No model or provider is selected and no network request occurs here.

A small edit can reverse the claim

Word replacement is risky when it changes negation or obligation. str_to_lower() makes the keyword checks ignore case. str_detect() asks whether each pattern is present.

meaning_check <- tibble(
  version = c("source", "unsafe edit"),
  text = c(
    "Paid training is provided.",
    "Paid training is not provided."
  )
) |>
  mutate(
    lower_text = str_to_lower(text),
    keyword_only_match =
      str_detect(lower_text, fixed("paid")) &
        str_detect(lower_text, fixed("training")) &
        str_detect(lower_text, fixed("provided")),
    negation_aware_offer =
      keyword_only_match &
        !str_detect(lower_text, "\\bnot\\b")
  ) |>
  select(-lower_text)

knitr::kable(
  meaning_check,
  col.names = c(
    "Version",
    "Text",
    "Keyword-only match",
    "Negation-aware offer rule"
  ),
  caption = "One added word reverses the training offer",
  row.names = FALSE
)
One added word reverses the training offer
Version Text Keyword-only match Negation-aware offer rule
source Paid training is provided. TRUE TRUE
unsafe edit Paid training is not provided. TRUE FALSE

Both sentences satisfy the keyword-only check. The small negation rule separates this pair, but it is not a general language-understanding method. Label preservation still requires review at the meaning level.

Augmentation can amplify a mistake

Check:

  1. whether the label and important details remain true;
  2. whether the wording is grammatical in the intended language;
  3. whether duplicates cross into the evaluation set;
  4. whether templates make one class unnaturally easy to recognize;
  5. whether generated text adds stereotypes or private information;
  6. whether one language or job family receives most synthetic rows; and
  7. how conclusions change when synthetic examples are removed.

Large language models can generate many variants, but scale does not settle these questions. Repeated training on generated text can also make a model’s representation of the underlying distribution less varied.

What to remember

  • Augmented text is synthetic data, not a newly observed case.
  • Record the source, method, label, and synthetic status.
  • A familiar word does not guarantee that meaning stayed the same.
  • Review every candidate and keep the rejected ones with their reasons.
  • Keep augmented rows out of evaluation data derived from their source.
  • Compare results with and without synthetic examples, and report a flat result as readily as a favorable one.

The Lab ends this stage with five accepted rows, two rejected ones, and a comparison that found no downstream difference. Their synthetic status and method remain attached. Keyword rules are the final shortcut. They face both the designed data and cases with negation or misleading word matches.

Sources