Skip to main content

Turning reading decisions into labels

Create a manual annotation codebook for workforce text

Learn how people label workforce text units without pretending that judgment is automatic or neutral.

The archived job page and checked flyer transcript leave the fictional Riverton Workforce Lab with 28 text units. To make them usable, the team must separate training offers, application requirements, schedules, named skills, and text that fits none of those categories. These distinctions affect which information a worker would find while comparing notices, but the notices do not label themselves.

A computer cannot learn these categories until people define them. Manual annotation is the work of reading an item and assigning a label according to written instructions. The hard part is not typing the label. It is deciding what the label means and applying that meaning consistently. The task records what a text unit explicitly says; it does not decide whether a worker qualifies or whether a program is a good option.

Note

The workforce text, labels, and research team are fictional. The examples were created to expose ordinary annotation choices without using worker or applicant records.

TipWhat you will learn

By the end of this lesson, you will be able to:

• define an annotation unit and label set;
• write rules for what each label includes and excludes;
• create a versioned codebook with foundryR;
• run a pilot and let disagreement change the rules;
• inspect ambiguous cases rather than hiding them; and
• explain why a human label is a documented judgment, not automatic truth.

Decide what one label describes

The annotation unit is one line of source text: a single job-detail list item from a posting, or a single line from the flyer. That keeps each decision small enough to check, while the document_id preserves the job posting or flyer that supplied the text.

Calling those lines sentences would be convenient and wrong. Two of them are headings in capital letters, three are noun phrases with no verb, and one is an imperative clause without terminal punctuation. The distinction matters because a rule written for sentences, such as “label the main claim of the text unit,” has to say what it does with a heading that makes no claim at all.

The next chunk loads readr for files, dplyr and tibble for tables, purrr for repeated checks, and stringr for text checks, then reads the CSV into a tibble.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

allowed_labels <- c(
  "training",
  "requirement",
  "schedule",
  "skill",
  "other"
)

first_units <- workforce_sentences |>
  select(
    sentence_id,
    document_id,
    text,
    reference_label,
    uncertainty
  ) |>
  slice_head(n = 8)

knitr::kable(
  first_units,
  col.names = c(
    "Unit ID",
    "Document ID",
    "Text",
    "Reference label",
    "Uncertainty"
  ),
  caption = "The first eight manually labeled text units",
  row.names = FALSE
)

The first eight manually labeled text units

Unit ID	Document ID	Text	Reference label	Uncertainty
s001	J001	Paid 12-week training is provided.	training	certain
s002	J001	No prior data experience is required.	requirement	certain
s003	J001	Evening schedules are available.	schedule	certain
s004	J001	Applicants need basic spreadsheet skills.	skill	certain
s005	J002	A high school diploma is required.	requirement	certain
s006	J002	The employer pays for certification training.	training	certain
s007	J002	Rotating night shifts are part of the job.	schedule	certain
s008	J002	Workers must be able to lift 50 pounds.	requirement	certain

The file calls the reviewed result a reference label, not a ground-truth label. The label records how this codebook was applied. Another defensible codebook could divide the concepts differently.

Count what the units actually are

The claim that these are not all sentences is checkable, but punctuation alone cannot classify grammar. Apply by October 15 is an imperative clause even without a full stop. The six flyer lines are therefore classified explicitly rather than inferred from their last character.

unit_shapes <- workforce_sentences |>
  mutate(
    shape = case_when(
      sentence_id %in% c("s023", "s024") ~ "heading in capitals",
      sentence_id %in% c("s025", "s026", "s027") ~ "phrase without a verb",
      sentence_id == "s028" ~ "imperative clause",
      TRUE ~ "complete sentence"
    )
  )

shape_counts <- unit_shapes |>
  count(shape, name = "units")

knitr::kable(
  shape_counts,
  col.names = c("Shape of the unit", "Units"),
  caption = "Twenty-two sentences, two headings, three phrases, one imperative clause",
  row.names = FALSE
)

Twenty-two sentences, two headings, three phrases, one imperative clause

Shape of the unit	Units
complete sentence	22
heading in capitals	2
imperative clause	1
phrase without a verb	3

Five of the 28 units are not sentences. A sixth, Apply by October 15, is an imperative clause whose flyer typography omits terminal punctuation. All six come from a notice laid out for a reader walking past a noticeboard. Paid training stipend has no verb; RIVERTON SKILLS OPEN HOUSE announces an event.

Both still receive a label, and the codebook has to say how. That is the reason the rules below talk about the unit’s main explicit claim rather than the sentence’s subject.

The six flyer rows match its six-line checked transcript. No wording has been expanded or interpreted in the annotation file.

flyer_rows <- workforce_sentences |>
  filter(document_id == "F001")
flyer_transcript <- read_lines(
  "data/workforce/training-flyer-ground-truth.txt"
)

knitr::kable(
  flyer_rows |>
    select(
      sentence_id,
      source_line,
      text,
      reference_label
    ),
  col.names = c(
    "Unit ID",
    "Flyer line",
    "Verbatim text",
    "Reference label"
  ),
  caption = "Row-level lineage from the flyer transcript",
  row.names = FALSE
)

Row-level lineage from the flyer transcript

Unit ID	Flyer line	Verbatim text	Reference label
s023	1	RIVERTON SKILLS OPEN HOUSE	other
s024	2	DATA SUPPORT CERTIFICATE	other
s025	3	Paid training stipend	training
s026	4	Evening classes	schedule
s027	5	No prior experience required	requirement
s028	6	Apply by October 15	other

Write the codebook before scaling up

A codebook records the label definitions, instructions, examples, and version. The complete instrument lives in R/workforce-codebook.R. Its rules are:

• training: the text unit offers or describes learning, mentoring, certification training, or an apprenticeship;
• requirement: the text unit states or prefers experience, education, credentials, a license, a portfolio, or a physical condition;
• schedule: the text unit describes when work or classes occur;
• skill: the text unit names an ability used in the work; and
• other: none of the four definitions applies; headings, deadlines, location, work setting, and travel belong here unless the text unit explicitly states another category.

The text unit’s main explicit claim receives one label. Reviewers do not infer a training offer from a job title or assume that a preferred credential is mandatory. A required shift receives schedule because timing is the main claim. Reviewers mark needs_review when more than one label remains defensible.

foundryR can store the codebook as a versioned object with a strict label schema and a content hash.

source("R/workforce-codebook.R")
annotation_codebook <- workforce_codebook()
recorded_codebook <- workforce_codebook("1.0.0")

annotation_codebook

foundry codebook: workforce-information
version: 1.1.0
hash: 7f4213ccadd8
variables:
  - label: string [training, requirement, schedule, skill, other] (Primary workforce-information label)
  - rationale: string (Short explanation using only the text unit)
  - uncertainty: string [certain, needs_review] (Whether the item needs adjudication)
examples: 7

The printed hash changes when the instructions, schema, examples, or version changes. Version 1.1.0 corrects the unit name. The archived labels keep their original 1.0.0 fingerprint because changing an instrument does not retroactively change which instrument produced earlier records. A new annotation run should use 1.1.0. The code above does not call an AI service; foundryR defines and fingerprints the measurement instrument.

Count labels without interpreting the count

A label distribution can expose a missing category or a data-entry problem. It cannot establish how common these ideas are across the real labor market.

label_counts <- workforce_sentences |>
  mutate(
    reference_label = factor(
      reference_label,
      levels = allowed_labels
    )
  ) |>
  count(reference_label, name = "units", .drop = FALSE) |>
  transmute(
    label = as.character(reference_label),
    units
  )

knitr::kable(
  label_counts,
  col.names = c("Reference label", "Text units"),
  caption = "Label counts in the teaching dataset",
  row.names = FALSE
)

Label counts in the teaching dataset

Reference label	Text units
training	7
requirement	9
schedule	5
skill	2
other	5

The counts describe this designed example only. They were not sampled from all jobs in Riverton or from any real population.

Keep ambiguity visible

Unit s013 mentions a certificate, a word that also appears in training offers. Its main claim is that a certificate is preferred, so version 1.1.0 of the codebook labels it as a requirement.

ambiguous_case <- workforce_sentences |>
  filter(sentence_id == "s013")

knitr::kable(
  ambiguous_case |>
    select(
      sentence_id,
      document_id,
      text,
      reference_label,
      uncertainty,
      rationale
    ),
  col.names = c(
    "Unit ID",
    "Document ID",
    "Text",
    "Reference label",
    "Uncertainty",
    "Rationale"
  ),
  caption = "A text unit that needs the codebook rule",
  row.names = FALSE
)

A text unit that needs the codebook rule

Unit ID	Document ID	Text	Reference label	Uncertainty	Rationale
s013	J004	A medical records certificate is preferred.	requirement	needs_review	Preference is the main claim despite the training-related noun

A reviewer who chose training may have followed a defensible reading rather than worked carelessly. The team should record the disagreement, discuss the rule, and issue a new codebook version if the definition changes. Overwriting the label would erase the evidence needed to improve the instrument.

Run a pilot before labeling everything

The rules above did not arrive fully formed. They came from a pilot: two reviewers labeled six units independently, without discussion, and the pairs that disagreed decided what the codebook still had to say.

pilot_round <- tibble(
  sentence_id = c("s001", "s004", "s010", "s013", "s020", "s024"),
  reviewer_p = c(
    "training",
    "skill",
    "requirement",
    "training",
    "requirement",
    "training"
  ),
  reviewer_q = c(
    "training",
    "skill",
    "other",
    "requirement",
    "schedule",
    "other"
  )
) |>
  left_join(
    workforce_sentences |>
      select(sentence_id, text, reference_label),
    by = join_by(sentence_id)
  ) |>
  mutate(agreed = reviewer_p == reviewer_q)

pilot_agreement <- mean(pilot_round$agreed)

knitr::kable(
  pilot_round |>
    select(sentence_id, text, reviewer_p, reviewer_q, agreed),
  col.names = c(
    "Unit",
    "Text",
    "Reviewer P",
    "Reviewer Q",
    "Agreed"
  ),
  caption = "The six-unit pilot, before any rule was written",
  row.names = FALSE
)

The six-unit pilot, before any rule was written

Unit	Text	Reviewer P	Reviewer Q	Agreed
s001	Paid 12-week training is provided.	training	training	TRUE
s004	Applicants need basic spreadsheet skills.	skill	skill	TRUE
s010	Six months of experience is preferred.	requirement	other	FALSE
s013	A medical records certificate is preferred.	training	requirement	FALSE
s020	Weekend shifts are required.	requirement	schedule	FALSE
s024	DATA SUPPORT CERTIFICATE	training	other	FALSE

Two of six matched in this designed pilot. Six items cannot establish a typical first-pass agreement rate. Here, the four disagreements identify wording that the initial instructions did not settle.

Turn each disagreement into a rule

The Lab wrote one rule per disagreement, then adjudicated the four units under those rules. The adjudicator was a third reader who had not labeled the pilot.

adjudication_log <- tibble(
  sentence_id = c("s010", "s013", "s020", "s024"),
  question = c(
    "Is a preferred condition still an entry condition?",
    "Does naming a credential make a unit a training offer?",
    "Does a required shift describe timing or an entry condition?",
    "Does a heading claim what its words name?"
  ),
  rule_added = c(
    "A preference is still an entry condition.",
    "A credential that is required or preferred is not training unless learning is offered.",
    "Timing takes precedence when a shift is required.",
    "Headings are other unless the unit explicitly states another category."
  ),
  adjudicated_label = c(
    "requirement",
    "requirement",
    "schedule",
    "other"
  )
)

instructions <- annotation_codebook$instructions
rule_clauses <- c(
  "main claim states or prefers",
  "is not training",
  "schedule takes precedence",
  "Headings, deadlines, location, work setting, and travel are other"
)

adjudication_check <- adjudication_log |>
  left_join(
    workforce_sentences |>
      select(sentence_id, reference_label, uncertainty),
    by = join_by(sentence_id)
  ) |>
  mutate(
    matches_file = adjudicated_label == reference_label,
    clause_in_codebook = map_lgl(
      rule_clauses,
      \(clause) str_detect(instructions, fixed(clause))
    )
  )

knitr::kable(
  adjudication_check |>
    select(sentence_id, question, rule_added, adjudicated_label, uncertainty),
  col.names = c(
    "Unit",
    "Question raised",
    "Rule written in response",
    "Decision",
    "Uncertainty kept"
  ),
  caption = "Four pilot disagreements, four rules, four recorded decisions",
  row.names = FALSE
)

Four pilot disagreements, four rules, four recorded decisions

Unit	Question raised	Rule written in response	Decision	Uncertainty kept
s010	Is a preferred condition still an entry condition?	A preference is still an entry condition.	requirement	certain
s013	Does naming a credential make a unit a training offer?	A credential that is required or preferred is not training unless learning is offered.	requirement	needs_review
s020	Does a required shift describe timing or an entry condition?	Timing takes precedence when a shift is required.	schedule	needs_review
s024	Does a heading claim what its words name?	Headings are other unless the unit explicitly states another category.	other	needs_review

Every rule in that table is present in the current instructions, and the check above reads them back out of the instrument rather than trusting the description. Every adjudicated decision matches the archived label. Version 1.0.0 remains on those rows; version 1.1.0 records the corrected unit wording for future work.

Two details stay visible. Three of the four disputed units retain needs_review; s010 is marked certain, so the file does not pretend every disagreement remained unresolved. Version 1.1.0 of the codebook now names one source-text line as the annotation unit. The historical sentence_id column remains a stable identifier, not a claim that every row is a grammatical sentence.

This pilot shows how disagreement can produce explicit rules. Its designed six items do not estimate how much a pilot would change a larger annotation job or how long that work would take.

Protect the people doing the work

Manual annotation can involve sensitive, repetitive, or distressing material. Before assigning work:

1. explain the purpose and allowed use of the data;
2. remove personal information that reviewers do not need;
3. pay fairly for reading, training, and disagreement review;
4. give reviewers a way to skip harmful material;
5. separate quality feedback from surveillance; and
6. record which languages and lived experience the task requires.

Expertise depends on the question. A worker, career counselor, employer, and labor researcher may interpret the same text unit differently. Those perspectives should be designed into the study rather than treated as noise.

What to remember

• Annotation labels are measurements created from written rules.
• Name the unit precisely; a heading and a phrase are not sentences.
• One unit per row keeps the decision small and traceable.
• A codebook needs a version, examples, and a change record.
• Run a pilot and let its disagreements write the rules.
• Ambiguity should lead to review, not an unrecorded correction.
• Human labels require ethical working conditions and relevant expertise.

The Lab has defined the categories and preserved its reading decisions. To test selection without exposing known answers, the active-learning exercise starts with five complete job postings and hides two other documents from a small model.

Sources

• Data Statements for Natural Language Processing
• Datasheets for Datasets
• Data Collection and Quality Challenges in Deep Learning
• foundryR codebook documentation