Turning reading decisions into labels

Create a manual annotation codebook for workforce text

training data
annotation
workforce research
Learn how people label workforce text units without pretending that judgment is automatic or neutral.

The archived job page and checked flyer transcript leave the fictional Riverton Workforce Lab with 28 text units. To make them usable, the team must separate training offers, application requirements, schedules, named skills, and text that fits none of those categories. These distinctions affect which information a worker would find while comparing notices, but the notices do not label themselves.

A computer cannot learn these categories until people define them. Manual annotation is the work of reading an item and assigning a label according to written instructions. The hard part is not typing the label. It is deciding what the label means and applying that meaning consistently. The task records what a text unit explicitly says; it does not decide whether a worker qualifies or whether a program is a good option.

Note

The workforce text, labels, and research team are fictional. The examples were created to expose ordinary annotation choices without using worker or applicant records.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • define an annotation unit and label set;
  • write rules for what each label includes and excludes;
  • create a versioned codebook with foundryR;
  • run a pilot and let disagreement change the rules;
  • inspect ambiguous cases rather than hiding them; and
  • explain why a human label is a documented judgment, not automatic truth.

Decide what one label describes

The annotation unit is one line of source text: a single job-detail list item from a posting, or a single line from the flyer. That keeps each decision small enough to check, while the document_id preserves the job posting or flyer that supplied the text.

Calling those lines sentences would be convenient and wrong. Two of them are headings in capital letters, three are noun phrases with no verb, and one is an imperative clause without terminal punctuation. The distinction matters because a rule written for sentences, such as “label the main claim of the text unit,” has to say what it does with a heading that makes no claim at all.

The next chunk loads readr for files, dplyr and tibble for tables, purrr for repeated checks, and stringr for text checks, then reads the CSV into a tibble.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

allowed_labels <- c(
  "training",
  "requirement",
  "schedule",
  "skill",
  "other"
)

first_units <- workforce_sentences |>
  select(
    sentence_id,
    document_id,
    text,
    reference_label,
    uncertainty
  ) |>
  slice_head(n = 8)
The first eight manually labeled text units
Unit ID Document ID Text Reference label Uncertainty
s001 J001 Paid 12-week training is provided. training certain
s002 J001 No prior data experience is required. requirement certain
s003 J001 Evening schedules are available. schedule certain
s004 J001 Applicants need basic spreadsheet skills. skill certain
s005 J002 A high school diploma is required. requirement certain
s006 J002 The employer pays for certification training. training certain
s007 J002 Rotating night shifts are part of the job. schedule certain
s008 J002 Workers must be able to lift 50 pounds. requirement certain

The file calls the reviewed result a reference label, not a ground-truth label. The label records how this codebook was applied. Another defensible codebook could divide the concepts differently.

Count what the units actually are

The claim that these are not all sentences is checkable, but punctuation alone cannot classify grammar. Apply by October 15 is an imperative clause even without a full stop. The six flyer lines are therefore classified explicitly rather than inferred from their last character.

unit_shapes <- workforce_sentences |>
  mutate(
    shape = case_when(
      sentence_id %in% c("s023", "s024") ~ "heading in capitals",
      sentence_id %in% c("s025", "s026", "s027") ~ "phrase without a verb",
      sentence_id == "s028" ~ "imperative clause",
      TRUE ~ "complete sentence"
    )
  )

shape_counts <- unit_shapes |>
  count(shape, name = "units")
Twenty-two sentences, two headings, three phrases, one imperative clause
Shape of the unit Units
complete sentence 22
heading in capitals 2
imperative clause 1
phrase without a verb 3

Five of the 28 units are not sentences. A sixth, Apply by October 15, is an imperative clause whose flyer typography omits terminal punctuation. All six come from a notice laid out for a reader walking past a noticeboard. Paid training stipend has no verb; RIVERTON SKILLS OPEN HOUSE announces an event.

Both still receive a label, and the codebook has to say how. That is the reason the rules below talk about the unit’s main explicit claim rather than the sentence’s subject.

The six flyer rows match its six-line checked transcript. No wording has been expanded or interpreted in the annotation file.

flyer_rows <- workforce_sentences |>
  filter(document_id == "F001")
flyer_transcript <- read_lines(
  "data/workforce/training-flyer-ground-truth.txt"
)
Row-level lineage from the flyer transcript
Unit ID Flyer line Verbatim text Reference label
s023 1 RIVERTON SKILLS OPEN HOUSE other
s024 2 DATA SUPPORT CERTIFICATE other
s025 3 Paid training stipend training
s026 4 Evening classes schedule
s027 5 No prior experience required requirement
s028 6 Apply by October 15 other

Write the codebook before scaling up

A codebook records the label definitions, instructions, examples, and version. The complete instrument lives in R/workforce-codebook.R. Its rules are:

  • training: the text unit offers or describes learning, mentoring, certification training, or an apprenticeship;
  • requirement: the text unit states or prefers experience, education, credentials, a license, a portfolio, or a physical condition;
  • schedule: the text unit describes when work or classes occur;
  • skill: the text unit names an ability used in the work; and
  • other: none of the four definitions applies; headings, deadlines, location, work setting, and travel belong here unless the text unit explicitly states another category.

The text unit’s main explicit claim receives one label. Reviewers do not infer a training offer from a job title or assume that a preferred credential is mandatory. A required shift receives schedule because timing is the main claim. Reviewers mark needs_review when more than one label remains defensible.

foundryR can store the codebook as a versioned object with a strict label schema and a content hash.

source("R/workforce-codebook.R")
annotation_codebook <- workforce_codebook()
recorded_codebook <- workforce_codebook("1.0.0")

annotation_codebook
foundry codebook: workforce-information
version: 1.1.0
hash: 7f4213ccadd8
variables:
  - label: string [training, requirement, schedule, skill, other] (Primary workforce-information label)
  - rationale: string (Short explanation using only the text unit)
  - uncertainty: string [certain, needs_review] (Whether the item needs adjudication)
examples: 7

The printed hash changes when the instructions, schema, examples, or version changes. Version 1.1.0 corrects the unit name. The archived labels keep their original 1.0.0 fingerprint because changing an instrument does not retroactively change which instrument produced earlier records. A new annotation run should use 1.1.0. The code above does not call an AI service; foundryR defines and fingerprints the measurement instrument.

Count labels without interpreting the count

A label distribution can expose a missing category or a data-entry problem. It cannot establish how common these ideas are across the real labor market.

label_counts <- workforce_sentences |>
  mutate(
    reference_label = factor(
      reference_label,
      levels = allowed_labels
    )
  ) |>
  count(reference_label, name = "units", .drop = FALSE) |>
  transmute(
    label = as.character(reference_label),
    units
  )
Label counts in the teaching dataset
Reference label Text units
training 7
requirement 9
schedule 5
skill 2
other 5

The counts describe this designed example only. They were not sampled from all jobs in Riverton or from any real population.

Keep ambiguity visible

Unit s013 mentions a certificate, a word that also appears in training offers. Its main claim is that a certificate is preferred, so version 1.1.0 of the codebook labels it as a requirement.

ambiguous_case <- workforce_sentences |>
  filter(sentence_id == "s013")
A text unit that needs the codebook rule
Unit ID Document ID Text Reference label Uncertainty Rationale
s013 J004 A medical records certificate is preferred. requirement needs_review Preference is the main claim despite the training-related noun

A reviewer who chose training may have followed a defensible reading rather than worked carelessly. The team should record the disagreement, discuss the rule, and issue a new codebook version if the definition changes. Overwriting the label would erase the evidence needed to improve the instrument.

Run a pilot before labeling everything

The rules above did not arrive fully formed. They came from a pilot: two reviewers labeled six units independently, without discussion, and the pairs that disagreed decided what the codebook still had to say.

pilot_round <- tibble(
  sentence_id = c("s001", "s004", "s010", "s013", "s020", "s024"),
  reviewer_p = c(
    "training",
    "skill",
    "requirement",
    "training",
    "requirement",
    "training"
  ),
  reviewer_q = c(
    "training",
    "skill",
    "other",
    "requirement",
    "schedule",
    "other"
  )
) |>
  left_join(
    workforce_sentences |>
      select(sentence_id, text, reference_label),
    by = join_by(sentence_id)
  ) |>
  mutate(agreed = reviewer_p == reviewer_q)

pilot_agreement <- mean(pilot_round$agreed)
The six-unit pilot, before any rule was written
Unit Text Reviewer P Reviewer Q Agreed
s001 Paid 12-week training is provided. training training TRUE
s004 Applicants need basic spreadsheet skills. skill skill TRUE
s010 Six months of experience is preferred. requirement other FALSE
s013 A medical records certificate is preferred. training requirement FALSE
s020 Weekend shifts are required. requirement schedule FALSE
s024 DATA SUPPORT CERTIFICATE training other FALSE

Two of six matched in this designed pilot. Six items cannot establish a typical first-pass agreement rate. Here, the four disagreements identify wording that the initial instructions did not settle.

Turn each disagreement into a rule

The Lab wrote one rule per disagreement, then adjudicated the four units under those rules. The adjudicator was a third reader who had not labeled the pilot.

adjudication_log <- tibble(
  sentence_id = c("s010", "s013", "s020", "s024"),
  question = c(
    "Is a preferred condition still an entry condition?",
    "Does naming a credential make a unit a training offer?",
    "Does a required shift describe timing or an entry condition?",
    "Does a heading claim what its words name?"
  ),
  rule_added = c(
    "A preference is still an entry condition.",
    "A credential that is required or preferred is not training unless learning is offered.",
    "Timing takes precedence when a shift is required.",
    "Headings are other unless the unit explicitly states another category."
  ),
  adjudicated_label = c(
    "requirement",
    "requirement",
    "schedule",
    "other"
  )
)

instructions <- annotation_codebook$instructions
rule_clauses <- c(
  "main claim states or prefers",
  "is not training",
  "schedule takes precedence",
  "Headings, deadlines, location, work setting, and travel are other"
)

adjudication_check <- adjudication_log |>
  left_join(
    workforce_sentences |>
      select(sentence_id, reference_label, uncertainty),
    by = join_by(sentence_id)
  ) |>
  mutate(
    matches_file = adjudicated_label == reference_label,
    clause_in_codebook = map_lgl(
      rule_clauses,
      \(clause) str_detect(instructions, fixed(clause))
    )
  )
Four pilot disagreements, four rules, four recorded decisions
Unit Question raised Rule written in response Decision Uncertainty kept
s010 Is a preferred condition still an entry condition? A preference is still an entry condition. requirement certain
s013 Does naming a credential make a unit a training offer? A credential that is required or preferred is not training unless learning is offered. requirement needs_review
s020 Does a required shift describe timing or an entry condition? Timing takes precedence when a shift is required. schedule needs_review
s024 Does a heading claim what its words name? Headings are other unless the unit explicitly states another category. other needs_review

Every rule in that table is present in the current instructions, and the check above reads them back out of the instrument rather than trusting the description. Every adjudicated decision matches the archived label. Version 1.0.0 remains on those rows; version 1.1.0 records the corrected unit wording for future work.

Two details stay visible. Three of the four disputed units retain needs_review; s010 is marked certain, so the file does not pretend every disagreement remained unresolved. Version 1.1.0 of the codebook now names one source-text line as the annotation unit. The historical sentence_id column remains a stable identifier, not a claim that every row is a grammatical sentence.

This pilot shows how disagreement can produce explicit rules. Its designed six items do not estimate how much a pilot would change a larger annotation job or how long that work would take.

Protect the people doing the work

Manual annotation can involve sensitive, repetitive, or distressing material. Before assigning work:

  1. explain the purpose and allowed use of the data;
  2. remove personal information that reviewers do not need;
  3. pay fairly for reading, training, and disagreement review;
  4. give reviewers a way to skip harmful material;
  5. separate quality feedback from surveillance; and
  6. record which languages and lived experience the task requires.

Expertise depends on the question. A worker, career counselor, employer, and labor researcher may interpret the same text unit differently. Those perspectives should be designed into the study rather than treated as noise.

When a language model reads the codebook too

A language model can act as one more annotator: it reads the codebook and proposes a label for each unit. The evidence is mixed. On some tasks models match or beat crowd workers; careful replications find them inconsistent from one dataset to the next. Models also follow a full codebook, with its definitions and edge cases, more reliably than a bare category name, which is one more reason to write the codebook first.

Treat a model’s labels the way this lesson treats any annotator’s. Record the codebook version, the prompt, and the model revision, compare a random sample with human decisions, and report where they disagree. The guide Using a language model in a reproducible analysis works through one example, including how to correct a count for the model’s errors.

What to remember

  • Annotation labels are measurements created from written rules.
  • Name the unit precisely; a heading and a phrase are not sentences.
  • One unit per row keeps the decision small and traceable.
  • A codebook needs a version, examples, and a change record.
  • Run a pilot and let its disagreements write the rules.
  • Ambiguity should lead to review, not an unrecorded correction.
  • Human labels require ethical working conditions and relevant expertise.
  • A language model used as an annotator needs the same codebook and checks.

The Lab has defined the categories and preserved its reading decisions. To test selection without exposing known answers, the active-learning exercise starts with five complete job postings and hides two other documents from a small model.

Sources