Labeling text with written rules

Measure rule coverage, conflicts, and mistakes

training data

weak supervision

workforce research

Learn how rule-based weak labels can expand workforce training data without becoming ground truth.

Five fictional sentence variants remain marked as synthetic. For the rule test, the Riverton Workforce Lab returns to its 28 reviewed fictional text units. Several phrases appear repeatedly: training, required, shift, and skill. The team turns those patterns into rules, then checks whether familiar words preserve distinctions a worker may use when comparing notices.

Rule-based training data uses written conditions to assign labels. These proposed labels are often called weak labels because a rule is an imperfect source of supervision. The team will measure where its rules label, abstain, conflict, and disagree with human review.

Note

The sentences and team are fictional. The rules were designed after inspecting this teaching dataset, so results on these rows are not an independent performance estimate.

What you will learn

By the end of this lesson, you will be able to:

express labeling rules as text patterns;
keep each rule’s vote and avoid hiding conflicts;
calculate coverage, conflict, and abstention;
report each rule’s own coverage and error;
compare usable weak labels with reference labels; and
stress-test rules on cases they were not designed to fit.

Write rules that can abstain

Each rule searches for words tied to one codebook category. A rule returns TRUE when it votes for its category and FALSE otherwise. The next chunk loads package helpers for files, tables, reshaping, repeated rule checks, strings, and agreement, then reads the file with declared column types. Inside the chunk, map() makes one TRUE/FALSE list per rule and as_tibble() turns that list into columns.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(foundryR)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

rule_patterns <- c(
  training = str_c(
    "training",
    "certificat",
    "apprenticeship",
    "mentor",
    sep = "|"
  ),
  requirement = str_c(
    "required",
    "preferred",
    "license",
    "diploma",
    "portfolio",
    "must be able",
    sep = "|"
  ),
  schedule = "schedule|shift|classes",
  skill = "skill|spreadsheet|communication"
)
rule_labels <- names(rule_patterns)
sentence_text_lower <- str_to_lower(workforce_sentences$text)

rule_votes <- bind_cols(
  tibble(sentence_id = workforce_sentences$sentence_id),
  as_tibble(
    map(rule_patterns, \(pattern) {
      str_detect(sentence_text_lower, pattern)
    })
  )
)

head(rule_votes, 8)

# A tibble: 8 × 5
  sentence_id training requirement schedule skill
  <chr>       <lgl>    <lgl>       <lgl>    <lgl>
1 s001        TRUE     FALSE       FALSE    FALSE
2 s002        FALSE    TRUE        FALSE    FALSE
3 s003        FALSE    FALSE       TRUE     FALSE
4 s004        FALSE    FALSE       FALSE    TRUE 
5 s005        FALSE    TRUE        FALSE    FALSE
6 s006        TRUE     FALSE       FALSE    FALSE
7 s007        FALSE    FALSE       TRUE     FALSE
8 s008        FALSE    TRUE        FALSE    FALSE

The rule-vote tibble preserves one column per rule. Combining the votes too early would hide why a label was proposed.

Mark abstentions and conflicts

A row can receive:

one vote, which produces a usable weak label;
no votes, which produces an abstention; or
several votes, which produces a conflict.

rule_counts <- rule_votes |>
  mutate(
    rule_count = as.integer(rowSums(rule_votes[rule_labels]))
  ) |>
  select(sentence_id, rule_count)
voted_labels <- rule_votes |>
  pivot_longer(
    cols = all_of(rule_labels),
    names_to = "weak_label",
    values_to = "rule_vote"
  ) |>
  filter(rule_vote) |>
  summarise(
    voted_label = first(weak_label),
    .by = sentence_id
  )

weak_results <- workforce_sentences |>
  select(sentence_id, text, reference_label) |>
  left_join(rule_counts, by = join_by(sentence_id)) |>
  left_join(voted_labels, by = join_by(sentence_id)) |>
  mutate(
    weak_label = case_when(
      rule_count == 0L ~ "abstain",
      rule_count > 1L ~ "conflict",
      TRUE ~ voted_label
    )
  ) |>
  select(-voted_label)

result_order <- c(
  "abstain",
  "conflict",
  "requirement",
  "schedule",
  "skill",
  "training"
)
result_counts <- tibble(weak_label = result_order) |>
  left_join(
    weak_results |>
      count(weak_label, name = "rows"),
    by = join_by(weak_label)
  ) |>
  mutate(rows = coalesce(rows, 0L))

knitr::kable(
  weak_results |>
    filter(weak_label %in% c("abstain", "conflict")),
  col.names = c(
    "Sentence ID",
    "Text",
    "Reference label",
    "Rule votes",
    "Weak-label result"
  ),
  caption = "The five sentences that abstain or conflict",
  row.names = FALSE
)

The five sentences that abstain or conflict
Sentence ID	Text	Reference label	Rule votes	Weak-label result
s011	Remote work is available two days each week.	other	0	abstain
s013	A medical records certificate is preferred.	requirement	2	conflict
s018	The work is outdoors and includes local travel.	other	0	abstain
s020	Weekend shifts are required.	schedule	2	conflict
s028	Apply by October 15	other	0	abstain

Three sentences receive no vote. Two receive conflicting votes. Sentence s013, for example, contains both certificate and preferred, so the training and requirement rules disagree.

Report coverage before accuracy

Coverage is the share of rows receiving at least one vote. Conflict rate is the share receiving more than one. Usable coverage counts rows with exactly one vote.

rule_summary <- tibble(
  measure = c(
    "coverage",
    "conflict rate",
    "usable coverage"
  ),
  value = c(
    mean(weak_results$rule_count > 0L),
    mean(weak_results$rule_count > 1L),
    mean(weak_results$rule_count == 1L)
  )
)

knitr::kable(
  rule_summary |>
    mutate(
      value = round(value, 3)
    ),
  col.names = c("Measure", "Share of 28 text units"),
  caption = "Coverage and conflict in the teaching dataset",
  row.names = FALSE
)

Coverage and conflict in the teaching dataset
Measure	Share of 28 text units
coverage	0.893
conflict rate	0.071
usable coverage	0.821

Coverage alone does not measure correctness. A rule can label every row and be wrong every time.

Compare usable labels with human review

The 23 rows with one rule vote include two errors. We can summarize that comparison with foundryR.

agreement_data <- weak_results |>
  filter(rule_count == 1L) |>
  select(weak_label, reference_label)
error_sentences <- weak_results |>
  filter(
    rule_count == 1L,
    weak_label != reference_label
  )
error_sentence_ids <- error_sentences$sentence_id
rule_agreement <- foundryR::foundry_agreement(
  agreement_data,
  estimate = "weak_label",
  truth = "reference_label"
)

knitr::kable(
  rule_agreement,
  col.names = c("Measure", "Value", "Compared sentences"),
  caption = "In-sample agreement for usable weak labels",
  row.names = FALSE
)

In-sample agreement for usable weak labels
Measure	Value	Compared sentences
accuracy	0.9130435	23
precision_macro	0.8854167	23
recall_macro	0.8000000	23
f1_macro	0.7466667	23
cohen_kappa	0.8811370	23
krippendorff_alpha	0.8832685	23

The rule labels 21 of the 23 usable rows the same way as the reference annotation. It incorrectly labels RIVERTON SKILLS OPEN HOUSE as skill and DATA SUPPORT CERTIFICATE as training. The trigger words are present, but the headings alone do not make either claim.

The full accounting keeps that error beside unresolved rows.

all_row_accounting <- tibble(
  outcome = c(
    "usable and matches reference",
    "usable and differs from reference",
    "unresolved: abstain or conflict"
  ),
  rows = c(
    weak_results |>
      filter(
        rule_count == 1L,
        weak_label == reference_label
      ) |>
      nrow(),
    weak_results |>
      filter(
        rule_count == 1L,
        weak_label != reference_label
      ) |>
      nrow(),
    weak_results |>
      filter(rule_count != 1L) |>
      nrow()
  )
)

knitr::kable(
  all_row_accounting,
  col.names = c("Outcome", "Text units"),
  caption = "Accounting for all 28 rule results",
  row.names = FALSE
)

Accounting for all 28 rule results
Outcome	Text units
usable and matches reference	21
usable and differs from reference	2
unresolved: abstain or conflict	5

These are in-sample results. The rules were written after reading the sentences, so they may encode the examples rather than the broader concept.

Look at each rule on its own

The 21-of-23 figure is a pooled number, and pooled numbers hide which rule is carrying the work. Each rule has its own coverage and its own error, and the error looks different depending on whether the rule fired alone or alongside another.

reference_labels <- workforce_sentences$reference_label
votes_per_row <- as.integer(
  rowSums(rule_votes[rule_labels])
)

rule_diagnostics <- map(
  rule_labels,
  \(rule_name) {
    fired <- rule_votes[[rule_name]]
    fired_alone <- fired & votes_per_row == 1L
    matches_reference <- reference_labels == rule_name

    tibble(
      rule = rule_name,
      rows_fired = sum(fired),
      rows_fired_alone = sum(fired_alone),
      alone_and_correct = sum(fired_alone & matches_reference),
      correct_whenever_fired = sum(fired & matches_reference)
    )
  }
) |>
  list_rbind() |>
  mutate(
    precision_when_alone = alone_and_correct / rows_fired_alone
  )

knitr::kable(
  rule_diagnostics |>
    mutate(
      precision_when_alone = round(
        precision_when_alone,
        2
      )
    ) |>
    select(
      rule,
      rows_fired,
      rows_fired_alone,
      alone_and_correct,
      precision_when_alone
    ),
  col.names = c(
    "Rule",
    "Rows it fired on",
    "Fired alone",
    "Alone and correct",
    "Precision when alone"
  ),
  caption = "Each rule's coverage and error, separated",
  row.names = FALSE
)

Each rule’s coverage and error, separated
Rule	Rows it fired on	Fired alone	Alone and correct	Precision when alone
training	9	8	7	0.88
requirement	10	8	8	1.00
schedule	5	4	4	1.00
skill	3	3	2	0.67

The pooled figure of 21 out of 23 becomes four different stories. The requirement and schedule rules were right every time they spoke alone. The training rule missed once in eight. The skill rule, which fired on only three rows, was wrong on one of them, and a precision of two thirds on three rows is a number to treat as a warning rather than an estimate.

A rule that fires rarely is where in-sample results are least trustworthy. There is no way to distinguish a rule that is genuinely reliable from one that has not yet met the case that breaks it.

Separate conflicting rules from correlated support

Rules can fire on the same row because they conflict or because they repeat the same evidence. Those cases have different consequences. The example below contains opposing-label conflicts. It does not estimate correlation or show confidence inflation.

vote_matrix <- as.matrix(rule_votes[rule_labels])
rule_pairs <- expand.grid(
  first = rule_labels,
  second = rule_labels,
  stringsAsFactors = FALSE
) |>
  as_tibble() |>
  filter(first < second) |>
  mutate(
    fired_together = map2_int(
      first,
      second,
      \(left, right) {
        sum(vote_matrix[, left] & vote_matrix[, right])
      }
    )
  ) |>
  arrange(desc(fired_together), first, second)

knitr::kable(
  rule_pairs,
  col.names = c("Rule", "Other rule", "Rows where both fired"),
  caption = "How often each pair of rules spoke about the same row",
  row.names = FALSE
)

How often each pair of rules spoke about the same row
Rule	Other rule	Rows where both fired
requirement	schedule	1
requirement	training	1
requirement	skill	0
schedule	skill	0
schedule	training	0
skill	training	0

Only two pairs ever overlap here, and both overlaps are the conflicts already seen. s013 triggers training and requirement because a certificate is a credential and a credential looks like training. s020 triggers requirement and schedule because a required shift is both a demand and a time.

These are conflict overlaps, not duplicate supporting votes. Correlated support would require two rules that tend to cast the same label because they reuse the same signal, such as two training rules built from nearly identical keyword lists. A label model can estimate rule accuracy and dependence under stronger data and modeling assumptions, but this table does not supply that evidence.

This lesson does not fit such a model: four rules, 28 rows, and two conflicting overlaps do not contain enough information to estimate accuracies and correlations. The scope here stops at measuring where the rules agree, where they collide, and where they say nothing.

What a conflict policy would cost

A team that needs a label for every row usually writes a priority order. Here is one, and here is the price it charges.

priority_order <- c(
  "schedule",
  "requirement",
  "training",
  "skill"
)

resolve_by_priority <- function(row_votes) {
  chosen <- priority_order[priority_order %in% rule_labels[row_votes]]

  if (length(chosen) == 0L) {
    NA_character_
  } else {
    chosen[1]
  }
}

conflict_rows <- which(votes_per_row > 1L)
conflict_outcomes <- tibble(
  sentence_id = rule_votes$sentence_id[conflict_rows],
  reference_label = reference_labels[conflict_rows],
  resolved_label = map_chr(
    conflict_rows,
    \(row) resolve_by_priority(vote_matrix[row, ])
  )
) |>
  mutate(policy_agrees = resolved_label == reference_label)

abstain_rows <- which(votes_per_row == 0L)
abstain_outcomes <- tibble(
  sentence_id = rule_votes$sentence_id[abstain_rows],
  reference_label = reference_labels[abstain_rows]
)

knitr::kable(
  conflict_outcomes,
  col.names = c(
    "Sentence ID",
    "Reference label",
    "Priority policy label",
    "Policy agrees"
  ),
  caption = "What a priority order does with the two conflicts",
  row.names = FALSE
)

What a priority order does with the two conflicts
Sentence ID	Reference label	Priority policy label	Policy agrees
s013	requirement	requirement	TRUE
s020	schedule	schedule	TRUE

knitr::kable(
  abstain_outcomes,
  col.names = c("Sentence ID", "Reference label"),
  caption = "What the three silent rows actually were",
  row.names = FALSE
)

What the three silent rows actually were
Sentence ID	Reference label
s011	other
s018	other
s028	other

The policy gets both conflicts right, and that is close to meaningless as evidence. The order was chosen by someone who had read the codebook rule saying timing takes precedence, and it was applied to two rows. Two successes chosen after the fact do not establish that the order generalizes.

The three rows where every rule abstained are more informative. All three carry the reference label other, which is the one category no rule was written for. The abstentions are correct behavior expressed badly: the rules have nothing to say, and a fifth rule for other would be worse, since other is defined by the absence of the other four rather than by any words of its own.

Every one of the 28 rows now has an account. Twenty-one usable and correct, two usable and wrong, two in conflict and resolved by a stated policy, and three abstentions whose true label was outside the rule set. Nothing is left in a category called “the rest”.

Try cases the rules did not shape

Before applying the rules to another batch, the team writes four fictional adversarial unit tests. Their expected labels were chosen to expose known shortcuts after the rules were written. They are not an independent evaluation sample.

stress_cases <- tibble(
  case_id = sprintf("challenge-%d", 1:4),
  text = c(
    "The training department is hiring an accountant.",
    "Employees learn from experienced coworkers.",
    "Certification is not required.",
    "Flexible hours are available."
  ),
  reference_label = c(
    "other",
    "training",
    "requirement",
    "schedule"
  ),
  reference_status = "designed challenge expectation"
)

stress_text_lower <- str_to_lower(stress_cases$text)
stress_votes <- bind_cols(
  tibble(case_id = stress_cases$case_id),
  as_tibble(
    map(rule_patterns, \(pattern) {
      str_detect(stress_text_lower, pattern)
    })
  )
)
stress_counts <- stress_votes |>
  mutate(
    rule_count = as.integer(rowSums(stress_votes[rule_labels]))
  ) |>
  select(case_id, rule_count)
stress_voted_labels <- stress_votes |>
  pivot_longer(
    cols = all_of(rule_labels),
    names_to = "weak_label",
    values_to = "rule_vote"
  ) |>
  filter(rule_vote) |>
  summarise(
    voted_label = first(weak_label),
    .by = case_id
  )

stress_results <- stress_cases |>
  left_join(stress_counts, by = join_by(case_id)) |>
  left_join(stress_voted_labels, by = join_by(case_id)) |>
  mutate(
    weak_label = case_when(
      rule_count == 0L ~ "abstain",
      rule_count > 1L ~ "conflict",
      TRUE ~ voted_label
    )
  ) |>
  select(-rule_count, -voted_label)

knitr::kable(
  stress_results,
  col.names = c(
    "Challenge ID",
    "Challenge text",
    "Reference label",
    "Reference status",
    "Rule result"
  ),
  caption = "Four designed adversarial failures",
  row.names = FALSE
)

Four designed adversarial failures
Challenge ID	Challenge text	Reference label	Reference status	Rule result
challenge-1	The training department is hiring an accountant.	other	designed challenge expectation	training
challenge-2	Employees learn from experienced coworkers.	training	designed challenge expectation	abstain
challenge-3	Certification is not required.	requirement	designed challenge expectation	conflict
challenge-4	Flexible hours are available.	schedule	designed challenge expectation	abstain

training department triggers a false positive. Employees learn and Flexible hours use words the rules do not cover. Certification is not required creates a conflict. These failures reveal missing context, synonyms, and negation.

Keep weak labels weak

When rules scale:

version every pattern and codebook;
preserve each rule vote and abstention;
measure coverage and conflict by source, language, and group;
reserve independently reviewed data for evaluation;
sample apparent successes for human audit;
test negation, polysemy, and unseen wording; and
compare conclusions with and without weak labels.

Rules can provide useful training signals. They do not create facts, repair a biased source, or remove the need for people who understand the domain.

What to remember

A weak label is a rule’s proposal, not ground truth.
Rules should be allowed to abstain and expose conflicts.
Coverage, conflict, and agreement answer different questions.
Report each rule separately; a pooled score hides the weak one.
Rules that share trigger words are not independent votes.
In-sample success can reflect rules written around the examples.
Designed challenge cases expose known shortcuts; independent evaluation asks whether they generalize.

The investigation ends with the questions that opened it: what does each fictional notice explicitly say about training, schedules, requirements, and skills? The Lab can trace an answer through source capture, OCR review, a codebook version, individual labels, synthetic status, and rule votes. None of that makes the notices representative or the rules ready for new data. It makes the limits and unresolved cases inspectable.

Write rules that can abstain

Mark abstentions and conflicts

Report coverage before accuracy

Compare usable labels with human review

Look at each rule on its own

Separate conflicting rules from correlated support

What a conflict policy would cost

Try cases the rules did not shape

Keep weak labels weak

What to remember

Sources