Working with several annotators

Measure disagreement before combining labels

training data
crowdsourcing
agreement
Learn how to compare workforce labels from several reviewers without treating majority vote as truth.

The outside provider review kept unlike records separate. The Riverton Workforce Lab now checks whether the people assigning labels make comparable decisions. Three fictional paid reviewers apply the same codebook independently to 11 local sentences. Nine rows match; two do not.

Crowdsourcing distributes tasks to a group of people, often through an online marketplace. Access to more reviewers can increase coverage, but a marketplace does not create expertise, fair working conditions, or valid labels. The team must design those conditions.

Note

The reviewers, annotations, and research team are fictional. The disagreements were designed for this lesson and do not evaluate real workers.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • keep reviewer labels separate;
  • calculate pairwise agreement with foundryR;
  • distinguish observed agreement from chance-corrected measures;
  • test how much one item moves an agreement statistic;
  • find items that need adjudication and record the decision; and
  • explain why majority vote is not automatic truth.

Preserve each person’s answer

The annotation file stores one column per reviewer. The next chunk loads file, table, reshaping, repeated-work, agreement, and reliability helpers, then reads the CSV files with declared column types. No vote has been combined or overwritten.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(foundryR)
library(irr)
library(stringr)

crowd_labels <- read_csv(
  "data/workforce/crowd_annotations.csv",
  col_types = cols(
    sentence_id = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    reviewer_a = col_character(),
    reviewer_a_uncertainty = col_character(),
    reviewer_a_rationale = col_character(),
    reviewer_b = col_character(),
    reviewer_b_uncertainty = col_character(),
    reviewer_b_rationale = col_character(),
    reviewer_c = col_character(),
    reviewer_c_uncertainty = col_character(),
    reviewer_c_rationale = col_character()
  )
)
sentence_text <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

reviewer_columns <- c(
  "reviewer_a",
  "reviewer_b",
  "reviewer_c"
)
uncertainty_columns <- paste0(
  reviewer_columns,
  "_uncertainty"
)
rationale_columns <- paste0(
  reviewer_columns,
  "_rationale"
)
allowed_labels <- c(
  "training",
  "requirement",
  "schedule",
  "skill",
  "other"
)
source("R/workforce-codebook.R")
crowd_codebook <- workforce_codebook("1.0.0")

knitr::kable(
  crowd_labels |>
    select(sentence_id, all_of(reviewer_columns)),
  col.names = c(
    "Sentence ID",
    "Reviewer A",
    "Reviewer B",
    "Reviewer C"
  ),
  caption = "Independent labels from three fictional reviewers",
  row.names = FALSE
)
Independent labels from three fictional reviewers
Sentence ID Reviewer A Reviewer B Reviewer C
s001 training training training
s002 requirement requirement requirement
s003 schedule schedule schedule
s004 skill skill skill
s005 requirement requirement requirement
s006 training training training
s010 requirement other requirement
s011 other other other
s013 requirement training other
s016 training training training
s017 requirement requirement requirement

Nine rows have complete agreement. Sentence s010 has one different answer. Sentence s013 has three different answers. Keeping the columns separate makes those disagreements visible, and the sample includes all five labels.

Measure all reviewer comparisons

Simple agreement is the share of answers that match. Cohen’s kappa adjusts for the agreement expected from the reviewers’ label frequencies. Krippendorff’s alpha is another chance-corrected measure that can support more coders and missing labels under suitable settings.

foundryR calculates several measures together. Here we retain observed agreement and Cohen’s kappa for every reviewer pair. Directional classification measures such as precision and recall are not shown because no reviewer is treated as the truth.

reviewer_pairs <- tibble(
  pair = c("A and B", "A and C", "B and C"),
  first = c(
    "reviewer_a",
    "reviewer_a",
    "reviewer_b"
  ),
  second = c(
    "reviewer_b",
    "reviewer_c",
    "reviewer_c"
  )
)

pairwise_agreement <- pmap(
  reviewer_pairs,
  \(pair, first, second) {
    measures <- foundryR::foundry_agreement(
      crowd_labels,
      estimate = second,
      truth = first
    )

    tibble(
      pair = pair,
      observed_agreement = filter(measures, metric == "accuracy")$value,
      cohen_kappa = filter(measures, metric == "cohen_kappa")$value
    )
  }
) |>
  list_rbind()

knitr::kable(
  pairwise_agreement |>
    mutate(
      observed_agreement = round(
        observed_agreement,
        2
      ),
      cohen_kappa = round(cohen_kappa, 2)
    ),
  col.names = c(
    "Reviewer pair",
    "Observed agreement",
    "Cohen's kappa"
  ),
  caption = "Pairwise agreement across all three reviewers",
  row.names = FALSE
)
Pairwise agreement across all three reviewers
Reviewer pair Observed agreement Cohen’s kappa
A and B 0.82 0.76
A and C 0.91 0.88
B and C 0.82 0.76

The B-and-C comparison is weaker than A and C. Reporting only one pair would hide that difference.

The table shows two decimals on purpose. With 11 items, one reviewer changing one answer moves observed agreement by about nine points, so a third decimal describes arithmetic rather than measurement. Publishing 0.818 invites a reader to compare it with 0.812 from another study as though the difference meant something.

Krippendorff’s alpha summarizes all three reviewers at once. pivot_longer() puts the reviewer columns into one label column, then pivot_wider() builds the matrix that irr needs. The calculation converts label names to category numbers; the numbers carry no order.

numeric_ratings <- crowd_labels |>
  select(sentence_id, all_of(reviewer_columns)) |>
  pivot_longer(
    cols = all_of(reviewer_columns),
    names_to = "reviewer",
    values_to = "label"
  ) |>
  mutate(category_number = match(label, allowed_labels)) |>
  select(sentence_id, reviewer, category_number) |>
  pivot_wider(
    names_from = reviewer,
    values_from = category_number
  ) |>
  arrange(sentence_id)

rating_matrix <- numeric_ratings |>
  select(all_of(reviewer_columns)) |>
  as.matrix()
all_rater_alpha <- irr::kripp.alpha(
  t(rating_matrix),
  method = "nominal"
)$value
unanimity_by_sentence <- crowd_labels |>
  select(sentence_id, all_of(reviewer_columns)) |>
  pivot_longer(
    cols = all_of(reviewer_columns),
    names_to = "reviewer",
    values_to = "label"
  ) |>
  summarise(
    unanimous = n_distinct(label) == 1L,
    .by = sentence_id
  ) |>
  arrange(sentence_id)
unanimity_share <- unanimity_by_sentence |>
  summarise(value = mean(unanimous))
unanimity_share <- unanimity_share$value

tibble(
  measure = c(
    "Krippendorff's alpha",
    "Unanimous share"
  ),
  value = round(
    c(all_rater_alpha, unanimity_share),
    2
  )
)
# A tibble: 2 × 2
  measure              value
  <chr>                <dbl>
1 Krippendorff's alpha  0.8 
2 Unanimous share       0.82

The 11-item example is too small for a precise population estimate. These statistics describe this annotation exercise and help locate problems; they do not certify the codebook or any reviewer.

Ask how much one item is worth

“Too small for a precise estimate” is a phrase readers skim past. The way to make it concrete is to remove one item at a time and recompute, which for 11 items means 11 recalculations and no new data.

alpha_without_item <- map_dbl(
  seq_len(nrow(rating_matrix)),
  \(dropped) {
    irr::kripp.alpha(
      t(rating_matrix[-dropped, , drop = FALSE]),
      method = "nominal"
    )$value
  }
)

sensitivity <- tibble(
  sentence_id = numeric_ratings$sentence_id,
  alpha_without_this_item = alpha_without_item
) |>
  arrange(desc(alpha_without_this_item))

alpha_range <- range(alpha_without_item)

knitr::kable(
  sensitivity |>
    mutate(
      alpha_without_this_item = round(
        alpha_without_this_item,
        2
      )
    ),
  col.names = c("Item removed", "Alpha for the other ten"),
  caption = "Krippendorff's alpha after dropping each item in turn",
  row.names = FALSE
)
Krippendorff’s alpha after dropping each item in turn
Item removed Alpha for the other ten
s013 0.91
s010 0.87
s002 0.78
s005 0.78
s017 0.78
s001 0.78
s006 0.78
s016 0.78
s011 0.77
s003 0.76
s004 0.76

Alpha is 0.80 on all 11 items. Drop one item and it lands anywhere between 0.76 and 0.91. The high end is not a coincidence: removing s013, the item all three reviewers read differently, is what produces it.

This is not a confidence interval, and it should not be reported as one. It is a cruder statement and a sufficient one here: a summary this sensitive to a single item cannot support a claim that one codebook, task, or reviewer pool is more reliable than another. Reporting the range beside the estimate is honest; reporting 0.797 alone is not.

A larger sample can reduce single-item sensitivity, but sample size alone does not determine how much alpha moves. Label prevalence, clustered documents, which item is removed, and the pattern of disagreement all matter. For a larger study, resample whole source documents rather than individual rows and report the resulting bootstrap distribution; this lesson’s 11 items cannot predict what its width would be for 500.

Find the disagreements

An item needs review when the three labels are not identical. left_join() adds the sentence text by matching the shared sentence identifier.

disagreements <- crowd_labels |>
  left_join(
    unanimity_by_sentence,
    by = join_by(sentence_id)
  ) |>
  filter(!unanimous) |>
  left_join(
    sentence_text |>
      select(sentence_id, text),
    by = join_by(sentence_id)
  ) |>
  arrange(sentence_id)

knitr::kable(
  disagreements |>
    select(
      "sentence_id",
      "text",
      "reviewer_a",
      "reviewer_a_uncertainty",
      "reviewer_b",
      "reviewer_b_uncertainty",
      "reviewer_c",
      "reviewer_c_uncertainty"
    ),
  col.names = c(
    "Sentence ID",
    "Text",
    "Reviewer A",
    "A uncertainty",
    "Reviewer B",
    "B uncertainty",
    "Reviewer C",
    "C uncertainty"
  ),
  caption = "Sentences sent to adjudication",
  row.names = FALSE
)
Sentences sent to adjudication
Sentence ID Text Reviewer A A uncertainty Reviewer B B uncertainty Reviewer C C uncertainty
s010 Six months of experience is preferred. requirement certain other needs_review requirement certain
s013 A medical records certificate is preferred. requirement needs_review training needs_review other needs_review

The first sentence says experience is preferred; one reviewer chose other. The second mentions a certificate and receives three different labels. Those choices place the same notices under different questions for a person comparing options. The codebook needs to explain the boundary rather than hide it in a vote.

Calculate a majority label carefully

A majority label can organize the next review step. It should not erase the individual answers or replace adjudication.

label_counts <- crowd_labels |>
  select(sentence_id, all_of(reviewer_columns)) |>
  pivot_longer(
    cols = all_of(reviewer_columns),
    names_to = "reviewer",
    values_to = "label"
  ) |>
  count(sentence_id, label, name = "votes")

top_vote_count <- label_counts |>
  summarise(top_votes = max(votes), .by = sentence_id)
top_vote_table <- label_counts |>
  left_join(top_vote_count, by = join_by(sentence_id)) |>
  filter(votes == top_votes) |>
  summarise(
    tied_top_labels = n(),
    top_label = paste(sort(label), collapse = ", "),
    .by = c(sentence_id, top_votes)
  )
majority_label <- top_vote_table |>
  mutate(
    majority_label = case_when(
      top_votes > length(reviewer_columns) / 2 &
        tied_top_labels == 1L ~ top_label,
      TRUE ~ "adjudicate"
    )
  ) |>
  select(sentence_id, majority_label)

combined_labels <- crowd_labels |>
  select(sentence_id) |>
  left_join(
    majority_label,
    by = join_by(sentence_id)
  ) |>
  left_join(
    unanimity_by_sentence,
    by = join_by(sentence_id)
  )

knitr::kable(
  top_vote_table,
  col.names = c(
    "Sentence ID",
    "Top votes",
    "Labels tied for top",
    "Top label or labels"
  ),
  caption = "Top vote count before majority assignment",
  row.names = FALSE
)
Top vote count before majority assignment
Sentence ID Top votes Labels tied for top Top label or labels
s001 3 1 training
s002 3 1 requirement
s003 3 1 schedule
s004 3 1 skill
s005 3 1 requirement
s006 3 1 training
s010 2 1 requirement
s011 3 1 other
s013 1 3 other, requirement, training
s016 3 1 training
s017 3 1 requirement 
knitr::kable(
  combined_labels,
  col.names = c(
    "Sentence ID",
    "Majority label",
    "Unanimous"
  ),
  caption = "Majority labels with unanimity retained",
  row.names = FALSE
)
Majority labels with unanimity retained
Sentence ID Majority label Unanimous
s001 training TRUE
s002 requirement TRUE
s003 schedule TRUE
s004 skill TRUE
s005 requirement TRUE
s006 training TRUE
s010 requirement FALSE
s011 other TRUE
s013 adjudicate FALSE
s016 training TRUE
s017 requirement TRUE

The unanimous column prevents an easy majority from looking like complete agreement. The tie check stays because teams may change the reviewer count; any tie among top labels goes to adjudicate. Sentence s013 returns adjudicate because three different answers do not form a majority. The team can review the sentence, codebook, and source labels before choosing a reference label.

Finish the adjudication

An adjudication queue that stays a queue is not a quality process. Both flagged items were decided by a fourth reader who had not labeled them, working from the codebook text rather than from the vote counts.

adjudication <- tibble(
  sentence_id = c("s010", "s013"),
  question = c(
    "Is a preferred condition an entry condition or something else?",
    "Does naming a credential make the unit a training offer?"
  ),
  rule_applied = c(
    "requirement: the main claim states or prefers an entry condition",
    "training: a credential required or preferred is not training"
  ),
  decision = c("requirement", "requirement"),
  decided_by = "fourth reader, not one of the three reviewers"
)

adjudication_outcome <- adjudication |>
  left_join(
    sentence_text |>
      select(sentence_id, text, reference_label),
    by = join_by(sentence_id)
  ) |>
  left_join(
    crowd_labels |>
      select(sentence_id, all_of(reviewer_columns)),
    by = join_by(sentence_id)
  ) |>
  mutate(
    matches_reference_file = decision == reference_label,
    reviewers_still_disagree = reviewer_a != reviewer_b |
      reviewer_b != reviewer_c
  )

knitr::kable(
  adjudication_outcome |>
    select(
      sentence_id,
      text,
      reviewer_a,
      reviewer_b,
      reviewer_c,
      decision
    ),
  col.names = c(
    "Sentence ID",
    "Text",
    "Reviewer A",
    "Reviewer B",
    "Reviewer C",
    "Adjudicated label"
  ),
  caption = "Both flagged items resolved, with the reviewer answers kept",
  row.names = FALSE
)
Both flagged items resolved, with the reviewer answers kept
Sentence ID Text Reviewer A Reviewer B Reviewer C Adjudicated label
s010 Six months of experience is preferred. requirement other requirement requirement
s013 A medical records certificate is preferred. requirement training other requirement

Both items resolve to requirement, and both decisions rest on a clause the adjudicator can point to in the codebook. The reviewer columns are still in the table with their disagreements intact, which is why the check above can confirm that the reviewers still differ after the item was settled.

Notice what the adjudication of s013 did not do. It did not make the item easy. Three trained readers reached three answers, and the sensitivity analysis showed that this one item carries most of the distance between an alpha of 0.80 and one of 0.91. A decision closes the item; it does not retire the question, which belongs in the next codebook revision as a worked example.

Agreement does not prove validity

Reviewers can agree because:

  • the codebook is clear;
  • the item is easy;
  • they share the same misunderstanding;
  • the task excludes a relevant perspective; or
  • the platform trained everyone toward the same shortcut.

Validity requires evidence that the labels measure the intended idea. Compare annotations with the research question, consult people with relevant knowledge, and test how conclusions change under reasonable label choices.

Treat crowd work as work

A responsible plan states:

  1. payment for training, annotation, and adjudication time;
  2. expected task length and rejection rules;
  3. privacy and data-handling requirements;
  4. access to support and the ability to skip harmful material;
  5. the expertise and language knowledge required; and
  6. an appeal path for disputed quality decisions.

Hidden attention checks and automatic rejection can shift measurement errors onto workers. Quality control should diagnose the task and instructions as well as individual answers.

What to remember

  • Keep individual reviewer labels before combining them.
  • Agreement measures summarize consistency, not truth.
  • Report agreement to a precision the sample size can support.
  • Show how far a statistic moves when one item is removed.
  • Inspect disagreement items and revise the codebook when needed.
  • Finish the adjudication and record which rule decided it.
  • Preserve whether a majority label was unanimous.
  • Fair pay, privacy, expertise, and appeal processes are part of data quality.

The comparison leaves nothing in the queue: two boundary cases were adjudicated against named rules, and the reviewer answers that produced them are still on the page. The following augmentation exercise keeps that annotation record intact and creates fictional variants. Each is marked as synthetic and checked against the same codebook before it can join training data.

Sources