Reviewing documents for discovery and alerts

Estimate what search and ranked review leave unseen

systems
e-discovery
media monitoring
Learn how keyword terms, ranked review, an elusion sample, and a small monitoring stream expose missed records.

A fictional request asks for records about the Calder Yard contract. It is not asking for a few good examples. It asks for the responsive records in a fixed collection, including the ones that do not use the words a searcher guesses first.

E-discovery is electronic discovery: finding records that must be reviewed or produced for a legal request. Technology-assisted review uses a model to rank unreviewed records so people can review likely responsive records first. Media monitoring watches new items as they arrive and sends alerts for items that matter.

Grossman and Cormack report that Blair and Maron’s legal-search study found searchers believed they had found at least 75 percent of relevant documents, while average recall was about 20 percent for the studied requests. Lesson 9’s active learning chooses uncertain examples to improve a model; ranked review here chooses high-scoring unreviewed records to find responsive material. The elusion rate is the share of unreviewed records that are responsive, and a low elusion rate can still hide many misses. Suppose a review of 600 records from a 10,000-record collection finds 70 of its 100 responsive records. The 9,400 unreviewed records still hold 30 responsive records, so the elusion rate is 30 / 9,400, about 0.32 percent, while recall is 70 / 100, or 70 percent.

Note

The records in this lesson are constructed teaching examples. They do not describe a real legal matter, company, person, news source, or monitoring target.

TipWhat you will learn

This lesson shows how to:

  • predeclare keyword terms and read their precision and recall;
  • train a keyword-seeded ranked review model without scoring its training rows;
  • stop review by a fixed rule chosen before sampling;
  • estimate missed responsive records with one finite-population sample;
  • explain why the fictional answer key is stronger than real reviewers usually have; and
  • process a short monitoring stream in arrival order with duplicate suppression.

Load the constructed review collection

The builder created a short review collection for one request. It also wrote a monitoring stream and metadata fingerprints. Each row carries a fiction notice. The reference labels were written by the page author, so they are labels for this constructed collection rather than ground truth.

suppressPackageStartupMessages({
  library(digest)
  library(dplyr)
  library(ggplot2)
  library(glmnet)
  library(knitr)
  library(purrr)
  library(readr)
  library(stringdist)
  library(stringr)
  library(tibble)
  library(tidymodels)
  library(textrecipes)
  library(tidyr)
})

hash_lines <- function(path) {
  digest(
    paste(read_lines(path), collapse = "\n"),
    algo = "sha256",
    serialize = FALSE
  )
}

collection_path <- "data/riverton/riverton-review-collection.csv"
stream_path <- "data/riverton/riverton-monitoring-stream.csv"
metadata_path <- "data/riverton/riverton-review-collection-metadata.csv"

review_records <- read_csv(
  collection_path,
  na = c("", "NA"),
  col_types = cols(
    doc_id = col_character(),
    record_type = col_character(),
    doc_date = col_date(),
    source_type = col_character(),
    sender_role = col_character(),
    subject = col_character(),
    review_request = col_character(),
    reference_responsive = col_logical(),
    reference_reason = col_character(),
    construction_subtype = col_character(),
    near_duplicate_of = col_character(),
    text = col_character(),
    author_note = col_character(),
    construction_id = col_character(),
    file_position = col_integer()
  )
)

monitoring_stream <- read_csv(
  stream_path,
  na = c("", "NA"),
  col_types = cols(
    stream_id = col_character(),
    record_type = col_character(),
    arrival_order = col_integer(),
    doc_date = col_date(),
    source_type = col_character(),
    text = col_character(),
    reference_alert = col_logical(),
    reference_reason = col_character(),
    near_duplicate_of = col_character(),
    author_note = col_character()
  )
)

review_metadata <- read_csv(
  metadata_path,
  na = character(),
  col_types = cols(
    artifact = col_character(),
    description = col_character(),
    source = col_character(),
    license = col_character(),
    created_on = col_character(),
    purpose = col_character(),
    rows = col_integer(),
    class_counts = col_character(),
    fingerprint = col_character(),
    rng_kind = col_character(),
    seed = col_character(),
    effective_seeds = col_character(),
    review_request = col_character(),
    shortcut_sweep = col_character()
  )
)

actual_fingerprints <- tibble(
  artifact = c(
    "riverton-review-collection.csv",
    "riverton-monitoring-stream.csv"
  ),
  fingerprint = c(hash_lines(collection_path), hash_lines(stream_path))
)

collection_summary <- review_records |>
  count(reference_responsive, name = "records") |>
  mutate(label = if_else(reference_responsive, "responsive", "not responsive"))

knitr::kable(
  collection_summary |>
    select(label, records),
  col.names = c("Reference label", "Records"),
  caption = "Constructed review collection labels for one fictional request",
  row.names = FALSE
)
Constructed review collection labels for one fictional request
Reference label Records
not responsive 204
responsive 36 
knitr::kable(
  tibble(review_request = review_metadata$review_request[[1]]),
  format = "html",
  escape = TRUE,
  col.names = "Review request",
  caption = "Fictional request used to define responsiveness",
  row.names = FALSE
)
Fictional request used to define responsiveness
Review request
Find records about hiding safety inspection delays, changing inspection logs, or deleting inspection-related messages for the Calder Yard contract.

The constructed prevalence is 15%, much higher than many legal-review matters. It was chosen so a small teaching page has enough responsive examples to inspect. Each record is a short note wrapped in one of 12 frames, such as “Reminder:” at the start or “Thanks.” at the end, and every frame holds the same number of responsive records and the same number of non-responsive records. Before writing the file, the builder tested whether the surface features it lists could separate the two classes. It would have refused to write the file if a check reached an F1 of 0.60 or a balanced accuracy of 0.75; the frame checks within keyword groups use only the balanced-accuracy limit. Passing means that no listed check separated the classes that well, not that surface features carry no signal. Both scores rise as a check separates the classes better: balanced accuracy, from lesson 44, averages the share of each class that a check gets right, and F1 combines the precision and recall defined in the keyword section below. The table below is the builder’s own record of those checks.

knitr::kable(
  tibble(builder_checks = review_metadata$shortcut_sweep[[1]]),
  format = "html",
  escape = TRUE,
  col.names = "Surface checks recorded by the builder",
  caption = "Checks the builder ran before writing the review collection",
  row.names = FALSE
)
Checks the builder ran before writing the review collection
Surface checks recorded by the builder
Refuses to write if any check reaches positive F1 >= 0.60 or balanced accuracy >= 0.75, and stops on any NA diagnostic. Checks: single values and held-out value sets (20 split-halves) of record features (length, punctuation, pronouns, digits, month, weekday, ID digit, ID order, date rank, file position, construction ID order, first three tokens, first word, first two words, sentence count, colon, source type, sender role, subject) and of core and frame features; the 20 most frequent non-topic tokens; out-of-fold glmnet ridge and lasso rankers over snowball stop-word counts and over format cues (length, punctuation, capitalization, digits), stratified 5-fold under three fold seeds, top 36; a frame-only ranker; a 15 percent cap on any frame or core opening among non-responsive records; CK-D1, label-free rarity rules (k = 1 to 4) and a held-out ranker over the pair of frame and core opening word; and CK-D2, frame rules and a held-out frame ranker within the keyword-match and no-match strata, thresholded on balanced accuracy.

The table uses the builder’s own shorthand. A held-out check learns a rule from some of the records and tests it on records it did not see. Ridge and lasso are two kinds of penalized regression, the same family as the ranker later on this page, and snowball stop words are a standard list of common words such as “the” and “to”. CK-D1 and CK-D2 are the builder’s names for its two frame checks. CK-D1 looks at each frame together with the first word of the note inside it, and CK-D2 compares the frames separately among records that contain a keyword and records that do not.

Search predeclared keywords

The keyword terms below are fixed before scoring. They are plausible terms for the written request, but they are still guesses. Keyword recall is knowable here only because the collection is fictional and the reference labels are already in the file.

keyword_terms <- c(
  "inspection",
  "inspect",
  "delete",
  "deleted",
  "delay",
  "delayed",
  "log",
  "Calder Yard"
)

keyword_pattern <- regex(
  paste0(
    "\\b(",
    paste(keyword_terms[keyword_terms != "Calder Yard"], collapse = "|"),
    ")\\b|\\bCalder Yard\\b"
  ),
  ignore_case = TRUE
)

review_scored <- review_records |>
  mutate(
    keyword_hit = str_detect(text, keyword_pattern),
    label = factor(
      if_else(reference_responsive, "responsive", "not responsive"),
      levels = c("responsive", "not responsive")
    )
  )

keyword_metrics <- review_scored |>
  summarise(
    hits = sum(keyword_hit),
    responsive_hits = sum(keyword_hit & reference_responsive),
    responsive_total = sum(reference_responsive),
    precision = responsive_hits / hits,
    recall = responsive_hits / responsive_total
  )

keyword_examples <- bind_rows(
  review_scored |>
    filter(keyword_hit, reference_responsive) |>
    arrange(doc_id) |>
    slice_head(n = 4),
  review_scored |>
    filter(keyword_hit, !reference_responsive) |>
    arrange(doc_id) |>
    slice_head(n = 4)
) |>
  transmute(
    doc_id,
    reference_label = if_else(reference_responsive, "responsive", "not responsive"),
    construction_subtype,
    text
  )

knitr::kable(
  keyword_metrics |>
    mutate(across(c(precision, recall), \(value) round(value, 3))),
  col.names = c("Keyword hits", "Responsive hits", "Responsive records", "Precision", "Recall"),
  caption = "Predeclared keyword terms on the constructed collection",
  row.names = FALSE
)
Predeclared keyword terms on the constructed collection
Keyword hits Responsive hits Responsive records Precision Recall
39 22 36 0.564 0.611 
knitr::kable(
  keyword_examples,
  format = "html",
  escape = TRUE,
  col.names = c("Document", "Reference label", "Construction note", "Text"),
  caption = "Keyword hits include both responsive records and false hits",
  row.names = FALSE
)
Keyword hits include both responsive records and false hits
Document Reference label Construction note Text
RYD-0008 responsive keyword responsive Per scheduling: The contract file should show the inspection as pending, not delayed.
RYD-0028 responsive keyword responsive Following up: The safety inspection delay should stay out of the weekly Calder Yard summary.
RYD-0043 responsive keyword responsive Remove the delayed inspection line from the packet before records reviews it. Call me if anything is unclear.
RYD-0045 responsive keyword responsive Forwarding from intake: The draft says the inspection was late; delete that line before the packet goes out.
RYD-0006 not responsive keyword false hit Forwarding from intake: The classroom inspection drill note belongs in the training folder, not the Calder Yard file.
RYD-0048 not responsive keyword false hit Please note: Please delete the duplicate lunch order from the event spreadsheet.
RYD-0049 not responsive keyword false hit Ticket note: The delayed chair shipment arrived at the front desk this afternoon.
RYD-0054 not responsive keyword false hit Following up: Calder Yard is listed on the volunteer cleanup map, not the contract file.

Precision is the share of reviewed hits that are responsive. Recall is the share of all responsive records that were found. The terms match whole words, plus the phrase Calder Yard; for example, log does not match chronology. In this fixture, the terms find 39 records. Of those, 22 are responsive under the constructed labels. That precision, 22 divided by 39, is visible after reviewing the hits. The recall, 22 divided by 36, is not normally visible; the fictional labels let the lesson show what the keyword search missed.

Rank the rest from the reviewed seed

The reviewed seed is the keyword-hit set. It has at least eight records of each class, which avoids the glmnet warning for very small classes under options(warn = 2). The model uses tf-idf features and penalized logistic regression. It trains only on reviewed records and scores only records not yet reviewed.

The stopping rule is also fixed before the model runs: review the keyword-hit seed, then review two ranked batches of ten. That rule has no statistical guarantee. It is a budget rule, and the elusion sample is what estimates the remaining miss.

fit_review_model <- function(training_rows) {
  class_counts <- table(training_rows$label)
  stopifnot(min(class_counts) >= 8L)

  review_recipe <- recipe(label ~ text, data = training_rows) |>
    step_tokenize(text) |>
    step_tokenfilter(text, max_tokens = 80) |>
    step_tfidf(text)

  review_model <- logistic_reg(penalty = 0.01, mixture = 1) |>
    set_engine("glmnet") |>
    set_mode("classification")

  workflow() |>
    add_recipe(review_recipe) |>
    add_model(review_model) |>
    fit(data = training_rows)
}

base_columns <- names(review_scored)
reviewed_records <- review_scored |>
  filter(keyword_hit)
unreviewed_records <- review_scored |>
  filter(!keyword_hit)

review_batches <- list()
ranked_batches <- 2L
batch_size <- 10L
tie_rows <- integer(ranked_batches)
pool_rows <- integer(ranked_batches)
batch_tied_rows <- integer(ranked_batches)
tie_crosses_batch_edge <- logical(ranked_batches)

for (batch_id in seq_len(ranked_batches)) {
  fitted_review <- fit_review_model(reviewed_records)
  scored_pool <- unreviewed_records |>
    bind_cols(
      predict(
        fitted_review,
        new_data = unreviewed_records,
        type = "prob"
      )
    ) |>
    mutate(score = .pred_responsive) |>
    arrange(desc(score), doc_id)

  tied_in_pool <- duplicated(scored_pool$score) |
    duplicated(scored_pool$score, fromLast = TRUE)
  tie_rows[[batch_id]] <- sum(tied_in_pool)
  pool_rows[[batch_id]] <- nrow(scored_pool)
  batch_tied_rows[[batch_id]] <- sum(tied_in_pool[seq_len(batch_size)])
  tie_crosses_batch_edge[[batch_id]] <- identical(
    scored_pool$score[[batch_size]],
    scored_pool$score[[batch_size + 1L]]
  )

  batch_rows <- scored_pool |>
    slice_head(n = batch_size) |>
    mutate(batch = batch_id)

  review_batches[[batch_id]] <- batch_rows
  reviewed_records <- bind_rows(
    reviewed_records,
    batch_rows |> select(all_of(base_columns))
  )
  unreviewed_records <- anti_join(
    unreviewed_records,
    batch_rows |> select(doc_id),
    by = join_by(doc_id)
  )
}

ranked_reviewed <- bind_rows(review_batches)

ranked_review_display <- ranked_reviewed |>
  transmute(
    batch,
    doc_id,
    score = round(score, 3),
    reference_label = if_else(reference_responsive, "responsive", "not responsive"),
    construction_subtype,
    text
  )

batch_summary <- ranked_reviewed |>
  count(batch, reference_responsive, name = "records") |>
  complete(
    batch,
    reference_responsive = c(FALSE, TRUE),
    fill = list(records = 0L)
  ) |>
  mutate(label = if_else(reference_responsive, "responsive", "not responsive")) |>
  select(batch, label, records)

review_progress <- bind_rows(
  tibble(
    stage = "keyword seed",
    reviewed_this_stage = nrow(review_scored |> filter(keyword_hit)),
    responsive_found = sum(review_scored$keyword_hit & review_scored$reference_responsive)
  ),
  ranked_reviewed |>
    group_by(batch) |>
    summarise(
      stage = paste0("ranked batch ", first(batch)),
      reviewed_this_stage = n(),
      responsive_found = sum(reference_responsive),
      .groups = "drop"
    ) |>
    select(stage, reviewed_this_stage, responsive_found)
) |>
  mutate(
    cumulative_reviewed = cumsum(reviewed_this_stage),
    cumulative_responsive_found = cumsum(responsive_found),
    fiction_only_recall = cumulative_responsive_found /
      sum(review_scored$reference_responsive)
  )

review_stop <- tibble(
  reviewed = nrow(reviewed_records),
  responsive_found = sum(reviewed_records$reference_responsive),
  unreviewed = nrow(unreviewed_records),
  fiction_known_missed = sum(unreviewed_records$reference_responsive)
)

missed_records <- unreviewed_records |>
  filter(reference_responsive) |>
  transmute(
    doc_id,
    construction_subtype,
    text
  )

tie_edge_sentence <- if (any(tie_crosses_batch_edge)) {
  paste0(
    "In ranked batch ", paste(which(tie_crosses_batch_edge), collapse = " and "),
    ", a tie crossed the batch edge, so the document-ID rule decided which of the tied records were reviewed."
  )
} else {
  "No tie crossed the edge of a ranked batch, so the tie rule did not decide which records were reviewed."
}

mostly_tied_batches <- which(batch_tied_rows > batch_size / 2)
tied_batch_sentence <- if (length(mostly_tied_batches) > 0L) {
  paste0(
    "Tied records fill most of ranked batch ", paste(mostly_tied_batches, collapse = " and "), ": ",
    paste(batch_tied_rows[mostly_tied_batches], collapse = " and "), " of its ", batch_size,
    " records share an exact score with another record, so their order says nothing about which is more likely to be responsive."
  )
} else {
  ""
}

gain_alt_text <- paste0(
  "Line chart of responsive records found against records reviewed; both axes start at zero. ",
  "The line runs from ", review_progress$cumulative_responsive_found[[1]],
  " found at ", review_progress$cumulative_reviewed[[1]], " reviewed to ",
  paste(
    paste0(
      review_progress$cumulative_responsive_found[-1], " at ",
      review_progress$cumulative_reviewed[-1]
    ),
    collapse = " and "
  ),
  "."
)

knitr::kable(
  batch_summary,
  col.names = c("Ranked batch", "Reference label", "Records"),
  caption = "Ranked review batches after the keyword seed",
  row.names = FALSE
)
Ranked review batches after the keyword seed
Ranked batch Reference label Records
1 not responsive 6
1 responsive 4
2 not responsive 9
2 responsive 1 
knitr::kable(
  review_progress |>
    mutate(fiction_only_recall = round(fiction_only_recall, 3)),
  col.names = c(
    "Stage",
    "Reviewed at stage",
    "Responsive found at stage",
    "Cumulative reviewed",
    "Cumulative responsive found",
    "Fiction-only recall"
  ),
  caption = "Review progress, with recall visible only because the collection is invented",
  row.names = FALSE
)
Review progress, with recall visible only because the collection is invented
Stage Reviewed at stage Responsive found at stage Cumulative reviewed Cumulative responsive found Fiction-only recall
keyword seed 39 22 39 22 0.611
ranked batch 1 10 4 49 26 0.722
ranked batch 2 10 1 59 27 0.750 
knitr::kable(
  review_stop,
  col.names = c("Reviewed", "Responsive found", "Unreviewed", "Fiction-known missed"),
  caption = "Predeclared stopping point before the elusion sample",
  row.names = FALSE
)
Predeclared stopping point before the elusion sample
Reviewed Responsive found Unreviewed Fiction-known missed
59 27 181 9 
knitr::kable(
  ranked_review_display,
  format = "html",
  escape = TRUE,
  col.names = c(
    "Batch",
    "Document",
    "Review score",
    "Reference label",
    "Construction note",
    "Text"
  ),
  caption = "Ranked records reviewed after the keyword seed",
  row.names = FALSE
)
Ranked records reviewed after the keyword seed
Batch Document Review score Reference label Construction note Text
1 RYD-0134 1.000 not responsive not responsive For today: If catering calls back, confirm the allergy cards are ready and ask about serving spoons.
1 RYD-0227 1.000 responsive vocabulary mismatch responsive Monday update: The draft route sheet tells too much about the missed check.
1 RYD-0230 1.000 not responsive not responsive Following up: Tell catering that the afternoon seminar needs vegetarian wraps and chilled water.
1 RYD-0012 1.000 not responsive not responsive For today: If the printer stalls, ask Omar about the paper tray and switch to plain stock.
1 RYD-0076 0.997 responsive near duplicate responsive original Please note: Keep the rough yard timeline out of the shared packet today.
1 RYD-0042 0.997 not responsive not responsive Move the portable whiteboard back after the job fair and wipe off the employer notes. Thanks.
1 RYD-0157 0.996 responsive vocabulary mismatch responsive Keep the missed-check note out of the records room until after the walkthrough. Thanks.
1 RYD-0029 0.996 not responsive not responsive Monday update: Ask someone to check whether the lab tablets charged overnight before learners sign them out.
1 RYD-0089 0.994 responsive vocabulary mismatch responsive Reminder: Tell records the rough chronology is a working copy and not ready to share.
1 RYD-0095 0.993 not responsive not responsive Reminder: Please ask maintenance about the squeaky hinge in the east hallway before tonight's open clinic.
2 RYD-0078 0.976 not responsive not responsive Keep the orange table signs with outreach materials. They match the flyers.
2 RYD-0122 0.973 responsive near duplicate responsive copy Please note: Keep the rough yard timeline away from the shared packet today.
2 RYD-0194 0.910 not responsive not responsive Use my desk copy of the table layout if the printed plan disappears. Thanks.
2 RYD-0079 0.891 not responsive not responsive If Fiona confirms availability, update the mentor roster before the newsletter draft closes. Call me if anything is unclear.
2 RYD-0187 0.880 not responsive not responsive Please keep the forklift practice cones by the south wall so the morning route stays clear.
2 RYD-0101 0.856 not responsive not responsive Following up: Tell the opening team about the lobby heater if it is still running at 7:45.
2 RYD-0203 0.847 not responsive not responsive If the portal times out, save the student form again; call IT after that. Thanks.
2 RYD-0130 0.847 not responsive not responsive Chat at noon: I answered the vendor's question about where to unload snacks during the rain.
2 RYD-0056 0.836 not responsive not responsive Reminder: Use the lapel microphone if the instructor needs it during the auditorium welcome.
2 RYD-0044 0.779 not responsive not responsive The shuttle can carry the evening mentors today if they meet by the library. Thanks. 
knitr::kable(
  missed_records,
  format = "html",
  escape = TRUE,
  col.names = c("Document", "Construction note", "Text"),
  caption = "Fiction-only responsive records still unreviewed at the stop",
  row.names = FALSE
)
Fiction-only responsive records still unreviewed at the stop
Document Construction note Text
RYD-0016 vocabulary mismatch responsive Chat at noon: Do not put the Tuesday problem in the customer packet yet.
RYD-0110 vocabulary mismatch responsive Keep the red binder off the shared table until the visitor leaves. Call me if anything is unclear.
RYD-0123 vocabulary mismatch responsive Ticket note: The orange tab has the gap; send the white tab if anyone asks today.
RYD-0127 near duplicate responsive copy For today: Set aside the marked route sheet until after the customer visit.
RYD-0146 vocabulary mismatch responsive Forwarding from intake: Please scrub the yard note before the noon packet goes out.
RYD-0154 vocabulary mismatch responsive Following up: Use the clean timeline in the client packet and park the rough one elsewhere.
RYD-0165 near duplicate responsive original For today: Set the marked route sheet aside until the customer visit is over.
RYD-0177 vocabulary mismatch responsive The side folder has the version with the missing Tuesday stop.
RYD-0236 vocabulary mismatch responsive Per scheduling: The file export should hide the late safety check until we settle the wording.

The team can see how many responsive records it has found, because reviewers code those records as they review them. It cannot see recall in a real matter without more assessment. The recall column is labeled as fiction-only because it uses the hidden labels for records that were not reviewed. In the two scoring rounds, 137 of 392 scored-pool rows were part of exact score ties; ties are ordered by document ID so the run is reproducible, not because lower IDs are more relevant. In ranked batch 2, a tie crossed the batch edge, so the document-ID rule decided which of the tied records were reviewed.

ggplot(
  review_progress,
  aes(x = cumulative_reviewed, y = cumulative_responsive_found)
) +
  geom_line(linewidth = 0.8) +
  geom_point(size = 2) +
  scale_x_continuous(
    breaks = c(0, review_progress$cumulative_reviewed),
    limits = c(0, max(review_progress$cumulative_reviewed))
  ) +
  scale_y_continuous(
    breaks = c(0, review_progress$cumulative_responsive_found),
    limits = c(0, max(review_progress$cumulative_responsive_found))
  ) +
  labs(
    x = "Records reviewed",
    y = "Responsive records found"
  ) +
  theme_minimal()
Line chart of responsive records found against records reviewed; both axes start at zero. The line runs from 22 found at 39 reviewed to 26 at 49 and 27 at 59.
Figure 1: Responsive records found as review proceeds through the keyword seed and two ranked batches.

The gain curve is the reviewer-visible curve: responsive found versus records reviewed. It does not claim that ranked review beats keyword search. The collection was written by the same person who wrote the lesson, so comparative claims from it would be weak evidence.

Sample the unreviewed pile once

An elusion sample reviews a random sample from the documents that were not reviewed. The seed and sample size are declared before sampling. The sample is drawn once and is not fed back into the model or the stopping decision.

The interval below treats the unreviewed pile as a finite population sampled without replacement. It tests every possible number of missed responsive records, keeps the values that would not put the observed sample count in either 2.5 percent tail under phyper(), then converts missed-record bounds to recall bounds.

elusion_seed <- 7401L
elusion_n <- 50L

set.seed(elusion_seed)
elusion_sample <- unreviewed_records |>
  slice_sample(n = elusion_n) |>
  arrange(doc_id)

observed_missed_in_sample <- sum(elusion_sample$reference_responsive)
unreviewed_total <- nrow(unreviewed_records)
found_total <- sum(reviewed_records$reference_responsive)

possible_missed <- 0:unreviewed_total
kept_missed <- possible_missed[
  vapply(
    possible_missed,
    \(missed) {
      lower_tail <- phyper(
        observed_missed_in_sample,
        missed,
        unreviewed_total - missed,
        elusion_n
      )
      upper_tail <- 1 - phyper(
        observed_missed_in_sample - 1L,
        missed,
        unreviewed_total - missed,
        elusion_n
      )
      lower_tail >= 0.025 && upper_tail >= 0.025
    },
    logical(1)
  )
]

missed_interval <- range(kept_missed)
recall_interval <- found_total / (found_total + rev(missed_interval))
point_missed <- unreviewed_total *
  observed_missed_in_sample / elusion_n
point_recall <- found_total / (found_total + point_missed)
known_recall <- found_total /
  (found_total + sum(unreviewed_records$reference_responsive))

elusion_result <- tibble(
  unreviewed = unreviewed_total,
  sample_size = elusion_n,
  responsive_in_sample = observed_missed_in_sample,
  point_recall = point_recall,
  recall_low = recall_interval[1],
  recall_high = recall_interval[2],
  fiction_known_recall = known_recall
)

binomial_interval <- stats::binom.test(
  observed_missed_in_sample,
  elusion_n
)$conf.int
binomial_missed <- unreviewed_total * binomial_interval
binomial_recall <- found_total / (found_total + rev(binomial_missed))
maximum_possible_recall <- found_total /
  (found_total + observed_missed_in_sample)

binomial_warning <- tibble(
  method = "binomial Clopper-Pearson on sampled elusion rate",
  printed_recall_low = binomial_recall[1],
  printed_recall_high = binomial_recall[2],
  maximum_possible_recall = maximum_possible_recall
)

knitr::kable(
  elusion_result |>
    mutate(across(where(is.numeric), \(value) round(value, 3))),
  col.names = c(
    "Unreviewed",
    "Sample size",
    "Responsive in sample",
    "Point recall",
    "Interval low",
    "Interval high",
    "Fiction-known recall"
  ),
  caption = "One finite-population elusion sample after the stop",
  row.names = FALSE
)
One finite-population elusion sample after the stop
Unreviewed Sample size Responsive in sample Point recall Interval low Interval high Fiction-known recall
181 50 2 0.789 0.551 0.931 0.75 
knitr::kable(
  binomial_warning |>
    mutate(across(where(is.numeric), \(value) round(value, 3))),
  col.names = c(
    "Method",
    "Printed recall low",
    "Printed recall high",
    "Maximum possible recall after the sample"
  ),
  caption = "A binomial interval treats the fixed unreviewed pile as if it were very large",
  row.names = FALSE
)
A binomial interval treats the fixed unreviewed pile as if it were very large
Method Printed recall low Printed recall high Maximum possible recall after the sample
binomial Clopper-Pearson on sampled elusion rate 0.521 0.968 0.931

The elusion seed is 7401, the builder’s base seed, fixed by rule before the sample was drawn. The sample found 2 responsive records in 50. The point estimate scales that to about 7.2 missed records in the unreviewed pile, and the exact finite-population interval still allows 2 to 22 missed records. That becomes recall from about 0.551 to 0.931. The fiction reveals that the true missed count is 9.

A binomial Clopper-Pearson interval treats the pile as if it were very large and samples with replacement. Here it gives a wider lower recall bound than the finite-population interval. Confidence intervals describe a repeated-sampling procedure; they are not a 95 percent chance statement about this one interval.

Repeatedly sampling, reviewing more records, and sampling again until the result looks acceptable would bias the report. The sample belongs after the stop, once.

Monitor new items in arrival order

Media monitoring has a different shape. Items arrive over time, duplicate wire copies appear, and a name like Riverton can refer to more than one place. This short stream watches organizations and program names, not individual people. It stores item IDs, decisions, and duplicate links; real review sets may contain personal or privileged material, which this lesson leaves out.

This use of “monitoring” is not lesson 47’s deployed-model monitoring, and it is not burst detection from lesson 56. It is an alerting rule over incoming text.

The watch phrases below were written after reading this constructed stream, so the alert count shows bookkeeping rather than transferable precision. The pattern watches Calder Yard, contract records, inspection messages, route notes, and missed check. A repeated alert is suppressed only when its character-trigram Jaccard distance to a prior alert is at most 0.35.

normalize_stream_text <- function(text) {
  text |>
    str_to_lower(locale = "en") |>
    str_replace_all("[^a-z0-9]+", " ") |>
    str_squish()
}

stream_decisions <- monitoring_stream |>
  arrange(arrival_order) |>
  mutate(
    normalized_text = normalize_stream_text(text),
    topic_hit = str_detect(
      text,
      regex("Calder Yard|contract records|inspection messages|route notes|missed check", ignore_case = TRUE)
    ),
    ambiguous_riverton = str_detect(text, regex("\\bRiverton\\b", ignore_case = TRUE)) &
      !str_detect(text, regex("Calder|Skills Centre|Workforce Lab", ignore_case = TRUE)),
    raw_alert = topic_hit & !ambiguous_riverton
  )

previous_alerts <- character()
previous_alert_ids <- character()
duplicate_of <- rep(NA_character_, nrow(stream_decisions))
send_alert <- rep(FALSE, nrow(stream_decisions))

for (row in seq_len(nrow(stream_decisions))) {
  if (!stream_decisions$raw_alert[[row]]) {
    next
  }

  if (length(previous_alerts) == 0L) {
    send_alert[[row]] <- TRUE
    previous_alerts <- c(previous_alerts, stream_decisions$normalized_text[[row]])
    previous_alert_ids <- c(previous_alert_ids, stream_decisions$stream_id[[row]])
    next
  }

  distances <- stringdist(
    stream_decisions$normalized_text[[row]],
    previous_alerts,
    method = "jaccard",
    q = 3
  )
  if (min(distances) <= 0.35) {
    duplicate_of[[row]] <- previous_alert_ids[[which.min(distances)]]
  } else {
    send_alert[[row]] <- TRUE
    previous_alerts <- c(previous_alerts, stream_decisions$normalized_text[[row]])
    previous_alert_ids <- c(previous_alert_ids, stream_decisions$stream_id[[row]])
  }
}

stream_decisions <- stream_decisions |>
  mutate(
    matched_prior_alert = duplicate_of,
    alert_sent = send_alert,
    decision = case_when(
      alert_sent ~ "alert sent",
      raw_alert & !is.na(matched_prior_alert) ~ "suppressed duplicate",
      ambiguous_riverton ~ "needs review: ambiguous Riverton",
      TRUE ~ "no alert"
    )
  )

monitoring_results <- tibble(
  alerts_sent = sum(stream_decisions$alert_sent),
  relevant_alerts = sum(stream_decisions$alert_sent & stream_decisions$reference_alert),
  false_alerts = sum(stream_decisions$alert_sent & !stream_decisions$reference_alert),
  relevant_items_missed = sum(!stream_decisions$alert_sent & stream_decisions$reference_alert & is.na(stream_decisions$matched_prior_alert)),
  suppressed_duplicates = sum(!is.na(stream_decisions$matched_prior_alert)),
  repeated_alerts = sum(stream_decisions$alert_sent & !is.na(stream_decisions$near_duplicate_of)),
  alert_precision = relevant_alerts / alerts_sent
)

stream_display <- stream_decisions |>
  select(
    arrival_order,
    stream_id,
    near_duplicate_of,
    matched_prior_alert,
    text,
    decision,
    reference_alert
  ) |>
  mutate(
    near_duplicate_of = coalesce(near_duplicate_of, "not a copy"),
    matched_prior_alert = coalesce(matched_prior_alert, "not suppressed")
  )

knitr::kable(
  stream_display,
  format = "html",
  escape = TRUE,
  col.names = c("Arrival", "Item", "Fixture duplicate of", "Matched prior alert", "Text", "Rule decision", "Reference alert"),
  caption = "Monitoring items are processed in arrival order",
  row.names = FALSE
)
Monitoring items are processed in arrival order
Arrival Item Fixture duplicate of Matched prior alert Text Rule decision Reference alert
1 MON-008 not a copy not suppressed Riverton Skills Centre posts a reminder about fall enrollment. no alert FALSE
2 MON-003 not a copy not suppressed Calder Yard contractor confirms a safety walkthrough next week. alert sent TRUE
3 MON-024 MON-003 MON-003 Syndicated brief: Calder Yard contractor confirms safety walkthrough next week. suppressed duplicate TRUE
4 MON-004 not a copy not suppressed Riverton Tidewater hosts a harbour festival unrelated to the Calder city program. no alert FALSE
5 MON-012 not a copy not suppressed A vendor post says a delayed inspection log at Calder Yard is under review. alert sent TRUE
6 MON-010 not a copy not suppressed Community calendar lists a classroom inspection drill at Riverton Skills Centre. no alert FALSE
7 MON-015 MON-012 not suppressed Wire copy: delayed inspection log at Calder Yard is under review. alert sent TRUE
8 MON-002 not a copy not suppressed City notice says bus passes remain available for workforce classes. no alert FALSE
9 MON-006 not a copy not suppressed A post mentions Riverton without enough context to pick the city or the town. needs review: ambiguous Riverton FALSE
10 MON-021 not a copy not suppressed Maintenance bulletin reports new guardrail signs at Calder Yard. alert sent FALSE
11 MON-018 MON-021 MON-021 Syndicated maintenance bulletin reports new guardrail signs at Calder Yard. suppressed duplicate FALSE
12 MON-014 not a copy not suppressed Local brief says contract records will be audited after a missed check. alert sent TRUE
13 MON-017 not a copy not suppressed Training newsletter asks volunteers to inspect donated laptops. no alert FALSE
14 MON-007 MON-014 not suppressed Vendor post repeats that contract records will be audited after a missed check. alert sent TRUE
15 MON-013 not a copy not suppressed Riverton Calder council posts a meeting agenda about yard procurement. no alert FALSE
16 MON-019 not a copy not suppressed Harbour blog writes about Riverton Tidewater seafood trucks. needs review: ambiguous Riverton FALSE
17 MON-011 not a copy not suppressed Safety clerk account says old route notes should not be shared. alert sent TRUE
18 MON-009 not a copy not suppressed Civic notice announces a parking delay near the Calder library. no alert FALSE
19 MON-023 MON-009 not suppressed Syndicated civic notice announces a parking delay near the Calder library. no alert FALSE
20 MON-001 not a copy not suppressed Workforce Lab posts a non-alert reminder about child care during evening classes. no alert FALSE
21 MON-005 not a copy not suppressed Calder Yard contractor denies deleting any inspection messages. alert sent TRUE
22 MON-016 MON-005 MON-005 Wire copy: Calder Yard contractor denies deleting any inspection messages. suppressed duplicate TRUE
23 MON-022 not a copy not suppressed Riverton school board posts a classroom inspection checklist. needs review: ambiguous Riverton FALSE
24 MON-020 not a copy not suppressed A regional item says Riverton won a rowing race, with no contract context. needs review: ambiguous Riverton FALSE 
knitr::kable(
  monitoring_results |>
    mutate(alert_precision = round(alert_precision, 3)),
  col.names = c(
    "Alerts sent",
    "Relevant alerts",
    "False alerts",
    "Relevant items missed",
    "Suppressed duplicates",
    "Repeated alerts",
    "Alert precision"
  ),
  caption = "Alert precision uses alerts sent after duplicate suppression as the denominator",
  row.names = FALSE
)
Alert precision uses alerts sent after duplicate suppression as the denominator
Alerts sent Relevant alerts False alerts Relevant items missed Suppressed duplicates Repeated alerts Alert precision
8 7 1 0 3 2 0.875

Alert precision is 7 relevant alerts out of 8 alerts sent. The denominator is alerts after duplicate suppression. Suppressed copies still count in the audit trail, but they do not consume a new alert slot. In a real stream, missed items are usually found by sampling non-alerts, the same idea as the elusion sample.

The ambiguity rule is crude. It sends MON-019, the Tidewater seafood item, to review even though the text itself points away from Calder Yard.

What this does not show

This page does not give legal advice, prove a review is defensible, or estimate a real system’s recall. It also leaves out privilege review, redaction, email threading, family coding, dense retrieval, and language-model relevance labeling. The labels are perfectly consistent because the author wrote them; real reviewers disagree.

The review model’s scores are ranks for this workflow, not calibrated probabilities of responsiveness. The monitoring rule is a small constructed example and is not suitable for watching individual people.

What to remember

  • Keyword precision is visible after reviewing hits; keyword recall is not visible in real work without more assessment.
  • Ranked review must train on reviewed records and score only unreviewed records.
  • A gain curve shows responsive records found, not recall, unless a complete fictional answer key is being used.
  • Draw an elusion sample once, after the stop.
  • A finite-population interval matches a fixed pile sampled without replacement.
  • Monitoring streams need arrival order, duplicate suppression, ambiguity handling, and a named precision denominator.

The review log goes forward with 2 sampled responsive records and a wide interval. The uncomfortable range is the lesson: stopping review does not make the unreviewed pile disappear.

Sources