Selecting sentences as a summary

Test TextRank against simple nulls

signals and discovery
extractive summarization
inaugural addresses
Learn how extractive summarization ranks existing sentences and why chance baselines still matter.

Jon has ten minutes before a reading-group meeting and one speech still open on his laptop. He does not need a new paragraph written for him. He needs to know which existing sentences deserve a first look.

That task has a name. Extractive summarization selects sentences already present in a text. It does not write new sentences, so its output can be checked against the source line by line.

TipWhat you will learn

This lesson shows how to:

  • define extractive summarization;
  • build the two input tables required by textrank_sentences();
  • rank sentences from one speech;
  • compare the result with random-position baselines; and
  • state what cannot be measured without a reference summary.

Build the sentence and term tables

TextRank is a graph-ranking algorithm. It treats sentences as nodes, the items in a graph that can be connected to other items. Sentences connect when they share terms. The function needs a data table with textrank_id and sentence, plus a terminology table with textrank_id and term.

library(dplyr)
library(tidyr)
library(tibble)
library(stringr)
library(tidytext)
library(tokenizers)
library(textrank)
library(knitr)

source("R/inaugural-corpus.R")
source("R/permutation-null.R")

paragraphs <- inaugural_paragraphs()

speeches <- paragraphs |>
  summarise(text = paste(paragraph, collapse = "\n"), .by = c(speech_id, year, president, party))

speech <- speeches |>
  filter(speech_id == "1933-Roosevelt")

sentence_vec <- tokenize_sentences(
  speech$text,
  strip_punct = FALSE,
  simplify = TRUE
)

sentence_data <- tibble(
  sentence_position = seq_along(sentence_vec),
  textrank_id = sprintf("s%02d", sentence_position),
  sentence = sentence_vec
) |>
  mutate(sentence_words = str_count(sentence, "[A-Za-z]+"))

terminology <- sentence_data |>
  select(textrank_id, sentence) |>
  unnest_tokens(term, sentence) |>
  filter(str_detect(term, "^[a-z]+$"), !term %in% stop_words$word)

input_summary <- tibble(
  table = c("data", "terminology", "distinct terms"),
  rows = c(nrow(sentence_data), nrow(terminology), n_distinct(terminology$term))
)

kable(
  input_summary,
  col.names = c("Input", "Rows"),
  caption = "Sentence and term inputs for TextRank",
  row.names = FALSE
)
Sentence and term inputs for TextRank
Input Rows
data 81
terminology 700
distinct terms 511

The data table has 81 sentences. The terminology table has 700 sentence-term rows and 511 distinct terms after stopwords are removed.

Rank the sentences

The algorithm gives each sentence a score based on its connections to other sentences. A sentence can rank high because it shares terms with many other sentences, or because it connects to other sentences that also rank high.

textrank_result <- textrank_sentences(
  data = sentence_data |>
    select(textrank_id, sentence),
  terminology = terminology |>
    select(textrank_id, term)
)

ranked_sentences <- textrank_result$sentences |>
  as_tibble() |>
  left_join(
    sentence_data |>
      select(textrank_id, sentence_position, sentence_words),
    by = join_by(textrank_id)
  ) |>
  arrange(desc(textrank), sentence_position)

textrank_summary <- ranked_sentences |>
  slice_head(n = 3) |>
  transmute(
    rank = row_number(),
    sentence_position,
    sentence_words,
    score = round(textrank, 4),
    sentence
  )

kable(
  textrank_summary,
  col.names = c("Rank", "Sentence position", "Sentence words", "TextRank score", "Selected sentence"),
  caption = "The three highest-ranked sentences in the 1933 speech",
  row.names = FALSE
)
The three highest-ranked sentences in the 1933 speech
Rank Sentence position Sentence words TextRank score Selected sentence
1 6 30 0.0328 In every dark hour of our national life a leadership of frankness and vigor has met with that understanding and support of the people themselves which is essential to victory.
2 67 24 0.0270 I am prepared under my constitutional duty to recommend the measures that a stricken nation in the midst of a stricken world may require.
3 60 25 0.0239 With this pledge taken, I assume unhesitatingly the leadership of this great army of our people dedicated to a disciplined attack upon our common problems.

The selected sentences come from positions 6, 67, and 60. That spread looks different from the opening of the speech, but a visible difference is not yet evidence that the algorithm found something rare.

Ask what random positions do

There is no reference summary for this speech in the corpus. That means this page cannot report accuracy. It can test checkable behavior: which sentences were selected, where they appear, and whether those positions look unusual compared with random sentence choices.

If human reference summaries existed, an overlap measure such as ROUGE could compare selected words or word sequences with those references. ROUGE would measure overlap, not whether the summary is complete, factual, or useful to a particular reader.

A lead baseline takes the first sentences. This baseline is often hard to beat in news writing because opening paragraphs often carry the main facts. Here it is only a comparison point.

A permutation null gives the same statistic after random shuffling. The p-value is the share of shuffled selections at least as extreme as the observed selection.

lead_baseline <- sentence_data |>
  slice_head(n = 3) |>
  transmute(
    baseline_rank = row_number(),
    sentence_position,
    sentence
  )

position_check <- tibble(
  method = c("TextRank", "Lead baseline"),
  selected_positions = c(
    paste(textrank_summary$sentence_position, collapse = ", "),
    paste(lead_baseline$sentence_position, collapse = ", ")
  ),
  earliest_position = c(
    min(textrank_summary$sentence_position),
    min(lead_baseline$sentence_position)
  ),
  latest_position = c(
    max(textrank_summary$sentence_position),
    max(lead_baseline$sentence_position)
  )
)

overlap_count <- sum(
  textrank_summary$sentence_position %in% lead_baseline$sentence_position
)

lead_overlap_null <- permutation_null(
  observed = overlap_count,
  replicate_fn = function() sum(sample(sentence_data$sentence_position, 3) %in% lead_baseline$sentence_position),
  replicates = 1000L,
  seed = 5401L,
  alternative = "less"
)

spread_observed <- max(textrank_summary$sentence_position) - min(textrank_summary$sentence_position)

spread_null <- permutation_null(
  observed = spread_observed,
  replicate_fn = function() {
    random_positions <- sample(sentence_data$sentence_position, 3)
    max(random_positions) - min(random_positions)
  },
  replicates = 1000L,
  seed = 5402L,
  alternative = "greater"
)

mean_length_observed <- mean(textrank_summary$sentence_words)

length_null <- permutation_null(
  observed = mean_length_observed,
  replicate_fn = function() mean(sentence_data$sentence_words[sample(sentence_data$sentence_position, 3)]),
  replicates = 1000L,
  seed = 5403L,
  alternative = "greater"
)

null_checks <- bind_rows(
  lead_overlap_null |> mutate(statistic = "Overlap with lead-3"),
  spread_null |> mutate(statistic = "Position range"),
  length_null |> mutate(statistic = "Mean selected sentence words")
) |>
  select(statistic, observed, null_mean, null_low, null_high, p_value, replicates, alternative) |>
  mutate(across(c(observed, null_mean, null_low, null_high, p_value), \(x) round(x, 3)))

kable(
  lead_baseline,
  col.names = c("Lead rank", "Sentence position", "Selected sentence"),
  caption = "The first three sentences as a lead baseline",
  row.names = FALSE
)
The first three sentences as a lead baseline
Lead rank Sentence position Selected sentence
1 1 I am certain that my fellow Americans expect that on my induction into the Presidency I will address them with a candor and a decision which the present situation of our Nation impels.
2 2 This is preeminently the time to speak the truth, the whole truth, frankly and boldly.
3 3 Nor need we shrink from honestly facing conditions in our country today. 
kable(
  position_check,
  col.names = c("Method", "Selected positions", "Earliest", "Latest"),
  caption = "A position check for the extracted sentences",
  row.names = FALSE
)
A position check for the extracted sentences
Method Selected positions Earliest Latest
TextRank 6, 67, 60 6 67
Lead baseline 1, 2, 3 1 3 
kable(
  null_checks,
  col.names = c("Statistic", "Observed", "Null mean", "Null low", "Null high", "p-value", "Replicates", "Alternative"),
  caption = "Random-position checks for the TextRank selection",
  row.names = FALSE
)
Random-position checks for the TextRank selection
Statistic Observed Null mean Null low Null high p-value Replicates Alternative
Overlap with lead-3 0.000 0.113 0.000 1.00 0.889 1000 less
Position range 61.000 41.263 12.000 70.00 0.189 1000 greater
Mean selected sentence words 26.333 22.761 11.667 38.35 0.297 1000 greater

The two selections share 0 sentence positions. That says almost nothing by itself: random three-sentence selections from an 81-sentence speech usually share no positions with sentences 1 to 3. In 1,000 shuffles, the null mean was 0.113 and the p-value for unusually low overlap was 0.889.

The more useful checks ask whether TextRank spread its picks through the speech or chose longer sentences. The observed range was 61 sentence positions, against a random mean of 41.346 (p = 0.189). The selected sentences averaged 26.33 words, against a random mean of 22.771 (p = 0.297). Neither check clears a chance baseline, so the honest conclusion is modest: TextRank gives a reading list, not evidence that these three sentences are better.

Summarizers always return sentences. A null check asks what the same selection statistic looks like after document position is stripped away. Without that comparison, a neat list can look more meaningful than it is.

Nothing here cleared a baseline, and that is still a result worth having. The method ran, the output is readable, and three separate checks told you how much weight it will carry. Knowing that before you put a summary in front of someone is worth more than a number that flatters the tool. Extractive summarization earns its place when a person is going to read the selected sentences and judge them; these checks say it has not earned the right to run unattended on this material.

What to remember

  • Extractive summarization selects sentences already in the text.
  • textrank_sentences() needs a sentence table and a term table.
  • The 1933 speech produced 81 sentence rows and 700 sentence-term rows.
  • Zero overlap with the lead-3 baseline is what random selection usually produces here.
  • Without a reference summary, accuracy is not available.

Use the ranked sentences as a reading aid. Do not turn them into a score for what the speech means.

Sources