Finding paragraph boundaries

Keep layout units before text conversion removes them

sentences and paragraphs
paragraph segmentation
workforce research
Learn how blank lines and HTML block elements define paragraphs in different file formats.

Help articles and job-board HTML reach the coordinator as readable text, yet their paragraph breaks live in different places. Comparing them requires keeping those boundaries before cleanup erases them.

In a plain-text help article, a blank line can mark a new paragraph. HTML gives the team block elements it can use as paragraph boundaries. The standard’s own definition of a paragraph is looser than a <p> element, so this is a working rule for the lesson rather than the specification’s answer.

Note

The help articles and job board are local fictional fixtures.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • define a paragraph as a layout unit;
  • split plain text on blank lines;
  • collect paragraph-like blocks from HTML with rvest;
  • compare paragraph counts across file formats; and
  • explain why block boundaries should be captured before tag removal.

Split plain text on blank lines

This section reads files with readr, repeats file work with purrr, splits text with stringr, expands list columns with tidyr, shapes tables with dplyr and tibble, and asks rvest for HTML. A paragraph is a block of text grouped by layout. It may contain one sentence or several.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tidyr)
library(rvest)

help_files <- list.files(
  "data/help_articles",
  pattern = "[.]txt$",
  full.names = TRUE
) |>
  sort()

split_blank_lines <- function(text) {
  str_split(str_trim(text), "\\r?\\n\\s*\\r?\\n+")[[1]] |>
    str_trim() |>
    discard(\(piece) identical(piece, ""))
}

help_articles <- tibble(
  file = basename(help_files),
  text = map_chr(help_files, read_file)
) |>
  mutate(paragraph = map(text, split_blank_lines))

help_paragraphs <- help_articles |>
  select(file, paragraph) |>
  unnest_longer(paragraph) |>
  group_by(file) |>
  mutate(paragraph_number = row_number()) |>
  ungroup()

help_counts <- help_paragraphs |>
  count(file, name = "paragraphs")

total_help_paragraphs <- sum(help_counts$paragraphs)
help_blank_line_counts <- str_count(help_articles$text, "\\r?\\n\\s*\\r?\\n+")

knitr::kable(
  help_counts,
  col.names = c("Help article", "Paragraphs from blank-line rule"),
  caption = "Paragraph counts in the plain-text help articles",
  row.names = FALSE
)
Paragraph counts in the plain-text help articles
Help article Paragraphs from blank-line rule
account-access.txt 1
delivery.txt 1
returns.txt 1 
knitr::kable(
  help_paragraphs,
  col.names = c("Help article", "Paragraph", "Paragraph number"),
  caption = "Plain-text paragraphs after splitting on blank lines",
  row.names = FALSE
)
Plain-text paragraphs after splitting on blank lines
Help article Paragraph Paragraph number
account-access.txt Use the password reset link on the sign-in page. The link expires after 30 minutes for your security. 1
delivery.txt Standard delivery takes three to five business days. Tracking appears after the parcel leaves our warehouse. 1
returns.txt You can return an unused item within 30 days. Keep the receipt and use the prepaid label included with your order. 1

None of these help files contains a blank line, so the rule has nothing to split on and returns each file whole. That is a fair result but an empty demonstration. The HTML section below is where the rule has to make a choice.

Take blocks from HTML

HTML records layout with elements. The code below reads the job-board page and keeps headings, paragraphs, and list items. The selector h1, h2, p, li names the block types the team wants to treat as paragraph-like units.

job_page <- read_html("data/workforce/job-board.html")
block_nodes <- html_elements(job_page, "h1, h2, p, li")

html_blocks <- tibble(
  tag = html_name(block_nodes),
  text = html_text2(block_nodes)
) |>
  mutate(block_number = row_number())

html_tag_counts <- html_blocks |>
  count(tag, name = "blocks") |>
  arrange(tag)

expected_tag_counts <- tibble(
  tag = c("h1", "h2", "li", "p"),
  blocks = c(1L, 6L, 22L, 12L)
)

knitr::kable(
  html_tag_counts,
  col.names = c("HTML tag", "Blocks captured"),
  caption = "Paragraph-like blocks captured from the job-board HTML",
  row.names = FALSE
)
Paragraph-like blocks captured from the job-board HTML
HTML tag Blocks captured
h1 1
h2 6
li 22
p 12 
knitr::kable(
  html_blocks |>
    slice_head(n = 8),
  col.names = c("HTML tag", "Text", "Block number"),
  caption = "The first eight block elements from the job board",
  row.names = FALSE
)
The first eight block elements from the job board
HTML tag Text Block number
h1 Riverton workforce opportunities 1
h2 Data analyst trainee 2
p Riverton Community Health 3
p Riverton 4
li Paid 12-week training is provided. 5
li No prior data experience is required. 6
li Evening schedules are available. 7
li Applicants need basic spreadsheet skills. 8

The HTML rule finds 41 blocks: 1 page heading, 6 job headings, 12 paragraph elements, and 22 list items. The job-detail sentences in the Riverton CSV came from those list items.

Show what tag removal loses

A common conversion step removes tags and keeps only the visible words. That can be useful for word counts, but it is too late for paragraph segmentation if the block marks are gone.

html_source <- read_file("data/workforce/job-board.html")
stripped_before_squish <- html_source |>
  str_remove_all("<[^>]+>")

paragraphs_after_tag_removal <- split_blank_lines(stripped_before_squish)
paragraphs_after_squish <- stripped_before_squish |>
  str_squish() |>
  split_blank_lines()

boundary_loss <- tibble(
  source = c(
    "HTML block elements",
    "Tags removed, whitespace preserved",
    "Tags removed, whitespace squished"
  ),
  paragraph_like_units = c(
    nrow(html_blocks),
    length(paragraphs_after_tag_removal),
    length(paragraphs_after_squish)
  )
)

knitr::kable(
  boundary_loss,
  col.names = c("Rule", "Paragraph-like units"),
  caption = "Whitespace cleanup collapses the tag-stripped text",
  row.names = FALSE
)
Whitespace cleanup collapses the tag-stripped text
Rule Paragraph-like units
HTML block elements 41
Tags removed, whitespace preserved 14
Tags removed, whitespace squished 1

Tag removal alone leaves 14 blank-line chunks because the source file still has line breaks and indentation. Those chunks are not paragraphs; they are source layout around titles, job cards, and lists. str_squish() removes the remaining blank-line boundaries and leaves one long text string. For this file, paragraph structure has to be captured before whitespace cleanup.

What to remember

  • A paragraph is a layout block, so the right rule depends on the file format.
  • These help files have no blank lines, so the blank-line rule cannot show much.
  • The chosen HTML selector captures 41 block-like units, including headings and list items.
  • Squishing whitespace after tag removal collapses the text to one unit.

The help files and the job-board HTML need different boundary rules. Here the reported counts are layout counts, not a grammar claim about where paragraphs must begin.

Sources