Collecting text from a web page

Scrape a saved job board without losing its source

source data loading
scraping
workforce research
Learn how to extract job titles and training details from structured HTML with R.

The fictional Riverton Workforce Lab begins with six job postings saved from a job board. A worker comparing the notices would need to keep each training offer, schedule, experience requirement, and named skill attached to the right job. Copying the lines into one block would make omissions and source mix-ups hard to audit.

Web scraping uses code to collect information from web pages. The team will scrape the saved page, keep the job and sentence identifiers, and check every record before it becomes research data. This first stage collects what the notices say; it does not classify the sentences or generalize from them.

The whole lesson works on static archived HTML: a file whose text is already in the markup when it is saved. That is the scope of what follows, and the last sections show two ordinary pages where the method stops working.

Note

The employers, postings, and Riverton Workforce Lab are fictional. The HTML file was written for this lesson and contains no personal information.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain how HTML gives parts of a page names and structure;
  • select repeated job cards with rvest;
  • keep source identifiers beside extracted text;
  • compare scraped output with an expected fixture;
  • notice a renamed class and a page whose text is not in the markup; and
  • identify legal, ethical, and technical checks needed before live scraping.

Begin with an archived page

Live websites change. A saved source file lets another researcher inspect the same page later. The team records the file path, size, and fingerprint before extracting anything.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(digest)
library(rvest)

job_page_path <- "data/workforce/job-board.html"
job_page_lines <- read_lines(job_page_path)
job_page_fingerprint <- digest::digest(
  paste(job_page_lines, collapse = "\n"),
  algo = "md5",
  serialize = FALSE
)
job_page_metadata <- read_csv(
  "data/workforce/job-board-metadata.csv",
  col_types = cols(
    source_file = col_character(),
    created_for_lesson = col_character(),
    retrieval_status = col_character(),
    records = col_character(),
    md5 = col_character(),
    rights_note = col_character()
  )
)
# A size read from disk counts the line endings the local copy happens to
# use, so it changes between systems. Measuring the newline-joined text, the
# same text the fingerprint covers, describes the content instead.
job_page_size <- nchar(
  paste(job_page_lines, collapse = "\n"),
  type = "bytes"
)

tibble(
  source_file = job_page_path,
  bytes = job_page_size,
  fingerprint = job_page_fingerprint
)
# A tibble: 1 × 3
  source_file                   bytes fingerprint                     
  <chr>                         <int> <chr>                           
1 data/workforce/job-board.html  3441 2df73bc39f05311b6b59d24f52cb8f7b

The fingerprint is compared with a stored value. It changes if the archived HTML changes and can reveal an accidental edit. It cannot prove that the postings are true or complete.

Read the page structure

HTML uses elements such as headings, paragraphs, and lists to organize a page. The example page places every posting inside an element whose class is job-card. A CSS selector is a short pattern that names the elements to collect.

job_page <- rvest::read_html(job_page_path)
job_cards <- rvest::html_elements(
  job_page,
  ".job-card"
)

length(job_cards)
[1] 6

R found six job cards. This count is the first signal that the selector matches the archived page we expected.

Extract one field at a time

Each card contains a job ID, posting date, title, employer, and location. The small helper below extracts one text field from each card.

card_text <- function(cards, selector) {
  map_chr(
    cards,
    \(card) {
      elements <- card |>
        rvest::html_elements(selector)
      stopifnot(identical(length(elements), 1L))

      elements |>
        rvest::html_text2()
    }
  )
}

jobs <- tibble(
  job_id = rvest::html_attr(
    job_cards,
    "data-job-id"
  ),
  posted = as.Date(
    rvest::html_attr(
      job_cards,
      "data-posted"
    )
  ),
  title = card_text(job_cards, ".job-title"),
  employer = card_text(job_cards, ".employer"),
  location = card_text(job_cards, ".location")
)

knitr::kable(
  jobs,
  col.names = c(
    "Job ID",
    "Posted",
    "Title",
    "Employer",
    "Location"
  ),
  caption = "Six postings extracted from the archived job board",
  row.names = FALSE
)
Six postings extracted from the archived job board
Job ID Posted Title Employer Location
J001 2026-08-01 Data analyst trainee Riverton Community Health Riverton
J002 2026-08-03 Maintenance technician Riverton Housing Cooperative Riverton
J003 2026-08-05 Junior web developer Riverton Digital Services Hybrid
J004 2026-08-08 Medical records clerk Riverton Family Clinic Riverton
J005 2026-08-10 Solar installation apprentice Riverton Solar Works Riverton region
J006 2026-08-12 Customer support specialist Riverton Transit Riverton

The job ID remains attached to every title. Later, a schedule or experience requirement must still point back to the posting that stated it.

Keep sentences connected to their postings

The details in each card are list items with the class job-detail. We extract them in page order and add stable sentence IDs.

scraped_sentences <- map2(
  job_cards,
  jobs$job_id,
  \(card, job_id) {
    details <- card |>
      rvest::html_elements(".job-detail") |>
      rvest::html_text2()

    tibble(
      document_id = job_id,
      text = details
    )
  }
) |>
  list_rbind() |>
  mutate(
    sentence_id = sprintf("s%03d", row_number()),
    .before = document_id
  )

expected_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)
expected_job_sentences <- expected_sentences |>
  filter(document_id != "F001") |>
  select(sentence_id, document_id, text) |>
  as_tibble()

knitr::kable(
  scraped_sentences |>
    slice_head(n = 8),
  col.names = c("Sentence ID", "Job ID", "Text"),
  caption = "The first eight extracted job-detail sentences",
  row.names = FALSE
)
The first eight extracted job-detail sentences
Sentence ID Job ID Text
s001 J001 Paid 12-week training is provided.
s002 J001 No prior data experience is required.
s003 J001 Evening schedules are available.
s004 J001 Applicants need basic spreadsheet skills.
s005 J002 A high school diploma is required.
s006 J002 The employer pays for certification training.
s007 J002 Rotating night shifts are part of the job.
s008 J002 Workers must be able to lift 50 pounds.

The exact comparison succeeds: all 22 sentences match the fixture in the same order. This check would stop the build if a selector dropped, duplicated, or reordered text.

Both the HTML and expected table were written for this lesson, so the comparison tests parser behavior rather than independent truth. A live collection would need a separately captured audit sample.

When the page changes shape

The archived file is stable because it is a file. A live board is maintained by people who redesign it. The two pages below are written into the lesson as short HTML strings, and both would break the code above in ways worth telling apart.

The first is a redesign. The employer’s new template renames job-card to posting and moves the title into a heading.

redesigned_html <- str_c(
  "<html><body>",
  "<div class='posting' data-job-id='J007'>",
  "<h3 class='posting-title'>Warehouse trainee</h3>",
  "<ul><li class='posting-detail'>Paid training is provided.</li></ul>",
  "</div>",
  "</body></html>"
)

redesigned_page <- rvest::read_html(redesigned_html)
old_selector_hits <- length(
  rvest::html_elements(redesigned_page, ".job-card")
)
new_selector_hits <- length(
  rvest::html_elements(redesigned_page, ".posting")
)

collect_cards <- function(page, selector, expected) {
  cards <- rvest::html_elements(page, selector)
  if (!identical(length(cards), expected)) {
    stop(
      str_c(
        "selector ",
        selector,
        " found ",
        length(cards),
        " cards; expected ",
        expected
      ),
      call. = FALSE
    )
  }

  list(
    selector = selector,
    found = length(cards),
    expected = expected,
    status = "counts agree"
  )
}

drift_error <- tryCatch(
  collect_cards(redesigned_page, ".job-card", 1L),
  error = conditionMessage
)

knitr::kable(
  tibble(
    page = c("archived board", "redesigned page"),
    selector = ".job-card",
    cards_found = c(
      length(job_cards),
      old_selector_hits
    ),
    cards_expected = c(6L, 1L)
  ),
  col.names = c("Page", "Selector", "Cards found", "Cards expected"),
  caption = "The same selector against two versions of one site",
  row.names = FALSE
)
The same selector against two versions of one site
Page Selector Cards found Cards expected
archived board .job-card 6 6
redesigned page .job-card 0 1

Zero matches is the dangerous result, because zero matches is also what an empty page produces. Without the expected count, the scraper would write an empty table and the day’s collection would look like a day with no postings. An expected count turns an unreported redesign into a stop.

The second page is harder. Its visible container has no job text. The same file stores a JSON payload in a script element that application code could read and render later. This fixture does not execute that JavaScript; it separates visible markup from embedded data.

dynamic_html <- str_c(
  "<html><body>",
  "<div id='board'></div>",
  "<script type='application/json' id='board-data'>",
  '{"jobs": [{"id": "J008", "title": "Night stocker"}]}',
  "</script>",
  "</body></html>"
)

dynamic_page <- rvest::read_html(dynamic_html)
visible_container_text <- rvest::html_element(
  dynamic_page,
  "#board"
) |>
  rvest::html_text2()
payload_text <- rvest::html_element(
  dynamic_page,
  "#board-data"
) |>
  rvest::html_text2()

tibble(
  source = c("visible container", "script payload"),
  characters = c(
    str_length(visible_container_text),
    str_length(payload_text)
  ),
  mentions_the_job = c(
    str_detect(visible_container_text, fixed("Night stocker")),
    str_detect(payload_text, fixed("Night stocker"))
  )
)
# A tibble: 2 × 3
  source            characters mentions_the_job
  <chr>                  <int> <lgl>           
1 visible container          0 FALSE           
2 script payload            52 TRUE

The container holds no characters. The job title exists only in the embedded script payload, which rvest can read as text but does not render into the container.

This example is deliberately gentle because the payload is already in the file. Many sites fetch data separately after the page loads. A saved copy of their initial HTML may contain nothing to extract, and an HTML parser cannot recover a response that was never saved.

Three routes exist from there, and they differ in cost and in what they promise. A documented API returns the same data with stated terms, and is the first thing to look for. The underlying data request the page itself makes can sometimes be called directly, which is faster but relies on an interface nobody promised to keep. A headless browser, driven from R through chromote or from Python through Playwright or Selenium, runs the page as a browser would; it is the general answer and by far the most expensive to run and maintain.

Whichever route is chosen, record it. Text recovered from a rendered page and text read from a saved file are not the same evidence, and a reader deserves to know which one produced a number.

A saved page is still incomplete evidence

The archived HTML shows what was captured, not every posting that existed. Before scraping a live site, the team must check:

  1. the site’s terms and robots policy;
  2. whether collection is lawful and expected in the relevant jurisdiction;
  3. whether personal or sensitive information appears;
  4. how often requests may be sent;
  5. whether page content is loaded later by JavaScript;
  6. which dates, filters, and geographic limits shaped the results; and
  7. how the saved HTML and retrieval time will be preserved.

The Robots Exclusion Protocol communicates crawler preferences. It is not a grant of legal permission, and permission does not settle every ethical question. A public page can still contain information that should not be collected or republished.

What to remember

  • Scraping turns page structure into data; it does not establish that the data is true or representative.
  • Archive the source and record when it was collected.
  • Keep source IDs attached to extracted text.
  • Assert expected counts and content so selector failures cannot return zero rows without an error.
  • Zero matches and an empty page look identical without an expected count.
  • Some pages hold their text outside the markup, where an HTML parser cannot reach it.
  • Review terms, rate limits, privacy, and research ethics before live collection.

The team can now inspect 22 sentences without detaching them from the six notices. One source remains outside that table: a fictional training flyer saved as an image. That image shifts the investigation from HTML structure to OCR. Its source must still be preserved, and the extracted wording must be checked rather than treated as certain.

Sources