Loading data from an API

Ask an online service for information and check its reply

source data loading
API
JSON
Learn how an API request becomes a checked data table in R.

A librarian gathering possible books about natural language processing could copy search results from a catalog into a table. Search results are candidates, not recommendations. The librarian still needs to judge whether each record is relevant, accurate, and useful.

An application programming interface, usually called an API, gives software a defined way to ask another service for information. This lesson reconstructs a query sent to the Open Library Search API and checks the saved reply. The displayed table comes from a dated response so the published output can be reproduced even when the service is unavailable or its ranking changes.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain the request-and-response pattern used by an API;
  • add search choices to an API request;
  • check whether the request succeeded;
  • turn a JSON response into a table in R;
  • replay a rate limit, a second page, and a changed schema from saved responses; and
  • handle changing results, service limits, and secret API keys.

Start with a request and a reply

Every API exchange in this lesson has two parts:

  1. R sends a request to a web address.
  2. The service sends a response back.

The request records the librarian’s question. The response contains a status code, descriptive details called headers, and a body holding the requested data.

Our request uses this endpoint, the web address assigned to book searches: https://openlibrary.org/search.json.

Build the request

We build the request before sending it so each choice remains visible. The httr2 package adds those choices one step at a time.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(httr2)
library(digest)
library(htmltools)

book_request <- httr2::request(
  "https://openlibrary.org/search.json"
) |>
  httr2::req_url_query(
    q = "natural language processing",
    fields = str_c(
      c("key", "title", "author_name", "first_publish_year"),
      collapse = ","
    ),
    limit = 5
  ) |>
  httr2::req_user_agent(
    str_c(
      "periodic-table-nlp-r/1.0 ",
      "(https://github.com/farach/periodic_table_nlp)"
    )
  ) |>
  httr2::req_timeout(20) |>
  httr2::req_retry(max_tries = 3)

book_request
<httr2_request>
GET https://openlibrary.org/search.json?q=natural%20language%20processing&fields=key%2Ctitle%2Cauthor_name%2Cfirst_publish_year&limit=5
Body: empty
Options:
* useragent     : "periodic-table-nlp-r/1.0 (https://github.com/farach/periodic_table_nlp)"
* timeout_ms    : 20000
* connecttimeout: 0
Policies:
* retry_max_tries        : 3
* retry_on_failure       : FALSE
* retry_failure_threshold: Inf
* retry_failure_timeout  : 30
* retry_realm            : "openlibrary.org"

The printed request begins with GET, the standard method used here to ask a service to send information. Printing it lets us inspect the destination and search choices without contacting the service again.

The choices added to the address are called query parameters:

Query parameters used in the Open Library request
Parameter What it asks for
q Books related to the words “natural language processing.”
fields Only the four pieces of information used in this lesson.
limit No more than five results.

The user-agent identifies the project making the request. The 20-second timeout keeps R from waiting indefinitely. The retry setting permits up to three attempts when a temporary connection problem interrupts the exchange; it does not guarantee that an unavailable service will respond.

Understand what happened when the request was sent

The original request was sent once on the retrieval date recorded below. During this lesson, req_perform() receives the same request but uses a local mock that replays the saved response bytes. That exercises the request-to-response step without contacting Open Library again. A normal req_perform() call would stop on an unsuccessful HTTP status such as 404 or 500.

JSON, short for JavaScript Object Notation, is a text format that can hold named values, lists, and records. Many programming languages can read it, which makes it common in API responses.

Keep a dated response

One successful request cannot guarantee that the service will be available tomorrow or return the same ranking. The repository keeps the JSON response with its retrieval time, request address, HTTP status, content type, record level, licensing reference, and fingerprint.

fixture_path <- "data/openlibrary-nlp-search.json"
metadata_path <- "data/openlibrary-nlp-search-metadata.csv"

fixture_metadata <- read_csv(
  metadata_path,
  col_types = cols(
    retrieved_at_utc = col_character(),
    request_url = col_character(),
    http_status = col_character(),
    content_type = col_character(),
    record_level = col_character(),
    api_documentation = col_character(),
    licensing_reference = col_character(),
    md5 = col_character()
  )
)
fixture_lines <- read_lines(fixture_path)
fixture_md5 <- digest::digest(
  paste(fixture_lines, collapse = "\n"),
  algo = "md5",
  serialize = FALSE
)

knitr::kable(
  tibble(
    detail = c(
      "Retrieved at",
      "HTTP status",
      "Content type",
      "Record level",
      "File fingerprint"
    ),
    value = c(
      fixture_metadata$retrieved_at_utc,
      fixture_metadata$http_status,
      fixture_metadata$content_type,
      fixture_metadata$record_level,
      fixture_md5
    )
  ),
  col.names = c("Fixture detail", "Recorded value"),
  caption = "Provenance for the saved API response",
  row.names = FALSE
)
Provenance for the saved API response
Fixture detail Recorded value
Retrieved at 2026-08-28T12:18:04Z
HTTP status 200
Content type application/json
Record level work-level catalog record
File fingerprint d65543b9f71a664cf568c25a436ad614

The MD5 fingerprint detects an accidental change to the saved file. It is not proof that the source is accurate or trustworthy. The request address and API documentation remain in the metadata file for a later audit.

Perform the request against the saved reply

A mock replaces the network exchange with a local function. httr2 still passes the prepared request into req_perform() and returns an httr2_response; the mock supplies the dated status, header, and JSON body.

replay_saved_response <- function(req) {
  httr2::response(
    status_code = as.integer(fixture_metadata$http_status),
    url = req$url,
    method = "GET",
    headers = list(
      `content-type` = fixture_metadata$content_type
    ),
    body = charToRaw(paste(fixture_lines, collapse = "\n"))
  )
}

fixture_response <- httr2::req_perform(
  book_request,
  mock = replay_saved_response
)

response_summary <- tibble(
  detail = c("Response class", "HTTP status", "Content type"),
  value = c(
    class(fixture_response)[[1]],
    as.character(httr2::resp_status(fixture_response)),
    httr2::resp_header(fixture_response, "content-type")
  )
)

knitr::kable(
  response_summary,
  col.names = c("Response detail", "Value"),
  caption = "The prepared request returns a replayed httr2 response",
  row.names = FALSE
)
The prepared request returns a replayed httr2 response
Response detail Value
Response class httr2_response
HTTP status 200
Content type application/json

Turn the saved response into a table

Open Library places search results inside a JSON section named docs. resp_body_json() translates that section into rows and columns because this response has the same fields for each catalog record.

fixture_data <- httr2::resp_body_json(
  fixture_response,
  simplifyVector = TRUE
)
book_records <- as_tibble(fixture_data$docs)

format_authors <- function(author_names) {
  if (length(author_names) == 0 || all(is.na(author_names))) {
    return("Not listed")
  }

  str_c(author_names, collapse = ", ")
}

books <- book_records |>
  transmute(
    title = title,
    authors = map_chr(author_name, format_authors),
    first_published = first_publish_year,
    catalog_url = str_c("https://openlibrary.org", key)
  )

escape_html <- function(value) {
  as.character(htmltools::htmlEscape(value))
}

catalog_links <- map2_chr(
  books$catalog_url,
  books$title,
  \(url, title) {
    safe_url <- as.character(
      htmltools::htmlEscape(url, attribute = TRUE)
    )
    safe_title <- as.character(
      htmltools::htmlEscape(
        title,
        attribute = TRUE
      )
    )
    sprintf(
      '<a href="%s" aria-label="View Open Library record for %s">View record</a>',
      safe_url,
      safe_title
    )
  }
)

display_books <- books |>
  transmute(
    title = escape_html(title),
    authors = escape_html(authors),
    first_published = first_published,
    catalog_link = catalog_links
  )

knitr::kable(
  display_books,
  col.names = c(
    "Title",
    "Author or authors",
    "First published",
    "Open Library record"
  ),
  caption = str_c(
    "Five candidate work-level catalog records retrieved",
    " ",
    fixture_metadata$retrieved_at_utc
  ),
  row.names = FALSE,
  escape = FALSE
)
Five candidate work-level catalog records retrieved 2026-08-28T12:18:04Z
Title Author or authors First published Open Library record
Natural language processing and information retrieval Tanveer Siddiqui, U. S. Tiwary 2008 View record
Natural Language Processing With Python Edward Loper, Steven Bird, Ewan Klein 2009 View record
Practical Natural Language Processing Sowmya Vajjala, Bodhisattwa Majumder, Anuj Gupta, Harshit Surana 2020 View record
Natural Language Processing with Transformers Lewis Tunstall, Leandro von Werra, Thomas Wolf 2022 View record
Natural language processing Randall Rustin 1973 View record

Open Library returns work-level catalog records by default. A row can represent many editions of the same work. The table contains candidate search results, not a checked reading list or a judgment about a book’s quality.

The authors and titles are escaped before display so API text cannot become page markup. Only the validated Open Library addresses are rendered as links.

Live data can change

Open Library may add records, correct dates, or reorder results. Nothing in this page contacts the service while it builds, so the displayed table comes from the saved response and stays stable.

When the request is sent again, the checks worth applying describe a bounded structure rather than five specific titles:

  • the service returned a successful status;
  • the response is JSON;
  • the response contains between one and five records; and
  • every returned record has a title.

The saved fixture makes the displayed table repeatable. Its metadata records when and how it was retrieved. Open Library states that the Internet Archive does not assert new proprietary rights over the database, while warning that existing rights may vary by contribution and jurisdiction. Reusers should read the current licensing statement rather than assume a blanket license.

Respect the service’s rate limit and stated preference for low-volume, human-facing requests.

Replay the awkward responses

A successful reply is the easy case. The three awkward ones are a refusal, a result set too large for one reply, and a reply whose fields have moved. Each is reconstructed below from the status codes and body shapes the service documents, built inside the lesson with httr2::response(). No request leaves the machine.

saved_response <- function(
  status,
  body,
  url = book_request$url,
  extra_headers = list()
) {
  httr2::response(
    status_code = status,
    url = url,
    method = "GET",
    headers = c(
      list("content-type" = "application/json"),
      extra_headers
    ),
    body = charToRaw(body)
  )
}

make_docs <- function(keys, titles) {
  str_c(
    sprintf(
      '{"key": "%s", "title": "%s"}',
      keys,
      titles
    ),
    collapse = ", "
  )
}

page_one_request <- book_request |>
  httr2::req_url_query(offset = 0)
page_two_request <- book_request |>
  httr2::req_url_query(offset = 5)

rate_limited <- saved_response(
  429L,
  '{"error": "too many requests"}',
  url = page_one_request$url,
  extra_headers = list("retry-after" = "30")
)
date_limited <- saved_response(
  429L,
  '{"error": "too many requests"}',
  url = page_one_request$url,
  extra_headers = list(
    "retry-after" = "Wed, 21 Oct 2037 07:28:00 GMT"
  )
)

page_one <- saved_response(
  200L,
  str_c(
    '{"numFound": 7, "start": 0, "docs": [',
    make_docs(
      sprintf("/works/OL%d", 1:5),
      sprintf("Candidate title %d", 1:5)
    ),
    "]}"
  ),
  url = page_one_request$url
)

page_two <- saved_response(
  200L,
  str_c(
    '{"numFound": 7, "start": 5, "docs": [',
    make_docs(
      c("/works/OL5", "/works/OL6"),
      c("Candidate title 5", "Candidate title 6")
    ),
    "]}"
  ),
  url = page_two_request$url
)

field_renamed <- saved_response(
  200L,
  str_c(
    '{"numFound": 1, "start": 0, "docs": [',
    '{"key": "/works/OL7", "name": "Title moved to a new field"}',
    "]}"
  ),
  url = page_one_request$url
)

docs_missing <- saved_response(
  200L,
  '{"numFound": 1, "start": 0}',
  url = page_one_request$url
)

httr2::resp_status(rate_limited)
[1] 429

The reader now has several saved replies and one function that has to survive all of them.

required_fields <- c("key", "title")
required_top_fields <- c("numFound", "start", "docs")
retry_reference_time <- as.POSIXct(
  "2037-10-21 07:27:30",
  tz = "GMT"
)

parse_retry_after <- function(value, now) {
  if (length(value) == 0L || is.na(value) || identical(value, "")) {
    return(NA_integer_)
  }
  if (str_detect(value, "^[0-9]+$")) {
    return(as.integer(value))
  }

  parsed <- suppressWarnings(curl::parse_date(value))
  if (length(parsed) == 0L || is.na(parsed)) {
    stop("Retry-After is neither delay-seconds nor an HTTP-date")
  }

  max(
    0L,
    as.integer(ceiling(as.numeric(difftime(parsed, now, units = "secs"))))
  )
}

read_saved_page <- function(
  response,
  page_size,
  retry_now = retry_reference_time
) {
  status <- httr2::resp_status(response)

  if (identical(status, 429L)) {
    retry_after <- httr2::resp_header(response, "retry-after")
    return(list(
      outcome = "rate limited",
      request_url = response$url,
      wait_seconds = parse_retry_after(retry_after, retry_now),
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  if (!identical(status, 200L)) {
    return(list(
      outcome = "http error",
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  body <- httr2::resp_body_json(response)
  missing_top_fields <- setdiff(required_top_fields, names(body))
  if (length(missing_top_fields) > 0L) {
    return(list(
      outcome = str_c(
        "schema failure: missing top-level ",
        str_c(missing_top_fields, collapse = ", ")
      ),
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  top_level_types_valid <-
    is.numeric(body$numFound) &&
      length(body$numFound) == 1L &&
      is.numeric(body$start) &&
      length(body$start) == 1L &&
      is.list(body$docs)
  if (!top_level_types_valid) {
    return(list(
      outcome = "schema failure: invalid top-level types",
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  documents <- body$docs
  absent <- required_fields[
    !map_lgl(
      required_fields,
      \(field) all(map_lgl(documents, \(record) field %in% names(record)))
    )
  ]

  if (length(absent) > 0) {
    return(list(
      outcome = str_c(
        "schema failure: missing ",
        str_c(absent, collapse = ", ")
      ),
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = as.integer(body$numFound),
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  start <- as.integer(body$start)
  reported_total <- as.integer(body$numFound)
  records_read <- length(documents)
  more_pages <- start + records_read < reported_total
  if (records_read == 0L && more_pages) {
    return(list(
      outcome = "schema failure: empty page before reported total",
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = reported_total,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  list(
    outcome = if (more_pages) "more pages" else "last page",
    request_url = response$url,
    wait_seconds = NA_integer_,
    reported_total = reported_total,
    next_start = if (more_pages) start + records_read else NA_integer_,
    records = tibble(
      key = map_chr(documents, "key"),
      title = map_chr(documents, "title")
    )
  )
}

page_size <- 5L
refused <- read_saved_page(rate_limited, page_size)
date_refused <- read_saved_page(date_limited, page_size)
first_page <- read_saved_page(page_one, page_size)
second_page <- read_saved_page(page_two, page_size)
drifted <- read_saved_page(field_renamed, page_size)
missing_docs <- read_saved_page(docs_missing, page_size)
missing_retry_header <- saved_response(
  429L,
  '{"error": "too many requests"}',
  url = page_one_request$url
)
missing_wait <- read_saved_page(missing_retry_header, page_size)

collected <- bind_rows(
  first_page$records,
  second_page$records
)
distinct_records <- collected |>
  distinct(key, .keep_all = TRUE)
shortfall <- first_page$reported_total - nrow(distinct_records)

knitr::kable(
  tibble(
    saved_reply = c(
      "429 with delay-seconds",
      "429 with HTTP-date",
      "page 1 of results",
      "page 2 of results",
      "title field renamed",
      "docs field missing"
    ),
    outcome = c(
      refused$outcome,
      date_refused$outcome,
      first_page$outcome,
      second_page$outcome,
      drifted$outcome,
      missing_docs$outcome
    ),
    records_returned = c(
      nrow(refused$records),
      nrow(date_refused$records),
      nrow(first_page$records),
      nrow(second_page$records),
      nrow(drifted$records),
      nrow(missing_docs$records)
    )
  ),
  col.names = c("Saved reply", "Outcome", "Records used"),
  caption = "Four saved replies and what the reading function did with each",
  row.names = FALSE
)
Four saved replies and what the reading function did with each
Saved reply Outcome Records used
429 with delay-seconds rate limited 0
429 with HTTP-date rate limited 0
page 1 of results more pages 5
page 2 of results last page 2
title field renamed schema failure: missing title 0
docs field missing schema failure: missing top-level docs 0

Six saved replies exercised two retry formats, two result pages, and two schema failures. None produced an empty result table without an error.

The refusal returned no records and a waiting time taken from the Retry-After header, which is the service telling the client how long to pause. Guessing that number, or retrying immediately, is how a temporary limit becomes a ban.

The first page reports start = 0, returns five records, and says seven exist, so start + records returned < numFound supplies the next offset. The second page reports start = 5; adding its two records reaches the reported total, so the client stops. A full page alone would not prove that another page exists.

The renamed field is harder to catch. The record still parsed as JSON and still had a key. Only the explicit list of required fields turned it into a reported failure instead of a table with a missing column.

Do not let the arithmetic pass unexamined

The service reported seven matching records. Seven rows arrived across two pages. Six of them were distinct, because one work appeared on both pages, which happens when ranking shifts between two requests.

pagination_report <- tibble(
  measure = c(
    "reported by the service",
    "rows received",
    "distinct works kept",
    "unexplained shortfall"
  ),
  value = c(
    first_page$reported_total,
    nrow(collected),
    nrow(distinct_records),
    shortfall
  )
)

knitr::kable(
  pagination_report,
  col.names = c("Measure", "Count"),
  caption = "Reported, received, and kept records after paging",
  row.names = FALSE
)
Reported, received, and kept records after paging
Measure Count
reported by the service 7
rows received 7
distinct works kept 6
unexplained shortfall 1

The shortfall of one is the useful number. A collection script that reported six records without it would look complete. Record the totals the service claimed, the rows you received, and the rows you kept, and treat any gap as a question about the collection rather than a rounding detail.

Keep API keys private

Open Library’s search endpoint does not require a key. Many other APIs do. An API key is a secret value that identifies an account and may permit billable or private access.

Never place a real key in:

  • an R or Quarto file;
  • a public repository;
  • a screenshot; or
  • printed output.

Store secrets outside the project, usually in an environment variable, a named setting that R can read at run time. If a key is exposed, revoke it and create a replacement.

When an API request fails

A failed request leaves the librarian without a checked list. Turning that failure into an empty table would hide the difference between “no books found” and “the service did not answer.” Report the failure and keep the reported error. Then ask:

  1. Is the device connected to the internet?
  2. Is the endpoint address correct?
  3. Does the service require a key or other permission?
  4. Did the request exceed a rate limit?
  5. Did the response format change?

The code in this lesson stops when the request fails or the response lacks the expected structure. The interruption is visible rather than being published as an apparently complete result.

What to remember

  • An API request asks a service for structured information.
  • Query parameters describe what to return and how much.
  • Check the status and response type before using the body.
  • Some JSON structures can be reshaped into rows and columns in R.
  • A refusal, a second page, and a renamed field can all be rehearsed from saved replies before any request is sent.
  • Honor the waiting time a service sends instead of guessing one.
  • Compare the total a service reports with the rows you actually kept.
  • Live API results may change even when the code stays the same.
  • A dated fixture can preserve the output used in a published example.
  • Keys and other secrets never belong in published code.

The librarian ends with five candidate records, not five recommendations. The request can be tried again, while the dated fixture preserves the reply shown here. Human judgment still decides which books belong on the reading list.

Sources