Turn separate documents into an organized collection
A person locked out of an account needs the right help article. The help-center editor has three separate files about deliveries, returns, and account access. The password-reset instructions are somewhere in those files, but each search result also needs to retain its topic and source. That becomes harder as loose documents accumulate.
The editor needs an organized collection called a corpus. Its plural is corpora. A corpus keeps text documents together and usually stores information describing each one. We will build a three-document corpus, check for empty or duplicate records, and use it to find the password-reset article.
The help articles in this lesson were written for this example. They do not come from a real support system.
By the end of this lesson, you will be able to:
Before collecting anything, the editor must choose the document unit, the amount of text that will count as one document. Depending on the question, it might be:
The right unit depends on the question. Because the editor wants to compare and search help articles, each complete article will be one document. Treating each sentence as a separate document would detach the sentence from the rest of its article.
Before loading real documents, confirm the right to use them. Record the license or permission, collection method, expected audience, privacy risk, and whether the people represented could reasonably expect this use. The files below are synthetic, so they contain no personal records or third-party text.
The example uses three plain-text files, files that store characters without page layout or styling. list.files() asks R to find file names ending in .txt.
[1] "account-access.txt" "delivery.txt" "returns.txt" R prints all three file names in alphabetical order. We keep those names as the first labels in the collection so each result can still be traced to the file that supplied its text.
Finding the file names does not yet load their contents. read_lines() reads the text inside one file. These files use UTF-8, a common way to store characters from many languages. A function is a named set of steps. The small function below preserves line breaks in the source copy.
expected_line_counts <- c(
"account-access.txt" = 1L,
"delivery.txt" = 1L,
"returns.txt" = 1L
)
read_document <- function(path) {
lines <- read_lines(path)
file_name <- basename(path)
stopifnot(identical(length(lines), unname(expected_line_counts[file_name])))
str_c(lines, collapse = "\n")
}
source_documents <- article_paths |>
set_names(basename(article_paths)) |>
map_chr(read_document)
analysis_documents <- str_replace_all(
source_documents,
"[\r\n]+",
" "
)
source_documents account-access.txt
"Use the password reset link on the sign-in page. The link expires after 30 minutes for your security."
delivery.txt
"Standard delivery takes three to five business days. Tracking appears after the parcel leaves our warehouse."
returns.txt
"You can return an unused item within 30 days. Keep the receipt and use the prepaid label included with your order." map_chr() runs read_document() once for each path and returns a character vector. The result contains three source copies, and the file names remain attached. A separate analysis_documents object replaces line breaks with spaces. The source copy is left untouched so paragraph boundaries can be recovered later.
The check using str_length() confirms that none of the documents is empty. An empty file might mean that loading failed or that the source itself was blank. Either case should be investigated before analysis.
The text is present, but the values alone have little room for context. Each document also needs metadata, information about the document rather than its main text. A source file, topic, publication date, or author can be metadata.
We will place the text and its labels in a tibble, with one row per document.
hash_document <- function(path) {
lines <- read_lines(path)
file_name <- basename(path)
stopifnot(identical(length(lines), unname(expected_line_counts[file_name])))
digest::digest(
paste(lines, collapse = "\n"),
algo = "md5",
serialize = FALSE
)
}
corpus <- tibble(
document_id = str_remove(
basename(article_paths),
"[.]txt$"
),
topic = c("account access", "delivery", "returns"),
source_file = basename(article_paths),
source_hash = map_chr(
article_paths,
hash_document
),
source_text = unname(source_documents),
analysis_text = unname(analysis_documents)
)
knitr::kable(
corpus |>
select(document_id, topic, source_file),
col.names = c("Document ID", "Topic", "Source file"),
caption = "The documents and their metadata",
row.names = FALSE
)| Document ID | Topic | Source file |
|---|---|---|
| account-access | account access | account-access.txt |
| delivery | delivery | delivery.txt |
| returns | returns | returns.txt |
The document_id gives every row a stable label. The topic identifies the article’s subject without requiring someone to open the full text. The source file records where that text came from. source_hash is a fingerprint that changes if a file changes; it helps detect accidental edits but is not proof of authorship or trustworthiness. Keeping these fields on the same row is the corpus equivalent of keeping a label attached to its folder.
A corpus is always a sample of something, even when nobody chose a sample. The honest version of that sentence names the frame: the population the collection was drawn from, the rule used to draw it, and what the rule left out.
For these three articles the frame is small and the answer is unusual. The collection is not a sample within its stated frame. It is every active, non-archived article this fictional help center published as of the collection date, which makes it a census of that tiny frame rather than a sample of the support questions people actually ask.
That distinction changes what the corpus can support later. A count over a census of published articles describes what the help center wrote. It does not describe what readers needed and did not find.
collection_record <- tibble(
field = c(
"frame",
"selection rule",
"collected on",
"excluded by rule",
"known coverage gap",
"language",
"unit"
),
value = c(
"active, non-archived articles from one fictional help center",
"all active articles on the collection date, no sampling",
"2026-08-28",
"drafts, archived articles, and reader messages",
"questions people asked that no article answers",
"English (en)",
"one complete article"
)
)
knitr::kable(
collection_record,
col.names = c("Field", "Recorded value"),
caption = "What the collection covers and what it leaves out",
row.names = FALSE
)| Field | Recorded value |
|---|---|
| frame | active, non-archived articles from one fictional help center |
| selection rule | all active articles on the collection date, no sampling |
| collected on | 2026-08-28 |
| excluded by rule | drafts, archived articles, and reader messages |
| known coverage gap | questions people asked that no article answers |
| language | English (en) |
| unit | one complete article |
Write this record before the analysis, not after it. Once results exist, the frame tends to be described in whatever terms make the results sound broader.
A quick count can reveal a document that loaded as empty or unexpectedly long. The function below separates text wherever it finds whitespace, then counts the resulting pieces. Here, whitespace means spaces, tabs, line breaks, and non-breaking spaces matched by [[:space:]]+.
count_whitespace_tokens <- function(text) {
tokens <- str_split_1(str_trim(text), "[[:space:]]+")
sum(str_length(tokens) > 0)
}
corpus <- corpus |>
mutate(
whitespace_token_count = map_int(
analysis_text,
count_whitespace_tokens
)
)
knitr::kable(
corpus |>
select(document_id, topic, whitespace_token_count),
col.names = c("Document ID", "Topic", "Whitespace-token count"),
caption = "Whitespace-token counts for the three documents",
row.names = FALSE
)| Document ID | Topic | Whitespace-token count |
|---|---|---|
| account-access | account access | 18 |
| delivery | delivery | 16 |
| returns | returns | 21 |
The three articles contain 18, 16, and 21 whitespace-separated pieces. These are often called tokens, not universal word counts. Languages without spaces between words, contractions, punctuation, and specialized tokenizers can produce different results. The counts are quick checks, not quality scores.
The editor now has rows, labels, and text. Before trusting the collection, we confirm that:
quality_checks <- tibble(
check = c(
"source_files_exist",
"no_empty_text",
"unique_document_ids",
"no_exact_duplicate_text",
"metadata_is_complete"
),
passed = c(
all(file.exists(article_paths)),
all(str_length(str_trim(corpus$source_text)) > 0),
n_distinct(corpus$document_id) == nrow(corpus),
n_distinct(corpus$source_text) == nrow(corpus),
!any(
is.na(
corpus |>
select(topic, source_file, source_hash)
)
)
)
)
quality_checks# A tibble: 5 × 2
check passed
<chr> <lgl>
1 source_files_exist TRUE
2 no_empty_text TRUE
3 unique_document_ids TRUE
4 no_exact_duplicate_text TRUE
5 metadata_is_complete TRUE Every passed value is TRUE, so the corpus passes these structural checks. They do not establish that the articles are accurate, useful, or representative. They show only that the three expected files produced three labeled, nonempty, distinct records.
The collection is now ready for the editor’s original question. Because every document shares one structure, the same search can be applied to all three. This one looks for the exact phrase "password reset".
| Document ID | Topic | Text |
|---|---|---|
| account-access | account access | Use the password reset link on the sign-in page. The link expires after 30 minutes for your security. |
The account-access article is the single match. Its topic, document ID, and text arrive together, so the editor can inspect the result and return to the source file. That connection becomes easier to lose as a corpus grows.
An R class tells R what kind of object it is handling and which operations belong with it.
Packages such as quanteda and tm provide dedicated corpus objects with tools for metadata and text processing. A tibble is enough for this lesson because its rows and columns are visible and easy to inspect. The same records can be converted to a package-specific corpus later.
A fourth file arrives. The support team copied the account-access article into a seasonal FAQ, changed the capitalization, replaced a hyphen, and added a courtesy line. To a reader it is the same article. To an exact comparison it is a new document.
The candidate below is written into the lesson so its wording is visible.
candidate_text <- paste(
"Use the password reset link on the sign in page.",
"The link expires after 30 minutes for your security.",
"Contact the help desk if the message does not arrive."
)
incoming <- tibble(
document_id = c(corpus$document_id, "seasonal-faq-account"),
analysis_text = c(corpus$analysis_text, candidate_text)
)
exact_duplicate_found <- n_distinct(incoming$analysis_text) < nrow(incoming)
normalize_for_matching <- function(text) {
text |>
str_to_lower() |>
str_replace_all("[^a-z0-9]+", " ") |>
str_squish()
}
normalized <- normalize_for_matching(incoming$analysis_text)
similarity <- function(left, right) {
distance <- as.integer(adist(left, right))
1 - distance / max(str_length(left), str_length(right))
}
containment <- function(left, right) {
left_words <- unique(str_split_1(left, " "))
right_words <- unique(str_split_1(right, " "))
shorter <- if (length(left_words) <= length(right_words)) {
left_words
} else {
right_words
}
longer <- if (length(left_words) <= length(right_words)) {
right_words
} else {
left_words
}
mean(shorter %in% longer)
}
pair_scores <- tibble(
left = incoming$document_id[
c(1L, 1L, 1L, 2L, 2L, 3L)
],
right = incoming$document_id[
c(2L, 3L, 4L, 3L, 4L, 4L)
],
similarity = map2_dbl(
normalized[c(1L, 1L, 1L, 2L, 2L, 3L)],
normalized[c(2L, 3L, 4L, 3L, 4L, 4L)],
similarity
),
containment = map2_dbl(
normalized[c(1L, 1L, 1L, 2L, 2L, 3L)],
normalized[c(2L, 3L, 4L, 3L, 4L, 4L)],
containment
)
)
knitr::kable(
pair_scores |>
mutate(
similarity = round(similarity, 3),
containment = round(containment, 3)
) |>
arrange(desc(similarity)),
col.names = c(
"Document",
"Compared with",
"Character similarity",
"Word containment"
),
caption = "All six document pairs, most similar first",
row.names = FALSE
)| Document | Compared with | Character similarity | Word containment |
|---|---|---|---|
| account-access | seasonal-faq-account | 0.651 | 1.000 |
| account-access | delivery | 0.311 | 0.125 |
| delivery | seasonal-faq-account | 0.283 | 0.125 |
| delivery | returns | 0.277 | 0.125 |
| returns | seasonal-faq-account | 0.276 | 0.200 |
| account-access | returns | 0.268 | 0.250 |
The exact-duplicate check returns FALSE: no two documents are byte-identical. The two comparisons that follow both single out the same pair, and they disagree about how strong the resemblance is.
Character similarity reaches 0.65 for the reprint. The added courtesy line holds it down, because every extra character counts as a difference. Word containment asks whether one document’s words all appear in the other, and returns 1.00. When a copy is extended rather than edited, containment is the measure that keeps working.
The thresholds used here, 0.6 and 0.9, are decisions rather than discoveries. Any threshold belongs in the record with the measure that produced it, because the same pair can be a near duplicate under one measure and a distinct document under another. Both comparisons are quadratic in the number of documents, which is invisible at four and prohibitive at four million.
A near duplicate creates a second problem. If one copy trains a model and the other tests it, the overlap may inflate the evaluation by letting the model reuse wording or memorize examples. This lesson does not train a model, so it cannot determine which mechanism would cause the inflation. The preventive step is to group related documents into a family and let a split move the whole family at once.
families <- incoming |>
mutate(
family_id = if_else(
document_id %in% c("account-access", "seasonal-faq-account"),
"account-access-family",
document_id
),
kept = document_id != "seasonal-faq-account"
)
family_split <- families |>
distinct(family_id) |>
mutate(
split = if_else(
family_id == "returns",
"evaluation",
"development"
)
)
assigned <- families |>
left_join(family_split, by = join_by(family_id))
removed <- assigned |>
filter(!kept)
split_is_family_consistent <- assigned |>
summarise(splits = n_distinct(split), .by = family_id) |>
pull(splits) |>
max()
knitr::kable(
assigned |>
select(document_id, family_id, split, kept),
col.names = c("Document", "Family", "Split", "Kept"),
caption = "Four candidates, three kept, one removed with its family recorded",
row.names = FALSE
)| Document | Family | Split | Kept |
|---|---|---|---|
| account-access | account-access-family | development | TRUE |
| delivery | delivery | development | TRUE |
| returns | returns | evaluation | TRUE |
| seasonal-faq-account | account-access-family | development | FALSE |
[1] "Use the password reset link on the sign in page. The link expires after 30 minutes for your security. Contact the help desk if the message does not arrive."Four candidates went in, three came out, and the removed text is printed rather than summarized. Reading what a cleaning step deleted is the only way to notice that it deleted the wrong thing. Here the discarded copy carries one sentence the original does not, about contacting the help desk, so a careful editor might merge that line instead of dropping the file.
split_is_family_consistent is the falsifiable part. It counts the distinct splits any family received, and the corpus is only safe to evaluate on when that number is one.
The successful search does not make the collection neutral. Someone decided which files belong, where a document ends, which metadata to keep, and which text to exclude. Every later result inherits those choices.
Record:
The password-reset answer is now findable without losing the article around it. The seasonal reprint is out of the collection, its family is recorded, and the sentence it added is still on the page for someone to judge. For a larger corpus, keep the same small label beside every document: what this text is, where it came from, and why it belongs in the collection.