Cleaning raw text

Repair common damage without pretending nothing was lost

documents
text cleaning
workforce research
Learn how HTML remnants, whitespace, invisible characters, and mojibake change raw text in R.

Saved job-board text, flyer lines, and one accented customer-service comment make up this cleaning set. The comment comes from an unrelated project, kept here because it contains an accented word.

Text from the wild carries damage. Tags remain after a copy, spacing carries page layout, invisible characters hide inside words, and accented letters can be read with the wrong encoding.

Cleaning means changing raw text so a later step can compare, count, or read it more consistently. Each repair needs a record of what it changed and what it cannot recover.

Note

The Riverton sources are fictional. The customer-service comments come from a separate teaching fixture.

TipWhat you will learn

This lesson prepares you to:

  • remove HTML while preserving readable text;
  • explain what whitespace squishing keeps and discards;
  • detect zero-width and control characters; and
  • describe why mojibake repair can fail.

Remove HTML remnants

The HTML example starts with the comment file, display tables, text checks, and an HTML parser. HTML is the markup language that names page parts such as headings, paragraphs, and lists.

library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(rvest)

feedback <- read_csv(
  "data/customer_feedback.csv",
  na = character(),
  col_types = cols(
    feedback_id = col_character(),
    submitted_at = col_character(),
    channel = col_character(),
    rating = col_character(),
    comment = col_character()
  )
)

entity_html <- "<p>Training &amp; tools&nbsp;provided.</p>"
regex_tag_stripped <- str_remove_all(entity_html, "<[^>]+>")
html_text <- rvest::html_text2(rvest::read_html(entity_html))
Entity handling after two HTML-cleaning approaches
Method Result
Remove tags with a regular expression Training & tools provided.
Read HTML with rvest Training & tools provided.

html_text2() turns HTML into readable text and decodes entities such as &amp; and &nbsp;. A regular expression can remove visible tags, but it leaves those entities behind. Keep the raw HTML when element boundaries still matter.

Squish whitespace

Whitespace includes spaces, tabs, and line breaks. Lesson 21 covers the comparison side of str_squish(). Here the point is layout loss: the function collapses runs of whitespace to one ordinary space and trims the ends.

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

flyer_lines <- workforce_sentences |>
  filter(document_id == "F001") |>
  pull(text)

flyer_block <- paste(flyer_lines, collapse = "\n")
squished_flyer <- str_squish(flyer_block)
Whitespace squishing removes the flyer line layout
Version Text
Flyer lines RIVERTON SKILLS OPEN HOUSE

DATA SUPPORT CERTIFICATE Paid training stipend Evening classes No prior experience required Apply by October 15 | |After str_squish() |RIVERTON SKILLS OPEN HOUSE DATA SUPPORT CERTIFICATE Paid training stipend Evening classes No prior experience required Apply by October 15 |

Squishing helps when accidental spaces would make two records look different. It also discards deliberate layout. The flyer heading, certificate heading, stipend, class time, requirement, and deadline become one long line.

Remove invisible characters

Unicode sorts characters into categories. Cf is the format category, including zero-width spaces that can sit inside a string without appearing on screen. Cc is the control category. In a stringr pattern, \\p{Cf} and \\p{Cc} name those categories.

zero_width_space <- intToUtf8(0x200B)
clean_label <- "data support"
damaged_label <- paste0("data", zero_width_space, " support")
label_repaired <- str_remove_all(damaged_label, "[\\p{Cf}]")

control_text <- str_c("shift", intToUtf8(0x0001), "starts soon")
control_marked <- str_replace_all(control_text, "[\\p{Cc}]", " ")
control_repaired <- str_squish(control_marked)
control_squished <- str_squish(control_text)
Invisible characters change a string even when it looks familiar
Example Text Character count
Clean label data support 12
With zero-width space data​ support 13
After repair data support 12
Control after whitespace squish shiftstarts soon 17
Control after category repair shift starts soon 17

The two labels look alike to a hurried reader, but R does not treat them as identical. Whitespace squishing does not remove the control character in the second example; the category repair does. Keep an audit field if hidden characters help explain source damage.

Repair mojibake carefully

Mojibake is text damage caused by reading bytes with the wrong character encoding. UTF-8 is a common encoding for many languages. Windows-1252 is another encoding often seen in older Western-language files.

Lesson 21 showed café stored in two legal Unicode normal forms. Both forms are readable and both preserve the word. This section is about a separate failure: bytes read with the wrong decoder, or bytes replaced after a lossy read.

accented_word <- str_extract(feedback$comment[2], fixed("café"))
mojibake_word <- iconv(
  accented_word,
  from = "Windows-1252",
  to = "UTF-8"
)
repaired_word <- iconv(
  mojibake_word,
  from = "UTF-8",
  to = "Windows-1252"
)
replacement_text <- "caf\uFFFD"
encoding_examples <- tibble(
  version = c(
    "Correct UTF-8",
    "Read as Windows-1252",
    "Repaired",
    "Typed by hand to show a lossy read"
  ),
  text = c(accented_word, mojibake_word, repaired_word, replacement_text)
) |>
  mutate(
    code_points = vapply(
      text,
      \(value) str_c(utf8ToInt(value), collapse = ", "),
      character(1)
    )
  )
The word café after encoding damage and repair
Version Text Unicode code points
Correct UTF-8 café 99, 97, 102, 233
Read as Windows-1252 café 99, 97, 102, 195, 169
Repaired café 99, 97, 102, 233
Typed by hand to show a lossy read caf� 99, 97, 102, 65533

Reading UTF-8 bytes as Windows-1252 turns café into café. Both bytes survive, so reversing the mistake restores the word. A different failure is lossy: when a read leaves �, the string contains U+FFFD rather than the source character. The repair cannot infer which letter belonged there. Mojibake and U+FFFD are separate problems.

Cleaning is a recorded choice

Cleaning makes later comparison easier. It can also remove source structure, spacing, hidden characters, and evidence of encoding damage. A collection should record each cleaning step, keep the raw source, and make the cleaned version a separate field.

Production pipelines track per-step changed and removed counts. Analysts perform a removed-side sample audit to ensure they aren’t accidentally erasing meaningful data. This is crucial because aggressive text cleaning can have a disparate impact, disproportionately destroying names or phrases from specific languages or dialects that use characters a naive rule did not expect.

When mojibake repair is in scope, production NLP typically relies on robust ecosystem tools (like the Python package ftfy — “fixes text for you”) rather than manual iconv() conversions.

What to remember

  • HTML parsers decode entities that tag-stripping leaves behind.
  • Whitespace squishing fixes accidental spacing and erases layout.
  • Invisible characters and encoding damage can make matching strings unequal.
  • Mojibake can be reversible, while U+FFFD means information was lost.
  • Keep the raw source and record every cleaning step.

Cleaned fields belong beside raw fields. Comparisons can use the cleaned copy, while the raw text keeps the evidence needed to explain the repair.

Sources