Skip to main content

Extracting text from an image

Run OCR and compare the result with a known transcript

Learn what optical character recognition can recover from a training flyer and how to check its errors.

The saved job board supplied digital text. The Riverton Workforce Lab’s next fictional notice is a training flyer available only as an image. Its wording covers a stipend, a class schedule, prior experience, and an application date. Losing or reversing a phrase would change what a person understood about the program.

Optical character recognition, shortened to OCR, estimates which text appears in an image. The team will run OCR, compare the result with a known reference transcript, and keep both the image and extracted text. Recovery comes before any attempt to classify what the flyer says.

Note

The flyer and program are fictional. The image was generated for this lesson, and the transcript was written from the source text that created it.

TipWhat you will learn

By the end of this lesson, you will be able to:

• explain the difference between digital text and text inside an image;
• run the Tesseract OCR engine from R;
• compare OCR output with a known reference transcript;
• interpret character-level edit distance;
• report character and word error rates for a damaged image; and
• identify conditions that make OCR output less reliable.

Inspect the source image

Figure 1: Fictional training flyer used for the OCR example

RIVERTON SKILLS OPEN HOUSE
DATA SUPPORT CERTIFICATE
Paid training stipend
Evening classes
No prior experience required
Apply by October 15

The adjacent text is the complete flyer transcript. It is needed even when OCR is available because automated extraction can be wrong or may not run in a reader’s browser. The image remains the source; OCR is one reading of it.

The checked transcript contains six lines.

library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(digest)
library(png)
library(tesseract)

transcript_path <- str_c(
  "data/workforce/",
  "training-flyer-ground-truth.txt"
)
reference_lines <- read_lines(transcript_path)
transcript_md5 <- digest::digest(
  paste(reference_lines, collapse = "\n"),
  algo = "md5",
  serialize = FALSE
)
flyer_metadata <- read_csv(
  "data/workforce/training-flyer-metadata.csv",
  col_types = cols(
    artifact = col_character(),
    width_pixels = col_integer(),
    height_pixels = col_integer(),
    md5 = col_character(),
    source = col_character(),
    rendering_note = col_character()
  )
)
image_metadata <- flyer_metadata |>
  filter(artifact == "training-flyer.png")
transcript_metadata <- flyer_metadata |>
  filter(artifact == "training-flyer-ground-truth.txt")
recorded_ocr_metadata <- flyer_metadata |>
  filter(artifact == "training-flyer-ocr-5.3.2.txt")
degraded_ocr_metadata <- flyer_metadata |>
  filter(artifact == "training-flyer-degraded-ocr-5.3.2.txt")
image_path <- "data/workforce/training-flyer.png"
image_array <- png::readPNG(
  image_path,
  info = TRUE
)
image_dimensions <- dim(image_array)[1:2]
image_md5 <- digest::digest(
  image_path,
  algo = "md5",
  serialize = FALSE,
  file = TRUE
)

reference_lines

[1] "RIVERTON SKILLS OPEN HOUSE"   "DATA SUPPORT CERTIFICATE"    
[3] "Paid training stipend"        "Evening classes"             
[5] "No prior experience required" "Apply by October 15"         

The transcript is the comparison source for this controlled example. In real research, the reference should be checked by people who can read the document and understand its language and layout.

Run OCR

The tesseract package connects R to the open-source Tesseract OCR engine. We select English because the flyer is written in English.

ocr_information <- tesseract::tesseract_info()
ocr_engine <- tesseract::tesseract("eng")
diagnostic_ocr_text <- tesseract::ocr(
  "data/workforce/training-flyer.png",
  engine = ocr_engine
)
diagnostic_ocr_lines <- diagnostic_ocr_text |>
  str_split_1(fixed("\n")) |>
  str_trim()
diagnostic_ocr_lines <- diagnostic_ocr_lines[
  str_length(diagnostic_ocr_lines) > 0
]
ocr_lines <- read_lines(
  "data/workforce/training-flyer-ocr-5.3.2.txt"
)

ocr_lines

[1] "RIVERTON SKILLS OPEN HOUSE"   "DATA SUPPORT CERTIFICATE"    
[3] "Paid training stipend"        "Evening classes"             
[5] "No prior experience required" "Apply by October 15"         

The published transcript is the committed output recorded with native Tesseract 5.3.2. The host engine still runs as a diagnostic, but its result does not change the displayed transcript or metrics. The exact match belongs to this image and recorded software version; it is not a general OCR accuracy claim.

ocr_provenance <- tibble(
  component = c(
    "R tesseract package",
    "Native Tesseract engine",
    "Language data",
    "Image MD5"
  ),
  value = c(
    as.character(
      utils::packageVersion("tesseract")
    ),
    ocr_information$version,
    "eng",
    image_md5
  )
)

knitr::kable(
  ocr_provenance,
  col.names = c("Component", "Recorded value"),
  caption = "Software and image details for this OCR run",
  row.names = FALSE
)

Software and image details for this OCR run

Component	Recorded value
R tesseract package	5.2.5
Native Tesseract engine	5.3.4
Language data	eng
Image MD5	5dcfe9b824159b34c95b1d11e861ed54

The lockfile pins the R package. The native engine and language data come from the operating system, so the lesson reports them at run time. The repository also preserves the raw output recorded with native Tesseract 5.3.2 in data/workforce/training-flyer-ocr-5.3.2.txt.

Measure the difference

Edit distance counts the smallest number of character insertions, deletions, or substitutions needed to turn one text into another. We normalize case and punctuation before comparing because the research question concerns the words, not the flyer styling.

normalize_for_comparison <- function(text) {
  text |>
    str_c(collapse = " ") |>
    str_to_lower() |>
    str_replace_all("[^a-z0-9]+", " ") |>
    str_squish()
}

reference_text <- normalize_for_comparison(
  reference_lines
)
observed_text <- normalize_for_comparison(
  ocr_lines
)
normalized_edit_distance <- as.integer(
  adist(reference_text, observed_text)
)
normalized_distance_share <-
  normalized_edit_distance /
    str_length(reference_text)

tibble(
  normalized_edit_distance =
    normalized_edit_distance,
  reference_characters = str_length(reference_text),
  normalized_distance_share =
    normalized_distance_share
)

# A tibble: 1 × 3
  normalized_edit_distance reference_characters normalized_distance_share
                     <int>                <int>                     <dbl>
1                        0                  138                         0

An edit distance of zero means that the normalized texts match exactly. This is not a universal character-error rate: normalization removed case, punctuation, and line boundaries, and the denominator includes spaces. The code separately requires every raw line and the critical experience statement to match exactly.

Damage the image and measure what is lost

A flyer generated at 1200 pixels wide and read at full size is the best case this pipeline will ever see. Real material arrives photographed at an angle, faxed, scanned at low resolution, or printed with a failing toner cartridge.

The image below is degraded inside the lesson so the damage is reproducible. Two things happen to it. Every third row and column is kept, which reduces it to a third of its size, and every third remaining row is lightened, which imitates the banding a tired scanner produces. Both steps are arithmetic on the pixel array, with no randomness, so the same page produces the same file every time it renders.

keep_every <- 3L
row_positions <- seq(1L, dim(image_array)[1], by = keep_every)
column_positions <- seq(1L, dim(image_array)[2], by = keep_every)
degraded_array <- image_array[
  row_positions,
  column_positions, ,
  drop = FALSE
]
banded_rows <- seq(1L, dim(degraded_array)[1], by = 3L)
degraded_array[banded_rows, , ] <- pmin(
  degraded_array[banded_rows, , ] + 0.35,
  1
)

degraded_path <- tempfile(fileext = ".png")
png::writePNG(degraded_array, degraded_path)

degraded_diagnostic_text <- tesseract::ocr(
  degraded_path,
  engine = ocr_engine
)
degraded_diagnostic_lines <- degraded_diagnostic_text |>
  str_split_1(fixed("\n")) |>
  str_trim()
degraded_diagnostic_lines <- degraded_diagnostic_lines[
  str_length(degraded_diagnostic_lines) > 0
]
degraded_lines <- read_lines(
  "data/workforce/training-flyer-degraded-ocr-5.3.2.txt"
)
diagnostic_matches_fixture <- c(
  identical(diagnostic_ocr_lines, ocr_lines),
  identical(degraded_diagnostic_lines, degraded_lines)
)

tibble(
  version = c("source image", "degraded copy"),
  width_pixels = c(
    dim(image_array)[2],
    dim(degraded_array)[2]
  ),
  height_pixels = c(
    dim(image_array)[1],
    dim(degraded_array)[1]
  ),
  lines_returned = c(
    length(ocr_lines),
    length(degraded_lines)
  ),
  diagnostic_matches_fixture
)

# A tibble: 2 × 5
  version       width_pixels height_pixels lines_returned diagnostic_matches_f…¹
  <chr>                <int>         <int>          <int> <lgl>                 
1 source image          1200           800              6 TRUE                  
2 degraded copy          400           267              6 FALSE                 
# ℹ abbreviated name: ¹​diagnostic_matches_fixture

The final column reports whether the host engine reproduced each committed transcript. A FALSE value is a local-version difference, while the published metrics below remain tied to the recorded 5.3.2 output.

Character and word error rates

Two rates are standard for this comparison. Character error rate, or CER, divides the character-level edit distance by the number of characters in the reference. Word error rate, or WER, does the same with whole words, so a word that is wrong in one letter counts once rather than once per letter.

Word-level distance needs its own small calculation, because adist() compares characters. The function below fills a table of costs, one row per reference word and one column per observed word, and reads the answer from the corner.

word_edit_distance <- function(reference_words, observed_words) {
  costs <- matrix(
    0L,
    nrow = length(reference_words) + 1L,
    ncol = length(observed_words) + 1L
  )
  costs[, 1L] <- seq.int(0L, length(reference_words))
  costs[1L, ] <- seq.int(0L, length(observed_words))

  for (row in seq_along(reference_words)) {
    for (column in seq_along(observed_words)) {
      substitution <- if (
        identical(reference_words[row], observed_words[column])
      ) {
        0L
      } else {
        1L
      }
      costs[row + 1L, column + 1L] <- min(
        costs[row, column + 1L] + 1L,
        costs[row + 1L, column] + 1L,
        costs[row, column] + substitution
      )
    }
  }

  costs[
    length(reference_words) + 1L,
    length(observed_words) + 1L
  ]
}

error_rates <- function(reference, observed) {
  reference_words <- str_split_1(reference, " ")
  observed_words <- if (str_length(observed) == 0L) {
    character()
  } else {
    str_split_1(observed, " ")
  }
  character_distance <- as.integer(adist(reference, observed))
  word_distance <- word_edit_distance(
    reference_words,
    observed_words
  )

  tibble(
    reference_characters = str_length(reference),
    reference_word_count = length(reference_words),
    cer = character_distance / str_length(reference),
    wer = word_distance / length(reference_words)
  )
}

degraded_observed <- normalize_for_comparison(degraded_lines)
rate_table <- bind_rows(
  error_rates(reference_text, observed_text) |>
    mutate(version = "source image", .before = 1),
  error_rates(reference_text, degraded_observed) |>
    mutate(version = "degraded copy", .before = 1)
)

knitr::kable(
  rate_table |>
    mutate(
      cer = round(cer, 3),
      wer = round(wer, 3)
    ) |>
    select(version, reference_characters, reference_word_count, cer, wer),
  col.names = c(
    "Version",
    "Reference characters",
    "Reference words",
    "CER",
    "WER"
  ),
  caption = "Error rates for the source image and its degraded copy",
  row.names = FALSE
)

Error rates for the source image and its degraded copy

Version	Reference characters	Reference words	CER	WER
source image	138	20	0.000	0.0
degraded copy	138	20	0.558	1.2

The source image scores zero on both rates. The degraded copy does not. Both rates come from committed Tesseract 5.3.2 transcripts, so a native engine upgrade cannot rewrite the published result. The diagnostic comparison above reports whether the local engine produced the same transcripts.

comparison <- tibble(
  line = seq_len(max(length(reference_lines), length(degraded_lines))),
  reference = c(
    reference_lines,
    rep(NA_character_, max(0L, length(degraded_lines) - length(reference_lines)))
  )[seq_len(max(length(reference_lines), length(degraded_lines)))],
  degraded_reading = c(
    degraded_lines,
    rep(NA_character_, max(0L, length(reference_lines) - length(degraded_lines)))
  )[seq_len(max(length(reference_lines), length(degraded_lines)))]
)

knitr::kable(
  comparison,
  col.names = c("Line", "Reference transcript", "Degraded reading"),
  caption = "The transcript beside what the degraded image produced",
  row.names = FALSE
)

The transcript beside what the degraded image produced

Line	Reference transcript	Degraded reading
1	RIVERTON SKILLS OPEN HOUSE	AIVS SPCR SKILLS OREM FUL Sk
2	DATA SUPPORT CERTIFICATE	LAI SJZRCRE CEAUIF CATE
3	Paid training stipend	viel tras oe, esis
4	Evening classes	Logs se senate
5	No prior experience required	8 he ee eis ites
6	Apply by October 15	Apps ey Oelaber 15

Read the table beside the rate. A CER of 0.558 means 0.558 character edits per reference character; it is not a fraction of information preserved and has no meaningful midpoint. Insertions can even push CER or WER above one. Here, the remaining words are not a faithful partial transcript of the flyer.

Two limits belong with these numbers. Both rates are computed against one transcript of one image, so they describe this pair and estimate nothing about OCR in general. The word rate also assumes that words are separated by spaces, which is true for this English flyer and false for Chinese, Japanese, and Thai, where a character rate or a script-aware segmentation is needed instead.

OCR errors are not evenly distributed

OCR becomes harder when a source has:

• low resolution, blur, shadows, or skew;
• handwriting or decorative fonts;
• tables, columns, stamps, or text over images;
• uncommon symbols or several languages;
• damaged pages; or
• a reading order that is not visually obvious.

An average error rate can hide the error that matters most. Changing "No prior experience required" to "prior experience required" would reverse the meaning while affecting only a few characters. The team must inspect important fields directly, not rely on one summary score.

Preserve the source and the correction trail

Keep:

1. the source image;
2. the OCR engine and language settings;
3. the unedited OCR output;
4. the reviewed and corrected transcript;
5. the person or process that checked it; and
6. any uncertainty about layout or unreadable text.

Do not overwrite raw OCR with corrections. Separate fields make later edits visible and let another reviewer return to the source.

What to remember

• Text inside an image must be recognized before ordinary text analysis.
• OCR output is an estimate, not a transcription guarantee.
• Compare important output with a human-checked reference.
• Report character and word error rates, and say which image produced them.
• One clean image cannot estimate accuracy on damaged material.
• Measure errors and inspect meaning-changing cases.
• Preserve the image, raw output, corrected text, and review history.

The archived job cards and checked flyer transcript now provide traceable text about training, schedules, requirements, and skills. Extraction has not decided what any sentence means. Human classification follows, using a versioned annotation codebook to record those reading decisions.

Sources

• tesseract R package documentation
• Tesseract OCR documentation
• Unicode Text Segmentation, UAX #29
• Web Content Accessibility Guidelines: text alternatives