How computers store text

A friendly guide to bits, bytes, Unicode, and UTF-8 in R

source data loading
encoding
Unicode
Learn why text sometimes becomes garbled and how to protect it when working in R.

Imagine opening the attendance file for a community class and finding Anaïs where Anaïs should be. A person can infer the name, but a search, badge printer, or duplicate check may treat it as something else. The error is easy to miss until the damaged name reaches a badge or a lookup fails.

One possible cause is a wrong character encoding, the rules that connect stored byte values with written text. We will recreate that failure from known bytes, decode the bytes correctly, and then handle text that looks identical but still fails an exact comparison.

Note

The attendance file and incident are constructed for this lesson. The encoding failure is real, but the story does not describe an actual person or class.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • distinguish a visible text unit, a code point, and a byte;
  • explain the different jobs of Unicode and UTF-8;
  • convert text to UTF-8 in R;
  • recognize bytes that no UTF-8 decoder can read, and a byte-order mark; and
  • check text that looks the same but behaves differently.

What happens when a computer saves a word

Checking Anaïs’s name begins with a precise account of what the computer saved. People see letters, punctuation, and emoji. A computer stores numbers, so saving text takes a few steps:

  1. Unicode assigns numbers called code points to abstract text elements.
  2. UTF-8 turns Unicode scalar values into one or more bytes.
  3. Those bytes are saved in a file or sent across a network.
  4. Software uses the same encoding to turn the bytes back into text.

What a reader sees as one character is often called a grapheme cluster. A grapheme cluster can contain one code point or several. The difference becomes visible later when we compare two ways to store é.

Consider two characters:

Unicode code points and UTF-8 bytes for two characters
Character Unicode code point Bytes used by UTF-8
A U+0041 41
é U+00E9 C3 A9

The values are written in hexadecimal, a compact way to write numbers. There is no need to memorize them. For now, notice that A needs one UTF-8 byte while é needs two.

Bits and bytes

A bit records one binary choice, 0 or 1. Eight bits make one byte, which has 256 possible combinations numbered from 0 through 255.

NoteWhy eight bits make a byte

A byte did not always have a fixed size. Early computers grouped bits according to the characters and numbers each machine needed, so a byte could mean different things on different systems.

Two standards helped settle the size:

  1. ASCII defined 128 text and control values, which fit in seven bits. RFC 20, written for early network interchange in 1969, put each seven-bit ASCII value inside an eight-bit byte and set the extra high bit to zero.
  2. IBM’s System/360 family, announced in 1964, made an eight-bit byte the basic building block of its storage formats. Its compatible machines became widely used, and their byte size travelled with the software and hardware built around them.

Eight bits provide (2^8 = 256) patterns: enough to hold every ASCII value with one bit left over. Today, a byte almost always means eight bits. Technical standards sometimes use octet when they need to rule out the older, variable-size meaning.

We will start with A and é, two characters small enough to inspect by eye. R’s charToRaw() function shows bytes. The word “raw” means that R should display the stored values without interpreting them as letters. map() from purrr applies that byte check to each text value and returns a list, which can hold byte results of different lengths.

library(tibble)
library(purrr)
library(stringi)

text <- c(
  letter_A = "A",
  e_with_accent = "\u00E9"
)

bytes <- map(enc2utf8(text), charToRaw)

bytes
$letter_A
[1] 41

$e_with_accent
[1] c3 a9

R prints raw bytes as two-digit hexadecimal values. The output shows 41 for A and c3 a9 for é. Hexadecimal digits may appear in upper- or lowercase without changing the values. The bytes c3 a9 become é only when software reads them using UTF-8.

Recreate the garbled name

We know that the source intended to store Anaïs as UTF-8. The next example keeps one byte sequence and asks R to interpret it two ways.

expected_name <- "Ana\u00EFs"
name_bytes <- charToRaw(enc2utf8(expected_name))

misread_name <- rawToChar(name_bytes)
Encoding(misread_name) <- "latin1"
garbled_name <- enc2utf8(misread_name)

utf8_name <- rawToChar(name_bytes)
Encoding(utf8_name) <- "UTF-8"
decoded_name <- enc2utf8(utf8_name)

c(
  read_as_latin1 = garbled_name,
  read_as_utf8 = decoded_name
)
read_as_latin1   read_as_utf8 
      "Anaïs"        "Anaïs"

Both results came from the same bytes. Reading the bytes as Latin-1 produced Anaïs; reading them as UTF-8 produced Anaïs. The correct choice comes from the source’s encoding declaration or other trustworthy metadata, not from guessing based on appearance.

Unicode and UTF-8 do different jobs

Unicode and UTF-8 answer different questions. Unicode is a shared catalog that assigns code points for text from many writing systems, along with punctuation, symbols, and emoji.

UTF-8 describes how to store Unicode scalar values as bytes. It uses:

  • one byte for common ASCII characters;
  • two bytes for many accented letters;
  • three bytes for many other characters; and
  • four bytes for characters such as many emoji.

UTF-16 and UTF-32 encode the same Unicode code points in other ways. The reader of a file still needs to know which encoding the writer used.

NoteWhere ASCII and Latin-1 fit

ASCII grew from the need for incompatible machines to exchange text. The standard uses seven bits for 128 values covering English letters, digits, punctuation, and control instructions such as a line break. UTF-8 deliberately keeps those same values, so an ASCII file is also valid UTF-8.

Latin-1 came later and uses all eight bits, giving it 256 values. It adds characters used by several Western European languages, but it still cannot represent most of the world’s writing systems. Its byte values above 127 do not mean the same thing in UTF-8. Confusing Latin-1 with UTF-8 is a common cause of garbled text.

Unicode took a different approach: assign a common number to each text element, then let encodings such as UTF-8 decide how those numbers become bytes. That separation is why one system can cover far more than any single eight-bit code page.

What R knows about a string

A string is one text value. R may attach a label to a string saying that it is Latin-1, UTF-8, or a sequence of bytes. Encoding() reports that label, so we can check what R believes about the text it received. We will display the answer as a tibble, a table that prints its size and each column’s type.

strings <- c(
  plain_letter = "A",
  accented_letter = "\u00E9"
)

declared_encoding <- Encoding(strings)

knitr::kable(
  tibble(
    text = unname(strings),
    what_R_reports = unname(declared_encoding)
  ),
  col.names = c("Text", "What R reports"),
  caption = "How R labels two character strings",
  row.names = FALSE
)
How R labels two character strings
Text What R reports
A unknown
é UTF-8

R reports the plain letter as "unknown". The letter is still valid. ASCII text uses the same byte values in the encodings R supports, so R does not need a special label here.

An encoding label describes how R should interpret bytes; it does not convert them. Assigning a different label can leave every byte untouched. When the bytes need to change, use a conversion function.

Transcode known Latin-1 text

Some files are correctly stored as Latin-1 and need to be converted for a UTF-8 workflow. This operation is called transcoding. The next example creates "café" from known Latin-1 bytes, then uses iconv() to convert them.

latin1_text <- rawToChar(as.raw(c(0x63, 0x61, 0x66, 0xE9)))
Encoding(latin1_text) <- "latin1"

utf8_text <- iconv(
  latin1_text,
  from = "latin1",
  to = "UTF-8",
  sub = NA_character_
)

result <- list(
  text = utf8_text,
  latin1_bytes = charToRaw(latin1_text),
  utf8_bytes = charToRaw(utf8_text)
)

result
$text
[1] "café"

$latin1_bytes
[1] 63 61 66 e9

$utf8_bytes
[1] 63 61 66 c3 a9

The converted text still reads "café", but its final byte changes from e9 to the two-byte UTF-8 sequence c3 a9. sub = NA_character_ tells R to return a missing value rather than replace a byte without reporting the failure. That visible missing value gives us a chance to investigate the source.

Bytes that are not well-formed UTF-8

The attendance file arrived from a system that had already lost track of its own encoding, so part of it is not valid UTF-8 at all. UTF-8 has a strict shape: a byte such as c3 promises that a continuation byte follows. When the promised byte is missing, the sequence is malformed, and a UTF-8 decoder must reject or replace it. Another encoding, such as Latin-1, can assign characters to the same bytes, but the bytes alone cannot recover which interpretation the author intended.

The example below builds Hi followed by c3 28. The byte 28 is an opening parenthesis, not a continuation byte, so the pair is broken.

damaged_bytes <- as.raw(c(0x48, 0x69, 0x20, 0xC3, 0x28))
damaged_text <- rawToChar(damaged_bytes)
Encoding(damaged_text) <- "UTF-8"

prefix_is_valid <- map_lgl(
  seq_along(damaged_bytes),
  \(position) validUTF8(rawToChar(damaged_bytes[seq_len(position)]))
)
first_broken_byte <- which(!prefix_is_valid)[1]

strict_result <- iconv(
  damaged_text,
  from = "UTF-8",
  to = "UTF-8",
  sub = NA_character_
)
marked_result <- iconv(
  damaged_text,
  from = "UTF-8",
  to = "UTF-8",
  sub = "\uFFFD"
)

Three behaviors are worth separating. validUTF8() answers yes or no. Strict conversion returns NA, which is loud and easy to count. Replacement produces U+FFFD, the replacement character, which keeps the row readable while destroying the evidence of what was there.

Position four is where the prefix stops being decodable, and that number is worth keeping. Report the file, the offset, and the raw bytes rather than the repaired string. A byte offset lets someone open the source and see the record that produced it; Hi \uFFFD( does not.

Never write the repaired text back over the source. Keep the original bytes, record the substitution rule, and count how many values needed it.

A byte-order mark and a file with two encodings

Two more arrival problems show up in the same class list. Some tools begin a UTF-8 file with the three bytes ef bb bf, a byte-order mark, usually shortened to BOM. It carries no text. Left in place it becomes an invisible prefix on the first field, so name stops matching name.

The file also mixes encodings, because rows were appended by two systems. That is common in exported data and it means one encoding rule cannot decode the whole file.

byte_order_mark <- as.raw(c(0xEF, 0xBB, 0xBF))
file_rows <- list(
  header = c(byte_order_mark, charToRaw("name")),
  row_from_utf8_system = charToRaw(enc2utf8("caf\u00E9")),
  row_from_latin1_system = as.raw(c(0x63, 0x61, 0x66, 0xE9))
)
declared_source_encoding <- c("UTF-8", "UTF-8", "latin1")

drop_byte_order_mark <- function(bytes) {
  if (
    length(bytes) >= 3L &&
      identical(bytes[1:3], byte_order_mark)
  ) {
    bytes[-(1:3)]
  } else {
    bytes
  }
}

decode_row <- function(bytes, declared) {
  text <- rawToChar(drop_byte_order_mark(bytes))
  Encoding(text) <- declared
  enc2utf8(text)
}

row_report <- tibble(
  row = names(file_rows),
  stored_bytes = unname(map_chr(
    file_rows,
    \(bytes) paste(as.character(bytes), collapse = " ")
  )),
  starts_with_bom = unname(map_lgl(
    file_rows,
    \(bytes) {
      length(bytes) >= 3L &&
        identical(bytes[1:3], byte_order_mark)
    }
  )),
  valid_as_utf8 = unname(map_lgl(
    file_rows,
    \(bytes) validUTF8(rawToChar(drop_byte_order_mark(bytes)))
  )),
  declared_by_source = declared_source_encoding,
  decoded = unname(map2_chr(
    file_rows,
    declared_source_encoding,
    decode_row
  ))
)

knitr::kable(
  row_report,
  col.names = c(
    "Row",
    "Stored bytes",
    "Starts with BOM",
    "Valid as UTF-8",
    "Declared by source",
    "Decoded text"
  ),
  caption = "Three rows with two encodings and one byte-order mark",
  row.names = FALSE
)
Three rows with two encodings and one byte-order mark
Row Stored bytes Starts with BOM Valid as UTF-8 Declared by source Decoded text
header ef bb bf 6e 61 6d 65 TRUE TRUE UTF-8 name
row_from_utf8_system 63 61 66 c3 a9 FALSE TRUE UTF-8 café
row_from_latin1_system 63 61 66 e9 FALSE FALSE latin1 café

The last two rows display the same word from different bytes. That is the point of the provenance column: declared_by_source records where the decoding rule came from, and it is metadata from the sending system rather than a guess made from appearance. The third row would have been lost had the whole file been decoded as UTF-8, since its bytes are not valid there.

Detecting an encoding by inspection is possible and sometimes necessary, but a detector returns a ranked guess. Keep the guess, its confidence, and the byte evidence beside the decoded text so that a later reader can disagree with it.

What current R does

This site builds and tests with R 4.6.1. That is the version that built this page, not a promise that it remains the newest release; CRAN’s R for Windows page lists the current installer.

On Windows 10 version 1903 and later, and on Windows Server 2022 and later, current 64-bit R builds use UTF-8 as their native encoding, matching the usual setup on macOS and Linux. This removed many older Windows limitations around file names, comments, symbols, and text passed to external libraries.

Recent releases have tightened the edges too. R 4.5 made package metadata files such as DESCRIPTION and PACKAGES UTF-8, added readLines(encoding = "bytes") for inspecting a file without creating invalid strings, and made iconv(from = "") prefer a string’s declared encoding.

Those improvements make modern R safer; they do not tell R how an outside file was written. Older software may still produce Windows-1252, Latin-1, or another encoding, and one file may still contain rows from several systems.

We can ask R about the current session:

encoding_info <- l10n_info()

encoding_info[c("UTF-8", "Latin-1")]
$`UTF-8`
[1] TRUE

$`Latin-1`
[1] FALSE

A value of TRUE next to UTF-8 means that the current R session uses UTF-8 as its native encoding. This describes the session. A file received from somewhere else may use different rules.

Why identical-looking text may not match

Correcting the encoding fixes the obvious damage, but an exact comparison can still fail. Unicode can represent some visible characters in more than one way. The accented letter é can be:

  • one code point for the complete letter; or
  • the letter e followed by a separate accent mark.

Both forms look the same on screen. A duplicate check may still treat them as different sequences.

same_appearance <- c(
  one_code_point = "\u00E9",
  letter_plus_accent = "e\u0301"
)

counts <- nchar(same_appearance, type = "chars")
exact_match_before <- identical(
  unname(same_appearance[1]),
  unname(same_appearance[2])
)
exact_match_after <- identical(
  unname(stri_trans_nfc(same_appearance[1])),
  unname(stri_trans_nfc(same_appearance[2]))
)

knitr::kable(
  tibble(
    form = c("one code point", "letter plus accent"),
    displayed_text = unname(same_appearance),
    code_point_count = unname(counts)
  ),
  col.names = c("Form", "Displayed text", "Code point count"),
  caption = "Two ways to represent the same visible character",
  row.names = FALSE
)
Two ways to represent the same visible character
Form Displayed text Code point count
one code point é 1
letter plus accent é 2 
c(
  exact_match_before_normalizing = exact_match_before,
  exact_match_after_normalizing = exact_match_after
)
exact_match_before_normalizing  exact_match_after_normalizing 
                         FALSE                           TRUE

Normalization changes equivalent Unicode sequences into a consistent form. The example uses Normalization Form C, usually shortened to NFC. Normalize a working copy when equivalent forms should compare as equal. Keep the source text when its exact representation matters.

A safe routine for text files

When source metadata says the attendance file is UTF-8, the same bytes can be decoded as UTF-8 without guessing. When text comes from a file, website, or another system:

  1. Find out which encoding the source promises to use.
  2. Read the text using that encoding.
  3. Remove a byte-order mark before comparing or joining on the first field.
  4. Convert the working copy to UTF-8.
  5. Count the values that failed to decode and keep their byte offsets.
  6. Check accented letters, non-Latin scripts, and emoji.
  7. Normalize only when the task requires equivalent forms to match.
  8. Keep conversion warnings visible.

Avoid useBytes = TRUE for normal text work. It tells R to work one byte at a time and can split a multi-byte character.

What to remember

  • A visible grapheme cluster can contain one Unicode code point or several.
  • UTF-8 stores Unicode scalar values as bytes.
  • Reading bytes with the wrong encoding is one common cause of garbled text, but appearance alone does not prove the cause.
  • Some byte sequences are valid in no encoding you were promised. Decide whether they become NA or a replacement character, and record which.
  • A byte-order mark is invisible and still breaks an exact match.
  • One file can hold rows written under two encodings.
  • An encoding label does not convert text.
  • UTF-8 is a practical working format for text from many languages.
  • Text that looks identical may need Unicode normalization before comparison.

With these checks, Anaïs can survive the trip from file to screen and match an equivalent spelling when the task requires it. Good text handling has a human-sized test: the person who entered a name should get the same name back.

Sources