Checking small grammar rules

Score three checks and keep the limits visible

sentences and paragraphs
grammar checking
workforce research
Learn how small grammar checks can find planted errors and why they are not a full grammar checker.

Correctly spelled sentences can still make a public summary sound broken, which is why the coordinator hesitates. Real words are not enough when the words do not fit together.

Spelling asks whether a word exists. Grammar asks whether the words fit together in a sentence. This lesson builds three small checks and scores them on sentences where the planted errors are known.

This project does not install a grammar checker, so the lesson builds three small checks by hand and treats them as a demonstration. Offline means the check runs from local files, without sending text to a web service.

Note

The Riverton sentences and marked grammar examples are invented for this lesson.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain the difference between spelling and grammar checks;
  • use UDPipe dependency output to compare a subject and a verb;
  • find doubled words with a stringr pattern;
  • test a simple a and an rule; and
  • explain why a demonstration check is not a working grammar checker.

Make a marked test set

The setup reads the Riverton CSV, prepares words with tokenizers and purrr, runs two pattern checks with stringr, and sends sentences to a local udpipe dependency parser. dplyr and tibble organise the results. A dependency parser labels how words relate to one another, such as which noun is the subject of a verb.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tokenizers)
library(udpipe)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

marked_sentences <- tibble(
  example_id = sprintf("g%02d", 1:14),
  text = c(
    "The worker checks the list.",
    "The worker check the list.",
    "The applicants submit forms.",
    "The applicants submits forms.",
    "Bring a badge to orientation.",
    "Bring a envelope to orientation.",
    "Apply for an evening class.",
    "Apply for a evening class.",
    "The session begins in an hour.",
    "A university partner sends trainers.",
    "The coordinator checks the the form.",
    "The team needs needs support.",
    "The employee uses a spreadsheet.",
    "The employee use a spreadsheet."
  ),
  planted_check = c(
    "none",
    "subject-verb number",
    "none",
    "subject-verb number",
    "none",
    "article letters",
    "none",
    "article letters",
    "none",
    "none",
    "doubled word",
    "doubled word",
    "none",
    "subject-verb number"
  )
) |>
  mutate(has_planted_error = planted_check != "none")

knitr::kable(
  marked_sentences,
  col.names = c("Example ID", "Sentence", "Planted check", "Has planted error"),
  caption = "Author-marked sentences before running the checks",
  row.names = FALSE
)
Author-marked sentences before running the checks
Example ID Sentence Planted check Has planted error
g01 The worker checks the list. none FALSE
g02 The worker check the list. subject-verb number TRUE
g03 The applicants submit forms. none FALSE
g04 The applicants submits forms. subject-verb number TRUE
g05 Bring a badge to orientation. none FALSE
g06 Bring a envelope to orientation. article letters TRUE
g07 Apply for an evening class. none FALSE
g08 Apply for a evening class. article letters TRUE
g09 The session begins in an hour. none FALSE
g10 A university partner sends trainers. none FALSE
g11 The coordinator checks the the form. doubled word TRUE
g12 The team needs needs support. doubled word TRUE
g13 The employee uses a spreadsheet. none FALSE
g14 The employee use a spreadsheet. subject-verb number TRUE

Seven of the 14 test sentences are marked as correct, and seven contain exactly one planted error. These examples describe only themselves. A checker that works on them has not solved grammar checking.

Run the three checks

The subject-verb check uses the UDPipe parser’s feats column, which can contain values such as Number=Sing or Number=Plur. It looks for an nsubj relation, short for nominal subject, pointing to a verb. In the code, \(tokens) is R shorthand for a small function applied to one sentence’s tokens.

extract_number <- function(feats) {
  str_match(coalesce(feats, ""), "Number=([^|]+)")[, 2]
}

find_subject_verb_flags <- function(annotated) {
  annotated_with_number <- annotated |>
    mutate(number = extract_number(feats))

  subjects <- annotated_with_number |>
    filter(dep_rel == "nsubj") |>
    transmute(
      doc_id,
      head_token_id,
      subject_token = token,
      subject_number = number
    )

  verbs <- annotated_with_number |>
    filter(upos %in% c("VERB", "AUX"), !is.na(number)) |>
    transmute(
      doc_id,
      verb_token_id = token_id,
      verb_token = token,
      verb_number = number
    )

  verbs |>
    inner_join(subjects, by = c("doc_id", "verb_token_id" = "head_token_id")) |>
    mutate(flag = !is.na(subject_number) & subject_number != verb_number)
}

score_check <- function(flag, expected) {
  tibble(
    true_positive = sum(flag & expected),
    false_positive = sum(flag & !expected),
    false_negative = sum(!flag & expected),
    true_negative = sum(!flag & !expected)
  )
}

parser <- udpipe_load_model("data/treebank/en_ewt-500-parser.udpipe")

marked_vertical <- marked_sentences$text |>
  tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
  map_chr(\(tokens) paste(tokens, collapse = "\n"))

marked_annotated <- udpipe_annotate(
  parser,
  x = marked_vertical,
  doc_id = marked_sentences$example_id,
  tokenizer = "vertical",
  tagger = "default",
  parser = "default"
) |>
  as.data.frame() |>
  as_tibble()

agreement_pairs <- find_subject_verb_flags(marked_annotated)
agreement_flags <- agreement_pairs |>
  summarise(agreement_flag = any(flag), .by = doc_id)

doubled_pattern <- regex("\\b([[:alpha:]]+)\\s+\\1\\b", ignore_case = TRUE)
article_pattern <- regex(
  "\\b(a)\\s+[aeiou][[:alpha:]]*|\\b(an)\\s+[^aeiou\\W][[:alpha:]]*",
  ignore_case = TRUE
)

scored_examples <- marked_sentences |>
  left_join(agreement_flags, by = c("example_id" = "doc_id")) |>
  mutate(
    agreement_flag = coalesce(agreement_flag, FALSE),
    doubled_flag = str_detect(text, doubled_pattern),
    article_flag = str_detect(text, article_pattern)
  )

check_scores <- bind_rows(
  score_check(
    scored_examples$agreement_flag,
    scored_examples$planted_check == "subject-verb number"
  ) |>
    mutate(check = "subject-verb number"),
  score_check(
    scored_examples$doubled_flag,
    scored_examples$planted_check == "doubled word"
  ) |>
    mutate(check = "doubled word"),
  score_check(
    scored_examples$article_flag,
    scored_examples$planted_check == "article letters"
  ) |>
    mutate(check = "article letters")
) |>
  select(check, everything())

expected_scores <- tibble(
  check = c("subject-verb number", "doubled word", "article letters"),
  true_positive = c(3L, 2L, 2L),
  false_positive = c(0L, 0L, 2L),
  false_negative = c(0L, 0L, 0L),
  true_negative = c(11L, 12L, 10L)
)

knitr::kable(
  agreement_pairs |>
    select(doc_id, subject_token, subject_number, verb_token, verb_number, flag),
  col.names = c("Example ID", "Subject", "Subject number", "Verb", "Verb number", "Flagged"),
  caption = "Subject-verb pairs found by the dependency parser",
  row.names = FALSE
)
Subject-verb pairs found by the dependency parser
Example ID Subject Subject number Verb Verb number Flagged
g01 worker Sing checks Sing FALSE
g02 worker Sing check Plur TRUE
g03 applicants Plur submit Plur FALSE
g04 applicants Plur submits Sing TRUE
g09 session Sing begins Sing FALSE
g10 partner Sing sends Sing FALSE
g11 coordinator Sing checks Sing FALSE
g12 team Sing needs Sing FALSE
g13 employee Sing uses Sing FALSE
g14 employee Sing use Plur TRUE 
knitr::kable(
  check_scores,
  col.names = c("Check", "True positives", "False positives", "False negatives", "True negatives"),
  caption = "Scores for the three grammar checks on marked examples",
  row.names = FALSE
)
Scores for the three grammar checks on marked examples
Check True positives False positives False negatives True negatives
subject-verb number 3 0 0 11
doubled word 2 0 0 12
article letters 2 2 0 10

The checks and expected answers share an author, so the score is a teaching check rather than an accuracy estimate. A true positive is a planted error that a check flags. A false positive is a correct sentence that a check flags. A false negative is a planted error that a check misses. A true negative is a correct sentence it leaves alone.

The subject-verb number check catches all three planted number errors in this small set. The doubled-word check catches both repeated-word examples. The article check catches the two planted a mistakes, but it also flags two correct sentences: an hour and A university. That rule uses the first letter, while English article choice depends on sound. Hour begins with a silent h, so it sounds like a vowel. University begins with a y sound, so it sounds like a consonant. The rule reads letters and cannot hear either word.

A real grammar checker uses thousands of rules, a trained model, or both. These three checks are a demonstration of the idea. The UDPipe model was trained here on 500 sentences and is deliberately weak, so every subject-verb result inherits its mistakes. The article check is wrong by design so the failure is visible.

Try the Riverton sentences

The same three checks can be run over the 28 Riverton sentences. The output is a screening list, not a proof that every unflagged sentence is correct.

riverton_vertical <- sentences$text |>
  tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
  map_chr(\(tokens) paste(tokens, collapse = "\n"))

riverton_annotated <- udpipe_annotate(
  parser,
  x = riverton_vertical,
  doc_id = sentences$sentence_id,
  tokenizer = "vertical",
  tagger = "default",
  parser = "default"
) |>
  as.data.frame() |>
  as_tibble()

riverton_agreement_pairs <- find_subject_verb_flags(riverton_annotated)
riverton_agreement_ids <- riverton_agreement_pairs |>
  filter(flag) |>
  pull(doc_id)
riverton_pair_sentence_count <- n_distinct(riverton_agreement_pairs$doc_id)

riverton_flags <- sentences |>
  transmute(
    sentence_id,
    text,
    agreement_flag = sentence_id %in% riverton_agreement_ids,
    doubled_flag = str_detect(text, doubled_pattern),
    article_flag = str_detect(text, article_pattern)
  ) |>
  mutate(any_flag = agreement_flag | doubled_flag | article_flag)

riverton_flag_summary <- tibble(
  check = c("subject-verb number", "doubled word", "article letters", "any check"),
  flagged_sentences = c(
    sum(riverton_flags$agreement_flag),
    sum(riverton_flags$doubled_flag),
    sum(riverton_flags$article_flag),
    sum(riverton_flags$any_flag)
  )
)

riverton_coverage <- tibble(
  measure = c("sentences with subject-verb pair", "sentences without subject-verb pair"),
  sentences = c(
    riverton_pair_sentence_count,
    nrow(sentences) - riverton_pair_sentence_count
  )
)

knitr::kable(
  riverton_flag_summary,
  col.names = c("Check", "Flagged Riverton sentences"),
  caption = "Grammar-check flags in the 28 Riverton sentences",
  row.names = FALSE
)
Grammar-check flags in the 28 Riverton sentences
Check Flagged Riverton sentences
subject-verb number 0
doubled word 0
article letters 0
any check 0 
knitr::kable(
  riverton_coverage,
  col.names = c("Coverage measure", "Sentences"),
  caption = "Subject-verb coverage in the Riverton sentences",
  row.names = FALSE
)
Subject-verb coverage in the Riverton sentences
Coverage measure Sentences
sentences with subject-verb pair 4
sentences without subject-verb pair 24

The three checks flag no Riverton sentence. The subject-verb check found only four subject-and-verb pairs in all 28 sentences, because it needs the tagger to mark a number feature on the verb and English verbs rarely carry one. On the other 24 sentences the check could not fire. A zero mostly measures coverage, not correctness.

Grammar checking in production

The three custom checks above demonstrate the concept, but real offline grammar checking requires a mature system. In the NLP ecosystem, open-source tools like LanguageTool are standard. Production grammar systems distinguish between:

  • Detection vs. correction: Highlighting a possible error (detection) is much safer than automatically rewriting the text (correction), especially when the engine’s coverage is low.
  • Normative context and auditability: What counts as “correct” grammar depends on the intended style (e.g., formal business English vs. informal messaging). Rule-based systems provide an auditable explanation for why text was flagged, whereas deep learning models can suggest fluent corrections but often struggle to explain their normative reasoning.
  • Rule/model/error-type comparison: Rule-based engines are highly precise for local stylistic and typographical errors (like doubled words or wrong articles). Neural models are generally used for catching semantic confusion where the grammar technically parses but the meaning is broken.

What to remember

  • Spelling checks words; grammar checks how words fit together.
  • UDPipe’s feats and nsubj labels can support a small subject-verb check.
  • A doubled-word pattern is simple and useful for one common typo.
  • The article-letter rule falsely flags an hour and A university.
  • These three checks are demonstrations, not an offline grammar checker for R.
  • No Riverton flags here means low coverage, not clean grammar.

The three checks can produce a review queue. They cannot approve public text on their own, especially when the most technical check fires on only four Riverton sentences.

Sources