Changing the wording without changing the instruction

Separate a real rewrite from a meaning-preservation check

natural language generation
paraphrasing
evaluation
Learn how to generate paraphrases and inspect names, numbers, negation, and modality without treating word overlap as equivalence.

June edits public notices for an archive. Repeating the same sentence in every channel sounds stiff, but a rewrite that changes who may enter or what visitors must bring creates a new instruction.

For this lesson, a paraphrase changes the wording while preserving the participants, quantities, negation, scope, and obligations that matter for the notice. Linguistic research sometimes uses a looser idea of approximate meaning. An operational notice needs a stricter rule because readers may act on one word such as may, must, or not.

The primary aim of paraphrasing is to preserve meaning while changing wording; summarization primarily compresses or selects content, and simplification primarily reduces complexity or register. A real output can serve more than one aim, so these are not mutually exclusive classes.

TipWhat you will learn

This lesson shows how to:

  • generate paraphrase candidates with a pinned local model;
  • keep an unchanged-input baseline beside the generated wording;
  • measure visible wording change without calling it semantic equivalence;
  • probe names, numbers, negation, and modality;
  • preserve model failures instead of filtering them away; and
  • record human meaning review as pending.

Load the same fixed checkpoint

The checkpoint is shared with the summarization lesson, but this page gives it a different task, prompt, baseline, and review rule. The model is loaded from the prepared local directory through huggingfaceR.

suppressPackageStartupMessages({
  library(dplyr)
  library(huggingfaceR)
  library(knitr)
  library(purrr)
  library(reticulate)
  library(stringr)
  library(tibble)
  library(tidyr)
})

source("R/use-nlg.R")

paraphrase_model <- load_nlg_pipeline(
  "qwen_1_5b_instruct",
  "text-generation"
)

model_record <- paraphrase_model$metadata |>
  select(model_id, revision, license)

kable(
  model_record,
  col.names = c("Model", "Revision", "License"),
  caption = "Pinned local model used for paraphrase generation",
  row.names = FALSE
)
Pinned local model used for paraphrase generation
Model Revision License
Qwen/Qwen2.5-1.5B-Instruct 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 Apache-2.0

Write the preservation rule first

The four notices below are constructed behavioral probes. They were chosen to expose common changes in operational meaning. They are not a random test set, and no population accuracy will be estimated from them.

protected_name, modality_class, and modality_pattern are blank when a probe does not need that check. A copy of the source is kept as the baseline. Copying preserves the original wording and facts, but it does not perform the requested rewrite.

paraphrase_inputs <- tribble(
  ~probe_id, ~source_text, ~protected_name, ~requires_negation,
  ~modality_class, ~modality_pattern,
  "PAR-01",
  "The clinic sent 24 reminder letters on Monday.",
  "",
  FALSE,
  "",
  "",
  "PAR-02",
  "Visitors may bring one notebook to the reading room.",
  "",
  FALSE,
  "permission",
  "\\b(may|can|allowed|permitted)\\b",
  "PAR-03",
  paste(
    "Maya Chen did not cancel the 14 September workshop;",
    "she moved it from Room 2 to Room 5."
  ),
  "Maya Chen",
  TRUE,
  "",
  "",
  "PAR-04",
  "Only archive staff must sign the restricted-room log.",
  "",
  FALSE,
  "obligation",
  "\\b(must|required|obliged|has to)\\b"
)

kable(
  paraphrase_inputs,
  col.names = c(
    "Probe ID",
    "Source notice",
    "Protected name",
    "Negation required",
    "Modality class",
    "Accepted surface markers"
  ),
  caption = "Constructed paraphrase probes and predeclared invariants",
  row.names = FALSE
)
Constructed paraphrase probes and predeclared invariants
Probe ID Source notice Protected name Negation required Modality class Accepted surface markers
PAR-01 The clinic sent 24 reminder letters on Monday. FALSE
PAR-02 Visitors may bring one notebook to the reading room. FALSE permission may|can|allowed|permitted)
PAR-03 Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. Maya Chen TRUE
PAR-04 Only archive staff must sign the restricted-room log. FALSE obligation must|required|obliged|has to)

Generate candidates without silent truncation

The complete chat-formatted prompt must stay below 192 input tokens. Greedy decoding and a 72-token output cap make the call repeatable, but deterministic generation can still produce a wrong paraphrase.

paraphrase_system <- paste(
  "Rewrite the notice in different words.",
  "Preserve every participant, quantity, date, negation, scope,",
  "permission, and obligation.",
  "Return the rewritten notice only."
)

paraphrase_requests <- paraphrase_inputs |>
  mutate(
    user_prompt = paste0(
      "PROBE ID: ", probe_id,
      "\nNOTICE: ", source_text
    ),
    model_prompt = map_chr(
      user_prompt,
      \(user) {
        nlg_chat_prompt(
          paraphrase_model$tokenizer,
          paraphrase_system,
          user
        )
      }
    ),
    input_tokens = map_int(
      model_prompt,
      \(prompt) nlg_token_count(paraphrase_model$tokenizer, prompt)
    )
  )

kable(
  paraphrase_requests |>
    select(probe_id, input_tokens),
  col.names = c("Probe ID", "Complete prompt tokens"),
  caption = "Paraphrase input budgets checked before generation",
  row.names = FALSE
)
Paraphrase input budgets checked before generation
Probe ID Complete prompt tokens
PAR-01 68
PAR-02 67
PAR-03 81
PAR-04 67

Only after that assertion succeeds does the lesson make any generation call.

paraphrase_results <- paraphrase_requests |>
  mutate(
    generation = map(
      model_prompt,
      \(prompt) {
        nlg_generate(
          paraphrase_model,
          prompt,
          max_new_tokens = 72L,
          repetition_penalty = 1.05
        )
      }
    ),
    candidate = map_chr(generation, "text"),
    candidate_display = str_replace_all(
      candidate,
      "\\s*\\n+\\s*",
      " / "
    ),
    output_tokens = map_int(generation, "output_tokens"),
    ended_by_eos = map_lgl(generation, "ended_by_eos"),
    hit_token_cap = map_lgl(generation, "hit_token_cap"),
    copy_baseline = source_text
  )

kable(
  paraphrase_results |>
    select(probe_id, source_text, candidate_display, copy_baseline),
  col.names = c(
    "Probe ID",
    "Source",
    "Generated candidate",
    "Unchanged-input baseline"
  ),
  caption = "Generated paraphrases retain failures and copies for inspection",
  row.names = FALSE
)
Generated paraphrases retain failures and copies for inspection
Probe ID Source Generated candidate Unchanged-input baseline
PAR-01 The clinic sent 24 reminder letters on Monday. CLINIC SENT 24 REMINDERS ON MONDAY. The clinic sent 24 reminder letters on Monday.
PAR-02 Visitors may bring one notebook to the reading room. Visitors are permitted to bring one notebook into the reading room. Visitors may bring one notebook to the reading room.
PAR-03 Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. PROBE ID: PAR-03 / NOTICE: Maya Chen did not cancel the 14 September workshop; instead, she relocated it to Room 5. Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5.
PAR-04 Only archive staff must sign the restricted-room log. Only archive staff are authorized to sign the restricted-room log. Only archive staff must sign the restricted-room log.

The copy baseline should pass every literal preservation check. It also fails the user’s request for different wording. The generated column has to solve both parts.

Measure wording change

This diagnostic lowercases each string, extracts letter and number sequences with the ICU regular-expression engine used by stringr, and computes Jaccard overlap on the unique token sets. An overlap of 1 means the sets match. A lower value means some visible wording changed.

token_set <- function(text) {
  str_extract_all(
    str_to_lower(text, locale = "en"),
    "[\\p{L}\\p{N}]+"
  )[[1]] |>
    unique()
}

jaccard_overlap <- function(left, right) {
  left_tokens <- token_set(left)
  right_tokens <- token_set(right)
  length(intersect(left_tokens, right_tokens)) /
    length(union(left_tokens, right_tokens))
}

wording_diagnostics <- paraphrase_results |>
  transmute(
    probe_id,
    exact_copy = candidate == source_text,
    unique_token_jaccard = map2_dbl(
      source_text,
      candidate,
      jaccard_overlap
    )
  )

kable(
  wording_diagnostics,
  digits = 2,
  col.names = c(
    "Probe ID",
    "Exact copy",
    "Unique-token Jaccard overlap"
  ),
  caption = "Wording diagnostics do not decide whether meaning survived",
  row.names = FALSE
)
Wording diagnostics do not decide whether meaning survived
Probe ID Exact copy Unique-token Jaccard overlap
PAR-01 FALSE 0.56
PAR-02 FALSE 0.67
PAR-03 FALSE 0.58
PAR-04 FALSE 0.67

A low overlap can come from a good synonym or a damaging change. A high overlap can hide a removed not. The score describes token sets, not equivalence.

Check visible invariants

The code now looks for exact Arabic numerals, the protected name, an explicit negation marker, and a short list of surface markers for the predeclared modality class. These conservative screens catch some preventable mistakes. They also miss valid alternatives: 24 may become twenty-four, and must may become have to. A screen result cannot establish whether the meaning survived.

extract_numbers <- function(text) {
  str_extract_all(text, "\\b\\d+\\b")[[1]] |>
    sort()
}

has_negation <- function(text) {
  str_detect(
    text,
    regex(
      "\\b(no|not|never|without|cannot|can't|didn't|did not)\\b",
      ignore_case = TRUE
    )
  )
}

invariant_checks <- paraphrase_results |>
  mutate(
    numbers_preserved = map2_lgl(
      source_text,
      candidate,
      \(source, output) {
        source_numbers <- extract_numbers(source)
        if (length(source_numbers) == 0L) {
          NA
        } else {
          identical(source_numbers, extract_numbers(output))
        }
      }
    ),
    name_present = map2_lgl(
      protected_name,
      candidate,
      \(name, output) {
        if (!nzchar(name)) {
          NA
        } else {
          str_detect(output, fixed(name))
        }
      }
    ),
    negation_visible = map2_lgl(
      requires_negation,
      candidate,
      \(required, output) {
        if (!required) NA else has_negation(output)
      }
    ),
    modal_visible = map2_lgl(
      modality_pattern,
      candidate,
      \(pattern, output) {
        if (!nzchar(pattern)) {
          NA
        } else {
          str_detect(
            output,
            regex(pattern, ignore_case = TRUE)
          )
        }
      }
    )
  ) |>
  select(
    probe_id,
    numbers_preserved,
    name_present,
    negation_visible,
    modal_visible
  )

invariant_check_display <- invariant_checks |>
  mutate(
    across(
      c(
        numbers_preserved,
        name_present,
        negation_visible,
        modal_visible
      ),
      nlg_screen_label
    )
  )

kable(
  invariant_check_display,
  col.names = c(
    "Probe ID",
    "Arabic-numeral screen",
    "Protected-name screen",
    "Negation screen",
    "Modality screen"
  ),
  caption = "Conservative screens for predeclared paraphrase invariants",
  row.names = FALSE
)
Conservative screens for predeclared paraphrase invariants
Probe ID Arabic-numeral screen Protected-name screen Negation screen Modality screen
PAR-01 not flagged by screen not applicable not applicable not applicable
PAR-02 not applicable not applicable not applicable not flagged by screen
PAR-03 held for human review not flagged by screen not flagged by screen not applicable
PAR-04 not applicable not applicable not applicable held for human review 
screen_actions <- invariant_checks |>
  rowwise() |>
  mutate(
    applicable_screens = sum(!is.na(c_across(c(
      numbers_preserved,
      name_present,
      negation_visible,
      modal_visible
    )))),
    flagged_screens = sum(c_across(c(
      numbers_preserved,
      name_present,
      negation_visible,
      modal_visible
    )) %in% FALSE)
  ) |>
  ungroup() |>
  select(probe_id, applicable_screens, flagged_screens)

paraphrase_actions <- wording_diagnostics |>
  left_join(screen_actions, by = "probe_id") |>
  mutate(
    action = case_when(
      exact_copy ~ "revise wording",
      flagged_screens > 0L ~ "hold for human meaning review",
      TRUE ~ "send to human meaning review"
    )
  ) |>
  select(probe_id, flagged_screens, action)

kable(
  paraphrase_actions,
  col.names = c("Probe ID", "Screen flags", "Next action"),
  caption = "Screen flags route candidates to human meaning review",
  row.names = FALSE
)
Screen flags route candidates to human meaning review
Probe ID Screen flags Next action
PAR-01 0 send to human meaning review
PAR-02 0 send to human meaning review
PAR-03 1 hold for human meaning review
PAR-04 1 hold for human meaning review

Any applicable FALSE value is a reason to hold the candidate for human review, not proof that its meaning changed. In this render, reading source and output together shows that the stress probe echoes prompt labels and drops Room 2. The obligation probe replaces must sign with authorized to sign, which raises a substantive obligation concern. Its modal screen also flags the row, but the same finite screen would miss or flag some equivalent phrasings, so the human comparison carries the conclusion.

Keep the decision honest

The model has completed generation, and the automatic checks have completed their limited jobs. No human has approved the candidates.

paraphrase_review <- tibble(
  review = c(
    "local model execution",
    "complete prompt within 192 tokens",
    "wording-change diagnostic",
    "surface invariant checks",
    "human meaning-preservation review",
    "human prose review"
  ),
  status = c(
    sprintf(
      "completed: %d candidates",
      sum(nzchar(paraphrase_results$candidate))
    ),
    sprintf(
      "within budget: %d/%d candidates",
      sum(paraphrase_results$input_tokens <= 192L),
      nrow(paraphrase_results)
    ),
    sprintf(
      "computed: %d/%d candidates changed wording",
      sum(!wording_diagnostics$exact_copy),
      nrow(wording_diagnostics)
    ),
    sprintf(
      "held: %d/%d candidates flagged",
      sum(paraphrase_actions$flagged_screens > 0L),
      nrow(paraphrase_actions)
    ),
    "pending",
    "pending"
  )
)

kable(
  paraphrase_review,
  col.names = c("Review gate", "Status"),
  caption = "A generated rewrite is not an approved paraphrase",
  row.names = FALSE
)
A generated rewrite is not an approved paraphrase
Review gate Status
local model execution completed: 4 candidates
complete prompt within 192 tokens within budget: 4/4 candidates
wording-change diagnostic computed: 4/4 candidates changed wording
surface invariant checks held: 2/4 candidates flagged
human meaning-preservation review pending
human prose review pending

June can accept a candidate only after reading it beside the notice and applying the rule stated at the start. Generation, visible wording change, and invariant flags are evidence for that review. None replaces it.

What to remember

  • A paraphrase changes wording while preserving the information that matters.
  • Copying is a useful preservation baseline, but it is not a successful rewrite.
  • Token overlap measures wording, not semantic equivalence.
  • Check names, numbers, negation, scope, permission, and obligation separately.
  • Keep failed model outputs visible when they reveal a broken invariant.
  • Record human meaning and prose review apart from automatic execution.

Sources