Drawing grammar links between words

Read dependency parses from a small trained model

word parsing
dependency parsing
workforce research
Learn how a dependency parser links words in workforce text and why attachment scores need limits.

Word labels are ready for the 28 Riverton sentences. The team then faces a harder decision: when a sentence says training is provided, which word is the thing provided, and does the sentence name a provider?

That question matters because labels alone do not say how words fit together. A sentence can contain experience and required while still saying the opposite of a simple keyword match. A passive sentence can also leave the provider unnamed, so the team needs links between words and a habit of checking them by eye.

A dependency parse says which word each token attaches to and what the link is called. The link points from one token to its head, the word it depends on. This lesson uses a local UDPipe model trained on the same 500-sentence English Web Treebank excerpt as the tagger in Lesson 17, with parsing turned on.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching. The parser and scoring files are local artifacts, so the lesson does not download a model while it runs.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • describe a dependency parse in plain language;
  • read root, compound, numeric modifier, passive subject, and punctuation links;
  • compare two parses of the same sentence; and
  • explain why a parser score from weblog posts needs a baseline and local checks.

Load the local parser inputs

The parser inputs come from local CSV files. dplyr, tibble, and purrr handle the table work; tokenizers and udpipe keep parsing tied to fixed tokens; digest records artifact fingerprints; and spacyr starts a comparison parse. The project helper use_project_spacy() starts the pinned spaCy pipeline.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(digest)
library(tokenizers)
library(udpipe)
library(spacyr)
source("R/use-spacy.R")

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

parser_score_path <- "data/treebank/parser-held-out-score.csv"
parser_model_path <- "data/treebank/en_ewt-500-parser.udpipe"
held_out_path <- "data/treebank/en_ewt-held-out.conllu"

parser_score <- read_csv(
  parser_score_path,
  na = character(),
  col_types = cols(
    measure = col_character(),
    value = col_double()
  )
)

metadata <- read_csv(
  "data/treebank/treebank-metadata.csv",
  na = character(),
  col_types = cols(
    artifact = col_character(),
    description = col_character(),
    source = col_character(),
    source_url = col_character(),
    retrieved_on = col_character(),
    license = col_character(),
    fingerprint = col_character()
  )
)

parser_score_fingerprint <- digest(
  paste(read_lines(parser_score_path), collapse = "\n"),
  algo = "sha256",
  serialize = FALSE
)
parser_model_fingerprint <- digest(
  parser_model_path,
  algo = "sha256",
  file = TRUE
)
expected_score_fingerprint <- metadata |>
  filter(artifact == "parser-held-out-score.csv") |>
  pull(fingerprint)
expected_model_fingerprint <- metadata |>
  filter(artifact == "en_ewt-500-parser.udpipe") |>
  pull(fingerprint)

held_out_lines <- read_lines(held_out_path)
held_out_tokens <- tibble(
  line = held_out_lines,
  sentence_index = cumsum(grepl("^# sent_id", held_out_lines))
) |>
  filter(!grepl("^#", line), nzchar(line)) |>
  mutate(fields = strsplit(line, "\t", fixed = TRUE)) |>
  filter(map_int(fields, length) >= 8L) |>
  transmute(
    sentence_index,
    token_id = map_chr(fields, 1),
    head_token_id = map_chr(fields, 7),
    dep_rel = map_chr(fields, 8)
  ) |>
  filter(
    !grepl("-", token_id, fixed = TRUE),
    !grepl(".", token_id, fixed = TRUE)
  )

stored_held_out_tokens <- as.integer(parser_score$value[parser_score$measure == "held_out_tokens"])
stored_correct_head <- as.integer(parser_score$value[parser_score$measure == "correct_head"])
stored_correct_head_and_relation <- as.integer(
  parser_score$value[parser_score$measure == "correct_head_and_relation"]
)
computed_unlabelled <- round(stored_correct_head / stored_held_out_tokens, 4)
computed_labelled <- round(stored_correct_head_and_relation / stored_held_out_tokens, 4)

pipeline <- use_project_spacy()
spacy_info <- spacy_pipeline_version()
spacy_info_table <- tibble(
  item = names(spacy_info),
  value = unname(unlist(spacy_info))
)

knitr::kable(
  parser_score,
  col.names = c("Measure", "Value"),
  caption = "Held-out parser scores bundled with the lesson",
  row.names = FALSE
)
Held-out parser scores bundled with the lesson
Measure Value
held_out_tokens 4007.0000
correct_head 2899.0000
unlabelled_attachment 0.7235
correct_head_and_relation 2629.0000
labelled_attachment 0.6561 
knitr::kable(
  spacy_info_table,
  col.names = c("Pipeline field", "Value"),
  caption = "Local spaCy pipeline used for the comparison table",
  row.names = FALSE
)
Local spaCy pipeline used for the comparison table
Pipeline field Value
name core_web_sm
version 3.8.0
lang en
license MIT
spacy 3.8.7

The chunk checks the score file against the held-out token count, recomputes both ratios, and matches the parser and score fingerprints in the metadata. The parser attaches 2,899 of 4,007 held-out tokens to the right head word, about 0.72, and gets both head and relation right for 2,629 tokens, about 0.66.

Parse fixed tokens

The parser receives one token per line because the model has no tokenizer. The vertical text block hands udpipe each sentence as a stack of fixed tokens.

parser <- udpipe_load_model(parser_model_path)

vertical <- sentences$text |>
  tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
  map_chr(\(tokens) paste(tokens, collapse = "\n"))

parsed <- udpipe_annotate(
  parser,
  x = vertical,
  doc_id = sentences$sentence_id,
  tokenizer = "vertical",
  tagger = "default",
  parser = "default"
) |>
  as.data.frame() |>
  as_tibble()

paid_parse <- parsed |>
  filter(doc_id == "s001") |>
  mutate(
    head_word = if_else(
      head_token_id == "0",
      "root",
      token[match(head_token_id, token_id)]
    )
  ) |>
  select(token, head_word, dep_rel)

knitr::kable(
  paid_parse,
  col.names = c("Token", "Head word", "Relation"),
  caption = "Dependency parse for 'Paid 12-week training is provided.'",
  row.names = FALSE
)
Dependency parse for ‘Paid 12-week training is provided.’
Token Head word Relation
Paid week compound
12 Paid flat
- Paid punct
week training compound
training provided nsubj:pass
is provided aux:pass
provided root root
. provided punct

The parse says provided is the root, the central word of this sentence. training attaches to provided as a passive subject, and is attaches as a passive helper verb. Inside Paid 12-week, the parser is wrong. It makes Paid part of week and hangs 12 off Paid. A reader would attach Paid to training and 12 to week.

The parse answers half the team’s question. training is the thing provided. Nothing in the sentence says who provides it, because English lets a passive drop the doer. There is no word for the parser to point at.

These tables use a few relation labels. root marks the central word. compound links words inside a larger noun phrase. flat is for flat name-like material, though it is a bad fit for 12 here. nummod marks a numeric modifier, and nmod marks a nominal modifier. nsubj:pass marks a passive subject, aux:pass marks a passive helper, ccomp marks a clausal complement, and punct marks punctuation.

Compare one sentence with spaCy

The same idea can come from a different tool. spaCy’s English parser does not use the Universal Dependencies label set. It uses another scheme, which is why nsubjpass will not be found in the UD documentation linked below.

spacy_s001 <- spacy_parse(
  c(s001 = "Paid 12-week training is provided."),
  pos = TRUE,
  lemma = TRUE,
  entity = TRUE,
  dependency = TRUE,
  nounphrase = TRUE
) |>
  as_tibble()

spacy_paid_parse <- spacy_s001 |>
  mutate(
    head_word = if_else(
      head_token_id == token_id,
      "ROOT",
      token[match(as.character(head_token_id), as.character(token_id))]
    )
  ) |>
  select(token, head_word, dep_rel, lemma, pos)

knitr::kable(
  spacy_paid_parse,
  col.names = c("Token", "Head word", "spaCy relation", "Lemma", "POS tag"),
  caption = "spaCy parse for the same sentence",
  row.names = FALSE
)
spaCy parse for the same sentence
Token Head word spaCy relation Lemma POS tag
Paid training nmod pay VERB
12 week nummod 12 NUM
- week punct - PUNCT
week training compound week NOUN
training provided nsubjpass training NOUN
is provided auxpass be AUX
provided ROOT ROOT provide VERB
. provided punct . PUNCT 
invisible(suppressMessages(spacy_finalize()))

The two tools produce different structures, not a spelling difference in the labels. spaCy attaches Paid to training and 12 to week; UDPipe attaches Paid to week and 12 to Paid. Both outputs still have tokens, heads, and labels, but they disagree about this sentence.

Read morphology in context

The parser output also includes the feats column. These are morphological features, compact labels for grammar carried by a token in this sentence.

paid_features <- parsed |>
  filter(doc_id == "s001", token %in% c("is", "provided")) |>
  select(token, feats)

knitr::kable(
  paid_features,
  col.names = c("Token", "Morphological features"),
  caption = "In-context features for two tokens in sentence s001",
  row.names = FALSE
)
In-context features for two tokens in sentence s001
Token Morphological features
is Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin
provided Tense=Past|VerbForm=Part|Voice=Pass

The feature string for is says it is a finite present-tense helper form. The feature string for provided says it is a past participle in passive voice. That is why the relation labels include passive forms.

Compare parsing with tagging

A parser score needs its own baseline. The chunk below uses a deliberately weak one: attach each token to the token before it, with the first token in a sentence attached to the root.

tagger_curve <- read_csv(
  "data/treebank/tagger-learning-curve.csv",
  na = character(),
  col_types = cols(
    training_sentences = col_integer(),
    training_tokens = col_integer(),
    held_out_tokens = col_integer(),
    correct_tokens = col_integer(),
    accuracy = col_double()
  )
)

tagging_score <- tagger_curve |>
  filter(training_sentences == 500L) |>
  transmute(
    system = "500-sentence tagger",
    correct = correct_tokens,
    tokens = held_out_tokens,
    accuracy
  )

attachment_baseline <- held_out_tokens |>
  group_by(sentence_index) |>
  mutate(previous_head = if_else(row_number() == 1L, "0", lag(token_id))) |>
  ungroup() |>
  summarise(
    system = "Previous-token baseline, head only",
    correct = sum(head_token_id == previous_head),
    tokens = n(),
    attachment = round(correct / tokens, 4),
    .groups = "drop"
  )

attachment_scores <- bind_rows(
  attachment_baseline,
  tibble(
    system = c(
      "500-sentence parser, head only",
      "500-sentence parser, head and relation"
    ),
    correct = c(stored_correct_head, stored_correct_head_and_relation),
    tokens = c(stored_held_out_tokens, stored_held_out_tokens),
    attachment = c(computed_unlabelled, computed_labelled)
  )
)

knitr::kable(
  tagging_score,
  col.names = c("System", "Correct tokens", "Tokens scored", "Accuracy"),
  caption = "Tagging accuracy on the held-out weblog tokens",
  row.names = FALSE
)
Tagging accuracy on the held-out weblog tokens
System Correct tokens Tokens scored Accuracy
500-sentence tagger 3641 4007 0.9087 
knitr::kable(
  attachment_scores,
  col.names = c("System", "Correct tokens", "Tokens scored", "Attachment score"),
  caption = "Parser attachment scores and a previous-token baseline",
  row.names = FALSE
)
Parser attachment scores and a previous-token baseline
System Correct tokens Tokens scored Attachment score
Previous-token baseline, head only 304 4007 0.0759
500-sentence parser, head only 2899 4007 0.7235
500-sentence parser, head and relation 2629 4007 0.6561

Attaching every token to the word before it gets 304 of 4,007 heads right. The trained parser gets 2,899 heads right, about 0.72. The tagger’s 0.91 belongs to a different task with a different set of choices, so the table keeps it separate. The head-and-relation row is stricter by construction: it can be right only when the head is right. These numbers come from one train-and-test split of thirteen weblog posts, and tokens inside a post are related because vocabulary, topic, and author repeat.

Check Riverton roots

The held-out score is a warning, not a local audit. Because the Riverton set has only 28 sentences, the lesson can compare each parser root with a hand-chosen root for the sentence.

parser_roots <- parsed |>
  filter(head_token_id == "0") |>
  transmute(sentence_id = doc_id, parser_root = token)

hand_roots <- tribble(
  ~sentence_id, ~hand_root,
  "s001", "provided",
  "s002", "required",
  "s003", "available",
  "s004", "need",
  "s005", "required",
  "s006", "pays",
  "s007", "part",
  "s008", "able",
  "s009", "required",
  "s010", "preferred",
  "s011", "available",
  "s012", "receives",
  "s013", "preferred",
  "s014", "uses",
  "s015", "provided",
  "s016", "apprenticeship",
  "s017", "required",
  "s018", "outdoors",
  "s019", "required",
  "s020", "required",
  "s021", "included",
  "s022", "skill",
  "s023", "HOUSE",
  "s024", "CERTIFICATE",
  "s025", "stipend",
  "s026", "classes",
  "s027", "required",
  "s028", "Apply"
)

root_audit <- hand_roots |>
  left_join(parser_roots, by = "sentence_id") |>
  mutate(root_matches = parser_root == hand_root)

root_summary <- root_audit |>
  summarise(
    right_root = sum(root_matches),
    wrong_root = sum(!root_matches),
    total = n(),
    .groups = "drop"
  )

root_mismatches <- root_audit |>
  filter(!root_matches)

knitr::kable(
  root_mismatches,
  col.names = c("Sentence ID", "Hand root", "Parser root", "Root matches"),
  caption = "Riverton sentences where the parser root needs review",
  row.names = FALSE
)
Riverton sentences where the parser root needs review
Sentence ID Hand root Parser root Root matches
s004 need skills FALSE
s017 required valid FALSE
s021 included Product FALSE
s023 HOUSE RIVERTON FALSE
s024 CERTIFICATE DATA FALSE
s026 classes Evening FALSE
s028 Apply October FALSE

Of the 28 Riverton sentences, the parser picks the hand-marked root in 21 and misses 7. The misses cluster in noun-heavy lines and flyer headings, where there is little sentence structure for the parser to hold on to.

included_parse <- parsed |>
  filter(doc_id == "s021") |>
  mutate(
    head_word = if_else(
      head_token_id == "0",
      "root",
      token[match(head_token_id, token_id)]
    )
  ) |>
  select(token, head_word, dep_rel)

knitr::kable(
  included_parse,
  col.names = c("Token", "Head word", "Relation"),
  caption = "A Riverton parse that needs human review",
  row.names = FALSE
)
A Riverton parse that needs human review
Token Head word Relation
Product root root
training included nsubj:pass
is included aux:pass
included Product ccomp
. Product punct

In Product training is included, the main claim is that training is included. The parser has made Product the head of the sentence and hung included beneath it as a subordinate clause, so the parser has the sentence inside out.

What to remember

  • A dependency parse links each token to a head word and names the relation.
  • UDPipe and spaCy disagree on the heads inside Paid 12-week.
  • A previous-token baseline gets 304 of 4,007 weblog heads right.
  • The trained parser gets 2,899 of 4,007 heads right, about 0.72.
  • The parser misses 7 of 28 hand-marked Riverton roots.
  • Parser links on job-posting text need direct inspection.

Use dependency links as leads for reading, especially around passives. On this corpus the useful warning is concrete: seven root choices need review before any downstream count trusts them.

Sources