library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(digest)
library(tokenizers)
library(udpipe)
library(spacyr)
source("R/use-spacy.R")
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
parser_score_path <- "data/treebank/parser-held-out-score.csv"
parser_model_path <- "data/treebank/en_ewt-500-parser.udpipe"
held_out_path <- "data/treebank/en_ewt-held-out.conllu"
parser_score <- read_csv(
parser_score_path,
na = character(),
col_types = cols(
measure = col_character(),
value = col_double()
)
)
metadata <- read_csv(
"data/treebank/treebank-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
source_url = col_character(),
retrieved_on = col_character(),
license = col_character(),
fingerprint = col_character()
)
)
parser_score_fingerprint <- digest(
paste(read_lines(parser_score_path), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)
parser_model_fingerprint <- digest(
parser_model_path,
algo = "sha256",
file = TRUE
)
expected_score_fingerprint <- metadata |>
filter(artifact == "parser-held-out-score.csv") |>
pull(fingerprint)
expected_model_fingerprint <- metadata |>
filter(artifact == "en_ewt-500-parser.udpipe") |>
pull(fingerprint)
held_out_lines <- read_lines(held_out_path)
held_out_tokens <- tibble(
line = held_out_lines,
sentence_index = cumsum(grepl("^# sent_id", held_out_lines))
) |>
filter(!grepl("^#", line), nzchar(line)) |>
mutate(fields = strsplit(line, "\t", fixed = TRUE)) |>
filter(map_int(fields, length) >= 8L) |>
transmute(
sentence_index,
token_id = map_chr(fields, 1),
head_token_id = map_chr(fields, 7),
dep_rel = map_chr(fields, 8)
) |>
filter(
!grepl("-", token_id, fixed = TRUE),
!grepl(".", token_id, fixed = TRUE)
)
stored_held_out_tokens <- as.integer(parser_score$value[parser_score$measure == "held_out_tokens"])
stored_correct_head <- as.integer(parser_score$value[parser_score$measure == "correct_head"])
stored_correct_head_and_relation <- as.integer(
parser_score$value[parser_score$measure == "correct_head_and_relation"]
)
computed_unlabelled <- round(stored_correct_head / stored_held_out_tokens, 4)
computed_labelled <- round(stored_correct_head_and_relation / stored_held_out_tokens, 4)
pipeline <- use_project_spacy()
spacy_info <- spacy_pipeline_version()
spacy_info_table <- tibble(
item = names(spacy_info),
value = unname(unlist(spacy_info))
)Drawing grammar links between words
Read dependency parses from a small trained model
word parsing
dependency parsing
workforce research
Learn how a dependency parser links words in workforce text and why attachment scores need limits.
Word labels are ready for the 28 Riverton sentences. The team then faces a harder decision: when a sentence says training is provided, which word is the thing provided, and does the sentence name a provider?
That question matters because labels alone do not say how words fit together. A sentence can contain experience and required while still saying the opposite of a simple keyword match. A passive sentence can also leave the provider unnamed, so the team needs links between words and a habit of checking them by eye.
A dependency parse says which word each token attaches to and what the link is called. The link points from one token to its head, the word it depends on. This lesson uses a local UDPipe model trained on the same 500-sentence English Web Treebank excerpt as the tagger in Lesson 17, with parsing turned on.
Note
The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching. The parser and scoring files are local artifacts, so the lesson does not download a model while it runs.
TipWhat you will learn
By the end of this lesson, you will be able to:
- describe a dependency parse in plain language;
- read root, compound, numeric modifier, passive subject, and punctuation links;
- compare two parses of the same sentence; and
- explain why a parser score from weblog posts needs a baseline and local checks.
Load the local parser inputs
The parser inputs come from local CSV files. dplyr, tibble, and purrr handle the table work; tokenizers and udpipe keep parsing tied to fixed tokens; digest records artifact fingerprints; and spacyr starts a comparison parse. The project helper use_project_spacy() starts the pinned spaCy pipeline.
| Measure | Value |
|---|---|
| held_out_tokens | 4007.0000 |
| correct_head | 2899.0000 |
| unlabelled_attachment | 0.7235 |
| correct_head_and_relation | 2629.0000 |
| labelled_attachment | 0.6561 |
| Pipeline field | Value |
|---|---|
| name | core_web_sm |
| version | 3.8.0 |
| lang | en |
| license | MIT |
| spacy | 3.8.7 |
The chunk checks the score file against the held-out token count, recomputes both ratios, and matches the parser and score fingerprints in the metadata. The parser attaches 2,899 of 4,007 held-out tokens to the right head word, about 0.72, and gets both head and relation right for 2,629 tokens, about 0.66.
Parse fixed tokens
The parser receives one token per line because the model has no tokenizer. The vertical text block hands udpipe each sentence as a stack of fixed tokens.
parser <- udpipe_load_model(parser_model_path)
vertical <- sentences$text |>
tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
map_chr(\(tokens) paste(tokens, collapse = "\n"))
parsed <- udpipe_annotate(
parser,
x = vertical,
doc_id = sentences$sentence_id,
tokenizer = "vertical",
tagger = "default",
parser = "default"
) |>
as.data.frame() |>
as_tibble()
paid_parse <- parsed |>
filter(doc_id == "s001") |>
mutate(
head_word = if_else(
head_token_id == "0",
"root",
token[match(head_token_id, token_id)]
)
) |>
select(token, head_word, dep_rel)| Token | Head word | Relation |
|---|---|---|
| Paid | week | compound |
| 12 | Paid | flat |
| - | Paid | punct |
| week | training | compound |
| training | provided | nsubj:pass |
| is | provided | aux:pass |
| provided | root | root |
| . | provided | punct |
The parse says provided is the root, the central word of this sentence. training attaches to provided as a passive subject, and is attaches as a passive helper verb. Inside Paid 12-week, the parser is wrong. It makes Paid part of week and hangs 12 off Paid. A reader would attach Paid to training and 12 to week.
The parse answers half the team’s question. training is the thing provided. Nothing in the sentence says who provides it, because English lets a passive drop the doer. There is no word for the parser to point at.
These tables use a few relation labels. root marks the central word. compound links words inside a larger noun phrase. flat is for flat name-like material, though it is a bad fit for 12 here. nummod marks a numeric modifier, and nmod marks a nominal modifier. nsubj:pass marks a passive subject, aux:pass marks a passive helper, ccomp marks a clausal complement, and punct marks punctuation.
Compare one sentence with spaCy
The same idea can come from a different tool. spaCy’s English parser does not use the Universal Dependencies label set. It uses another scheme, which is why nsubjpass will not be found in the UD documentation linked below.
spacy_s001 <- spacy_parse(
c(s001 = "Paid 12-week training is provided."),
pos = TRUE,
lemma = TRUE,
entity = TRUE,
dependency = TRUE,
nounphrase = TRUE
) |>
as_tibble()
spacy_paid_parse <- spacy_s001 |>
mutate(
head_word = if_else(
head_token_id == token_id,
"ROOT",
token[match(as.character(head_token_id), as.character(token_id))]
)
) |>
select(token, head_word, dep_rel, lemma, pos)
invisible(suppressMessages(spacy_finalize()))| Token | Head word | spaCy relation | Lemma | POS tag |
|---|---|---|---|---|
| Paid | training | nmod | pay | VERB |
| 12 | week | nummod | 12 | NUM |
| - | week | punct | - | PUNCT |
| week | training | compound | week | NOUN |
| training | provided | nsubjpass | training | NOUN |
| is | provided | auxpass | be | AUX |
| provided | ROOT | ROOT | provide | VERB |
| . | provided | punct | . | PUNCT |
The two tools produce different structures, not a spelling difference in the labels. spaCy attaches Paid to training and 12 to week; UDPipe attaches Paid to week and 12 to Paid. Both outputs still have tokens, heads, and labels, but they disagree about this sentence.
Read morphology in context
The parser output also includes the feats column. These are morphological features, compact labels for grammar carried by a token in this sentence.
paid_features <- parsed |>
filter(doc_id == "s001", token %in% c("is", "provided")) |>
select(token, feats)| Token | Morphological features |
|---|---|
| is | Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin |
| provided | Tense=Past|VerbForm=Part|Voice=Pass |
The feature string for is says it is a finite present-tense helper form. The feature string for provided says it is a past participle in passive voice. That is why the relation labels include passive forms.
Compare parsing with tagging
A parser score needs its own baseline. The chunk below uses a deliberately weak one: attach each token to the token before it, with the first token in a sentence attached to the root.
tagger_curve <- read_csv(
"data/treebank/tagger-learning-curve.csv",
na = character(),
col_types = cols(
training_sentences = col_integer(),
training_tokens = col_integer(),
held_out_tokens = col_integer(),
correct_tokens = col_integer(),
accuracy = col_double()
)
)
tagging_score <- tagger_curve |>
filter(training_sentences == 500L) |>
transmute(
system = "500-sentence tagger",
correct = correct_tokens,
tokens = held_out_tokens,
accuracy
)
attachment_baseline <- held_out_tokens |>
group_by(sentence_index) |>
mutate(previous_head = if_else(row_number() == 1L, "0", lag(token_id))) |>
ungroup() |>
summarise(
system = "Previous-token baseline, head only",
correct = sum(head_token_id == previous_head),
tokens = n(),
attachment = round(correct / tokens, 4),
.groups = "drop"
)
attachment_scores <- bind_rows(
attachment_baseline,
tibble(
system = c(
"500-sentence parser, head only",
"500-sentence parser, head and relation"
),
correct = c(stored_correct_head, stored_correct_head_and_relation),
tokens = c(stored_held_out_tokens, stored_held_out_tokens),
attachment = c(computed_unlabelled, computed_labelled)
)
)| System | Correct tokens | Tokens scored | Accuracy |
|---|---|---|---|
| 500-sentence tagger | 3641 | 4007 | 0.9087 |
| System | Correct tokens | Tokens scored | Attachment score |
|---|---|---|---|
| Previous-token baseline, head only | 304 | 4007 | 0.0759 |
| 500-sentence parser, head only | 2899 | 4007 | 0.7235 |
| 500-sentence parser, head and relation | 2629 | 4007 | 0.6561 |
Attaching every token to the word before it gets 304 of 4,007 heads right. The trained parser gets 2,899 heads right, about 0.72. The tagger’s 0.91 belongs to a different task with a different set of choices, so the table keeps it separate. The head-and-relation row is stricter by construction: it can be right only when the head is right. These numbers come from one train-and-test split of thirteen weblog posts, and tokens inside a post are related because vocabulary, topic, and author repeat.
Check Riverton roots
The held-out score is a warning, not a local audit. Because the Riverton set has only 28 sentences, the lesson can compare each parser root with a hand-chosen root for the sentence.
parser_roots <- parsed |>
filter(head_token_id == "0") |>
transmute(sentence_id = doc_id, parser_root = token)
hand_roots <- tribble(
~sentence_id, ~hand_root,
"s001", "provided",
"s002", "required",
"s003", "available",
"s004", "need",
"s005", "required",
"s006", "pays",
"s007", "part",
"s008", "able",
"s009", "required",
"s010", "preferred",
"s011", "available",
"s012", "receives",
"s013", "preferred",
"s014", "uses",
"s015", "provided",
"s016", "apprenticeship",
"s017", "required",
"s018", "outdoors",
"s019", "required",
"s020", "required",
"s021", "included",
"s022", "skill",
"s023", "HOUSE",
"s024", "CERTIFICATE",
"s025", "stipend",
"s026", "classes",
"s027", "required",
"s028", "Apply"
)
root_audit <- hand_roots |>
left_join(parser_roots, by = "sentence_id") |>
mutate(root_matches = parser_root == hand_root)
root_summary <- root_audit |>
summarise(
right_root = sum(root_matches),
wrong_root = sum(!root_matches),
total = n(),
.groups = "drop"
)
root_mismatches <- root_audit |>
filter(!root_matches)| Sentence ID | Hand root | Parser root | Root matches |
|---|---|---|---|
| s004 | need | skills | FALSE |
| s017 | required | valid | FALSE |
| s021 | included | Product | FALSE |
| s023 | HOUSE | RIVERTON | FALSE |
| s024 | CERTIFICATE | DATA | FALSE |
| s026 | classes | Evening | FALSE |
| s028 | Apply | October | FALSE |
Of the 28 Riverton sentences, the parser picks the hand-marked root in 21 and misses 7. The misses cluster in noun-heavy lines and flyer headings, where there is little sentence structure for the parser to hold on to.
included_parse <- parsed |>
filter(doc_id == "s021") |>
mutate(
head_word = if_else(
head_token_id == "0",
"root",
token[match(head_token_id, token_id)]
)
) |>
select(token, head_word, dep_rel)| Token | Head word | Relation |
|---|---|---|
| Product | root | root |
| training | included | nsubj:pass |
| is | included | aux:pass |
| included | Product | ccomp |
| . | Product | punct |
In Product training is included, the main claim is that training is included. The parser has made Product the head of the sentence and hung included beneath it as a subordinate clause, so the parser has the sentence inside out.
What to remember
- A dependency parse links each token to a head word and names the relation.
- UDPipe and spaCy disagree on the heads inside
Paid 12-week. - A previous-token baseline gets 304 of 4,007 weblog heads right.
- The trained parser gets 2,899 of 4,007 heads right, about 0.72.
- The parser misses 7 of 28 hand-marked Riverton roots.
- Parser links on job-posting text need direct inspection.
Use dependency links as leads for reading, especially around passives. On this corpus the useful warning is concrete: seven root choices need review before any downstream count trusts them.