Compare one word vector with one token occurrence vector
similarity
contextual embeddings
spaCy
Learn how spaCy returns a context-sensitive token representation and why two uses of the same word can have different vectors.
A student underlines bank twice in one sentence. The first bank raises interest rates; the second sits beside a river.
The spelling is identical. The surrounding words make the two occurrences different.
A contextualized word representation gives a vector to a token occurrence rather than to a word type. The same word can therefore receive two different vectors in the same document.
TipWhat you will learn
This lesson shows how to:
start the pinned local spaCy pipeline through the project helper;
confirm that the small pipeline has no static word-vector table;
read the token tensor produced for one sentence;
compare token occurrence vectors with an all-pair baseline;
check same-sense controls; and
separate contextual encoding from word-sense identification.
Load the local spaCy pipeline
spaCy is a Python library. This project reaches it through R/use-spacy.R, which pins the Python environment and prevents downloads during rendering.
The pinned small spaCy pipeline has no static word-vector table
Item
Value
pipeline
en_core_web_sm
static vector rows
0
static vector columns
0
The zero-row vector table matters. This small pipeline does not store one reusable vector for every word. It still produces an internal tensor for the tokens it reads.
Extract token occurrence vectors
A tensor is an array of numbers. For this sentence, spaCy returns one row per token and 96 columns per token.
text <-"The bank raised interest rates. He sat on the river bank and watched."doc <-nlp(text)tokens <-vapply(seq_len(length(doc)) -1L,function(i) doc[i]$text,character(1))tensor <-py_to_r(doc$tensor)bank_positions <-which(tokens =="bank")the_positions <-which(tokens %in%c("The", "the"))token_table <-tibble(position =seq_along(tokens),token = tokens,target =case_when( position %in% bank_positions ~"bank occurrence", position %in% the_positions ~"the occurrence",TRUE~"other token" ))kable( token_table,col.names =c("Position", "Token", "Role in this lesson"),caption ="Tokens in the sentence and the positions compared below",row.names =FALSE)
Tokens in the sentence and the positions compared below
Position
Token
Role in this lesson
1
The
the occurrence
2
bank
bank occurrence
3
raised
other token
4
interest
other token
5
rates
other token
6
.
other token
7
He
other token
8
sat
other token
9
on
other token
10
the
the occurrence
11
river
other token
12
bank
bank occurrence
13
and
other token
14
watched
other token
15
.
other token
The two bank tokens are at positions 2 and 12. They share spelling, but they do not share the same tensor row.
Compare occurrence vectors with a baseline
Cosine similarity compares vector direction. A score of 1 would mean the two rows point in exactly the same direction. The two bank occurrence vectors score about 0.3466.
Cosine similarities across all 105 token pairs in the example sentence
All-pair baseline
Cosine
mean
0.0476
median
-0.0074
5th percentile
-0.1594
95th percentile
0.4475
kable( comparison_table |>mutate(cosine_similarity =round(cosine_similarity, 4),percentile_of_all_pairs =round(percentile_of_all_pairs, 3) ),col.names =c("Comparison", "First position", "Second position", "Cosine similarity", "Percentile of all token pairs"),caption ="Repeated word strings compared with the all-pair baseline",row.names =FALSE)
Repeated word strings compared with the all-pair baseline
Comparison
First position
Second position
Cosine similarity
Percentile of all token pairs
bank / bank
2
12
0.3466
0.914
the / the
1
10
0.5322
0.971
The bank pair is higher than an arbitrary token pair from the same sentence: it sits around the 91st percentile of the all-pair baseline. That supports a modest claim. The encoder gives different vectors to occurrences, and the two bank rows are still closer than most random token pairs in this short text.
Check same-sense controls
The stronger claim would be that the distance isolates word sense. The controls below test that claim by changing sense and grammatical role separately.
tensor_for_text <-function(text) { doc <-nlp(text) tokens <-vapply(seq_len(length(doc)) -1L,function(i) doc[i]$text,character(1) )list(tokens = tokens, tensor =py_to_r(doc$tensor))}bank_pair_cosine <-function(text) { parsed <-tensor_for_text(text) positions <-which(parsed$tokens =="bank")if (length(positions) !=2L) {stop("Expected exactly two bank tokens.", call. =FALSE) }cosine(parsed$tensor[positions[1], ], parsed$tensor[positions[2], ])}bank_control_sentences <-tibble(comparison =c("different senses, different roles","same financial sense, same role","same river sense","same financial sense, different roles" ),text =c("The bank raised interest rates. He sat on the river bank and watched.","The bank raised interest rates. The bank approved the loan.","He sat on the river bank and watched. She walked along the muddy bank.","The bank approved the loan. She walked into the bank." ),cosine_similarity =c(bank_pair_cosine(text[1]),bank_pair_cosine(text[2]),bank_pair_cosine(text[3]),bank_pair_cosine(text[4]) ))role_gap <-abs( bank_control_sentences$cosine_similarity[bank_control_sentences$comparison =="different senses, different roles"] - bank_control_sentences$cosine_similarity[bank_control_sentences$comparison =="same financial sense, different roles"])kable( bank_control_sentences |>mutate(cosine_similarity =round(cosine_similarity, 4)),col.names =c("Comparison", "Sentences", "Cosine similarity"),caption ="Same-sense controls for two occurrences of bank",row.names =FALSE)
Same-sense controls for two occurrences of bank
Comparison
Sentences
Cosine similarity
different senses, different roles
The bank raised interest rates. He sat on the river bank and watched.
0.3466
same financial sense, same role
The bank raised interest rates. The bank approved the loan.
0.9422
same river sense
He sat on the river bank and watched. She walked along the muddy bank.
0.5900
same financial sense, different roles
The bank approved the loan. She walked into the bank.
0.3447
The reversal is the lesson. Same financial sense in different grammatical roles scores about 0.3447, almost the same as different senses in different roles at about 0.3466. Same financial sense in the same role scores about 0.9422. In these four probes, the grammatical-role contrast is at least as large as the dictionary-sense contrast. This is an illustration, not a word-sense evaluation: there is no benchmark or independent sense labeling here.
Contrast with one vector per word
The word2vec lesson trains one vector for each kept word type. A one-vector-per-word model cannot give river bank and finance bank two rows in the same sentence. A contextual pipeline can.
That does not mean the vector knows which dictionary sense is meant. The en_core_web_sm tok2vec layer, short for token-to-vector, is the part of the pipeline that turns each token into numbers, and it is a shared encoder trained for tagging, parsing, and named-entity recognition. It has no word-sense objective, so it carries the information useful for those tasks. A representation that changes with context is not the same thing as a model that knows which sense you meant. This is a small pipeline’s encoder, not a large language model.