Training word vectors

Check whether neighbours carry information

similarity
word embeddings
inaugural addresses
Learn how distributed word representations are trained from local context and why a small corpus can make neighbour lists look better than they are.

A curator wants to tag a speech archive without reading every paragraph. She asks which words sit near freedom, union, and peace in the language of the collection.

A table of neighbours appears quickly. It looks meaningful, but every word-vector model can return a nearest neighbour for any word it kept.

A distributed word representation is one numeric vector per word, learned from the words that appear near it. This lesson loads one saved local model, then checks whether an easy related pair separates from arbitrary pairs.

TipWhat you will learn

This lesson shows how to:

  • read a pinned word2vec skip-gram model;
  • inspect the preprocessing and training settings stored with it;
  • explain why the fitted model is saved rather than trained during render;
  • compare an expected related pair with random pairs; and
  • summarize the full distribution of cosine similarities.

Load one saved word2vec model

Word2vec comes from work by Mikolov and colleagues on learning word vectors from nearby words. The skip-gram version uses the current word to predict words in its window. The R package splits input on whitespace, so the training strings should already be lowercased with punctuation and numbers removed.

library(dplyr)
library(tibble)
library(stringr)
library(readr)
library(quanteda)
library(word2vec)
library(ggplot2)
library(knitr)

source("R/inaugural-corpus.R")

within_tolerance <- function(observed, expected, tolerance = 1e-4) {
  abs(observed - expected) < tolerance
}

metadata <- read_csv(
  "data/word2vec/word2vec-metadata.csv",
  na = character(),
  col_types = cols(
    field = col_character(),
    value = col_character()
  )
)
metadata_values <- setNames(metadata$value, metadata$field)

paragraphs <- inaugural_paragraphs()
training_tokens <- tokens(
  paragraphs$paragraph,
  remove_punct = TRUE,
  remove_numbers = TRUE
) |>
  tokens_tolower()

model <- word2vec::read.word2vec("data/word2vec/inaugural-word2vec.bin")
embedding_matrix <- as.matrix(model)
punctuation_rows <- rownames(embedding_matrix)[grepl("[[:punct:]]", rownames(embedding_matrix))]

training_summary <- tibble(
  item = c(
    "training documents",
    "tokens after cleanup",
    "vocabulary rows",
    "dimensions",
    "punctuation rows",
    "unique word pairs"
  ),
  value = c(
    metadata_values[["training_documents"]],
    metadata_values[["training_tokens"]],
    metadata_values[["vocabulary_rows"]],
    metadata_values[["dimensions"]],
    metadata_values[["punctuation_rows"]],
    metadata_values[["unique_pairs"]]
  )
)

hyperparameter_table <- tibble(
  setting = c("algorithm", "dimensions", "window", "iterations", "minimum count", "threads"),
  value = metadata_values[c("algorithm", "dimensions", "window", "iterations", "minimum_count", "threads")]
)

model_fingerprint <- tibble(
  field = c("generator", "model SHA-256", "reproducibility note"),
  value = metadata_values[c("generator", "sha256_model", "reproducibility")]
)

kable(
  training_summary,
  col.names = c("Item", "Value"),
  caption = "Training text and vocabulary checks stored with the saved word2vec model",
  row.names = FALSE
)
Training text and vocabulary checks stored with the saved word2vec model
Item Value
training documents 1377
tokens after cleanup 134419
vocabulary rows 2709
dimensions 50
punctuation rows 1
unique word pairs 3667986 
kable(
  hyperparameter_table,
  col.names = c("Setting", "Value"),
  caption = "Hyperparameters for the pinned local word2vec skip-gram model",
  row.names = FALSE
)
Hyperparameters for the pinned local word2vec skip-gram model
Setting Value
algorithm skip-gram
dimensions 50
window 5
iterations 5
minimum count 5
threads 1 
kable(
  model_fingerprint,
  col.names = c("Field", "Value"),
  caption = "Provenance for the committed word2vec model",
  row.names = FALSE
)
Provenance for the committed word2vec model
Field Value
generator data-raw/build-word2vec-model.R
model SHA-256 b182100ecb33cb339f16fdf5df5cea783dd113843a3a89e7e65393b921e448dd
reproducibility note word2vec is not bitwise reproducible on this package version. Across five fits with the same seed the neighbour order changed every time, so this model is trained once here and committed.

The metadata records 1,377 training documents, 134,419 cleaned tokens, 2,709 vocabulary rows, and 3,667,986 unique word pairs. It also records one punctuation row, the trainer’s </s> marker. Punctuation was removed before training, so forms such as peace. and peace, do not compete with peace.

The same training code with the same seed and threads = 1L produced neighbour lists in a different order on each fit. The similarity values moved only slightly, about 0.03 here, but the ranking changed. This page loads one saved fit, and the metadata records its SHA-256 fingerprint. A result you cannot reproduce is a result you cannot check; pinning the artefact is the normal fix when exact words will be printed.

Show the similarity distribution

The distribution below uses every unique word pair in the cleaned vocabulary. It also marks the freedom / liberty pair with a vertical line.

normalized_embeddings <- embedding_matrix / sqrt(rowSums(embedding_matrix^2))
word_similarity_matrix <- tcrossprod(normalized_embeddings)
diag(word_similarity_matrix) <- NA_real_
word_similarities <- word_similarity_matrix[upper.tri(word_similarity_matrix)]
freedom_liberty_percentile <- mean(word_similarities <= freedom_liberty_cosine)

similarity_summary <- tibble(
  statistic = c("word pairs", "median", "95th percentile", "freedom / liberty percentile"),
  value = c(
    length(word_similarities),
    median(word_similarities),
    as.numeric(quantile(word_similarities, 0.95)),
    freedom_liberty_percentile
  )
)

kable(
  similarity_summary |>
    mutate(value = if_else(statistic == "word pairs", as.character(as.integer(value)), sprintf("%.3f", value))),
  col.names = c("Statistic", "Value"),
  caption = "Distribution of all pairwise word-vector cosine similarities",
  row.names = FALSE
)
Distribution of all pairwise word-vector cosine similarities
Statistic Value
word pairs 3667986
median 0.794
95th percentile 0.935
freedom / liberty percentile 0.444

freedom / liberty sits at the 44th percentile of the full pairwise distribution in this saved model. The pair was chosen to be easy, yet it lands below the median pair.

plot_similarities <- tibble(cosine_similarity = word_similarities)

ggplot(plot_similarities, aes(x = cosine_similarity)) +
  geom_histogram(bins = 40, boundary = 0, color = "white") +
  geom_vline(xintercept = freedom_liberty_cosine, linewidth = 1, linetype = "dashed") +
  labs(
    x = "Cosine similarity",
    y = "Word pairs",
    title = "Many arbitrary word pairs sit above an easy related pair"
  ) +
  theme_minimal()
A histogram concentrated toward high positive cosine values. A vertical line for freedom slash liberty falls left of the median rather than near the far right edge.
Figure 1: Pairwise cosine similarities among words kept by the pinned local word2vec model.

Published embeddings are trained on billions of tokens. Large training text gives the model many more contexts for separating near neighbours from accidental neighbours.

What to remember

  • A word2vec model gives one vector to each kept word.
  • The pinned model used skip-gram, 50 dimensions, window 5, 5 iterations, minimum count 5, and one thread.
  • The word2vec package expects text that is already cleaned before whitespace splitting.
  • Neighbour lists need a pinned artefact when exact words are printed.
  • In this saved fit, even freedom / liberty lands below the median of all pairwise similarities.

The curator keeps the neighbour table as a prompt for reading, not as evidence that the archive has revealed its themes.

Sources