Sorting messages by purpose

Train a multi-class intent classifier and question the labels

classification
intent classification
workforce research
Learn how intent classification routes messages and why the label schema controls what the system can do.

Luis is assigned the shared inbox while the front desk is closed. One message asks for a password reset, another asks whether a class accepts GED applicants, and a third reports a suspicious recruiter.

Routing the message is useful only if the label matches a real next step. A worker can reset access, check application status, answer eligibility questions, record a problem, or call someone back. Anything outside those actions strains the label set.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.

A dataset written by the same person who writes the classifier cannot measure whether the method works. This lesson uses the invented inbox to show mechanics; it does not estimate how an intake system would perform with real residents.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • define intent classification in plain language;
  • fit a small multi-class text model;
  • inspect multi-class probability output;
  • identify messages that strain one-label routing; and
  • explain why coarser taxonomies can make accuracy look better.

Load the intent labels

An intent is what the sender wants the system to do. In this fixture each message has exactly one recorded intent, even when a person might see more than one. readr opens the file, dplyr and tidyr reshape results, purrr selects probability columns, stringr checks simple rules, and tidymodels with textrecipes and glmnet fits the model.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(ggplot2)
library(tidymodels)
library(textrecipes)
library(glmnet)

inbox <- read_csv(
  "data/riverton/riverton-inbox.csv",
  na = character(),
  col_types = cols(
    message_id = col_character(),
    text = col_character(),
    is_spam = col_logical(),
    intent = col_character(),
    author_note = col_character()
  )
) |>
  mutate(intent = factor(intent))

intent_counts <- inbox |>
  count(intent, name = "messages")

knitr::kable(
  intent_counts,
  col.names = c("Recorded intent", "Messages"),
  caption = "Intent labels in the invented inbox",
  row.names = FALSE
)
Intent labels in the invented inbox
Recorded intent Messages
ask_eligibility 12
check_status 13
report_problem 13
request_callback 11
reset_access 11

The label schema is already a decision about what the Lab can act on. If no one can process a distinction, adding that distinction to the label list creates extra disagreement rather than better service.

Fit a multi-class model

Multi-class classification chooses one label from more than two possible labels. The seed before this fit is 5002. The model is fitted so the reader can see the mechanics of a multi-class prediction.

intent_recipe <- recipe(intent ~ text, data = inbox) |>
  step_tokenize(text) |>
  step_tokenfilter(text, max_tokens = 100) |>
  step_tfidf(text)

mechanics_rows <- inbox

intent_model <- multinom_reg(penalty = 0.01, mixture = 1) |>
  set_engine("glmnet") |>
  set_mode("classification")

set.seed(5002)
intent_fit <- workflow() |>
  add_recipe(intent_recipe) |>
  add_model(intent_model) |>
  # The lesson reports nothing performance-related from these rows; it shows the
  # shape of a multi-class fit and states plainly why it scores nothing.
  fit(data = mechanics_rows) # resubstitution-ok: no score is reported

intent_predictions <- mechanics_rows |>
  select(message_id, text, intent) |>
  bind_cols(predict(intent_fit, mechanics_rows, type = "prob")) |>
  bind_cols(predict(intent_fit, mechanics_rows))

probability_columns <- names(intent_predictions) |>
  keep(\(column) str_starts(column, ".pred_") & column != ".pred_class")

probability_check <- intent_predictions |>
  transmute(
    message_id,
    probability_sum = rowSums(pick(all_of(probability_columns)))
  )

probability_display <- intent_predictions |>
  slice_head(n = 3) |>
  select(message_id, all_of(probability_columns)) |>
  rename_with(
    \(column) str_remove(column, "^\\.pred_"),
    all_of(probability_columns)
  )

predicted_distribution <- intent_predictions |>
  count(.pred_class, name = "messages") |>
  arrange(.pred_class)

knitr::kable(
  probability_display |>
    mutate(across(-message_id, \(value) round(value, 3))),
  col.names = c(
    "Message",
    "Ask eligibility",
    "Check status",
    "Report problem",
    "Request callback",
    "Reset access"
  ),
  caption = "Five class probabilities for three fitted-row examples",
  row.names = FALSE
)
Five class probabilities for three fitted-row examples
Message Ask eligibility Check status Report problem Request callback Reset access
M001 0.014 0.940 0.039 0.003 0.004
M002 0.061 0.881 0.007 0.022 0.029
M003 0.016 0.934 0.034 0.007 0.009 
knitr::kable(
  predicted_distribution,
  col.names = c("Predicted intent", "Messages"),
  caption = "Predicted-class distribution from the fitted multi-class model",
  row.names = FALSE
)
Predicted-class distribution from the fitted multi-class model
Predicted intent Messages
ask_eligibility 12
check_status 13
report_problem 13
request_callback 11
reset_access 11

The fitted model has five classes. For each message it returns five probabilities, one for each class, and each row’s probabilities sum to 1. Those predictions are made on the same 60 rows the model was fitted to, so the table above shows output shape and nothing about how well the model works. No performance score is reported here. The messages were written by the same person who built the classifier, there are only 60 of them, and any score computed from them would measure this fixture rather than the method. A real number would require messages the author did not write, labels supplied by people who did not build the model, and a holdout set kept away from fitting.

Find labels that strain the taxonomy

The hard part is often the label schema, not the model. A taxonomy is the allowed list of labels and the rules for choosing among them. If the taxonomy does not match the downstream work, the classifier can be accurate and still route badly.

boundary_messages <- inbox |>
  filter(message_id %in% c("M034", "M055", "M044")) |>
  mutate(
    other_plausible_action = case_when(
      message_id == "M034" ~ "ask_eligibility or fraud review",
      message_id == "M055" ~ "ask_eligibility or fraud review",
      message_id == "M044" ~ "none without the missing request"
    ),
    why_it_strains_the_schema = case_when(
      message_id == "M034" ~
        "reports a suspicious listing but also mentions training and bank access",
      message_id == "M055" ~
        "asks whether an offer is real and reports a recruiter asking for ID",
      message_id == "M044" ~
        "gives call instructions without a full service request"
    )
  ) |>
  select(message_id, intent, other_plausible_action, why_it_strains_the_schema, text)

knitr::kable(
  boundary_messages,
  col.names = c(
    "Message",
    "Recorded intent",
    "Other plausible action",
    "Why one label is hard",
    "Text"
  ),
  caption = "Boundary cases for the intent taxonomy",
  row.names = FALSE
)
Boundary cases for the intent taxonomy
Message Recorded intent Other plausible action Why one label is hard Text
M034 report_problem ask_eligibility or fraud review reports a suspicious listing but also mentions training and bank access A listing says paid training, but the attachment asks for my bank password.
M044 request_callback none without the missing request gives call instructions without a full service request Please leave a voicemail if I miss the call.
M055 report_problem ask_eligibility or fraud review asks whether an offer is real and reports a recruiter asking for ID I am not sure whether this offer is real; the recruiter wants a photo of my ID.

Messages M034 and M055 genuinely fit two actions. A staff member may both record a problem and answer whether the offer is safe. If the Lab has a separate fraud review queue, these two messages should probably go there. If it does not, the taxonomy forces the worker to choose the nearest available bucket and record the loss.

M044 fits none of the main service questions by itself; it is a call preference clipped away from the reason for the call. A good intake design would ask for the missing purpose before routing. Lesson 11, “Working with several annotators,” shows why multiple reviewers should inspect these boundary cases before a label becomes a reference answer.

An other label absorbs what the designer did not anticipate. It is useful as a safety valve, but it also hides new work unless someone reads the contents.

Coarser labels can flatter a system

If several intents are folded into broader queues, the same guesses can receive a higher score. The classifier did not get smarter; the scoring rule became easier. This lesson does not compute that score because a hand-fitted rule on these same 60 messages would only create another resubstitution number. The design point is enough: measured accuracy can rise when the taxonomy gets coarser without the system getting better. That may be acceptable if the broad queue is all the organization can act on. It is misleading if the report implies better understanding.

What to remember

  • Intent classification routes text by what the sender wants.
  • The allowed intent list must match actions the system can take.
  • Probability output shows the shape of a multi-class prediction, not its quality.
  • Boundary cases need human adjudication and clearer instructions.
  • Coarser categories can raise measured accuracy by making the task easier.

Sources