Words, Tokens, and Vector Semantics: How Machines Read and Represent Language

Notes from two lectures: one on words and tokenization, one on word meaning, distributional semantics, and word2vec

Before a model can learn what a word means, it first has to decide what counts as a “word” in the first place — and that turns out to be a much harder question than it looks. This post is in two parts. Part 1 covers words and tokens: why counting words is surprisingly slippery, why NLP systems abandoned words in favor of learned subword tokens, and how the dominant tokenization algorithm, Byte-Pair Encoding, actually works. Part 2 covers vector semantics: how, once text has been tokenized, machines learn what those tokens mean by turning them into vectors.

---

Part 1: Words and Tokens

How Many Words Are in a Sentence? Harder Than It Sounds

Take a plain sentence: “They picnicked by the pool, then lay back on the grass and looked at the stars.” Counting is 16 words if punctuation doesn’t count, 18 if it does — already an ambiguous start.

Spoken language complicates this further. An utterance like “I do uh main- mainly business data processing” contains disfluencies: fragments (main-) and filled pauses (uh, um). Whether these should count as words at all is a genuine modeling decision, not a settled fact.

Two more useful distinctions:

• Type vs. instance — a type is a distinct entry in the vocabulary V; an instance is one occurrence of that type in running text. The example sentence above has 14 types and 16 instances once punctuation is set aside. Vocabulary size |V| refers to the number of types.
• Orthographic vs. grammatical words — I’m is one word on the page but arguably two grammatically: the pronoun I and the verb ‘m (short for am). Similarly, should They and they count as the same word?

And underlying all of this: not every written language uses spaces. Chinese, Japanese, and Thai don’t mark word boundaries the way English does, so “count the words” isn’t even a well-defined operation across languages.

Tokenizing Chinese: No Agreed-Upon Answer

Chinese text is built from characters called hanzi (汉字), each representing a morpheme — a minimal meaning-bearing unit. A Chinese word contains 2.4 hanzi on average, but where word boundaries actually fall is contested. The sentence 姚明进入总决赛 (“Yao Ming reaches the finals”) can be segmented into 3, 5, or 7 “words” depending on which linguistic convention is applied — different treebanks (Chinese Treebank vs. Peking University) disagree. One common practical workaround: skip the segmentation debate entirely and just use individual characters as tokens.

There Are Simply Too Many Words

Even setting segmentation debates aside, vocabularies grow without bound. Larger corpora reveal more distinct word types, not fewer:

Corpus	Types (|V|)	Instances (N)
Shakespeare	31 thousand	884,000
Brown Corpus	38 thousand	1 million
Switchboard conversations	20 thousand	2.4 million
COCA	2 million	440 million
Google N-grams	13+ million	1 trillion

This relationship is captured by Heaps’ Law (also called Herdan’s Law): |V| = kN^β, where β is roughly 0.5 — vocabulary size grows with roughly the square root of text length. No matter how large the training corpus, there will always be unknown words the model has never seen.

Words also split functionally into function words (of, the, is, and — a closed, small set) and content words (mango, braise, snowy — an open, ever-growing set), which is part of why the open class keeps expanding vocabulary size indefinitely.

Words Have Parts: Morphology

A morpheme is the minimal meaning-bearing unit in a language — fox is one morpheme, cats is two (cat + -s). Morphology is the study of how morphemes combine. Every word has a root carrying the main meaning and, optionally, affixes adding further meaning (worked = root work + affix -ed).

Affixes come in two flavors:

• Inflectional morphemes — grammatical, often marking agreement: -ed for past tense, -s/-es for plural.
• Derivational morphemes — more idiosyncratic, often changing grammatical class entirely: care (noun) → careful (adjective) → carefully (adverb).

A related phenomenon is the clitic: a morpheme that behaves syntactically like a word but is phonologically reduced and attached to a neighbor — English ‘ve in I’ve (it can’t stand alone), ’s in the teacher’s book, French l’ in l’opéra, or Arabic b (“by/with”) and w (“and”).

Morphological Typology: Languages Vary Enormously

Two dimensions matter most for tokenization: how many morphemes a language typically packs per word, and how cleanly those morphemes can be separated.

On the Greenberg (1960) scale of morphemes-per-word, languages range from strongly analytic (Vietnamese ≈1.1, Farsi ≈1.5, English ≈1.7) through synthetic (Old English, Yakut, Swahili, Sanskrit, roughly 2.1–2.6) to polysynthetic (Greenlandic/Inuit ≈3.7). Cantonese sits at the analytic end — keoi5 waa6 cyun4 gwok3 zeoi3 daai6 gaan1 uk1 hai6 ni1 gaan1 (“he say entire country most big building house is this building”) uses roughly one morpheme per word. Koryak, at the polysynthetic end, can pack an entire sentence’s worth of meaning into a single word via many stacked morphemes.

Segmentability varies too: agglutinative languages like Turkish have clean, separable morpheme boundaries, while fusional languages conflate several morphemes into a single affix — Russian -om in stolom marks instrumental case, singular number, and first declension all at once, and English -s in “she reads” simultaneously marks third person and present tense. These are tendencies, not hard rules.

Why NLP Abandoned Words as Tokens

Three problems converge to rule out “words” as the unit NLP systems tokenize on: many languages have no orthographic word boundaries at all, defining a word post-hoc is contested even where boundaries exist, and vocabulary size grows without bound as more text is seen. Using raw Unicode characters as tokens avoids these problems but creates a new one — characters are too small a unit to carry useful meaning efficiently. Morphemes would be a natural middle ground, but they’re very hard to define algorithmically and consistently across languages.

The modern solution sidesteps the whole debate: let the data decide the tokenization, via a learned subword vocabulary.

Tokenizing at all — rather than working with raw strings — matters for several reasons: it lets systems agree on the length of an input, it’s a precondition for algorithms like perplexity that assume a fixed tokenization, and it eliminates the unknown-word problem, since any word unseen in training can still be decomposed into known subword pieces.

Byte-Pair Encoding (BPE)

The two dominant subword tokenization algorithms are Byte-Pair Encoding (Sennrich et al., 2016) and Unigram language modeling tokenization (Kudo, 2018, often loosely called “SentencePiece” after the library that implements it). Both have the same two-part structure: a token learner that induces a vocabulary from a raw training corpus, and a token encoder/segmenter that applies that learned vocabulary to tokenize new text.

The BPE token learner works by repeated merging:

function BYTE-PAIR ENCODING(strings C, number of merges k) returns vocab V
    V ← all unique characters in C          # initial set of tokens is characters
    for i = 1 to k do                        # merge tokens til k times
        tL, tR ← Most frequent pair of adjacent tokens in C
        tNEW ← tL + tR                        # make new token by concatenating
        V ← V + tNEW                          # update the vocabulary
        Replace each occurrence of tL, tR in C with tNEW   # and update the corpus
    return V

Concretely: start with vocabulary [A, B, C, D, E] and corpus A B D C A B E C A B. The most frequent adjacent pair is (A, B), so it merges into AB, giving vocabulary [A, B, C, D, E, AB] and corpus AB D C AB E C AB. Repeating with the next most frequent pair, (C, AB), yields token CAB, and so on for k total merges.

BPE is generally run separately within each whitespace-separated word — merges never cross word boundaries. A worked example on the toy corpus set new new renew reset renew (with a space marker ␣ prepended to each word):

1. Starting vocabulary is just the individual characters: ␣, e, n, r, s, t, w.
2. The most frequent adjacent pair is n e (count 4, from new + renew), merging to ne.
3. Next, ne w merges to new (count 4).
4. Then ␣ r merges to ␣r, and ␣r e merges to ␣re (each count 3) — the system has now effectively learned the prefix re-.
5. Further merges continue: (␣, new) → ␣new; (␣re, new) → ␣renew; (s, e) → se; (se, t) → set.

To tokenize a new test sentence, the encoder applies every merge rule learned during training, greedily, in the exact order they were learned (test-set frequencies play no role). The sentence is first split into raw characters, then each merge rule is applied in sequence. The result both reconstructs training-set words exactly and generalizes to unseen words that share learned subwords — e.g. a vocabulary that learned ␣re from renew and reset can also tokenize a novel word like rearrange.

Tokenization in Practice: GPT-4o

Looking at real GPT-4o tokenization (via Tat Dat Duong’s Tiktokenizer visualizer) on the string “Anyhow, she’s seen Jane’s 224123 flowers anyhow!” shows several patterns: most words become single tokens with a leading space; clitics like ’s are split off as a separate token for less-frequent words (Jane’s) but folded into the token for very frequent words (she’s); numbers are chunked into groups of three digits; and the same word gets different token IDs depending on whether it’s capitalized or has a leading space (Anyhow vs. ·anyhow). Some of this behavior comes from preprocessing — regular expressions that pre-split digit chunks and strip off clitics before BPE merges are even applied.

Tokenization Is Not Equally Good in Every Language

Even though BPE tokenizers are nominally multilingual, LLM training corpora remain overwhelmingly dominated by English text. Since BPE merges are learned from frequency in that corpus, most learned tokens end up serving English well, leaving comparatively few efficient tokens for other languages — so words in other languages get split into more, smaller pieces. A concrete comparison on GPT-4o: an English recipe sentence (“In a deep bowl, mix the orange juice with the sugar, ginger, and nutmeg.”) tokenizes into 18 tokens with no word split across multiple tokens, while the equivalent Spanish sentence (“En un recipiente hondo, mezclar el jugo de naranja con el azúcar, jengibre, y nuez moscada.”) takes 33 tokens, with 6 of its 16 words split across multiple tokens. This asymmetry has real downstream consequences — it means non-English text costs more tokens to represent and process for the same amount of content, a point especially relevant for lower-resource languages that see even less representation in training corpora than major world languages like Spanish.

Part 2: Vector Semantics & Embeddings

Once text has been tokenized, the next question is what those tokens actually mean. How does a computer know that “car” and “automobile” mean roughly the same thing, or that “king” relates to “queen” the way “man” relates to “woman”? None of this is hard-coded — it’s learned entirely from how words are used in text. This section walks through the core ideas behind vector semantics, the dominant approach to representing word meaning in modern NLP.

Word Meaning Is Slippery

Before getting to vectors, it helps to see why representing meaning is hard in the first place.

A lemma (a dictionary headword) can carry multiple senses. “Mouse” the rodent and “mouse” the computer peripheral are the same lemma but completely different concepts — this is polysemy. Beyond that, senses relate to each other in several distinct ways:

• Synonymy — different words, (nearly) the same meaning: couch/sofa, big/large. True synonyms are rare; even close pairs like water/H2O differ by register or context (nobody says “H2O” while surfing).
• Similarity — related but not interchangeable meaning: car and bicycle are similar without being synonyms. Human similarity judgments (like those in the SimLex-999 dataset) show this is a graded, measurable property, not a yes/no one.
• Relatedness — words that co-occur in a semantic field without being similar: coffee and cup are related, not similar; coffee and tea are similar.
• Antonymy — opposites along a single dimension of meaning (hot/cold, rise/fall) while remaining otherwise near-identical.
• Connotation — the affective coloring of a word. Copy and forgery can describe the same object while carrying opposite sentiment.

Any serious computational theory of meaning needs to capture at least some of this structure. That’s exactly what vector semantics sets out to do.

The Distributional Hypothesis: You Shall Know a Word by the Company It Keeps

The foundational idea, going back to linguist Zellig Harris (1954), is startlingly simple: define a word by the contexts it appears in.

A nice illustration is the word ongchoi, a recent English borrowing most readers won’t recognize. If you observe it appearing in sentences like:

Ongchoi is delicious sautéed with garlic. Ongchoi leaves with salty sauces.

and separately notice that spinach, chard, and collard greens show up in nearly identical company — “sautéed with garlic,” “leaves,” “delicious” — you can correctly infer that ongchoi is a leafy green, without ever seeing a definition. (It’s water spinach, Ipomoea aquatica, known elsewhere as kangkong or rau muống.)

This gives us two ideas that together define vector semantics:

1. Meaning is defined by linguistic distribution — the set of contexts a word occurs in.
2. Meaning is a point in multidimensional space — each word becomes a vector, and words with similar distributions end up nearby in that space.

From here on, computation happens over these vector representations — “meaning representations” — rather than over raw strings.

Building Vectors from Counts

The most direct way to turn distribution into a vector is the word-context matrix, a co-occurrence matrix where each row is a target word, each column is a context word, and each cell counts how often the two appear near each other (commonly within some fixed window, e.g. ±4 words).

For a full vocabulary this matrix is enormous — potentially 50,000 × 50,000 — and overwhelmingly sparse, since most word pairs never co-occur. That sparsity is actually useful: it means efficient sparse-matrix storage and computation are possible, and this “counting” approach to embeddings (later refined with tf-idf weighting) was the standard method for decades.

Measuring Similarity: From Dot Product to Cosine

Given two word vectors, how do we quantify how similar they are? The natural starting point is the dot product:

v · w = Σ vᵢwᵢ

The dot product is large when two vectors have large values in the same dimensions — a reasonable proxy for similarity. But it has a flaw: it’s biased by vector length, and frequent words (the, of, you) accumulate large raw counts across many dimensions simply by appearing everywhere. Raw dot product ends up overrating frequent words regardless of actual meaning overlap.

The fix is cosine similarity — the dot product normalized by the magnitude of each vector:

cos(v, w) = (v · w) / (|v| |w|)

Cosine ranges from −1 (opposite direction) to +1 (same direction), and since raw co-occurrence counts are never negative, in practice it ranges from 0 to 1 for count-based vectors. A concrete example from the lecture: computing cosine similarity between cherry and information using their counts across the contexts pie, data, and computer gives a similarity near 0.017 — they barely overlap in usage. Digital and information, by contrast, come out around 0.996 — nearly identical distributional behavior, which matches the intuition that both are used heavily in computing-related contexts.

From Sparse Counts to Dense Embeddings

Count vectors work, but they’re long (tens of thousands of dimensions) and sparse (mostly zeros). The modern alternative is to learn dense embeddings: short vectors (typically 50–1000 dimensions) where most entries are non-zero.

Dense vectors have real practical advantages:

• Far fewer parameters, which helps as inputs to downstream machine learning models.
• Better generalization than raw counts.
• Better capture of synonymy — with sparse counts, car and automobile are literally different dimensions, so a document mentioning one gets no “credit” for relating to a document mentioning the other. Dense embeddings collapse this gap.

Two families dominate the “learn dense vectors directly” approach: neural language-model-inspired methods (word2vec, GloVe) and matrix-factorization methods like Singular Value Decomposition (of which Latent Semantic Analysis, or LSA, is a well-known special case). A further evolution — contextual embeddings like ELMo and BERT — computes a distinct vector for each occurrence of a word depending on its surrounding context, rather than one fixed vector per word type. That’s a topic for another post; here we focus on the static, one-vector-per-word case.

Word2Vec: Learning Meaning by Predicting Neighbors

Word2vec (Mikolov et al.) reframes the whole problem. Instead of counting how often words co-occur, it trains a classifier to predict co-occurrence — and then discards the classifier, keeping only the weights it learned as the word embeddings. This is a form of self-supervision: the training signal (which words actually appeared nearby in the corpus) comes for free from raw text, with no human annotation required.

This post focuses on one popular variant: skip-gram with negative sampling (SGNS).

Setting Up the Prediction Task

Given a target word and a context window (say ±2 words), skip-gram training data looks like this:

…lemon, a [tablespoon c1 of c2 apricot of c3 jam c4, a] pinch…

Each (target, context-word) pair extracted from real text — like (apricot, jam) — becomes a positive example. To give the classifier something to contrast against, we also generate negative examples: pairs of the target word with words drawn randomly from the vocabulary that didn’t actually appear nearby, e.g. (apricot, aardvark).

The classifier’s job is simple: given a (word, context) pair, output the probability that this pair is a genuine, “nearby” pair rather than a random one.

Turning Similarity into Probability

Just as with count vectors, we measure similarity between the target embedding w and context embedding c as their dot product. But a dot product is an unbounded real number, not a probability — so it gets passed through a sigmoid, borrowed straight from logistic regression:

P(+ | w, c) = σ(c · w) = 1 / (1 + exp(−c · w))
P(− | w, c) = 1 − P(+ | w, c)

When there are multiple context words in a window rather than just one, skip-gram assumes independence across them and multiplies the individual probabilities (or, equivalently, sums their log-probabilities).

The Training Loop

Putting it together, learning skip-gram embeddings follows a clean recipe:

1. Initialize every word in the vocabulary with a random d-dimensional vector.
2. Extract positive (word, context) pairs from co-occurring words in the corpus.
3. Sample random (word, context) pairs as negatives.
4. Train a logistic-regression-style classifier to tell these apart, adjusting the embeddings via gradient updates as it goes.
5. Throw away the classifier itself — the embeddings it left behind are the deliverable.

It’s a slightly unusual framing: the “task” (predicting neighbors) is never actually useful on its own. It’s a pretext task whose only purpose is to force useful structure into the embeddings.

What These Embeddings Actually Capture

Window Size Shapes the Notion of “Similar”

The size of the context window changes what kind of similarity gets encoded:

• Small windows (±2) tend to surface syntactically similar words within the same taxonomy. The nearest neighbors of Hogwarts with a small window are other fictional schools — Sunnydale, Evernight, Blandings.
• Larger windows (±5) surface words from the same broader semantic field. With a wider window, Hogwarts’s neighbors shift toward the Harry Potter universe itself — Dumbledore, half-blood, Malfoy.

There’s no single “correct” window size — it’s a modeling choice that trades off syntactic versus topical similarity.

Analogies via Vector Arithmetic

One of the most striking emergent properties of dense embeddings is that simple vector arithmetic can solve analogies. This traces back to the parallelogram model of analogical reasoning (Rumelhart & Abrahamson, 1973): to solve “apple is to tree as grape is to ___,” compute tree − apple, add it to grape, and look for the nearest word — which turns out to be vine.

Word2vec and GloVe embeddings reproduce this surprisingly well:

king − man + woman ≈ queen
Paris − France + Italy ≈ Rome

Formally, for an analogy *a : a* :: b : b**, the parallelogram method solves:

b* = argmax_x  distance(x, a* − a + b)

This is elegant, but it isn’t magic. The method reliably works mainly for frequent words, short distances in embedding space, and a fairly narrow set of relation types (country–capital pairs, parts of speech); it breaks down for many other relations. Understanding exactly why analogical structure emerges — and where it fails — remains an open research question.

Embeddings as a Lens on Language Change and Culture

Because embeddings are trained from real text, they inherit whatever patterns exist in that text — including how meaning drifts over time and what biases were baked into the culture that produced the corpus.

Training separate embeddings on different decades of text (one study used ~30 million Google Books volumes spanning 1850–1990) shows word meanings visibly migrating through the vector space over time. Gay moves from a neighborhood of cheerful, pleasant, witty in the early 1900s toward homosexual, lesbian by the 1990s. Broadcast drifts from an agricultural sense (sow, scatter, seed) toward its now-dominant media sense (television, radio, BBC).

The same technique surfaces social bias, which is a genuine concern for any system built on top of these representations. Asking embeddings to complete “man : computer programmer :: woman : ” has notoriously returned homemaker; “father : doctor :: mother :” has returned nurse. Historical analysis of competence-related adjectives (smart, wise, brilliant) shows a measurable bias toward male-associated words that only slowly diminished between 1960 and 1990, while dehumanizing adjectives (barbaric, monstrous, bizarre) were more strongly associated with Asian names in the 1930s, with that association fading — but not disappearing — over the 20th century. These patterns line up with independent survey data from the same eras, which is part of why embeddings have become a research tool for quantifying historical bias, not just an NLP engineering tool. It’s also a caution: any downstream system — including something like a hiring-screening tool built on embeddings of resumes — can absorb and reproduce these same biases if left unchecked.

Summary

Together, these two lectures trace the pipeline from raw text to meaning. Part 1 showed why “words” are a surprisingly unstable unit to compute over — vocabularies grow without bound, morphology varies wildly across languages, and many languages don’t even mark word boundaries — and how Byte-Pair Encoding sidesteps all of that by learning subword tokens directly from corpus statistics, at the cost of representing some languages less efficiently than others. Part 2 picked up from there: once text is tokenized, vector semantics rests on one deceptively simple idea — a word (or token) is defined by the company it keeps — and builds an entire computational framework on top of it. Count-based approaches turn co-occurrence statistics directly into sparse vectors, compared via cosine similarity. Word2vec instead learns compact dense vectors as a side effect of training a lightweight classifier to distinguish real from fake (word, context) pairs. Either way, the resulting embeddings capture graded similarity, support surprisingly effective analogical reasoning, and — for better and worse — act as a mirror of the language and culture they were trained on.

Based on lecture material adapted from Dan Jurafsky’s Speech and Language Processing (SLP3), Stanford University.