Embedding and Semantic Similarity

If you have ever searched a document by keyword and missed the answer because the wording was different, you have already felt the limitation of exact text matching. Embeddings solve this by converting text into numbers that capture meaning. Semantic similarity then measures how close those meanings are, even when the words differ.

This post walks through that pipeline with concrete Python examples:

1. Word embeddings with GloVe (classic, static vectors)
2. Sentence/chunk embeddings with modern models via Ollama
3. Semantic similarity with cosine similarity on sample text

The running example is a Wikipedia-style article about Southern Methodist University (SMU), stored in SMU_Wiki.txt.

What Is an Embedding?

An embedding is a numeric representation of text. Instead of storing a sentence as words, we convert it into a vector — a list of numbers — that captures its meaning. Texts with similar meaning usually end up close together in this vector space.

Consider these two sentences. They share the word bank, but the meaning is different:

• “We had a picnic on the river bank.”
• “She works at a major banking company.”

A keyword search for “bank” matches both. An embedding-based system can place them in different regions of vector space, because the surrounding context differs.

Embeddings are the foundation of modern NLP applications including semantic search, retrieval-augmented generation (RAG), clustering, recommendation, and duplicate detection.

Word Embedding

A word embedding maps each word (or subword token) to a dense vector. The model learns these vectors from large text corpora so that words appearing in similar contexts receive similar vectors.

Word embeddings are useful for:

• Capturing local lexical relationships (synonyms, morphology, related concepts)
• Understanding word sense in a static way (one vector per word type)
• Classical NLP tasks: text classification features, NER, POS tagging
• Teaching the core idea before moving to contextual and sentence-level embeddings

Classic examples include Word2Vec, GloVe, and fastText.

Example: GloVe (glove-wiki-gigaword-50)

GloVe (Global Vectors for Word Representation) learns word vectors from word co-occurrence statistics in a large corpus. The model name breaks down as:

Load the model with Gensim:

import gensim.downloader as api

model = api.load("glove-wiki-gigaword-50")
model["bank"]

This returns a 50-number vector for the word bank:

array([ 0.66488 , -0.11391 ,  0.67844 ,  0.17951 ,  0.6828  , -0.47787 ,
       -0.30761 ,  0.17489 , -0.70512 , -0.55022 ,  0.1514  ,  0.10214 ,
       ... 50 values total ...], dtype=float32)

Each number is a learned feature. We rarely interpret individual dimensions by hand; what matters is the overall direction of the vector relative to other words.

Comparing word meanings

model.similarity("bank", "river")   # 0.414
model.similarity("bank", "credit")  # 0.765

Even though both pairs contain the word bank, the embedding space tells us which neighbor is closer in meaning.

Find the nearest words to bank:

model.most_similar([model["bank"]], topn=5)

[('bank', 1.00),
 ('banks', 0.87),
 ('securities', 0.80),
 ('banking', 0.80),
 ('investment', 0.78)]

This is a clear financial cluster. GloVe captured domain association from co-occurrence patterns in training data.

Limitation of static word embeddings

GloVe assigns one vector per word, regardless of context. The word bank has a single vector whether you mean a river bank or a financial institution. Modern transformer models produce contextual embeddings where the same word gets different vectors depending on surrounding text. Sentence-level embedding models (covered next) build on this idea at the document level.

Sentence and Chunk Embedding

A sentence embedding (or chunk embedding) maps an entire sentence or text passage to one fixed-length vector.

Sentence embeddings power:

• Semantic search — find passages by meaning, not exact keywords
• RAG — retrieve relevant context before generation
• Clustering — group similar documents
• Duplicate detection — find near-duplicate content
• Recommendation — match items by semantic profile

In this post, we compare several popular embedding models using Ollama to embed text from an SMU Wikipedia-style article.

Loading the sample document

with open("SMU_Wiki.txt", "r", encoding="utf-8") as f:
    text = f.read()

The file describes SMU’s history, academics, campus, athletics, and enrollment. It is long enough to illustrate an important practical issue: context length limits.

A quick word count:

text.split(" ")
len(text.split(" "))
# 2861 words (approximate; token count will differ)

Words are not tokens. Tokenizers split text into subword pieces, so token count is usually different from word count. For embedding models, context length is measured in tokens, not characters or words.

Comparing Popular Embedding Models

Before embedding, it helps to compare model properties. These four families are widely used in 2025–2026 workflows:

What each column means

• Number of parameters: model size (learned weights). Larger models are often more accurate but need more memory and compute.
• Embedding dimension: length of the output vector (len(embedding)). This is not the same as context length.
• Context length: maximum number of input tokens per embedding call. Longer context supports larger chunks; shorter context requires splitting documents first.

Critical distinction: embedding dim vs context length

These are often confused. Example with mxbai-embed-large on Ollama:

So len(emb_mxbai) == 1024 does not contradict a 512-token context limit. You always get a 1024-length vector; the model just reads at most 512 tokens of input.

Implication for our SMU text

The SMU article is roughly ~3000 tokens (and ~2861 words). If we embed the entire document in one call:

• embeddinggemma (2K context) → truncates to first ~2048 tokens
• mxbai-embed-large (512 context) → truncates to first 512 tokens
• nomic-embed-text (8K context) → likely fits the whole document
• qwen3-embedding (32K–40K) → easily fits the whole document

For mxbai and other short-context models, chunk the document before embedding.

Embedding with Ollama

Ollama provides a simple local API for running embedding models with direct GPU support. We use LangChain’s wrapper:

from langchain_ollama import OllamaEmbeddings

embedder_mxbai = OllamaEmbeddings(model="mxbai-embed-large:335m")
embedder_gemma = OllamaEmbeddings(model="embeddinggemma:300m")

Important: OllamaEmbeddings is not callable like a function. Use .embed_query() for a single string:

emb_mxbai = embedder_mxbai.embed_query(text)
emb_gemma = embedder_gemma.embed_query(text)

print(len(emb_mxbai))   # 1024
print(len(emb_gemma))   # 768

Each model returns one vector for the entire input (subject to context truncation):

Can you run mxbai on Hugging Face with GPU?

Yes. Ollama is convenient, but not required:

import torch
from sentence_transformers import SentenceTransformer

device = "cuda" if torch.cuda.is_available() else "cpu"
model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1", device=device)

emb = model.encode("The sky is blue because of Rayleigh scattering")
print(len(emb))  # 1024

Semantic Similarity

Semantic similarity measures how close two texts are in meaning, not in exact wording.

We already saw this at the word level with GloVe:

• bank vs river → 0.41
• bank vs credit → 0.76

At the sentence level, the idea is the same: convert each text to a vector, then measure closeness.

Cosine similarity

The most common metric is cosine similarity. It measures the angle between two vectors, ignoring magnitude:

Interpretation:

Rules when comparing embeddings

1. Compare within the same model when ranking passages (gemma with gemma, mxbai with mxbai).
2. Do not subtract vectors of different dimensions (emb_gemma is 768-dim, emb_mxbai is 1024-dim).
3. Respect context limits — truncate or chunk long documents before embedding.
4. You cannot reverse an embedding to text — vectors are a one-way semantic summary.

Hands-on example: paraphrase vs unrelated sentence

import numpy as np
from langchain_ollama import OllamaEmbeddings

def cosine_similarity(a, b):
    a = np.array(a)
    b = np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

embedder = OllamaEmbeddings(model="embeddinggemma:300m")

emb1 = embedder.embed_query("Southern Methodist University is in Dallas.")
emb2 = embedder.embed_query("SMU is located in Dallas, Texas.")
emb3 = embedder.embed_query("The moon orbits Earth.")

print("Similar sentences:", cosine_similarity(emb1, emb2))
print("Unrelated sentence:", cosine_similarity(emb1, emb3))

Results with embeddinggemma:

Similar sentences: 0.854
Unrelated sentence: 0.218

The paraphrased SMU sentences score 0.85 — high semantic similarity despite different wording. The unrelated astronomy sentence scores 0.22 — much lower. This is exactly the behavior semantic search relies on.

Summary

Three ideas worth carrying forward:

1. Embeddings turn language into geometry. Similar meanings land near each other in vector space.
2. Word and sentence embeddings solve different problems. GloVe teaches the intuition; modern models like embeddinggemma and mxbai power search and RAG.
3. Context length and embedding dimension are different knobs. Always check both before embedding long documents.

Everything downstream — semantic search, RAG, clustering — is built on these vectors and the similarity scores between them.