PathweysAI Engineer Track · Embeddings Vector Dbs
What Embeddings Are
What Embeddings Are
An embedding turns a piece of text, a word, a sentence, a whole document, into a list of numbers positioned in space such that texts with similar meaning end up near each other and texts with different meaning end up far apart. That is the bridge that lets a computer do math on meaning: search by similarity, group similar items together, or flag something that does not fit.
Why this matters
This is the substrate underneath search, recommendation, and every retrieval system in this field, but the similarity these numbers capture is only approximate. Two sentences can look similar in embedding space while actually meaning very different things, particularly around negation, exact numbers, or precise phrasing.
What embeddings are
An embedding is a dense vector, a list of numbers, that encodes the meaning of text, image, or audio into a fixed-size representation, where similar meanings map to nearby points in vector space. What makes this genuinely useful rather than just a numerical curiosity is that relationships between concepts tend to survive the mapping: analogies that hold in meaning tend to hold approximately in the geometry too, and semantic similarity becomes a fast nearest-neighbor search rather than an open-ended comparison problem. That is the foundation underneath retrieval-augmented generation, semantic search, deduplication, and clustering alike.
An embedding turns meaning into geometry. Everything downstream, search, retrieval, clustering, is ultimately just measuring distance in that space.
Why vectors: meaning as geometry
Embeddings earn their usefulness by making semantic similarity computable as geometric distance, which is exactly the kind of problem computers are fast at. Three properties matter in practice: dimensionality, typically in the hundreds to low thousands, where more dimensions buy more capacity at the cost of more storage and slower search; similarity as proximity, where the chosen distance metric defines what 'close' actually means; and a shared space, since every item has to be embedded by the same model to be comparable at all. That last point is the rule that trips people up most often: a vector from one model and a vector from another are not comparable, because they live in entirely different coordinate systems, so every item inside a single search index has to come from one consistent model.
Embeddings make meaning geometric, but only within one specific model's space. One model per index is a non-negotiable rule, not a style preference.
Worked example
Embedding the sentences 'the cat sat on the mat' and 'a feline rested on the rug' produces vectors that land close together despite sharing almost no words, because the embedding captures meaning rather than surface text, while embedding 'the cat sat on the mat' against 'interest rates rose fifty basis points' produces vectors that land far apart, correctly reflecting that the two sentences are unrelated.
Practice
Embed twenty short sentences with a real embedding model and visualize them in two dimensions using PCA or UMAP. Find one sentence pair that is surface-similar but far apart in the space, or the reverse, to make the approximate nature of the similarity concrete.
Interview checks
- Why can't you compare embeddings produced by two different models?
- What kind of meaning does an embedding typically fail to capture well?
- Why does dimensionality trade off against storage and search speed?
Common failure modes
- Treating embedding similarity as exact, when a cosine score of 0.8 versus 0.82 is not a reliable ranking signal on its own.
- Comparing embeddings produced by different models, or by the same model at different dimensions, which is simply meaningless.
- Assuming an embedding captures all of a text's meaning, when it reliably loses negation, exact numbers, and precise phrasing.
Resources
- Distributed Representations of Words and Phrases (word2vec) — The foundational idea behind continuous vector representations that encode semantics; its introduction is the clearest intuition available.
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Covers sentence-level embeddings and the contrastive training that makes them semantically meaningful.
- OpenAI Embeddings guide — A practical provider reference for how embeddings are used and the cosine-similarity convention.
Expert signals
Knowing that embeddings are approximate, that models and dimensions cannot be mixed within one index, and that the right embedding model is genuinely task-specific, a legal-document search needs a different model than a code search, is what separates someone who treats embeddings as a precise tool rather than a vague similarity gesture.
Confidence
high