What Are Embeddings?
Embeddings are the foundation of how AI understands meaning. They convert things (words, sentences, images) into lists of numbers (vectors) that capture their essence.
The magic: similar things get similar vectors.
"king" → [0.2, 0.8, 0.1, ...]
"queen" → [0.25, 0.78, 0.15, ...] // Similar vectors!
"banana" → [0.9, 0.1, 0.7, ...] // Different vectorWhy Embeddings Matter
Computers Only Understand Numbers
A computer can't directly understand "dog" or "cat". But it CAN compare numbers:
- •Is 5 close to 6? Yes.
- •Is 5 close to 1000? No.
Embeddings convert meaning into numbers so computers can:
- •Measure similarity between concepts
- •Find related items
- •Understand analogies
- •Power search, recommendations, and AI
The Alternative: One-Hot Encoding
Without embeddings, we might use one-hot encoding:
Vocabulary: [cat, dog, fish, bird, car, ...]
"cat" → [1, 0, 0, 0, 0, ...]
"dog" → [0, 1, 0, 0, 0, ...]
"fish" → [0, 0, 1, 0, 0, ...]Problems:
- •No similarity: distance(cat, dog) = distance(cat, car)
- •Huge vectors: vocabulary of 100K words = 100K-dimensional vectors
- •No meaning captured
Part 2: How Embeddings Capture Meaning
The Word2Vec Revelation
Word2Vec (2013) discovered that training on simple tasks produces meaningful embeddings.
Training task: Predict a word from its neighbors
Sentence: "The cat sat on the mat"
Given: ["The", "sat", "on", "the"]
Predict: "cat"After training on millions of sentences, the model learns:
- •Words in similar contexts get similar embeddings
- •"dog" and "cat" appear in similar contexts → similar embeddings
- •Relationships are captured as vector directions
The Famous Analogy
king - man + woman ≈ queenVector arithmetic:
[0.2, 0.8, 0.1, 0.5] // king
- [0.1, 0.3, 0.1, 0.8] // man
+ [0.15, 0.35, 0.15, 0.2] // woman
= [0.25, 0.85, 0.15, -0.1] // ≈ queenWhy this works:
- •"king" and "queen" share royalty features
- •"man" and "woman" differ by gender features
- •Subtracting "man" removes male features
- •Adding "woman" adds female features
- •Result: royalty + female = queen
Part 3: Types of Embeddings
Word Embeddings
One vector per word. Same embedding regardless of context.
from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5)
vector = model.wv['cat'] # 100-dimensional vectorProblem: "bank" has the same embedding whether it's a river bank or a money bank.
Contextual Embeddings (BERT, GPT)
Different vectors based on context.
"I went to the bank to deposit money"
↓
bank → [0.2, 0.5, 0.1, ...] (financial meaning)
"I sat by the river bank"
↓
bank → [0.8, 0.1, 0.7, ...] (geographical meaning)Sentence Embeddings
One vector for an entire sentence.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode([
"I love programming",
"Coding is my passion",
"The weather is nice"
])
# embeddings[0] and embeddings[1] will be similar
# embeddings[2] will be differentImage Embeddings
Same concept, but for images. A trained model (like CLIP) converts images to vectors.
Photo of cat → [0.1, 0.8, 0.3, ...]
Photo of dog → [0.15, 0.75, 0.35, ...] // Similar (both pets)
Photo of car → [0.9, 0.1, 0.2, ...] // DifferentPart 4: Measuring Similarity
Cosine Similarity
The most common similarity measure. Measures the angle between vectors.
B
╱
╱
╱ θ (small angle = similar)
╱
A ───────
cosine_similarity = cos(θ) = (A · B) / (|A| × |B|)
Range: -1 (opposite) to 1 (identical)from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
a = np.array([1, 2, 3])
b = np.array([1, 2, 4])
c = np.array([-1, -2, -3])
print(cosine_similarity([a], [b])) # 0.99 (very similar)
print(cosine_similarity([a], [c])) # -1.0 (opposite)Euclidean Distance
Measures straight-line distance between points.
B●
│╲
│ ╲ distance
│ ╲
A●──────
distance = √((x₁-x₂)² + (y₁-y₂)² + ...)When to use which:
- •Cosine: When direction matters (text similarity)
- •Euclidean: When magnitude matters (some image tasks)
Part 5: Practical Applications
Semantic Search
Traditional search: keyword matching Embedding search: meaning matching
Query: "How to fix a broken heart"
Keyword search matches:
❌ "Heart surgery procedures" (has "heart")
❌ "Fixing car engines" (has "fix")
Embedding search matches:
✓ "Dealing with breakup pain" (similar MEANING)
✓ "Moving on after a relationship" (similar MEANING)# Index documents
documents = [
"Dealing with breakup pain",
"Heart surgery procedures",
"Moving on after a relationship"
]
doc_embeddings = model.encode(documents)
# Search
query = "How to fix a broken heart"
query_embedding = model.encode(query)
# Find most similar
similarities = cosine_similarity([query_embedding], doc_embeddings)
most_similar_idx = similarities.argmax()
print(documents[most_similar_idx]) # "Dealing with breakup pain"RAG (Retrieval Augmented Generation)
Power ChatGPT-like systems with your own data:
1. Embed your documents → store in vector database
2. User asks question
3. Embed question → find similar documents
4. Send question + retrieved documents to LLM
5. LLM generates answer using your documentsRecommendations
Find similar items based on embeddings:
# User liked these movies
liked_movies = ["Inception", "Interstellar", "The Matrix"]
liked_embeddings = [get_embedding(m) for m in liked_movies]
# Average their embeddings to get "user taste"
user_taste = np.mean(liked_embeddings, axis=0)
# Find movies with similar embeddings
all_movie_embeddings = ...
similarities = cosine_similarity([user_taste], all_movie_embeddings)
# Top recommendations
top_indices = similarities.argsort()[0][-5:]
recommendations = [movies[i] for i in top_indices]Part 6: Vector Databases
Why Vector Databases?
Regular databases: exact matches (WHERE name = "John") Vector databases: similarity search (find vectors near this one)
Regular DB query: SELECT * FROM users WHERE age = 25
Vector DB query: Find 10 vectors closest to [0.1, 0.5, 0.3, ...]Popular Vector Databases
Pinecone - Managed, easy to use
import pinecone
pinecone.init(api_key="...")
index = pinecone.Index("my-index")
# Insert
index.upsert([("doc1", embedding1), ("doc2", embedding2)])
# Query
results = index.query(query_embedding, top_k=5)Weaviate - Open source, feature-rich
Milvus - Open source, scalable
Chroma - Lightweight, good for prototyping
pgvector - PostgreSQL extension (use existing DB)
Approximate Nearest Neighbor (ANN)
Finding exact nearest neighbors is slow for millions of vectors. ANN algorithms trade accuracy for speed:
Exact search: Compare with ALL vectors (slow but perfect)
ANN search: Smart indexing (fast but might miss some)
Accuracy trade-off:
1 million vectors
Exact: 1000ms, 100% recall
ANN: 10ms, 95% recallCommon ANN algorithms:
- •HNSW (Hierarchical Navigable Small Worlds)
- •IVF (Inverted File Index)
- •PQ (Product Quantization)
Part 7: Best Practices
Choosing Embedding Models
| Use Case | Model | Dimensions |
|---|---|---|
| General text | OpenAI text-embedding-3-small | 1536 |
| Fast & small | all-MiniLM-L6-v2 | 384 |
| Multilingual | multilingual-e5-large | 1024 |
| Images | CLIP | 512 |
| Code | CodeBERT | 768 |
Chunking for Documents
Long documents need to be split into chunks before embedding:
# Bad: One embedding for entire document
doc_embedding = embed(long_document) # Information gets averaged out
# Good: Split into meaningful chunks
chunks = split_into_chunks(long_document, chunk_size=500)
chunk_embeddings = [embed(chunk) for chunk in chunks]Chunking strategies:
- •Fixed size (500 tokens) with overlap (50 tokens)
- •Sentence-based
- •Paragraph-based
- •Semantic (split at topic changes)
Handling Updates
When documents change, you need to re-embed and re-index:
def update_document(doc_id, new_content):
# Delete old embeddings
vector_db.delete(filter={"doc_id": doc_id})
# Create new embeddings
chunks = chunk_document(new_content)
embeddings = embed(chunks)
# Insert new embeddings
for i, (chunk, emb) in enumerate(zip(chunks, embeddings)):
vector_db.insert(
id=f"{doc_id}_chunk_{i}",
vector=emb,
metadata={"doc_id": doc_id, "text": chunk}
)Key Takeaways
- 1Embeddings convert meaning to numbers - similar things get similar vectors
- 2Contextual embeddings understand context - same word, different contexts, different vectors
- 3Cosine similarity measures semantic similarity - most common for text
- 4Vector databases enable fast similarity search - essential for production systems
- 5Chunking matters for long documents - break into meaningful pieces before embedding
- 6Choose the right embedding model - balance quality, speed, and cost for your use case