Embeddings and Vector Spaces

Early approaches embedded individual words:

Key limitation: Same word gets same embedding regardless of context.

Transformer-based models produce context-dependent embeddings:

# Static embedding (Word2Vec)
embed("bank") = [0.1, 0.3, ...]  # Same vector always

# Contextual embedding (BERT)
embed("river bank") = [0.1, 0.3, ...]   # Geography sense
embed("bank account") = [0.5, 0.2, ...]  # Finance sense

For retrieval, we need to embed entire texts:

Modern embedding models are trained specifically for sentence/document embedding, not just adapted from language models.

Standard retrieval embedding architecture:

Document -> Encoder -> Document Embedding
Query    -> Encoder -> Query Embedding

Similarity = cosine(doc_embedding, query_embedding)

Properties:

• Documents can be pre-computed and indexed
• Query embedding computed at query time
• O(1) similarity computation per document (with ANN)

Alternative that processes query and document together:

[Query, Document] -> Encoder -> Relevance Score

Properties:

• More accurate (sees both together)
• Cannot pre-compute: O(n) inference for n documents
• Used for reranking, not retrieval

Most modern embedding models use contrastive learning:

# Simplified contrastive loss
def contrastive_loss(anchor, positive, negatives):
    pos_sim = cosine_similarity(anchor, positive)
    neg_sims = [cosine_similarity(anchor, neg) for neg in negatives]
    
    # Push positive close, negatives far
    loss = -log(exp(pos_sim / tau) / 
                (exp(pos_sim / tau) + sum(exp(neg_sim / tau) for neg_sim in neg_sims)))
    return loss

Key factors:

Recent models accept task instructions:

# Instruction-tuned (e.g., E5, BGE)
query_embedding = embed("Represent this query for retrieval: What is RAG?")
doc_embedding = embed("Represent this document for retrieval: RAG combines...")

This improves performance by specifying the intended use.

Most common for text embeddings:

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Properties:

• Range: [-1, 1] (for normalized vectors, [0, 1] if positive)
• Measures angle, not magnitude
• Invariant to vector length

When to use: Default choice for text embeddings.

def dot_product(a, b):
    return np.dot(a, b)

Properties:

• Magnitude matters
• Unbounded range
• Equivalent to cosine for normalized vectors

When to use: When embeddings are already normalized, or magnitude is meaningful.

def euclidean_distance(a, b):
    return np.linalg.norm(a - b)

Properties:

• Measures absolute difference
• Affected by magnitude
• For normalized vectors: sqrt(2 - 2 * cosine)

When to use: Rarely for text; more common for image embeddings.

MTEB scores represent late 2025 frontier standards.

MTEB scores are approximate and vary by benchmark subset. Always verify current values.

Matryoshka Representation Learning (MRL) trains embeddings such that prefixes of the full embedding are also meaningful:

full_embedding = model.encode(text)  # 1024 dimensions

# All these are valid embeddings with decreasing quality
dim_512 = full_embedding[:512]  
dim_256 = full_embedding[:256]
dim_128 = full_embedding[:128]
dim_64 = full_embedding[:64]

• OpenAI text-embedding-3-* (native)
• Nomic-embed-text-v1.5
• Several fine-tuned models

Traditional Chunking: Document -> Split into chunks -> Embed chunks individually

• Issue: Chunk 2 loses the context from Chunk 1.

Late Chunking (introduced by Jina AI/Voyage): Full Document -> Model Encoder -> Token-level Embeddings -> Pool into chunk boundaries

• Benefit: Each chunk's embedding contains information from the entire document because the transformer's self-attention was applied to the full sequence before pooling.
• Requirement: A model with long-context support (at least 8k+ tokens).

• Retrieval precision is critical
• Can afford storage overhead
• Query latency budget > 50ms

Long documents must be chunked before embedding:

def embed_document(document: str, max_tokens: int = 512) -> list[np.array]:
    chunks = chunk_document(document, max_tokens=max_tokens)
    embeddings = []
    for chunk in chunks:
        embedding = embed(chunk)
        embeddings.append(embedding)
    return embeddings

Considerations:

• Chunk size should be less than model max tokens
• Overlap helps preserve context across chunk boundaries
• Store chunk-to-document mapping for retrieval

Many systems expect normalized embeddings:

def normalize(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm

# Cosine similarity of normalized vectors = dot product
similarity = np.dot(normalize(a), normalize(b))

Most vector databases and embedding APIs handle normalization, but verify.

Embedding computation is expensive. Cache aggressively:

import hashlib

def get_embedding(text: str, cache: dict) -> np.array:
    key = hashlib.sha256(text.encode()).hexdigest()
    
    if key in cache:
        return cache[key]
    
    embedding = compute_embedding(text)
    cache[key] = embedding
    return embedding

Embeddings are not comparable across:

• Different models
• Different versions of the same model
• Sometimes different API calls (some APIs have non-determinism)

If you update your embedding model:

• All existing embeddings become incompatible
• Must re-embed entire corpus
• Search results will be inconsistent during migration

1. Version your embeddings:

embedding_metadata = {
    "model": "text-embedding-3-large",
    "model_version": "2024-01",
    "dimensions": 3072,
    "created_at": "2025-12-16"
}

2. Plan for re-embedding:

• Estimate cost and time for full re-embed
• Build pipelines that can run in background
• Test new embeddings before switching

3. Blue-green deployment:

Index A: Current embeddings
Index B: New embeddings (building)

Query -> Both indexes -> Merge or switch

4. Track embedding quality:

• Monitor retrieval metrics continuously
• Detect drift in embedding distributions
• Alert on quality degradation