ADR-0015: Centralized Embedding Store in AI Service¶

Status¶

PROPOSED

Date: 2026-02-15

Supersedes: ADR-0012 (Embedding Factory vs Vector Index Ownership)

Changelog¶

Context¶

ADR-0012 established that each service owns its own embedding vectors: the AI service owns conversation vectors, the KB service owns document vectors, both stored in pgvector tables within the shared loryth database.

Upon further analysis, this distributed ownership model creates a critical problem: embedding model migration becomes nearly impossible when vectors are fragmented across services. Changing the embedding model (e.g., from text-embedding-3-small to a newer model, or from an OpenAI model to a local Ollama model) would require:

• Coordinating schema migrations across every service that stores vectors
• Re-embedding all content in each service simultaneously
• Handling different vector dimensions per model across multiple databases/tables
• Accepting downtime or building parallel migration infrastructure in every service independently

Since no service currently has production embedding data, now is the ideal time to centralize before fragmentation occurs.

Additionally, pgvector — while convenient for V1 — may not be the optimal long-term solution for vector similarity search. A centralized store provides a single point where the underlying vector database can be swapped without affecting consuming services.

Decision¶

1. AI Service Owns All Embedding Storage and Search¶

The AI service becomes the single owner of all embedding vectors across the platform. Other services (KB, etc.) interact with embeddings exclusively through the AI service's API:

• POST /embeddings/store — Generate embedding for text and store it with metadata
• POST /embeddings/search — Similarity search across stored embeddings
• DELETE /embeddings/{uuid} — Remove an embedding

Each embedding record has: - namespace — Logical grouping owned by the calling service (e.g., kb.documents, ai.conversations) - source_id — The owning service's identifier for the embedded content (e.g., document UUID, conversation UUID) - text — Original text that was embedded (retained for re-embedding during model migration) - vector — The embedding vector - model_id — Which embedding model produced this vector - metadata — Arbitrary JSON for filtering (e.g., {"space_id": "...", "chunk_index": 3})

Services retain domain ownership of what to embed and when, but the AI service owns how (model selection) and where (storage).

2. Per-Model Table Strategy¶

Embedding vectors are stored in model-specific tables rather than a single table with mixed dimensions:

ai_embedding_<model_slug>
  id          BIGSERIAL PRIMARY KEY
  uuid        UUID NOT NULL UNIQUE
  namespace   VARCHAR(128) NOT NULL
  source_id   UUID NOT NULL
  text        TEXT NOT NULL
  vector      vector(<dim>)
  metadata    JSONB
  created_at  TIMESTAMP NOT NULL

Where <model_slug> is a normalized identifier (e.g., nomic_embed_text_768, text_embedding_3_small_1536) and <dim> matches the model's output dimension.

Benefits: - Each table has a homogeneous vector dimension — no padding or truncation - Indexes are optimized for a single dimension - Model migration creates a new table without touching the old one - Old tables can be dropped cleanly after migration

3. Embedding Model Migration Process¶

When switching from model A to model B (documented here for future implementation):

Phase 1 — Dual Write - New embedding requests generate vectors with both model A and model B - Model A vectors go to the existing table, model B vectors go to the new table - All reads continue using model A's table

Phase 2 — Backfill - A background job iterates over all rows in model A's table - For each row, re-embeds the stored text using model B and inserts into model B's table - Progress is tracked to allow resumption if interrupted

Phase 3 — Switch - Reads switch to model B's table (configuration change, no deployment required) - Writes switch to model B only (dual-write stops) - Verify search quality meets expectations

Phase 4 — Cleanup - Drop model A's table and associated indexes - Remove model A configuration

This process requires no downtime and no coordination with consuming services. The API contract (POST /embeddings/store, POST /embeddings/search) remains identical throughout.

Note: This migration process is documented for future reference. It will not be implemented in V1 — only the centralized store and per-model table structure are implemented initially.

4. Vector Database Strategy¶

V1 uses pgvector for simplicity (already deployed, no additional infrastructure). However, the centralized store architecture deliberately abstracts the storage backend so it can be replaced.

Current: pgvector (V1) - Co-located with relational data in PostgreSQL - IVFFlat indexing, adequate for up to ~1M vectors - Zero additional infrastructure - Limited: no built-in metadata filtering on vector search, index must be rebuilt after bulk inserts, HNSW support exists but is less mature than purpose-built solutions

Future candidates to evaluate:

The decision on which vector database to adopt (if any) is deferred until query patterns and data volume are better understood. The API contract between services and the AI embedding store remains the same regardless of the backend.

5. Impact on Existing Code¶

• ai_conversation_vector table and ConversationVector entity are replaced by the centralized embedding tables
• The EmbeddingProvider protocol moves from a per-service concern to an internal detail of the AI service's embedding store
• KB service (when it needs embeddings) will call the AI service's embedding API rather than managing its own vectors
• The /conversations/search endpoint continues to work but delegates to the embedding store internally (filtering by namespace=ai.conversations)

Consequences¶

Positive¶

• Embedding model migration is a single-service operation with a well-defined, zero-downtime process
• No vector fragmentation across services — one place to manage, monitor, and optimize
• Vector database can be swapped without affecting consuming services
• Original text is retained, enabling re-embedding without involving source services
• Per-model tables provide clean dimension isolation and index optimization

Negative¶

• AI service becomes a runtime dependency for any service that needs vector search
• Single point of failure for all embedding operations (mitigated by standard HA practices)
• Network hop for embedding operations that were previously local (KB querying its own pgvector)
• More complex AI service with additional API surface and storage responsibilities

Neutral¶

• The dual-write migration pattern adds temporary cost (two model invocations per request) during transitions
• Backfill duration scales linearly with the number of stored embeddings
• Vector database evaluation should be revisited when approaching 100K+ embeddings or when query latency requirements tighten

Alternatives Considered¶

1. Keep Distributed Ownership (ADR-0012 Status Quo)
2. Description: Each service continues to own and store its own vectors
3. Pros: No cross-service dependency, each service optimizes independently
4. Cons: Model migration requires coordinated changes across all services, vector fragmentation makes it impractical to switch models or databases

5. Why not chosen: The model migration problem is a fundamental limitation that worsens as more services adopt embeddings

6. Shared Library for Embedding Management

7. Description: A shared loryth-embeddings library that each service uses to manage its own vectors with consistent migration tooling
8. Pros: No runtime dependency on AI service, each service still owns its data
9. Cons: Migration coordination still required across services, library must handle multiple database backends, schema changes must be deployed to every service simultaneously

10. Why not chosen: Distributes complexity without solving the coordination problem; migration remains a multi-service operation

11. Dedicated Embedding Microservice (Separate from AI)

12. Description: A standalone loryth-embeddings service focused purely on vector storage and search
13. Pros: Clean separation of concerns, AI service stays focused on orchestration
14. Cons: Additional service to deploy and operate, the AI service already has model provider infrastructure that embedding generation depends on
15. Why not chosen: Premature service split; the AI service is the natural home for model-dependent operations. Can be extracted later if the embedding store grows into a distinct operational concern

References¶

• ADR-0012: Embedding Factory vs Vector Index Ownership (superseded)
• ADR-0014: Ollama as Local Model Provider (embedding model selection)
• pgvector Documentation
• Qdrant Documentation
• Milvus Documentation