ADR-0014: Ollama as Local Model Provider via OpenAI-Compatible Interface¶

Status¶

ACCEPTED

Date: 2026-02-15

Changelog¶

Implementation Status¶

Core provider hierarchy and binding resolution are implemented. OpenAICompatibleProvider, OllamaProvider, OpenAICompatibleEmbeddingProvider, and OllamaEmbeddingProvider are in place using the openai Python SDK. The resolve_model_provider() and resolve_embedding_provider() factory functions map ModelBinding records to provider instances. The RunOrchestrator resolves a model provider per-run from the space policy's bindings, with fallback to the DI-injected stub.

Not yet implemented: - Docker Compose Ollama service integration (Section 4) - Seed tool extension for configuring model bindings (Section 7) - Embedding provider resolution wired into the embedding pipeline (deferred to ADR-0015)

Context¶

The AI service has a fully implemented orchestration loop (RunOrchestrator), SSE streaming, tool execution, and web ingestion pipeline. However, all model interactions use StubModelProvider and StubEmbeddingProvider, which return canned responses. This means:

• The orchestration loop never exercises real model behavior (tool call decisions, multi-turn reasoning)
• Embedding search always returns zero vectors (no meaningful similarity results)
• E2E and smoke testing of the full pipeline is impossible without a real model
• Manual testing during development gives no insight into actual model interaction quality

Commercial providers (OpenAI, Anthropic) are unsuitable as the default development provider due to cost, rate limits, and the need for API keys in every developer environment.

Key factors: - Ollama runs locally, is free, and supports a wide range of open-weight models - Ollama exposes an OpenAI-compatible API (/v1/chat/completions, /v1/embeddings) - The openai Python SDK supports custom base_url, making it work with any OpenAI-compatible endpoint - The ModelBinding entity already has provider, model_id, and endpoint_url fields - The ProviderType enum already includes LOCAL and OPENAI values - The same OpenAI-compatible interface is used by vLLM, LM Studio, and other local inference servers

Decision¶

We will implement an OpenAI-compatible model provider using the official openai Python SDK, with Ollama as the primary local development target:

1. Provider Hierarchy¶

ModelProvider (ABC)
  -> OpenAICompatibleProvider    # Uses openai SDK with configurable base_url
       -> OllamaProvider         # Ollama-specific defaults (endpoint, model selection)
  -> StubModelProvider           # Existing stub for unit/integration tests

EmbeddingProvider (ABC)
  -> OpenAICompatibleEmbeddingProvider  # Uses openai SDK embeddings API
       -> OllamaEmbeddingProvider       # Ollama-specific defaults
  -> StubEmbeddingProvider              # Existing stub

OpenAICompatibleProvider accepts a base_url and api_key (defaulting to a dummy key for local servers that don't require auth). This same class works for OpenAI (https://api.openai.com/v1), Ollama (http://localhost:11434/v1), vLLM, or any other compatible endpoint.

OllamaProvider inherits from OpenAICompatibleProvider with Ollama-specific defaults (e.g., dummy API key, default model).

2. Provider Selection via ModelBinding¶

The existing ModelBinding entity drives provider selection at runtime:

Provider instances are resolved from the ModelBinding configuration on the space's policy. No code changes are needed to switch between providers — it's a data change on SpacePolicy.model_bindings.

3. Recommended Models for Development¶

These are small enough to run on developer machines without a GPU while still producing meaningful results for testing.

4. Docker Compose Integration (Optional)¶

Ollama can be added as an optional infrastructure service in docker-compose.yml for developers who don't have it installed locally:

ollama:
  image: ollama/ollama:latest
  volumes:
    - ollama-data:/root/.ollama
  networks:
    - loryth-network
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 30s

Developers who already have Ollama running on their host machine can skip the Docker service entirely and point model bindings at their local instance (http://host.docker.internal:11434/v1 from Docker, or http://localhost:11434/v1 when running the AI service directly on the host with make dev).

Models are pulled on first use (Ollama handles this automatically when a model is requested).

5. Embedding Dimensions¶

Ollama embedding models use different dimensions than OpenAI (e.g., nomic-embed-text produces 768 dimensions vs OpenAI's 1536). Embedding dimension management is handled by the centralized embedding store defined in ADR-0015, which uses per-model tables to support different dimensions natively.

6. Provider Resolution via ModelBinding¶

Provider instances are constructed at runtime from ModelBinding records on the space's SpacePolicy. A provider factory maps binding fields to the correct provider class:

def resolve_model_provider(binding: ModelBinding) -> ModelProvider:
    if binding.provider == ProviderType.LOCAL:
        return OllamaProvider(
            base_url=binding.endpoint_url or "http://localhost:11434/v1",
            model=binding.model_id,
            api_key=binding.api_key_ref,
        )
    if binding.provider == ProviderType.OPENAI:
        return OpenAICompatibleProvider(
            base_url=binding.endpoint_url or "https://api.openai.com/v1",
            model=binding.model_id,
            api_key=binding.api_key_ref,
        )
    return StubModelProvider()

This means provider configuration is entirely data-driven — different spaces can use different models and providers without code changes. The InfrastructureConfig beans for ModelProvider and EmbeddingProvider are no longer needed as static singletons; instead, the orchestrator resolves the provider from the space's model bindings when a run starts.

Tests continue to use StubModelProvider and StubEmbeddingProvider via the test DI container — no test changes needed.

7. Configuration via Seed Tool¶

The existing seed tool (server/tools/seed/) is extended with an AI seeder that configures model bindings for a space via the AI service API. This allows different developer setups without changing code:

Local Ollama (host machine):

# endpoint_url points to the developer's local Ollama instance
ModelBinding(provider="local", model_id="qwen2.5:3b",
             endpoint_url="http://localhost:11434/v1", modality="text")
ModelBinding(provider="local", model_id="nomic-embed-text",
             endpoint_url="http://localhost:11434/v1", modality="embeddings")

Docker Ollama:

ModelBinding(provider="local", model_id="qwen2.5:3b",
             endpoint_url="http://ollama:11434/v1", modality="text")

OpenAI (production or developer with API key):

ModelBinding(provider="openai", model_id="gpt-4o",
             api_key_ref="openai-key", modality="text")

The seed tool's SeedConfig gains SEED_AI_BASE_URL and optional SEED_OLLAMA_BASE_URL (defaulting to http://localhost:11434/v1) so each developer can point to their own Ollama instance. Seed data definitions live in server/tools/seed/src/seed/data/ following the established pattern.

Consequences¶

Positive¶

• Real model interactions during development without API costs
• Full pipeline testable end-to-end (orchestration, tool calls, embeddings, search)
• OpenAICompatibleProvider is reusable for OpenAI, Azure OpenAI, vLLM, and any other compatible endpoint
• Provider switching is a configuration change, not a code change
• The openai SDK handles streaming, retries, and error handling

Negative¶

• Ollama requires local resources (memory for model weights, ~2-4 GB for recommended models)
• Local model quality is lower than commercial providers (smaller models, less capable tool calling)
• First model pull adds startup latency (~1-2 minutes for small models)
• Embedding dimensions differ from OpenAI (handled by ADR-0015's per-model tables)

Neutral¶

• Commercial provider support (OpenAI, Anthropic) can be added later by configuring ModelBinding entries; OpenAICompatibleProvider already handles OpenAI directly
• Anthropic uses a different API format and will need its own provider class (not OpenAI-compatible)
• GPU acceleration is optional; CPU inference is sufficient for development with small models

Alternatives Considered¶

1. Implement OpenAI Provider First
2. Description: Start with the official OpenAI API as the first real provider
3. Pros: Best model quality, well-documented SDK, direct path to production
4. Cons: Requires API keys and incurs costs for every developer and test run, rate limits during heavy testing

5. Why not chosen: Adds friction to development; Ollama provides a zero-cost local alternative while using the same SDK

6. Use Raw httpx Instead of openai SDK

7. Description: Implement the OpenAI-compatible API calls directly with httpx
8. Pros: No additional dependency, full control over request/response handling
9. Cons: Must implement streaming SSE parsing, error handling, retries, and response deserialization manually; duplicates well-tested SDK logic

10. Why not chosen: The openai SDK is mature, well-maintained, and handles edge cases we'd have to reimplement

11. Use llama-cpp-python for Direct Inference

12. Description: Run models directly in-process using llama.cpp Python bindings
13. Pros: No separate server process, lower latency, simpler deployment
14. Cons: Tight coupling to llama.cpp, different API than production providers, model management complexity, build dependencies (C++ compilation)
15. Why not chosen: Ollama provides the same local inference with an API-compatible interface, keeping the provider abstraction clean

References¶

• Ollama Documentation
• Ollama OpenAI Compatibility
• OpenAI Python SDK
• ADR-0011: AI Run Orchestration Model (orchestration loop)
• ADR-0015: Centralized Embedding Store (embedding dimensions and model migration)