ADR-0011: AI Run Orchestration Model¶

Status¶

ACCEPTED

Date: 2026-02-14

Changelog¶

Implementation Status¶

The orchestration loop and tool execution are fully implemented. RunOrchestrator executes the complete model-call → tool-call → repeat loop with SSE streaming, cooperative cancellation, and all safety limits (max model calls, max tool calls, max wall time, loop breaker). Real HTTP tool execution is handled by HttpToolExecutor via httpx.

Not yet implemented: - Run queuing/prioritization for multi-tenant fairness - Cost attribution using token counts

See also: ADR-0014 — real LLM provider integrations (OpenAI-compatible, Ollama) are now implemented. The orchestrator resolves a model provider per-run from the space policy's ModelBinding records, falling back to StubModelProvider when no binding is configured.

Context¶

The AI service needs an orchestration model for executing multi-step LLM interactions. A "run" represents a single orchestration cycle: the system takes conversation context, calls an LLM, optionally executes tool calls, and produces a final response. This must support:

• Asynchronous execution (runs may take 10-120 seconds)
• Server-Sent Events (SSE) streaming of intermediate steps to clients
• Cancellation of in-progress runs
• Reproducibility (which tools and model were used)
• Safety limits to prevent runaway loops or excessive resource consumption

Key constraints: - Runs are long-lived relative to HTTP request timeout (may span many model + tool calls) - Clients need real-time visibility into run progress - The system must be resilient to model provider failures mid-run - Tool calls may fail independently and should not necessarily terminate the entire run

Decision¶

We will implement an async run orchestration model with the following design:

1. Run State Machine: Runs progress through states: pending -> running -> streaming -> completed|failed|cancelled. State transitions are validated on the Run aggregate with RunStateError raised for invalid transitions. Terminal states (completed, failed, cancelled) are immutable.

2. Run Steps: Each atomic operation within a run (model call, tool call) is recorded as a RunStep child entity with its own status (pending -> running -> completed|failed), timing, token counts, and tool I/O. Steps provide a complete audit trail.

3. Pinned Tool Catalog: At run creation time, the current ToolCatalogRevision is recorded on the run. All tool resolution during the run uses this pinned revision, ensuring consistent behavior even if tools are refreshed mid-run.

4. SSE Streaming: The GET /runs/{uuid}/stream endpoint returns an SSE event stream. Events include run.step.created, run.step.completed, run.completed, run.failed, run.cancelled. This gives clients real-time visibility without polling.

5. Cancellation: POST /runs/{uuid}/cancel sets a cancellation flag. The orchestration loop checks this flag between steps and transitions to cancelled state. In-flight model calls are not interrupted but the loop exits after the current step completes.

6. Safety Limits: Enforced per-run via SpacePolicy configuration:

7. max_model_calls (default 12): Maximum LLM invocations per run
8. max_tool_calls (default 20): Maximum tool executions per run
9. max_wall_time_seconds (default 120): Wall-clock timeout

10. Loop Breaker: Detects 3 consecutive identical tool calls (same tool_id + same arguments) and terminates the run with RunLimitExceededError

11. Pessimistic Locking: State transitions acquire a SELECT ... FOR UPDATE lock on the run row to prevent concurrent modifications (e.g., cancellation racing with completion).

Consequences¶

Positive¶

• Complete audit trail via RunSteps enables debugging and cost attribution
• Pinned catalog revision ensures reproducible runs
• SSE streaming gives clients real-time feedback without polling overhead
• Safety limits prevent unbounded resource consumption
• Loop breaker catches degenerate LLM behavior (repeated identical tool calls)
• Pessimistic locking prevents race conditions in state transitions

Negative¶

• SSE connections are long-lived, consuming server resources proportional to active runs
• Cancellation is cooperative (between steps), not preemptive; a slow model call must complete before cancellation takes effect
• Run step recording adds write amplification to the database during orchestration
• Pessimistic locking may cause brief contention if cancellation and completion race

Neutral¶

• Future work may add run queuing/prioritization for multi-tenant fairness
• Token counts on run steps enable future cost attribution and billing features

Alternatives Considered¶

1. Synchronous Request-Response Runs
2. Description: Block the HTTP request until the run completes, return full result
3. Pros: Simpler client integration, no SSE complexity
4. Cons: HTTP timeouts for long runs (30-120s), no intermediate progress, poor UX for multi-step orchestration

5. Why not chosen: Runs routinely exceed typical HTTP timeout windows; users need intermediate feedback

6. Polling-Based Status Checks

7. Description: Start run asynchronously, client polls GET /runs/{uuid} for status
8. Pros: Simpler server implementation (no SSE), works with all HTTP clients
9. Cons: Latency between steps (poll interval), unnecessary load on server, poor real-time experience

10. Why not chosen: SSE provides superior UX with lower overhead; polling can be added as fallback later

11. WebSocket-Based Streaming

12. Description: Use WebSocket connections for bidirectional run communication
13. Pros: Full-duplex (client can send cancellation inline), lower overhead than SSE for high-frequency updates
14. Cons: More complex server and client implementation, harder to load balance, not all proxies handle WebSockets well
15. Why not chosen: SSE is sufficient for our unidirectional streaming needs and has better proxy compatibility; cancellation via separate POST is adequate

References¶

• Server-Sent Events (SSE) Specification
• OpenAI Assistants API - Runs
• ADR-0010: AI Tool Registry and Manifest Discovery (catalog pinning)