Skip to content

ADR-0013: Web Ingestion in AI Service

Status

ACCEPTED

Date: 2026-02-14

Changelog

Date Revision Description Author
2026-02-14 1.0 Initial version souperk
2026-02-15 1.1 Update status; pipeline fully implemented souperk

Implementation Status

The web ingestion pipeline is fully implemented. IngestionService.fetch_url uses a three-phase transaction pattern: (1) read policy and check ingestion_enabled, (2) HTTP fetch via httpx, content type detection, SHA-256 hashing, text extraction (BeautifulSoup for HTML, PyMuPDF for PDF), and MinIO upload of raw + extracted content, (3) dedup check via find_by_content_hash and insert. The POST /web/fetch endpoint returns proper error codes: 403 for disabled ingestion, 413 for size limit exceeded, 422 for unsupported content types or network errors.

Not yet implemented: - Chunking of ingested content for LLM context windows - Migration to KB-based ingestion (when KB service provides URL-fetch API) - JavaScript rendering for dynamic pages - OCR for scanned/image-based PDFs

Context

The AI service needs the ability to fetch web content (URLs provided by users or referenced during tool execution) and make it available as context for LLM conversations. This involves downloading HTML/PDF content, extracting readable text, and storing both raw and extracted versions for audit and reprocessing.

Key factors: - Users may paste URLs into conversations that the LLM needs to read - Tool calls may reference external resources that need to be fetched - Raw content must be preserved for auditability and future re-extraction with improved parsers - Content size must be bounded to prevent abuse - The KB service already has sophisticated document processing capabilities

Decision

We will implement basic web ingestion in the AI service for V1 with a clear path to delegating to KB in the future:

  1. Fetch and Store: The AI service fetches URLs via httpx, stores the raw response body in MinIO under ai-ingest/{space_id}/raw/{content_hash}, extracts text, and stores the extracted version under ai-ingest/{space_id}/extracted/{content_hash}.

  2. Content Types: V1 supports HTML (via BeautifulSoup text extraction) and PDF (via PyMuPDF text extraction). Other content types are rejected with IngestionError.

  3. Deduplication: Content is identified by SHA-256 hash of the raw bytes. If the same content has already been ingested for a space, the existing SourceDocument record is returned without re-fetching or re-processing.

  4. Size Limits: Maximum ingest size is controlled by SpacePolicy.max_ingest_bytes (default 25MB). Content exceeding this limit is rejected before storage.

  5. SourceDocument Record: Each ingestion creates a SourceDocument aggregate tracking: source URL, content hash, document type, raw/extracted blob keys (MinIO paths), byte size, and space ownership.

  6. No Chunking in V1: The extracted text is returned as-is to the caller. Chunking and embedding of ingested content is deferred to future work (likely delegated to KB service).

Consequences

Positive

  • Simple, self-contained implementation with no cross-service dependencies
  • Content-hash deduplication prevents redundant fetches and storage
  • Raw content preservation enables future re-extraction with better parsers
  • MinIO storage separates blob data from the relational database
  • Size limits prevent abuse and unbounded resource consumption

Negative

  • Duplicates some capability that the KB service will eventually provide
  • HTML extraction via BeautifulSoup is basic (no JavaScript rendering, limited handling of complex layouts)
  • PDF extraction via PyMuPDF may struggle with scanned/image-based PDFs
  • No chunking means large documents may exceed LLM context windows

Neutral

  • The extraction pipeline is intentionally simple; production-quality extraction may require headless browser rendering or OCR
  • MinIO bucket lifecycle policies should be configured to manage storage growth
  • Future migration to KB-based ingestion will require updating the ingestion service to call KB's API instead of processing locally

Alternatives Considered

  1. Delegate to KB Service from Day One
  2. Description: AI service sends URLs to KB service for processing; KB returns extracted text
  3. Pros: Single extraction pipeline, KB has better document processing capabilities, no duplication
  4. Cons: Creates hard runtime dependency on KB service, KB service doesn't yet have a URL-fetch API, increases scope of KB changes needed

  5. Why not chosen: Adds cross-service coupling and blocks AI service delivery on KB feature work

  6. No Web Ingestion (User Pastes Text)

  7. Description: Users manually copy-paste web content into conversations
  8. Pros: Zero implementation effort, no security concerns from URL fetching
  9. Cons: Poor UX, loses formatting/structure, impractical for long documents, breaks tool-initiated content fetch

  10. Why not chosen: Fundamentally limits the AI service's utility; web content access is a core use case

  11. Proxy Through External Service (e.g., Jina Reader)

  12. Description: Use a third-party web content extraction API
  13. Pros: Better extraction quality, handles JavaScript-rendered pages, maintained by specialists
  14. Cons: External dependency, potential cost, data privacy concerns (content sent to third party), availability risk
  15. Why not chosen: Unacceptable data privacy trade-off for a self-hosted platform; may be offered as optional enhancement later

References