📘 Technical Design 031 — Handle Generation Strategy Protocol & Mappers¶

🧭 1. Summary¶

MemberBuilder couples discovery, enrichment, caching, and metrics in a monolithic class (~450 lines). TD-031 replaces it with composable strategy-based architecture:

• HandleGenerationStrategy protocol: Single interface for mapping sources to specs
• Immutable, replaceable specs: Build-time drafts representing handles before finalization (ModuleSpec, ClassSpec, etc.); immutable per generation, but replaceable wholesale by plugins before materialization
• Core mappers: 7 source-specific implementations (Module, Type, Function, Method, Parameter, Field, Property)
• Pipeline orchestration: HandleBuildingPipeline executes strategies in sequence with metrics
• Depth-first discovery: Stack-based processing minimizes memory footprint
• Integration with TD-033: ReflectionEngine uses strategies instead of MemberBuilder

At a glance: - Replace monolithic builder with pluggable mappers - Each mapper ≤100 lines, single responsibility - Specs are immutable per-generation; plugins can replace a spec (or inject a replacement handle) before materialization - Pipeline handles metrics, error wrapping, strategy composition - Full regression parity with MemberBuilder outputs - Examples are contract-first and concise (prefer ≤50 lines when needed)

📚 2. Context and Problem Statement¶

2.1 MemberBuilder Pain Points¶

From member_builder.py (~450 lines):

Issues: 1. No extensibility: ORM columns, descriptors require editing _process_callable_or_field_member() 2. Hardcoded heuristics: Method vs field detection embedded in private methods 3. Testing friction: Can't unit-test specific discovery logic without full builder 4. Complexity: Seven responsibilities (module, type, method, field, property, constructor, parameter) in one class 5. Missing composition: Can't combine core discovery with framework-specific logic

Example: Adding SQLAlchemy Column Support

Current (requires editing MemberBuilder internals):

# Must edit MemberBuilder._process_type_members() private method
def _process_type_members(self, type_, type_info, module_info):
    # ... existing code ...
    # INSERT: custom SQLAlchemy detection here
    if hasattr(type_, '__mapper__'):
        for col in type_.__mapper__.columns:
            # ... handle ORM column ...

Desired (pluggable strategy):

class SQLAlchemyMapper(HandleGenerationStrategy):
    def map(self, source, context):
        if hasattr(source, '__mapper__'):
            for col in source.__mapper__.columns:
                field_spec = FieldSpec(qualname=..., orm_column=col)
                context.ensure_member(field_spec)
        return context.spec

2.2 Goals¶

1. Single strategy protocol: map(source, context) -> spec for all member types
2. 7 core mappers: Module, Type, Function, Method, Parameter, Field, Property (each ≤100 lines)
3. Immutable, replaceable specs: Per-generation drafts, enriched via with_*() returning new instances; plugins may replace a spec or handle before materialization
4. Pipeline orchestration: Deterministic composition, metrics per strategy
5. Regression parity: Outputs match MemberBuilder golden fixtures
6. Zero public API changes: Strategies internal to engine; ReflectionContext unchanged (integration in TD-032)

2.3 Out of Scope¶

• ReflectionContext integration (TD-032)
• MemberBuilder removal (TD-032)
• Lazy loading (Phase C.4)
• Forward reference resolution (Phase F)

2.4 Assumptions¶

• TD-033 complete: ReflectionEngine + HandleRegistry available
• Python 3.13+: PEP 695 generics, modern typing
• Immutable specs: Per-generation drafts; enrichment returns new instances; plugins may replace the draft or resulting handle before materialization
• Depth-first discovery: LIFO stack minimizes peak memory

🧩 3. Design Overview¶

3.1 Strategy-Based Architecture¶

graph TB
    subgraph "TD-033: Foundation"
        Engine["ReflectionEngine<br/>Orchestration"]
        Registry["HandleRegistry<br/>Identity"]
    end

    subgraph "TD-031: Strategy Layer (THIS TD)"
        Protocol["HandleGenerationStrategy<br/>Protocol"]
        Pipeline["HandleBuildingPipeline<br/>Orchestrator"]
        Context["HandleGenerationContext<br/>Stack Manager"]

        subgraph "Specs (Immutable)"
            ModSpec["ModuleSpec"]
            TypeSpec["ClassSpec"]
            FuncSpec["FunctionSpec"]
        end

        subgraph "Core Mappers"
            ModMapper["ModuleHandleMapper"]
            TypeMapper["TypeHandleMapper"]
            FuncMapper["FunctionHandleMapper"]
            MethMapper["MethodHandleMapper"]
        end
    end

    Engine -->|Uses| Pipeline
    Pipeline -->|Executes| ModMapper
    Pipeline -->|Executes| TypeMapper
    Pipeline -->|Executes| FuncMapper
    Pipeline -->|Executes| MethMapper

    ModMapper -->|Implements| Protocol
    TypeMapper -->|Implements| Protocol
    FuncMapper -->|Implements| Protocol
    MethMapper -->|Implements| Protocol

    ModMapper -->|Creates| ModSpec
    TypeMapper -->|Creates| TypeSpec
    FuncMapper -->|Creates| FuncSpec

    Pipeline -->|Manages| Context
    Context -->|Registers specs| Registry

    classDef foundation fill:none,stroke:#9C27B0,stroke-width:2px,color:#9C27B0;
    classDef strategy fill:none,stroke:#1976D2,stroke-width:2px,color:#1976D2;
    classDef specs fill:none,stroke:#4CAF50,stroke-width:2px,color:#4CAF50;
    classDef mappers fill:none,stroke:#FF9800,stroke-width:2px,color:#FF9800;

    class Engine,Registry foundation;
    class Protocol,Pipeline,Context strategy;
    class ModSpec,TypeSpec,FuncSpec specs;
    class ModMapper,TypeMapper,FuncMapper,MethMapper mappers;

3.2 Key Components¶

3.3 Three-TD Integration¶

TD-033 (Complete first): Provides ReflectionEngine + HandleRegistry TD-031 (This TD): Builds strategy layer, integrates with engine TD-032 (After this): Wires strategies into ReflectionContext, removes MemberBuilder

3.4 Core Contracts (source of truth)¶

⚙️ 4. Detailed Design¶

4.1 Spec Hierarchy (Immutable, Replaceable Drafts)¶

• Specs are immutable per generation (e.g., frozen dataclasses or equivalent).
• Generation is idempotent: recreating a spec for the same source yields the same data.
• Plugins can replace a spec wholesale before materialization (including changing handle type) or inject a replacement handle.
• Specs are build-time only; once a handle is materialized, mutations happen on the handle itself.
• Required set (8 types): Module, Class, Function, Method, Parameter, Field, Property, Constructor.
• Enrichment uses with_*()/copy() helpers that return new spec instances.

4.2 HandleRef (Indirection for Inter-Handle Links)¶

• All inter-handle relationships (module → types, type → members, return types, parameter types) use HandleRef.
• Structure: HandleRef(qualname, kind, generation, resolver_fn, meta?).
• Resolution path:
• Check cached resolved handle + matching generation → cache_hit metric.
• On miss or generation mismatch → refresh via registry/engine → cache_miss metric.
• Registry (TD-033) owns generation counters; swaps increment generation so refs re-resolve cleanly.
• Supports lazy resolution: refs can carry a lazy thunk; engine forces resolution on access.

4.3 Lifecycle & Plugin Interception¶

1. Registry ensures a skeleton handle entry (identity + generation).
2. Strategies/plugins produce an immutable spec draft; plugins may replace the spec or veto/replace the target handle type before materialization.
3. Pipeline materializes a handle from the spec; registry entry points to the live handle.
4. Enrichment/mutation happens on the handle (not the spec).
5. Type change/swaps: plugin or engine can replace the handle instance; registry updates generation; refs re-resolve.

4.4 Lazy Resolution¶

• Handles may carry LazyValue wrappers for expensive/graph-expanding attributes (e.g., return types, annotations).
• Accessing a LazyValue triggers engine-mediated resolution (can build missing handles on demand).
• Metrics: record timing and hit/miss when lazy values resolve (shared pattern with HandleRef metrics).
• Flow (lazy attribute access):

graph LR
    H[Handle<br/>live instance] --> LV[LazyValue<br/>wrapper]
    LV --> |access| ENG[Engine resolve<br/>HandleRef + LazyValue]
    ENG --> |cache_hit| OUT[Handle/Value returned]
    ENG --> |cache_miss + build| REG[Registry<br/>generation update]
    REG --> OUT
    classDef node fill:none,stroke:#1976D2,stroke-width:2px,color:#1976D2;
    class H,LV,ENG,REG,OUT node;

4.5 Pipeline & Metrics (Refresh)¶

• HandleBuildingPipeline executes strategies with per-strategy timing.
• Metrics to add:
• handle_ref.resolve.cache_hit, handle_ref.resolve.cache_miss
• handle_ref.resolve.duration
• Strategy execution durations (existing)
• Swap events (registry generation increments)
• Verification matrix (target coverage):
• Unit: HandleRef cache hit/miss and generation mismatch re-resolve; LazyValue timing + error surfacing; strategy idempotency.
• Integration: Pipeline parity vs MemberBuilder golden fixtures; end-to-end registry swaps reflected via refs.
• Performance: <10% delta vs MemberBuilder baseline with lazy defaults; measure cache hit ratio.

4.6 Acceptance Criteria (Updated)¶

• ⏳ Specs: immutable, replaceable drafts; eight types defined; with_*() semantics documented.
• ⏳ HandleRef: standardized for all inter-handle links; generation-based cache with hit/miss metrics.
• ⏳ LazyValue: wrapper defined; access routes through engine; timing + hit/miss captured.
• ⏳ Plugin interception: can replace specs or handles before materialization; lifecycle documented.
• ⏳ Parity: outputs match MemberBuilder golden fixtures; performance within 10% with lazy resolution defaults.

4.7 Strategy Protocol¶

from typing import Protocol

from .mapping_context import MappingContext
from ..specs import BaseSpec


class HandleGenerationStrategy(Protocol):
    """Protocol for mapping source objects to enriched specs.

    Responsibilities:
    1. Identify: Determine if source is applicable (e.g., is it a module?)
    2. Enrich: Discover members, enqueue via context.ensure_member()
    3. Return: Enhanced spec with discovered metadata

    Multiple strategies execute in sequence via HandleBuildingPipeline.
    Later strategies refine earlier results (composition pattern).

    Contract:
    - MUST return a spec (never None; pass through if not applicable)
    - MUST NOT mutate input spec (use copy() for enrichment)
    - SHOULD enqueue discovered members via context.ensure_member()
    - MAY skip processing if source not applicable (return context.spec unchanged)
    """

    def map(self, source: object, context: MappingContext) -> BaseSpec:
        """Map source object to enriched spec.

        :param source: Python object being mapped (module, class, function, etc.).
        :type source: object
        :param context: Mapping context with spec, parent, module, ensure_member().
        :type context: MappingContext
        :return: Enhanced spec with discovered members and metadata.
        :rtype: BaseSpec
        :raises ValueError: If source is unsupported and strict validation required.

        Example::

            >>> def map(self, source, context):
            ...     if not isinstance(source, type):
            ...         return context.spec  # Not a class, pass through
            ...
            ...     # Discover methods
            ...     for name, obj in vars(source).items():
            ...         if inspect.isfunction(obj):
            ...             method_spec = MethodSpec(qualname=f"{source.__qualname__}.{name}")
            ...             context.ensure_member(method_spec)
            ...
            ...     return context.spec
        """
        ...

4.8 MappingContext (Contract-First)¶

@dataclass(frozen=True)
class MappingContext:
    """Immutable context provided to mapping strategies."""

    source: object
    spec: BaseSpec
    parent: ClassSpec | None
    module: ModuleSpec
    cache: HierarchicalCache
    _generation_context: "HandleGenerationContext"

    def ensure_member(self, member_spec: BaseSpec) -> BaseSpec: ...
    def get_spec(self, qualname: str) -> BaseSpec | None: ...

4.9 HandleGenerationContext (Contract-First)¶

class HandleGenerationContext:
    """Manages spec draft lifecycle during handle building (LIFO for memory efficiency)."""

    def register_spec(self, spec: BaseSpec) -> BaseSpec: ...
    def get_spec(self, qualname: str) -> BaseSpec | None: ...
    def next_pending(self) -> BaseSpec | None: ...
    def finalize_all(self) -> dict[str, BaseSpec]: ...

4.10 HandleBuildingPipeline (Contract-First)¶

class HandleBuildingPipeline:
    """Executes mapping strategies with timing and error handling."""

    def execute(self, source: object, context: MappingContext) -> BaseSpec: ...
    def walk(self, root_spec: BaseSpec, context: HandleGenerationContext) -> dict[str, BaseSpec]: ...

🧪 5. Test Strategy¶

5.1 Test Matrix¶

✅ 5. Acceptance Criteria¶

• ⏳ Specs immutable per-generation; with_*() helpers return new instances; drafts replaceable before materialization; eight spec types defined.
• ⏳ Strategy protocol + MappingContext frozen contract; contract tests in place.
• ⏳ HandleGenerationContext: LIFO stack, generation-aware registration, finalize_all; pipeline: strategy composition + timing metrics.
• ⏳ HandleRef: generation-aware cache hit/miss + swap response; LazyValue: resolve timing/hit/miss via engine mediation.
• ⏳ Core mappers use immutable specs and refs/lazy where applicable; unit coverage ≥80% per mapper.
• ⏳ Regression parity vs MemberBuilder (structure + metrics); performance within 10% with lazy defaults.

📋 6. Implementation Phases¶

Phase C.2a: Specs Foundation (⏳ Pending)¶

Tasks: 1. Create reflection/engine/specs/ package 2. Implement immutable BaseSpec and 8 spec types: Module, Class, Function, Method, Parameter, Field, Property, Constructor 3. Provide with_() helpers returning new instances; document replaceable draft semantics 4. Write unit tests (100% coverage): immutability, with_() helpers, replacement semantics

Acceptance: All spec tests pass with 100% coverage

Phase C.2b: Strategy Protocol (⏳ Pending)¶

Tasks: 1. Create reflection/engine/strategies/protocol.py 2. Define HandleGenerationStrategy protocol with map() signature 3. Create reflection/engine/strategies/mapping_context.py 4. Implement MappingContext dataclass (frozen=True) 5. Write contract tests: validate protocol compliance

Acceptance: Protocol defined; contract tests pass

Phase C.2c: Generation Context & Pipeline (⏳ Pending)¶

Tasks: 1. Create reflection/engine/handle_generation_context.py 2. Implement stack management (LIFO), register_spec(), next_pending(), finalize_all(), generation-aware registry 3. Create reflection/engine/strategies/pipeline.py 4. Implement strategy composition loop + timing metrics wrapping 5. Write unit tests (95% coverage): circular refs, finalization, metrics

Acceptance: Context + pipeline tests pass

Phase C.2d: Core Mappers (⏳ Pending)¶

Tasks: 1. Create reflection/engine/strategies/mappers/ package 2. Implement ModuleHandleMapper (≤100 lines) 3. Implement TypeHandleMapper (≤100 lines) 4. Implement FunctionHandleMapper (≤100 lines) 5. Implement MethodHandleMapper (≤100 lines) 6. Implement ParameterHandleMapper (≤100 lines) 7. Implement FieldHandleMapper (≤100 lines) 8. Implement PropertyHandleMapper (≤100 lines) 9. Write unit tests (≥80% per mapper): discovery logic, ensure_member() calls, immutable spec usage, HandleRef/LazyValue where appropriate

Acceptance: All mappers implemented; tests pass

Phase C.2e: Engine Integration (⏳ Pending)¶

Tasks: 1. Update ReflectionEngine.resolve_type() to use pipeline 2. Update ReflectionEngine.resolve_module() to use pipeline 3. Create default pipeline factory: create_default_pipeline() 4. Write regression tests: compare outputs with MemberBuilder golden fixtures 5. Write integration tests: full workflow (source → specs → handles → registry) 6. Performance benchmarks: validate <10% overhead vs MemberBuilder

Acceptance: Engine uses strategies; regression parity validated; performance acceptable

📝 7. Dependencies¶

Requires TD-033 Complete: - HandleRegistry for identity management - ReflectionEngine infrastructure - Cache abstraction

Blocks TD-032: - TD-032 needs working strategy-based building before ReflectionContext integration

📖 8. Changelog¶

• 2025-11-15 — Initial draft: Strategy protocol, eight spec types, seven mappers, pipeline orchestration, TD-033 integration plan.
• 2025-11-22 — Draft update: Locked specs as immutable/replaceable drafts; introduced HandleRef + generation + hit/miss metrics; added LazyValue wrapper and swap lifecycle; refreshed acceptance criteria and verification matrix.

End of TD-031