📘 Technical Design 004 — Split Oversized Reflection Modules¶

🧭 1. Summary¶

The reflection subsystem contains several oversized modules that combine multiple concerns, reducing readability and increasing maintenance risk. This design splits the largest modules into focused, cohesive internal modules with well-defined interfaces.

At a glance:

• Split high_fidelity_type_hints.py (963 lines) into forward reference processing, namespace derivation, and core provider logic
• Split member_info.py (654 lines) into base classes, specific member types, and builder utilities
• Extract 15+ private methods from HighFidelityTypeHintProvider into focused utility classes
• Create focused internal modules for common concerns: caching strategies, error models, and token processing
• Add extension points for custom augmentation strategies and cache key builders
• Maintain public API stability through re-exports and composition

📚 2. Context and Problem Statement¶

Several reflection modules have grown large and combine multiple concerns, making them difficult to understand, test, and maintain:

• high_fidelity_type_hints.py (963 lines): Combines forward reference resolution, namespace derivation, token parsing, augmentation heuristics, and the main provider implementation
• member_info.py (654 lines): Contains base MemberInfo class plus all specific member types and utility methods
• Mixed responsibilities reduce testability and increase the risk of introducing bugs during changes

2.1 Goals¶

1. Improve maintainability: Split oversized modules into focused, single-responsibility components
2. Enhance testability: Enable targeted unit testing of individual concerns
3. Reduce complexity: Lower cyclomatic complexity and cognitive load per module
4. Preserve stability: Maintain existing public APIs through composition and re-exports
5. Enable extension: Create clean extension points for future enhancements
6. Improve debuggability: Make internal state and flow more visible through better separation
7. Reduce coupling: Minimize dependencies between different concerns within large modules

2.2 Non-Goals (Out of Scope)¶

• Changing public APIs or behavior of existing components
• Performance optimizations beyond maintaining current bounds
• Adding new features or capabilities beyond modularization
• Modifying test interfaces or test data structures

2.3 Assumptions and Constraints¶

• Alpha stage allows internal reorganization without external compatibility concerns
• Existing test coverage must be preserved
• Public import paths and class names remain unchanged
• Module splitting should not introduce circular dependencies
• Performance characteristics must remain within current bounds

🧩 3. Design Overview¶

The design decomposes the two largest modules and extracts common utilities into focused internal modules. Each new module has a single, well-defined responsibility and clear interfaces.

3.1 New Internal Modules¶

3.2 Removed/Changed Interfaces (Before → After)¶

3.3 Removal/Deprecation Policy¶

• Internal reorganization only: No public API changes
• Alpha stage: Internal module restructuring without deprecation period
• Re-export strategy: Maintain existing import paths through __init__.py re-exports
• Test updates: Update import statements in tests but preserve test logic and coverage

📐 4. Detailed API Surface¶

4.1 Forward Reference Processing¶

# forward_refs/tokens.py
class TokenExtractor:
    def extract_forward_ref_tokens(self, annotation_str: str) -> list[str]: ...
    def is_bare_name(self, token: str) -> bool: ...

# forward_refs/augmentation.py
class NamespaceAugmenter:
    def __init__(self, import_resolver: ImportResolver): ...
    def augment_namespace(self, globalns: dict, tokens: list[str], origin_module: str) -> dict: ...
    def guess_sibling_modules(self, base_module: str, token: str) -> list[str]: ...

# forward_refs/namespaces.py
class NamespaceDeriver:
    def derive_namespaces(self, subject: Any, origin_module: str) -> tuple[dict, dict]: ...
    def build_localns(self, subject: Any) -> dict: ...

# forward_refs/postprocessing.py
class TypeHintPostprocessor:
    def __init__(self, *, strict: bool = False): ...
    def process_unresolved(self, hints: dict, subject_qualname: str) -> dict: ...

4.2 Member Type Organization¶

# member_types/base.py
class MemberInfo:
    # Core MemberInfo functionality remains the same
    pass

# member_types/callables.py
class CallableInfo(MemberInfo): ...
class FunctionInfo(CallableInfo): ...
class MethodInfo(CallableInfo): ...
class ConstructorInfo(CallableInfo): ...

# member_types/structural.py
class TypeInfo(MemberInfo): ...
class FieldInfo(MemberInfo): ...
class PropertyInfo(MemberInfo): ...

# member_types/containers.py
class ModuleInfo(MemberInfo): ...
class PackageInfo(MemberInfo): ...

4.3 Internal Utilities¶

# internal/caching.py
class CacheKeyBuilder:
    def __init__(self, strategy: Literal["identity", "canonical"]): ...
    def build_key(self, subject: Any, **context: Any) -> str: ...

# internal/errors.py
class ErrorPayloadBuilder:
    @staticmethod
    def build_type_hint_error_payload(subject: Any, slot: str | None, token: str | None, attempted: list[str]) -> dict: ...

# internal/annotation_extraction.py
class AnnotationExtractor:
    def extract_raw_annotations(self, obj: Any) -> dict[str, Any]: ...
    def extract_from_owner_module(self, module_name: str) -> dict[str, Any]: ...
    def extract_from_owner_class(self, member_qualname: str, import_resolver: ImportResolver) -> dict[str, Any]: ...

# internal/property_resolution.py
class PropertyTypeResolver:
    def resolve_property_type(self, prop: PropertyInfo, type_hints_provider: TypeHintProvider) -> Any | None: ...
    def apply_precedence_rules(self, getter_hint: Any, setter_hint: Any, class_hint: Any, module_hint: Any) -> Any: ...

⚙️ 5. Semantics¶

5.1 Module Organization Principles¶

• Single Responsibility: Each new module handles one specific concern
• Clear Dependencies: Dependencies flow in one direction, no circular imports
• Composition over Inheritance: Main provider classes compose smaller components
• Internal Interfaces: Well-defined interfaces between internal modules

5.2 Import Strategy¶

• Public APIs Unchanged: Existing imports continue to work through re-exports
• Internal Modules Private: New modules are implementation details, not public APIs
• Gradual Migration: Internal code can migrate to new structure incrementally

5.3 Testing Strategy¶

• Preserve Coverage: All existing tests continue to pass
• Add Focused Tests: New unit tests for individual internal modules
• Integration Tests: Maintain existing integration tests for full provider behavior

🧪 6. Examples (concise)¶

6.1 Forward Reference Processing (Internal API)¶

from .forward_refs.tokens import TokenExtractor
from .forward_refs.augmentation import NamespaceAugmenter

# Internal usage within HighFidelityTypeHintProvider
extractor = TokenExtractor()
tokens = extractor.extract_forward_ref_tokens("ForwardRef('MyClass')")

augmenter = NamespaceAugmenter(import_resolver)
enriched_globals = augmenter.augment_namespace(globalns, tokens, origin_module)

6.2 Member Type Usage (Public API Unchanged)¶

# Public API remains exactly the same
from loryth.common.reflection.member_info import TypeInfo, CallableInfo
# Implementation now uses composed member types internally

✅ 7. Acceptance Criteria¶

• high_fidelity_type_hints.py reduced to < 400 lines with core provider logic only
• member_info.py reduced to < 200 lines with base classes and re-exports only
• Forward reference processing split into 4 focused modules under forward_refs/
• Member types split into 4 focused modules under member_types/
• Internal utilities extracted to internal/ package
• All existing tests pass without modification to test logic
• Public import paths remain unchanged (re-exports working)
• No circular dependencies introduced between new modules
• Test coverage maintained or improved
• Cyclomatic complexity reduced by > 30% in split modules
• Extension points documented for custom augmentation strategies
• Performance benchmarks show no regression (< 5% overhead)
• Internal module dependency graph is acyclic and documented

🚀 8. Rollout and Migration¶

Phase 1: Extract Forward Reference Processing¶

1. Create forward_refs/ package with token extraction utilities
2. Create namespace derivation and augmentation modules
3. Update HighFidelityTypeHintProvider to use composed components
4. Add unit tests for new modules

Phase 2: Split Member Info Hierarchy¶

1. Create member_types/ package with base classes
2. Split member types into focused modules by category
3. Update member_info.py to re-export from new locations
4. Verify all existing imports continue working

Phase 3: Extract Internal Utilities¶

1. Create internal/ package for shared utilities
2. Extract caching and error handling components
3. Update consumers to use new utilities
4. Add focused unit tests

⚠️ 9. Risks and Mitigations¶

• Risk: Circular import dependencies during reorganization
• Mitigation: Design dependency graph upfront; use composition over inheritance
• Risk: Performance regression from additional indirection
• Mitigation: Benchmark critical paths; optimize hot paths if needed
• Risk: Breaking existing imports during transition
• Mitigation: Maintain re-exports throughout; validate imports in CI
• Risk: Test fragility during module moves
• Mitigation: Update imports only; preserve test logic and assertions

📈 10. Progress¶

Legend: [x] = complete, [ ] = pending

• Design dependency graph and module interfaces
• Create forward_refs/ package structure
• Extract token processing utilities
• Extract namespace derivation logic
• Extract augmentation heuristics
• Extract post-processing components
• Create member_types/ package structure
• Split callables, structural, and container types
• Create internal/ utilities package
• Extract caching strategies
• Extract error models and builders
• Update main provider classes to use composition
• Add comprehensive unit tests for new modules
• Verify performance benchmarks
• Update documentation and examples

📝 11. Changelog¶

• 2025-09-28 — TD-004 Complete: All phases implemented successfully. Created 12 focused modules across 3 packages (forward_refs/, member_types/, internal/). Reduced member_info.py from 654→37 lines (94% reduction). All 153 tests pass. Performance maintained. Public APIs preserved through re-exports.
• 2025-09-28 — Phase 3 Complete: Created internal/ utilities package with 4 focused modules (caching, errors, annotation_extraction, property_resolution). All internal utilities properly tested and documented with extension points.
• 2025-09-28 — Phase 2 Complete: Split member_info.py (654 lines) into organized member_types/ package with 4 focused modules (base, containers, structural, callables). All tests pass and public APIs preserved via re-exports.
• 2025-09-28 — Phase 1 Complete: Extracted forward reference processing into dedicated forward_refs/ package with 4 focused modules (tokens, namespaces, augmentation, postprocessing). Added TypeHintResolutionError to exceptions module.
• 2025-09-28 — Initial design using project template; identified specific modules and splitting strategy based on current codebase analysis

📖 12. Glossary¶

• Forward Reference: A string-based type annotation that refers to a type not yet defined
• Namespace Augmentation: Process of enriching global/local namespaces to resolve forward references
• Member Type: Specific subclass of MemberInfo representing different kinds of reflected members
• Provider Composition: Pattern where main provider classes delegate to smaller, focused components
• Re-export: Making internal modules available through existing public import paths