📘 Technical Design 024 — Test Suite Quality Improvement Meta¶

🧭 1. Summary¶

This meta technical design addresses critical test suite quality issues that impact library maintainability and evolution. It establishes universal quality principles across all test types (unit, integration) and tracks implementation through component-specific TDs (TD-025 for units, TD-026 for integration).

The design targets four key qualities: readability (tests as documentation), maintainability (minimize refactoring impact), discoverability (find relevant tests easily), and reliability (tests fail only on behavioral changes).

📚 2. Context and Problem Statement¶

2.1 Quality Crisis Indicators¶

The test suite has evolved with three critical pain points that make the library difficult to evolve:

1. Testing Internals Not Public API (~33% of tests)
2. Tests directly call private methods (_internal())
3. Breaks when implementation changes but behavior stays same

4. Example: test_high_fidelity_components.py tests provider._attempt_direct_resolution()

5. Fixture & Resource Disorganization

6. 22 inline fixtures scattered across 580 tests (3.8%)
7. No clear distinction: fixtures vs resources vs helpers
8. Duplication makes maintenance difficult

9. Example: tests/fixtures/ exists but not fully utilized

10. Test Brittleness & Overfitting

11. Tests fail when internal structure changes (even if behavior unchanged)
12. Example: RegistrationValidator refactoring broke tests checking _validator type
13. Tests too specific to implementation details

2.2 Analysis Data¶

2.3 Impact on Evolution¶

• Refactoring Cost: High - internal changes require test rewrites
• Discoverability: Low - hard to find tests for specific features
• Onboarding: Difficult - no clear patterns for writing new tests
• Maintenance Burden: Growing - more tests break with minor changes

2.4 Goals¶

1. Public API Testing: All tests validate observable public behavior (hard rule)
2. Clear Fixture Organization: Centralized fixtures with clear patterns
3. Reduced Brittleness: Tests only break when behavior changes
4. Better Discoverability: Tests easy to find and understand
5. Improved Readability: Tests serve as documentation

2.5 Non-Goals¶

• Rewriting all tests at once (module-by-module approach)
• Adding new testing frameworks or infrastructure
• Changing test execution behavior or performance
• Refactoring regression tests into other categories (separate effort)

2.6 Assumptions¶

• Components should be testable via public APIs (if not: code smell)
• Tests don't need shared state or setup (no service dependencies)
• Contributors will adopt new patterns for new tests
• Existing tests will be migrated incrementally

🧩 3. Design Overview¶

This meta TD defines universal principles and delegates implementation to specialized TDs:

TD-024: Test Suite Quality Meta
├── Universal Quality Principles
│   ├── Public API Testing (HARD RULE)
│   ├── Fixture Organization
│   ├── Test Independence
│   └── Documentation Requirements
│
├── Implementation Tracking
│   ├── TD-025: Unit Test Standards
│   ├── TD-026: Integration Test Standards
│   └── Migration Progress
│
└── Quality Metrics & Governance
    ├── Success Criteria
    ├── Migration Phases
    └── Module Priority Matrix

3.1 Universal Quality Principles¶

Principle 1: Public API Testing (HARD RULE)¶

✅ Test: Observable public behavior via public APIs
❌ Never: Call private methods (_internal) or protected methods (_protected)

Why: Internal changes shouldn't break tests unless behavior changes.

Principle 2: Behavior-Based Testing¶

✅ Assert: Observable outcomes (return value, state change, exception)
❌ Assert: Internal state (_cache.size), type of internal object (_validator type)

Why: Implementation details can change without affecting behavior.

Principle 3: Clear Test Naming¶

✅ test_component_behavior_when_condition()
❌ test_failure(), test_edge_case()

Why: Name should explain what's being tested and why.

Principle 4: Fixture Organization¶

✅ Centralized: tests/fixtures/ (shared fixtures)
✅ Localized: test file (specific fixtures)
❌ Scattered: Inline duplicates across many test files

Why: Reduces duplication and improves discoverability.

Principle 5: Test Independence¶

✅ Each test runs standalone
✅ No test state pollution or cross-test dependencies
✅ Tests can run in any order

Why: Enables parallel execution and reliable CI/CD.

3.2 Fixture vs Resource Distinction¶

3.3 Quality Metrics¶

📐 4. Universal Standards (Apply to All Test Types)¶

4.1 Rule: No Private API Testing¶

Statement: Never call _private() or _protected() methods in tests.

# ❌ FORBIDDEN
def test_resolution():
    hints = provider._attempt_direct_resolution(func)
    assert hints is not None

# ✅ REQUIRED: Test through public API
def test_resolution_success():
    hints = provider.for_callable(func)
    assert hints is not None

Rationale: Public APIs define contracts; implementation is internal.

Enforcement: Code review + linting (future).

Exception: None. If component isn't testable via public API, it's a code smell.

4.2 Rule: Behavior-Based Assertions¶

Statement: Assert observable outcomes, not internal state.

# ❌ FORBIDDEN: Checking internal state
def test_caching():
    assert len(obj._cache) == 1  # Tightly coupled to implementation

# ✅ REQUIRED: Test behavioral outcome
def test_caching_returns_same_result():
    result1 = obj.get(key)
    result2 = obj.get(key)
    assert result1 is result2  # Observable behavior: identity preserved

Rationale: Implementation details can change; behavior should remain stable.

Enforcement: Code review, test design discussion.

4.3 Rule: Every Test Has Explanatory Docstring¶

Statement: Every test function must have a docstring explaining why it exists.

# ✅ REQUIRED: Explains purpose
def test_container_rejects_duplicate_registration():
    """Test that container prevents duplicate service registration.

    Validates that attempting to register the same service type twice
    raises RegistrationError with appropriate messaging.
    """

Rationale: Tests document expected behavior; docstrings explain edge cases.

Enforcement: CLAUDE.md already requires this (verify compliance).

4.4 Rule: AAA Pattern¶

Statement: Use Arrange-Act-Assert pattern consistently.

def test_example():
    # ARRANGE: Set up test data/fixtures
    container = make_container()
    service = MockService()

    # ACT: Perform the action
    container.register(service)
    resolved = container.resolve(MockService)

    # ASSERT: Verify outcomes
    assert resolved is service

Rationale: Makes test structure clear and easy to follow.

Enforcement: Code review.

4.5 Rule: Test Naming Convention¶

Statement: Use test_component_behavior_when_condition() naming.

# ✅ GOOD: Clear what's tested and when
test_container_resolves_registered_service()
test_container_raises_error_when_dependency_missing()
test_decorator_preserves_class_identity_when_applied()

# ❌ BAD: Vague, doesn't explain scenario
test_container()
test_resolve_success()
test_error_case()

Rationale: Name communicates test purpose without reading code.

Enforcement: Code review (no blanket linting, as some exceptions exist).

✅ 5. Fixture Organization Strategy¶

5.1 Centralized Fixture Structure¶

tests/
├── conftest.py                           # Root conftest (imports from fixtures/)
├── fixtures/                             # Centralized fixtures & helpers
│   ├── __init__.py
│   ├── di/                              # DI-specific fixtures
│   │   ├── injection.py                 # Injection fixtures
│   │   ├── container.py                 # Container fixtures
│   │   └── bootstrap.py                 # Bootstrap fixtures
│   ├── reflection/                      # Reflection fixtures
│   │   ├── providers.py                 # Provider fixtures
│   │   └── context.py                   # Context fixtures
│   ├── markers/                         # Marker fixtures
│   │   └── common.py
│   └── common/                          # Shared test utilities
│       └── module_factory.py            # make_module (existing)
│
├── resources/                            # Code samples for library to parse
│   ├── reflection/                      # Reflection test resources
│   ├── di/                              # DI test resources
│   └── app_test_modules/                # Application examples
│
└── unit/                                # Test files
    ├── di/
    │   ├── conftest.py                  # DI-specific fixtures (if needed)
    │   └── test_*.py
    └── reflection/
        ├── conftest.py                  # Reflection-specific fixtures (if needed)
        └── test_*.py

5.2 Fixture Categorization¶

5.3 Migration Plan¶

Phase 1: Audit existing inline fixtures (22 instances) - Categorize as "shared" or "test-specific" - Move shared fixtures to tests/fixtures/

Phase 2: Reorganize by feature area - Create di/, reflection/, markers/ subdirectories - Group related fixtures together

Phase 3: Document fixture usage - Update CLAUDE.md with fixture patterns - Add examples to conftest.py

5.4 Test File Organization by Directory¶

Strategy: Group related test files into logical directories by component/module.

Structure for Reflection Tests (Example):

tests/unit/reflection/
├── __init__.py
├── conftest.py                              # Optional: shared reflection fixtures
├── type_resolution/                         # Phase 2: Type Resolution Module
│   ├── __init__.py
│   ├── conftest.py                          # Phase-specific fixtures (import_resolver)
│   ├── test_import_resolver.py              # Tests public ImportResolver API
│   ├── test_import_resolution.py            # Tests resolution workflows
│   ├── test_resolve_dotted_success.py       # Tests dotted path resolution
│   └── test_type_resolver.py                # Tests type resolution
├── type_hints/                              # Phase 3: Type Hints Components
│   ├── __init__.py
│   ├── conftest.py
│   ├── test_type_hints_annotations.py
│   ├── test_type_hints_fields.py
│   └── test_composite_providers.py
├── scanning/                                # Phase 4: Scanning (Component Layer)
│   ├── __init__.py
│   ├── conftest.py                          # scanner fixture
│   ├── test_scanner.py                      # PackageScanner component tests
│   └── test_scanner_component.py            # More component tests
├── context/                                 # Context/Integration Layer Tests
│   ├── __init__.py
│   ├── conftest.py                          # reflection_context fixture
│   ├── test_context_scanning.py             # ReflectionContext scanning behavior
│   └── test_context_error_handling.py       # Error handling through context
├── member_builder/                          # Phase 5: Member Builder
│   ├── __init__.py
│   ├── conftest.py
│   ├── test_member_builder.py
│   └── test_invocation.py
└── metrics/                                 # Phase 6: Metrics & Error Handling
    ├── __init__.py
    ├── conftest.py
    ├── test_metrics_components.py
    ├── test_metrics_hook.py
    └── test_error_payloads_schema.py

Benefits: - Tests for related components grouped together - Easier to find tests for specific features - Phase-specific fixtures isolated to conftest.py - Maximum 5 test files per directory prevents overcrowding

5.5 Fixture Scoping Hierarchy¶

Rule: Organize fixtures by scope (shared → phase-specific → inline)

Decision Tree:

Is fixture used by >1 phase or module?
├─ YES → Put in tests/fixtures/[area]/common.py
└─ NO → Is it used by >1 file in this phase?
    ├─ YES → Put in tests/unit/[module]/conftest.py
    └─ NO → Put inline in the single test file

Example from Phase 2 Implementation: - member_builder: Shared fixture in tests/fixtures/reflection_common.py (used by import_resolver, type_resolver) - null_metrics: Shared fixture in tests/fixtures/reflection_common.py (used by import_resolver) - import_resolver: Phase-specific in tests/unit/reflection/type_resolution/conftest.py (used by 2 test files) - type_resolver: Inline in test_type_resolver.py (used by 1 test file) - reflection_context: Inline in test_resolve_dotted_success.py (used by 1 test file)

Rules for Each Scope: 1. Shared Fixtures: Centralize in tests/fixtures/[area]/ - [ ] Document purpose and scope in docstring - [ ] Import in root tests/conftest.py - [ ] Never duplicate across phases

1. Phase-Specific Fixtures: Keep in tests/unit/[module]/conftest.py
2. Import shared fixtures from tests/fixtures/
3. Only define fixtures used by 2+ files in this phase

4. Add conftest.py to package (with __init__.py)

5. File-Specific Fixtures: Inline in test file

6. Define fixture in same file as tests using it
7. Prefix docstring with "Local fixture" to indicate scope
8. Use when fixture is only relevant to one test file

5.6 Unit vs Integration Test Classification¶

Critical Principle: Classify tests by WHAT IS BEING TESTED, not by directory structure.

Criteria for Unit Tests¶

Tests should be unit tests if: - Single Component Focus: Testing a single component in isolation - Component Has Own Logic: Component has behavior beyond delegation - Clear Responsibility: Component has a well-defined, independent job - Examples: - PythonImportResolver - resolves module imports - PackageScanner - scans packages and discovers members - SimpleTypeHintProvider - extracts type hints from callables - CompositeTypeHintProvider - orchestrates provider selection

Location: tests/unit/[module]/[component]/

Criteria for Integration Tests¶

Tests should be integration tests if: - Facade/Coordinator Pattern: Component's primary job is delegating to other components - Cross-Component Workflows: Tests require multiple components working together - Context/Orchestrator Tests: Tests of high-level APIs that coordinate lower-level components - Examples: - ReflectionContext - facade that coordinates import/type/scanning/etc (construct directly via the public constructor; fixtures like reflection_context_factory cover custom metrics/strict options) - Error propagation across component boundaries - Scanning policies and filtering across components

Location: tests/integration/[module]/

Facade Pattern Recognition¶

If a component's primary responsibility is coordinating or delegating to other components: - Test it in integration, NOT unit tests - Reason: Testing a facade in isolation requires mocking all delegates - Problem with mocks: Verifying "mock was called with X" is implementation testing - Better approach: Test the facade through integration tests where real components interact

Anti-Pattern:

# WRONG: Unit testing a facade (requires extensive mocking)
def test_reflection_context_delegates_correctly():
    mock_scanner = MagicMock()
    mock_type_resolver = MagicMock()
    ctx = ReflectionContext(
        scanner=mock_scanner,
        type_resolver=mock_type_resolver,
        ...
    )
    ctx.scan.module("foo")
    mock_scanner.scan_module.assert_called_once_with("foo")  # Implementation testing!

Good Pattern:

# GOOD: Integration testing a facade (uses real components)
def test_reflection_context_scanning_with_error_policy():
    ctx = ReflectionContext()
    with pytest.raises(ScanError):
        list(ctx.scan_module("nonexistent"))  # Tests observable behavior

5.7 Successful Patterns from Phases 1-4¶

Real Implementations Over Mocks¶

Pattern: Prefer lightweight real implementations over test doubles

Examples: - DictHierarchicalCache (test-only) vs ConcurrentHierarchicalCache (production) - ReflectionMetricsCollector (real metrics with snapshot retrieval) for all tests - Real MemberBuilder with test cache instead of mock introspection

Benefits: - Tests observable behavior, not implementation details - Catches real integration issues - Simpler test code (no mock verification) - More confidence in test results - Metrics snapshots validate actual metrics collection behavior

Fixture Factory Pattern¶

Pattern: Fixtures provide fully-configured, ready-to-use instances

Example:

@pytest.fixture
def scanner(member_builder: MemberBuilder) -> PackageScanner:
    """Provide configured PackageScanner."""
    metrics = ReflectionMetricsCollector()  # Real metrics collector
    import_resolver = PythonImportResolver(member_builder, metrics)
    return PackageScanner(import_resolver, member_builder, metrics)

Benefits: - Encapsulates component wiring (not in tests) - Tests focus on behavior, not setup - Fixtures define how components work together

Progressive Fixture Scoping¶

Pattern: Move fixtures from inline → phase-specific → shared as they're reused

1. Start inline: Define fixture in test file
2. Recognize reuse: Move to phase-specific conftest.py (2+ files in phase)
3. Promote to shared: Move to tests/fixtures/ (2+ phases use it)

Result: Natural, emergent organization that matches actual needs

Sphinx Docstring Requirement¶

Pattern: All tests and fixtures MUST have Sphinx-style docstrings

Requirements: - :param with type and description for each parameter - :type: for parameter types - :return: with description for return value - :rtype: for return type - Fixture scope noted: "Local fixture" vs "Shared fixture"

Example:

@pytest.fixture
def reflection_context() -> ReflectionContext:
    """Provide configured ReflectionContext for integration testing.

    Shared fixture used by multiple test phases for context-level workflows.

    :return: Configured reflection context instance.
    :rtype: ReflectionContext
    """
    return ReflectionContext()

Benefits: - Clear documentation of fixture purpose and scope - IDE autocomplete and type checking - Easier discovery of what fixtures do

🎯 6. Module-by-Module Migration¶

6.1 Priority Matrix¶

6.2 Migration Process¶

For each module:

1. Audit: Identify all private API calls
2. Refactor: Convert to test public APIs instead
3. Reorganize: Move fixtures to centralized location
4. Verify: Ensure tests still pass and add new ones if gaps found
5. Document: Update tests with clear docstrings explaining why they exist

6.3 Marking Non-Compliant Tests¶

During migration, mark tests that need work:

def test_something():
    """Test that something works.

    TODO: TD-025 - Refactor to test public API instead of _private()
    """
    # Current implementation that violates rules
    pass

This allows gradual migration without blocking other work.

📊 7. Success Criteria¶

• All tests comply with public API rule (0% private API calls)
• Fixtures organized in tests/fixtures/ structure
• Every test has docstring explaining why it exists
• No test breaks due to non-behavioral code changes
• Fixture duplication reduced by 50%+
• New tests follow all universal standards

⚠️ 8. Risks and Mitigations¶

📈 9. Progress Tracking¶

Legend: [x] = complete, [ ] = pending

Meta TD-024:¶

• Define universal quality principles
• Create Fixture organization strategy
• Document fixture scoping hierarchy and decision tree
• Provide test file organization by directory pattern
• Demonstrate with Phase 2 (Type Resolution) implementation
• Implement linting rules (future)
• Complete fixture reorganization across all phases

Implementation TDs:¶

• TD-025: Unit Test Standards (see TD-025)
• TD-026: Integration Test Standards (see TD-026)

Fixture Reorganization Phases:¶

• [~] Phase 1: Reflection module fixtures
• Create tests/fixtures/reflection_common.py
• Move shared fixtures (member_builder, null_metrics)
• Organize Phase 2 (Type Resolution) tests by directory
• Implement fixture scoping hierarchy
• Apply to Phases 3-6 (Type Hints, Scanning, Member Builder, Metrics)
• Phase 2: DI module fixtures
• Phase 3: Markers module fixtures
• Phase 4: Verification and documentation

Implementation Examples:¶

• ✅ Phase 2 (Type Resolution): Complete example of fixture organization
• Shared: tests/fixtures/reflection_common.py (member_builder, null_metrics)
• Phase-specific: tests/unit/reflection/type_resolution/conftest.py (import_resolver)
• File-specific: Inline fixtures in test_type_resolver.py, test_resolve_dotted_success.py

📝 10. Changelog¶

• 2025-10-19: Updated TD-024 with fixture scoping hierarchy and test directory organization
• Added section 5.4: Test File Organization by Directory
• Added section 5.5: Fixture Scoping Hierarchy with decision tree
• Referenced Phase 2 implementation as concrete example
• 2025-10-18: Initial meta TD created to address test suite quality issues

📖 11. Glossary¶

• Public API: Methods/properties without underscore prefix
• Private API: Methods/properties with underscore prefix (_internal)
• Protected API: Single underscore methods (should also be avoided in tests)
• Fixture: Test helper (factory, mock, builder)
• Resource: Code sample for library to parse (e.g., test app)
• Behavior-Based Testing: Asserting observable outcomes, not internal state
• Brittleness: Test breaking when implementation changes but behavior stays same