Technical Design — Centralized Metrics for Reflection¶

Status: Completed Owner: Reflection maintainers Last updated: 2025-09-27

Problem

• Ad-hoc metrics hooks and logger patterns exist; they are not standardized across components, hampering observability and long-term tracking.

Goals

• Provide a single ReflectionMetrics interface used across resolvers, scanners, invokers, and type-hints providers.
• Make it optional (no-op by default), dependency-free, and safe to call (never raises).

Non-goals

• Introducing a hard dependency on a metrics backend.
• Full tracing; we stick to lightweight counters and structured events.

Design

• Introduce interface:
  • class ReflectionMetrics(Protocol): def emit(self, event: str, payload: Mapping[str, Any]) -> None: ...
• Provide default implementation:
  • NullMetrics: does nothing.
• Provide optional adapters:
  • LoggingMetrics: emits via logging at DEBUG level.
  • HookMetrics: wraps a user-supplied callable for custom sinks.

Implementation summary

• A centralized ReflectionMetrics sink is injectable via the builder and propagated to import resolution, scanning, invocation, and type-hints providers.
• Providers use the sink to emit structured events around successes, retries, and failures; emission is best-effort and never raises.
• Default behavior is dependency-free (NullMetrics); optional adapters (LoggingMetrics, HookMetrics) are available for immediate observability or integration with custom sinks.

Event names (current)

• Core (as originally proposed)
  • type_hints.resolved {subject, status: ok|partial}
  • type_hints.first_fail {subject, priority: list[str]}
  • type_hints.retry_fail {subject, error}
  • type_hints.strict_error {subject, slot, token, attempted}
  • import.resolve {module, success}
  • scan.module {module, classes, functions}
  • invoke.call {qualname, success}
  • invoke.construct {qualname, success}
• Additional helpful signals currently emitted by the type-hints provider
  • type_hints.tokens {dotted: list[str], bare: list[str], priority: list[str]}
  • type_hints.import_attempt {module, symbol, injected: bool}
  • type_hints.sibling_attempt {module, symbol, injected: bool}
  • type_hints.sibling_reuse {module, symbol, injected: bool}
  • type_hints.unresolved {subject, slots: list[str], tokens: list[str], attempted: list[str]}
  • get_type_hints_retry_error {subject, error}

Thread-safety and performance

• Calls are fire-and-forget; do not block hot paths.
• Exceptions raised by adapters are swallowed in emit paths.

Wiring

• ReflectionContextBuilder accepts a metrics sink via with_metrics and propagates it to constructed providers.
• Providers default to NullMetrics when no sink is given.

Testing strategy

• Adapters
  • Ensure LoggingMetrics and HookMetrics never raise from emit even if the logger/hook fails.
• Providers (using a stub sink to capture events)
  • Type-hints: assert events for successful resolution, retry paths, partial resolution (tolerant mode), and strict errors (with slot/token/attempted payload).
  • Import resolution: assert import.resolve for both success and failure in module/package resolution and dotted-path try-resolve.
  • Invocation: assert invoke.call and invoke.construct events for success and failure, including signature mismatch and runtime exceptions.
  • Scanning: assert scan.module (and package-level, if emitted) with payload including module identity and member counts.

Migration

• Replace any ad-hoc per-provider hooks with the centralized metrics sink supplied through the builder.
• Keep constructors backward-friendly during transition by internally mapping deprecated hook parameters to the centralized sink, where present.

Acceptance criteria (status)

• Met:
  • A single metrics sink is injectable across reflection components.
  • Providers emit structured metrics via ReflectionMetrics; adapter exceptions are swallowed.
  • Centralized wiring via the builder is implemented; default behavior is no-op.
• Outstanding follow-ups to close the loop:
  • Documentation: Quickstart with with_metrics usage and a concise event reference (see “Remaining work”).
  • Testing: Ensure systematic event assertions exist across providers and adapters (see “Remaining work”).
  • Scanning metrics: Verify emission and payload for scan.module (and package-level if applicable); add tests and/or implementation detail as needed.

Remaining work

• Documentation
  • Add a Quickstart snippet showing how to inject a metrics sink with the builder and observe core events.
  • Produce an Event Reference page/table enumerating event names, when they emit, payload fields, and examples ( include both core and additional type-hints signals).
  • Write a brief migration note demonstrating replacement of ad-hoc hooks with with_metrics.
• Tests
  • Adapters: tests that emit never raises even when the underlying hook/logger fails.
  • Type-hints provider: tests for ok, retry, partial (tolerant), and strict error paths with payload assertions ( subject/slots/tokens/attempted).
  • Import resolver: success and failure branches for module and package resolution, and dotted-path try_resolve behavior.
  • Invoker: success/failure for call and construct, including signature mismatch and runtime error propagation while still emitting success=False.
  • Scanner: confirm scan.module (and package-level if used) metrics are emitted with stable payload shape; add tests or wire emission as necessary.
• Event schema consistency
  • Align and document payload keys across events (e.g., subject vs. qualname vs. module) and note any justified differences.
  • Keep a minimal shared reference for event names to avoid drift and ensure consistency in new code.