📘 Technical Design 018 — DI Benchmarking Suite¶

🧭 1. Summary¶

TD-018 proposes a structured performance benchmarking suite for the dependency injection (DI) container so we can measure the Phase 3 injection pipeline introduced in TD-011. The existing ad-hoc script lacks reproducibility, coverage of injector-heavy workloads, and trend tracking. This design introduces pytest-benchmark driven scenarios that cover cold bootstrap, warm resolution, and proxy-heavy method interception to give us reliable baselines before further optimization.

At a glance:

• Adopt pytest-benchmark as the canonical harness for DI performance testing.
• Capture scenarios that combine constructor, property, and proxy-based injectors across realistic service graphs.
• Produce repeatable, diffable results with documented budgets and follow-up guidance.

📚 2. Context and Problem Statement¶

The repository currently ships a standalone tests/performance/di_benchmark.py script that compares direct container wiring with the builder API. While useful for smoke checks, it omits core TD-011 behaviors (unified injector pipeline, proxy wrapping, optional dependency handling) and produces console output that cannot be tracked over time. TD-011’s remaining work plan highlights performance benchmarking as the last acceptance gap, so we need a reproducible suite that exercises the new injection surface.

2.1 Goals¶

• Deliver a pytest-driven benchmarking harness that runs locally with a single command and records stable metrics for TD-011 injector workflows.
• Model representative DI workloads: cold bootstrap of 50+ service graphs, warm singleton/transient resolution, and proxy-heavy interceptors.
• Store benchmark expectations alongside documentation so regressions surface during development and can be investigated quickly.

2.2 Non-Goals (Out of Scope)¶

• Integrating benchmarks into CI or release gates (manual/local runs only for now).
• Building automated optimizations; this TD focuses on measurement, not tuning.
• Benchmarking non-Python DI implementations or cross-process orchestration.

2.3 Assumptions and Constraints¶

• Developers have access to the managed .venv and can install pytest-benchmark via uv.
• Benchmarks must run within a few minutes on commodity developer hardware to encourage regular usage.
• Results should remain comparable across runs, so scenarios will minimize nondeterminism and external I/O.

🧩 3. Design Overview¶

We will refactor the current performance directory into a pytest module backed by pytest-benchmark. Each benchmark scenario will live in tests/performance/di/ and use shared helpers to assemble service graphs and injector mixes. Running pytest tests/performance/di --benchmark-only yields standardized reports that can be diffed against previous runs and stored locally for regression tracking.

3.1 Facades at a Glance¶

3.2 Removed/Changed Interfaces (Before → After)¶

3.3 Removal/Deprecation Policy¶

Alpha stage: replace the standalone script in the same change; no deprecation window required. Documentation and examples will be updated alongside the transition.

📐 4. Implementation Details¶

4.1 Project Layout¶

• Create tests/performance/di/__init__.py and tests/performance/di/test_benchmarks.py for pytest collection.
• Move helper types into tests/performance/di/scenarios.py to construct service graphs and injectors.
• Add tests/performance/di/benchmark_docs.md summarizing run commands and interpretation tips.

4.2 Benchmark Scenarios¶

• Cold bootstrap: Run container.bootstrap() on graphs with 10, 50, and 100 services, including property injectors and proxy-based custom injectors.
• Warm resolution: Measure repeated resolve() calls for singleton and transient services across the same graphs to quantify injector overhead.
• Proxy-heavy workflows: Apply reference injectors (transactional, caching) to method-rich services and benchmark invocation throughput.
• Deep optional chains: Include services with optional dependencies to measure reflection and annotation processing cost.

Each scenario will use parametrized pytest functions with benchmark fixture to record timings and optionally compare against stored baselines.

4.3 Metrics and Reporting¶

• Rely on pytest-benchmark’s histogram/statistics output; optionally persist JSON files under tests/performance/results/ (ignored by git) for trend tracking.
• Document target budgets (e.g., cold bootstrap of 50-service graph should stay under X ms) once baseline data is collected.

4.4 Tooling Integration¶

• Add pytest-benchmark to dev dependencies via pyproject.toml and uv. Update Makefile with make bench convenience target running pytest --benchmark-only tests/performance/di.
• Provide guidance for using --benchmark-compare against stored baseline directories.

⚙️ 5. Semantics¶

• Benchmarks always run with injectors enabled (Autowired + proxy-based) to reflect TD-011 workloads.
• Warm resolution scenarios discard first-run results to avoid bootstrap noise and focus on steady-state costs.
• Proxy scenarios include both method interception and property injection to measure combined overhead.
• All benchmarks enforce deterministic seeding for random data and avoid I/O beyond in-memory operations.

🧪 6. Examples (concise)¶

PYTHONPATH=src .venv/bin/python -m pytest tests/performance/di --benchmark-only

def test_cold_bootstrap_50_service_graph(benchmark: BenchmarkFixture) -> None:
    container = build_graph(size=50, include_proxies=True)
    benchmark(lambda: container.bootstrap())

✅ 7. Acceptance Criteria¶

• Pytest-based benchmarking suite covers cold bootstrap, warm resolution, and proxy-heavy scenarios.
• pytest-benchmark dependency and make bench entry point documented in README or benchmark_docs.md.
• Benchmark results can be compared across runs using --benchmark-compare with stored baselines.
• Existing standalone script is removed or updated to delegate to the pytest suite.

🚀 8. Rollout and Migration¶

• Phase 1: Introduce pytest suite alongside helper modules; gather initial baseline numbers and document runbook.
• Phase 2: Replace legacy script and update developer guides to reference the new command.
• Phase 3: Optionally add regression thresholds or alerts once baseline confidence is established.

⚠️ 9. Risks and Mitigations¶

• Noisy measurements: Mitigate by repeating runs, pinning CPU frequency guidance, and documenting best practices.
• High maintenance overhead: Keep scenarios focused on representative workloads; review annually to retire stale cases.
• Dependency drift: Track pytest-benchmark updates through uv and pin versions as needed.

📈 10. Progress¶

Legend: [x] = complete, [ ] = pending

• Add pytest suite and shared scenario builders.
• Integrate pytest-benchmark dependency and tooling support.
• Document run instructions and baseline storage.

📝 11. Changelog¶

• 2025-10-06: Initial draft capturing benchmarking suite design.

📖 12. Glossary¶

• Benchmark Scenario: Defined combination of service graph, injectors, and lifecycle used in performance tests.
• Cold Bootstrap: First-time container initialization including dependency analysis and injection.
• Warm Resolution: Repeated service resolution after initial bootstrap.
• Proxy-heavy Workflow: Scenario where method interception injectors wrap multiple methods, stressing proxy overhead.