Skip to content

ADR-0007: Testing Strategy

Status

PROPOSED

Date: 2025-07-05

Changelog

Date Revision Description Author
2025-07-05 1.0 Initial version Loryth Team

Context

The Loryth project has evolved into a multiservice distributed system with complex interactions. A comprehensive testing strategy is essential to:

  • Ensure the correctness of individual components
  • Verify service interactions and integration points
  • Validate end-to-end user journeys
  • Maintain performance under load
  • Detect regressions early in the development process
  • Provide confidence during continuous deployment
  • Ensure contract compliance between services

Without a well-defined testing strategy, we risk quality issues, production failures, and reduced development velocity as the system grows more complex.

Decision

We will implement a multi-layered testing strategy with the following components:

  1. Unit Testing:

    • Comprehensive unit tests for all business logic
    • Mocking of external dependencies
    • Target coverage of 80%+ for core business logic
    • Fast execution to enable frequent runs during development

  2. Integration Testing:

    • Tests for integration between components within a service
    • Database integration tests with test-specific database instances
    • Message broker integration using test containers
    • API integration tests for each service

  3. Contract Testing:

    • Consumer-driven contract tests for service interfaces
    • Schema validation for events and messages
    • API contract tests to ensure backward compatibility
    • Automated contract verification in CI pipeline

  4. End-to-End Testing:

    • Critical user journey tests across multiple services
    • Realistic test environments with all services running
    • Data setup and teardown automation
    • Selective E2E testing focused on high-value flows

  5. Performance Testing:

    • Load testing for critical APIs and operations
    • Performance benchmarks for key user journeys
    • Stress testing to identify breaking points
    • Regular performance regression testing

  6. Security Testing:

    • Automated security scanning for vulnerabilities
    • Penetration testing for critical components
    • Authentication and authorization specific tests

  7. Test Automation and Infrastructure:

    • CI/CD pipeline integration for all test levels
    • Test environment automation with infrastructure as code
    • Containerized test environments for consistency
    • Comprehensive test reporting and analytics

Consequences

Positive

  • Quality assurance: Comprehensive coverage across different test types
  • Early detection: Issues found earlier in the development process
  • Deployment confidence: Reduced risk during deployments
  • Documentation: Tests serve as executable documentation
  • Design feedback: Test-driven development improves design
  • Regression prevention: Automated tests prevent regressions

Negative

  • Development overhead: Significant effort required to maintain comprehensive tests
  • Execution time: More extensive testing increases build times
  • Environment complexity: Test environments for distributed systems are complex
  • Flaky tests risk: E2E tests in particular can be prone to intermittent failures

Neutral

  • Skill requirements: Team needs expertise in various testing approaches
  • Tool diversity: Different test types require different tools and frameworks
  • Maintenance commitment: Ongoing commitment needed to maintain test suite

Alternatives Considered

  1. Manual Testing Focus

    • Description: Rely primarily on manual testing with minimal automation
    • Pros:
      • Less upfront development time
      • More exploratory testing possible
      • No automated test maintenance overhead
    • Cons:
      • Not scalable as a system grows
      • Slow feedback cycle
      • High regression risk
      • Inconsistent test coverage
    • Why not chosen: Not sustainable for a complex distributed system

  2. End-to-End Testing Only

    • Description: Focus on E2E tests with minimal unit/integration tests
    • Pros:
      • Tests complete user journeys
      • More realistic test scenarios
      • Less mocking required
    • Cons:
      • Slow execution time
      • Difficult to isolate failures
      • Limited coverage of edge cases
      • Brittle tests that break with any change
    • Why not chosen: Too slow and inefficient for comprehensive testing

  3. Third-Party Testing Service

    • Description: Outsource a majority of testing to specialized testing service
    • Pros:
      • Reduced internal testing overhead
      • Specialized expertise
      • Independent quality assessment
    • Cons:
      • Reduced developer ownership of quality
      • Knowledge gap between development and testing
      • Potential communication overhead
    • Why not chosen: Quality should be owned by the development team

References