Technical Design TDD-0003 — Notification Service¶
| Field | Value |
|---|---|
| ID | TDD-0003 |
| Title | Notification Service |
| Status | Draft |
| Owner | @souperk |
| Audience | contributors · integrators |
| Created | 2026-02-15 |
| Last Updated | 2026-02-15 |
| Reviewers | — |
| Approvers | — |
| Tracking | — |
| Related | — |
1. Summary¶
The platform lacks a centralized mechanism for delivering notifications to users. Individual services (IAM, Grocery, Wallet, Issues) each have events that should trigger user-facing messages (welcome emails, activity alerts, security notices), but no service owns the delivery pipeline.
This document describes the design for loryth-notifications, a new microservice responsible for accepting notification requests (via RabbitMQ events or REST API), rendering templates, dispatching through channel adapters (email, in-app inbox), and tracking delivery lifecycle with retries and idempotency.
Expected impact: any service can trigger a notification with a single RabbitMQ message, without owning email/SMTP logic or delivery reliability concerns.
2. Context and Problem Statement¶
Currently, no service sends transactional emails or in-app messages. Events like account creation, password reset, or issue assignment have no user-facing notification path. Building email delivery into each service would duplicate SMTP configuration, retry logic, and template management across the codebase.
A dedicated notification service centralizes:
- Template rendering and channel dispatch
- Delivery tracking with retries and exponential backoff
- Idempotency to prevent duplicate sends
- An in-app inbox for realtime message delivery
2.1 Goals¶
Functional Requirements¶
- Accept notification requests from any service via RabbitMQ or REST API
- Render file-based templates with per-notification context
- Deliver via email (SMTP) and in-app inbox channels
- Track delivery status (pending, processing, sent, failed, cancelled) with full audit trail
- Deduplicate notifications using idempotency keys
- Retry failed deliveries with exponential backoff
- Provide a paginated inbox with read/unread/archive state
- Stream realtime inbox updates via SSE
Non-functional Requirements¶
- Throughput: <100 notifications/hour (initial target)
- Reliability: no message loss — outbox pattern ensures persistence before acknowledgement
- Observability: structured logging with notification_uuid for end-to-end tracing
- Latency: email delivery within 30 seconds of notification creation under normal conditions
2.2 Out of Scope¶
- Email attachments (adds MIME complexity, deferred to v1.1+)
- Multi-tenancy (matches existing single-tenant platform)
- DB-based template management API (deferred to v1.1)
- SES/SendGrid provider adapters (deferred to v1.1, when provider is chosen)
- Bounce/complaint webhook handling (deferred to v1.1)
- User notification preferences / opt-out (deferred to v1.1)
- Rate limiting (not needed at <100/hour throughput)
- Push notifications (mobile)
2.3 Assumptions and Constraints¶
| # | Assumption | Rationale |
|---|---|---|
| A1 | Single shared PostgreSQL database with dedicated tables for notifications | Matches existing platform pattern (all services share one PG instance) |
| A2 | User email addresses resolved via RabbitMQ event payload or lightweight IAM internal API | IAM owns user identity; notification service should not duplicate user data |
| A3 | No attachments in v1 | Low throughput, simple templates — attachments add significant complexity |
| A4 | Secrets (SMTP credentials) stored in environment variables | Matches existing DbConfig.from_env() / RabbitMqConfig.from_env() pattern |
| A5 | Single outbox consumer (consumer_id=1) is sufficient | <100 notifications/hour does not require parallel consumers |
| A6 | Templates are file-based (packaged with the service) | Simpler for v1; DB-based templates can be added in v1.1 |
| A7 | Notification preferences default to opt-in for all channels | Users receive notifications unless they explicitly opt out |
2.4 Open Questions¶
- Should the notification service own a copy of user email addresses, or always resolve them at send-time via IAM? (Assumed: resolve at send-time via event payload)
- Will there be admin APIs to manage templates, channel configs, or retry policies? (Deferred to v1.1)
- Should bounce/complaint webhooks from future providers (SES, SendGrid) update notification status? (Deferred to v1.1)
3. Design Overview¶
The service has two runtime components:
- API (FastAPI): REST endpoints for creating/querying notifications and inbox management
- Worker (Typer CLI): outbox consumer that picks up pending notifications, renders templates, and dispatches to channel adapters
Both share the domain layer and connect to the same PostgreSQL database.
flowchart TB
subgraph "Producer Services"
IAM["loryth-iam"]
GROCERY["loryth-grocery"]
WALLET["loryth-wallet"]
ISSUES["loryth-issues"]
end
subgraph "RabbitMQ"
EX_IAM["iam.events (topic)"]
EX_GROCERY["grocery.events (topic)"]
EX_NOTIFY["notifications.inbound (topic)"]
Q_NOTIFY["notifications.process (queue)"]
Q_DLQ["notifications.dlq (queue)"]
end
subgraph "loryth-notifications"
subgraph "API (FastAPI)"
API_CREATE["POST /notification"]
API_LIST["GET /notification"]
API_STATUS["GET /notification/{id}"]
API_INBOX["GET /inbox/*"]
API_SSE["GET /inbox/stream (SSE)"]
end
subgraph "Worker (Typer CLI)"
CONSUMER["EventConsumer (outbox poller)"]
HANDLER["NotificationHandler"]
DISPATCHER["ChannelDispatcher"]
end
subgraph "Domain"
SVC["NotificationService"]
TMPL["TemplateRenderer"]
REPO["NotificationRepository"]
ATTEMPT_REPO["DeliveryAttemptRepository"]
INBOX_REPO["InboxMessageRepository"]
end
subgraph "Channel Adapters"
EMAIL_ADAPTER["SmtpEmailAdapter"]
INBOX_ADAPTER["InboxChannelAdapter"]
end
DB[(PostgreSQL)]
end
subgraph "External"
SMTP["SMTP Server (MailHog / Postfix / SES)"]
end
IAM -->|"AccountCreatedEvent"| EX_IAM
GROCERY -->|"domain events"| EX_GROCERY
EX_IAM --> Q_NOTIFY
EX_GROCERY --> Q_NOTIFY
EX_NOTIFY --> Q_NOTIFY
Q_NOTIFY --> HANDLER
Q_NOTIFY -.->|"reject after max retries"| Q_DLQ
API_CREATE --> SVC
API_LIST --> REPO
API_STATUS --> REPO
API_INBOX --> INBOX_REPO
HANDLER --> SVC
SVC --> TMPL
SVC --> REPO
SVC --> DISPATCHER
DISPATCHER --> EMAIL_ADAPTER
DISPATCHER --> INBOX_ADAPTER
EMAIL_ADAPTER --> SMTP
INBOX_ADAPTER --> INBOX_REPO
INBOX_ADAPTER --> API_SSE
SVC --> DB
CONSUMER --> REPO
ATTEMPT_REPO --> DB 3.1 Key Components¶
| Component | Purpose |
|---|---|
| NotificationService | Core domain service — creates notifications, orchestrates dispatch |
| TemplateRenderer | Renders file-based templates (subject, HTML body, text body) from template key + context |
| ChannelDispatcher | Routes notifications to the appropriate channel adapter (email, in-app) |
| SmtpEmailAdapter | Sends email via SMTP; implements EmailProvider protocol |
| InboxChannelAdapter | Creates InboxMessage records for in-app delivery |
| NotificationRepository | CRUD + query for Notification aggregate |
| DeliveryAttemptRepository | Audit log of each delivery attempt with status and error details |
| InboxMessageRepository | CRUD + query for InboxMessage (read/unread/archive state) |
| EventConsumer | Outbox poller that picks up pending notifications for processing |
| InboxEventBroker | In-process fan-out for SSE connections (dict of recipient_id → asyncio.Queue set) |
3.2 Architecture¶
Data Model¶
Notification (Aggregate Root)¶
class NotificationStatus(StrEnum):
PENDING = "pending"
PROCESSING = "processing"
SENT = "sent"
FAILED = "failed"
CANCELLED = "cancelled"
class NotificationChannel(StrEnum):
EMAIL = "email"
IN_APP = "in_app"
class NotificationCategory(StrEnum):
ACCOUNT = "account"
SECURITY = "security"
SYSTEM = "system"
ACTIVITY = "activity"
class Notification(BaseAggregate):
__tablename__ = "notification"
recipient_id: Mapped[UUID] = mapped_uuid("recipient_id")
recipient_email: Mapped[str | None] = mapped_column("recipient_email", String(255), nullable=True)
channel: Mapped[NotificationChannel] = mapped_enum("channel", NotificationChannel)
category: Mapped[NotificationCategory] = mapped_enum("category", NotificationCategory)
status: Mapped[NotificationStatus] = mapped_enum("status", NotificationStatus)
template_key: Mapped[str] = mapped_column("template_key", String(128), nullable=False)
template_context: Mapped[dict[str, Any]] = mapped_column("template_context", JSONB(), nullable=False)
idempotency_key: Mapped[str | None] = mapped_column("idempotency_key", String(255), nullable=True, unique=True)
source_event_type: Mapped[str | None] = mapped_column("source_event_type", String(255), nullable=True)
attempt_count: Mapped[int] = mapped_column("attempt_count", Integer, nullable=False, default=0)
max_attempts: Mapped[int] = mapped_column("max_attempts", Integer, nullable=False, default=3)
sent_at: Mapped[datetime | None] = mapped_column("sent_at", DateTime, nullable=True)
failed_at: Mapped[datetime | None] = mapped_column("failed_at", DateTime, nullable=True)
next_retry_at: Mapped[datetime | None] = mapped_column("next_retry_at", DateTime, nullable=True)
DeliveryAttempt (Child Entity)¶
class DeliveryStatus(StrEnum):
SUCCESS = "success"
FAILED = "failed"
class DeliveryAttempt(AuditMixin, BaseEntity):
__tablename__ = "delivery_attempt"
id: Mapped[int] = mapped_primary_key("id")
notification_id: Mapped[int] = mapped_column(
"notification_id", BigInteger, ForeignKey("notification.id", ondelete="CASCADE"), nullable=False
)
attempt_number: Mapped[int] = mapped_column("attempt_number", Integer, nullable=False)
status: Mapped[DeliveryStatus] = mapped_enum("status", DeliveryStatus)
provider_response: Mapped[str | None] = mapped_column("provider_response", Text, nullable=True)
error_message: Mapped[str | None] = mapped_column("error_message", Text, nullable=True)
duration_ms: Mapped[int | None] = mapped_column("duration_ms", Integer, nullable=True)
InboxMessage (Aggregate Root — for in-app channel)¶
class InboxMessage(BaseAggregate):
__tablename__ = "inbox_message"
notification_id: Mapped[int] = mapped_column(
"notification_id", BigInteger, ForeignKey("notification.id", ondelete="CASCADE"), nullable=False
)
recipient_id: Mapped[UUID] = mapped_uuid("recipient_id")
title: Mapped[str] = mapped_column("title", String(255), nullable=False)
body: Mapped[str] = mapped_column("body", Text, nullable=False)
category: Mapped[str] = mapped_column("category", String(64), nullable=False)
is_read: Mapped[bool] = mapped_column("is_read", Boolean, nullable=False, default=False)
read_at: Mapped[datetime | None] = mapped_column("read_at", DateTime, nullable=True)
archived_at: Mapped[datetime | None] = mapped_column("archived_at", DateTime, nullable=True)
State Machine¶
┌─────────────┐
│ PENDING │ ◄──── (created / retry scheduled)
└──────┬──────┘
│ worker picks up
▼
┌─────────────┐
│ PROCESSING │
└──────┬──────┘
┌────┴────┐
│ │
success failure
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│ SENT │ │ attempt < max │──► back to PENDING
└──────────┘ │ attempt ≥ max │──► FAILED
└──────────────┘
┌──────────┐
│ CANCELLED │ (admin action)
└──────────┘
Valid transitions:
- PENDING → PROCESSING (worker claims it)
- PROCESSING → SENT (delivery succeeded)
- PROCESSING → PENDING (delivery failed, retries remain)
- PROCESSING → FAILED (delivery failed, max retries exhausted)
- PENDING → CANCELLED (admin cancellation)
3.3 Interface Impact (Before → After)¶
Not applicable — new service addition.
4. Implementation Details¶
4.1 Core Contracts and APIs¶
REST API¶
| Method | Path | Description | Response |
|---|---|---|---|
| POST | /notification | Create a notification (idempotent with idempotency_key) | SuccessResponse[NotificationModel] |
| GET | /notification | List notifications (paginated, filterable by recipient/status) | SuccessResponse[Page[NotificationModel]] |
| GET | /notification/{id} | Get notification detail with delivery attempts | SuccessResponse[NotificationDetailModel] |
| GET | /inbox | List inbox messages (paginated, filterable by read/unread) | SuccessResponse[Page[InboxMessageModel]] |
| GET | /inbox/unread-count | Get unread message count | SuccessResponse[UnreadCountModel] |
| GET | /inbox/{id} | Get inbox message detail | SuccessResponse[InboxMessageModel] |
| PUT | /inbox/{id}/read | Mark message as read | SuccessResponse[InboxMessageModel] |
| PUT | /inbox/read | Bulk mark as read | 204 No Content |
| DELETE | /inbox/{id} | Archive (soft-delete) message | 204 No Content |
| GET | /inbox/stream | SSE stream for realtime updates | text/event-stream |
| GET | /health | Health check | {"status": "healthy"} |
RabbitMQ Message Schemas¶
Inbound: SendNotificationTask (from any service)
{
"message_name": "loryth.notifications.domain.notification.events.SendNotificationTask",
"message_type": "task",
"recipient_id": "550e8400-e29b-41d4-a716-446655440000",
"recipient_email": "kostas@example.com",
"channel": "email",
"category": "account",
"template_key": "welcome",
"template_context": {
"username": "kostas",
"app_name": "Loryth"
},
"idempotency_key": "iam.account-created.550e8400-e29b-41d4-a716-446655440000"
}
Outbound: NotificationSentEvent
{
"message_name": "loryth.notifications.domain.notification.events.NotificationSentEvent",
"message_type": "event",
"notification_uuid": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
"recipient_id": "550e8400-e29b-41d4-a716-446655440000",
"channel": "email",
"sent_at": "2026-02-09T14:30:00Z"
}
Outbound: NotificationFailedEvent
{
"message_name": "loryth.notifications.domain.notification.events.NotificationFailedEvent",
"message_type": "event",
"notification_uuid": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
"recipient_id": "550e8400-e29b-41d4-a716-446655440000",
"reason": "Max retries exceeded: Connection refused",
"total_attempts": 3
}
Message Definitions (Python)¶
class NotificationCreatedEvent(BaseEvent):
notification_uuid: UUID
recipient_id: UUID
channel: NotificationChannel
class NotificationSentEvent(BaseEvent):
notification_uuid: UUID
recipient_id: UUID
channel: NotificationChannel
class NotificationFailedEvent(BaseEvent):
notification_uuid: UUID
recipient_id: UUID
reason: str
class SendNotificationTask(BaseTask):
__loryth_task_name__ = "loryth.notifications.send-notification"
recipient_id: UUID
recipient_email: str | None = None
channel: str # "email" | "in_app"
category: str # "account" | "security" | "system" | "activity"
template_key: str
template_context: dict[str, Any]
idempotency_key: str | None = None
4.2 Component Architecture¶
Email Channel¶
Provider Adapter Interface¶
@dataclass(frozen=True)
class EmailMessage:
to: str
subject: str
body_html: str
body_text: str | None = None
from_address: str | None = None # falls back to config default
reply_to: str | None = None
@dataclass(frozen=True)
class EmailResult:
success: bool
provider_message_id: str | None = None
error: str | None = None
class EmailProvider(Protocol):
def send(self, message: EmailMessage) -> EmailResult: ...
SMTP Configuration¶
@dataclass
class SmtpConfig:
host: str = "localhost"
port: int = 1025
username: str = ""
password: str = ""
use_tls: bool = False
from_address: str = "noreply@loryth.app"
@staticmethod
def from_env() -> "SmtpConfig":
return SmtpConfig(
host=os.environ.get("SMTP_HOST", "localhost"),
port=int(os.environ.get("SMTP_PORT", "1025")),
username=os.environ.get("SMTP_USERNAME", ""),
password=os.environ.get("SMTP_PASSWORD", ""),
use_tls=os.environ.get("SMTP_USE_TLS", "false").lower() == "true",
from_address=os.environ.get("SMTP_FROM_ADDRESS", "noreply@loryth.app"),
)
MailHog for Development¶
Add to docker-compose.override.yml:
mailhog:
image: mailhog/mailhog:v1.0.1
ports:
- "1025:1025" # SMTP
- "8025:8025" # Web UI
Service environment:
loryth-notifications-api:
environment:
SMTP_HOST: mailhog
SMTP_PORT: 1025
SMTP_USE_TLS: "false"
SMTP_FROM_ADDRESS: "noreply@loryth.local"
Template System¶
File-based templates stored in the service package:
loryth/notifications/templates/
welcome/
subject.txt → "Welcome to $app_name, $username!"
body.html → <html>...</html>
body.txt → Plain text version
password_reset/
subject.txt
body.html
body.txt
email_verification/
subject.txt
body.html
body.txt
@dataclass(frozen=True)
class RenderedMessage:
subject: str
body_html: str
body_text: str | None
@service()
class TemplateRenderer:
def render(self, template_key: str, context: dict[str, Any]) -> RenderedMessage:
base_path = importlib.resources.files("loryth.notifications.templates") / template_key
subject = Template((base_path / "subject.txt").read_text()).safe_substitute(context)
body_html = Template((base_path / "body.html").read_text()).safe_substitute(context)
body_txt_path = base_path / "body.txt"
body_text = Template(body_txt_path.read_text()).safe_substitute(context) if body_txt_path.is_file() else None
return RenderedMessage(subject=subject, body_html=body_html, body_text=body_text)
Uses Python stdlib string.Template ($variable / ${variable} syntax) — intentionally simple, no external dependency. Upgrade to Jinja2 if needed later.
In-App Inbox Channel¶
- In-app is a delivery channel — the dispatch pipeline treats it like email
inbox_messagetable stores the materialized, user-facing message with read/unread/archive state- Read/unread tracking via
is_readboolean +read_attimestamp - Realtime delivery via SSE (Server-Sent Events) — simpler than WebSockets, built-in reconnection
SSE Design¶
- In-process
InboxEventBrokermaintainsdict[UUID, set[asyncio.Queue]]for fan-out to multiple tabs/devices - 30-second heartbeat to keep connections alive
Last-Event-IDsupport for reconnection catch-up- SSE event types:
new_message,message_read,messages_read_bulk,message_archived,unread_count
Client Sync Strategy¶
- Initial load:
GET /inbox+GET /inbox/unread-countin parallel - Realtime: open SSE connection immediately after initial load
- Reconnection: EventSource auto-reconnects; server replays missed events via
Last-Event-ID - Visibility change: fetch authoritative unread count when tab becomes visible
- Badge count: server pushes
unread_countevent every 60s as drift correction
4.3 Integration Points¶
Idempotency Strategy¶
Database-level deduplication via unique constraint on idempotency_key:
@transactional
def create(self, task: SendNotificationTask) -> NotificationDto:
if task.idempotency_key:
existing = self._repository.find_by_idempotency_key(task.idempotency_key)
if existing is not None:
logger.info(f"Duplicate notification suppressed: {task.idempotency_key}")
return NotificationDto.map(existing)
notification = Notification(
recipient_id=task.recipient_id,
recipient_email=task.recipient_email,
channel=NotificationChannel(task.channel),
category=NotificationCategory(task.category),
template_key=task.template_key,
template_context=task.template_context,
idempotency_key=task.idempotency_key,
source_event_type=task.message_name,
)
self._repository.insert(notification)
return NotificationDto.map(notification)
Idempotency key convention: {service}.{event_type}.{entity_uuid} — e.g., iam.account-created.550e8400-...
Processing Algorithm¶
- Worker receives message from RabbitMQ (or outbox consumer picks up task)
- Deserialize to
SendNotificationTask - Check
idempotency_key→ if duplicate, ACK and skip - Create
Notificationentity (status=PENDING) - Persist to database (within
@transactional) - ACK the RabbitMQ message
- Delivery loop (same or next worker cycle): a. Query:
SELECT * FROM notification WHERE status='pending' AND (next_retry_at IS NULL OR next_retry_at <= NOW())b. For each notification:- Set status = PROCESSING
- Render template (
template_key+template_context) - Dispatch to channel adapter (email →
SmtpEmailAdapter, in_app →InboxChannelAdapter) - On SUCCESS: create
DeliveryAttempt(SUCCESS),notification.record_attempt_success() - On FAILURE: create
DeliveryAttempt(FAILED, error),notification.record_attempt_failure() - Flush changes (within transaction)
Cross-Service Integration (Phase 1.2)¶
- IAM worker publishes
SendNotificationTaskonAccountCreatedEvent - RabbitMQ bindings:
iam.events→notifications.processqueue - Welcome email and email verification templates
4.4 Error Handling and Edge Cases¶
Retry/Backoff + DLQ Policy¶
| Parameter | Value |
|---|---|
| Max attempts | 3 (configurable per notification) |
| Backoff | Exponential: min(2^attempt * 2s, 300s) → 4s, 8s, 16s... capped at 5min |
| Jitter | ±25% random jitter to avoid thundering herd |
| DLQ | notifications.dlq queue — notifications that exceed max attempts |
| DLQ retention | Manual review; admin API to retry or dismiss (v1.1) |
Failure Modes¶
| Failure Mode | Handling |
|---|---|
| RabbitMQ down | Outbox pattern ensures events are persisted in DB. Worker retries RabbitMQ connection with backoff. No messages lost. |
| Database down | Worker stops processing. FastAPI health check fails. Kubernetes restarts pod. Messages remain in RabbitMQ queue (unacked). |
| SMTP provider down | Delivery attempt fails → recorded in delivery_attempt → notification stays PENDING with next_retry_at set → retried on next cycle. |
| Worker crash mid-processing | Notification is in PROCESSING state. On worker restart, a recovery sweep resets stale PROCESSING notifications (older than 5 minutes) back to PENDING. |
| Partial failure (DB commit ok, email sent but ACK lost) | Idempotency key prevents duplicate notification creation. The delivery_attempt table provides audit trail. Re-processing the same event is safe. |
Recovery sweep (runs on worker startup):
@transactional
def recover_stale_processing(self, stale_threshold_minutes: int = 5) -> int:
cutoff = datetime.now(timezone.utc) - timedelta(minutes=stale_threshold_minutes)
stale = self._repository.find_stale_processing(cutoff)
for notification in stale:
notification.status = NotificationStatus.PENDING
logger.warning(f"Recovered stale notification: {notification.uuid}")
return len(stale)
Gotchas¶
- Duplicate sends: The biggest risk. Mitigated by idempotency keys + unique constraint, but operators must ensure all producers include meaningful keys.
- Partial failure: Notification created in DB but email fails → safe because retry picks it up. Email sent but DB commit fails → rare with local PG, but the idempotency key prevents re-creation on retry.
- SMTP timeouts: Default 30s timeout on
smtplib.SMTP. If the provider is slow, this blocks the worker thread. At <100/hour throughput this is acceptable; for higher throughput, use async SMTP (aiosmtplib). - Template injection:
string.Template.safe_substitutedoes not execute code, but template context should still be sanitized at the API boundary (Pydantic handles this). - MailHog in production: Ensure
SMTP_HOSTenvironment variable points to a real SMTP server in production. The dev defaults (localhost:1025) will silently fail if not overridden. - Provider rate limits: SES has a default 1 email/second for sandbox accounts. The SMTP adapter should respect rate limits — at <100/hour this is not a concern, but add rate limiting before scaling up.
4.5 Observability¶
Structured Logging¶
All log lines include notification context:
logger.info(
"Notification created",
extra={
"notification_uuid": str(notification.uuid),
"recipient_id": str(notification.recipient_id),
"channel": notification.channel,
"template_key": notification.template_key,
"idempotency_key": notification.idempotency_key,
},
)
Key log events:
| Event | Level | When |
|---|---|---|
notification.created | INFO | Notification persisted |
notification.duplicate_suppressed | INFO | Idempotency key match |
notification.delivery.started | INFO | Delivery attempt begins |
notification.delivery.success | INFO | Email sent successfully |
notification.delivery.failed | WARNING | Delivery attempt failed (will retry) |
notification.failed | ERROR | Max retries exhausted |
notification.recovery.stale | WARNING | Stale PROCESSING notification recovered |
worker.batch.processed | DEBUG | Outbox batch completed |
smtp.connection.error | ERROR | SMTP server unreachable |
Metrics (Prometheus-style names)¶
| Metric | Type | Description |
|---|---|---|
notifications_created_total | Counter | Total notifications created, labeled by channel, category |
notifications_sent_total | Counter | Successfully sent, labeled by channel |
notifications_failed_total | Counter | Failed after max retries, labeled by channel, reason |
notifications_delivery_attempts_total | Counter | All delivery attempts, labeled by status |
notifications_delivery_duration_seconds | Histogram | Time to deliver (per attempt) |
notifications_pending_count | Gauge | Notifications in PENDING state |
notifications_processing_count | Gauge | Notifications in PROCESSING state |
outbox_batch_size | Histogram | Events per outbox batch |
outbox_processing_duration_seconds | Histogram | Time per outbox batch |
inbox_unread_count | Gauge | Per-user unread count (sampled) |
sse_connections_active | Gauge | Active SSE connections |
Tracing¶
RequestIdMiddlewarealready provides per-request UUID- Add
notification_uuidto all log/metric contexts for end-to-end tracing - Answer "what happened to notification X?" via:
GET /notification/{uuid}→ shows status, all delivery attempts with timestamps and errors
Dashboard Suggestions¶
- Notification Throughput:
rate(notifications_created_total[5m])by channel - Delivery Success Rate:
notifications_sent_total / notifications_created_total - Delivery Latency: p50/p95/p99 of
notifications_delivery_duration_seconds - Failure Rate:
rate(notifications_failed_total[5m])— alert if > 0 sustained for 5 min - Pending Queue Depth:
notifications_pending_count— alert if > 50 sustained for 10 min - SMTP Errors:
rate(notifications_delivery_attempts_total{status="failed"}[5m])— alert if > 5/min
5. Semantics¶
Notification Lifecycle¶
- A notification is immutable once created — only status, attempt count, and timestamps change
- Template rendering happens at delivery time, not creation time — this allows template fixes to apply to pending retries
- Idempotency keys are globally unique and permanent — they are never recycled or expired
- In-app messages are independent records linked to their parent notification — archiving an inbox message does not affect the notification
Domain Events¶
NotificationCreatedEventis recorded on entity creation (for outbox dispatch)NotificationSentEventis recorded on successful delivery (published tonotifications.eventsexchange)NotificationFailedEventis recorded when max retries are exhausted
7. Test Matrix¶
7.1 Unit Coverage (8 tests)¶
| # | Test | What it validates |
|---|---|---|
| 1 | test_notification_initial_status_is_pending | New notification starts as PENDING |
| 2 | test_notification_mark_sent_transitions | PROCESSING → SENT with timestamp |
| 3 | test_notification_mark_failed_transitions | PROCESSING → FAILED after max retries |
| 4 | test_notification_retry_stays_pending | Failed attempt with retries remaining → PENDING with next_retry_at |
| 5 | test_template_renderer_renders_welcome | Welcome template produces correct subject/body with context vars |
| 6 | test_idempotency_key_prevents_duplicate | Same idempotency key returns existing, no insert |
| 7 | test_retry_backoff_calculation | Exponential backoff: 4s, 8s, 16s, capped at 300s |
| 8 | test_delivery_attempt_records_error | Failed attempt captures error message |
7.2 Integration Coverage (10 tests)¶
| # | Test | What it validates |
|---|---|---|
| 9 | test_notification_repo_insert_and_find | Persist and retrieve by ID from real PG |
| 10 | test_notification_repo_find_by_recipient | Query by recipient UUID returns correct set |
| 11 | test_idempotency_key_unique_constraint | DB rejects duplicate idempotency_key |
| 12 | test_delivery_attempt_repo_tracks_attempts | Multiple attempts persisted and ordered |
| 13 | test_create_notification_api_returns_200 | POST /notification creates and returns notification |
| 14 | test_list_notifications_api_filters | GET /notification filters by recipient |
| 15 | test_duplicate_idempotency_key_api | POST with same key returns same notification |
| 16 | test_smtp_adapter_sends_to_mailhog | Real SMTP send, verify via MailHog API |
| 17 | test_smtp_adapter_handles_connection_failure | Unreachable SMTP raises EmailDeliveryError |
| 18 | test_mark_inbox_read_api | PUT /inbox/{id}/read updates is_read and read_at |
7.3 E2E Coverage (4 tests)¶
| # | Test | What it validates |
|---|---|---|
| 19 | test_full_notification_email_delivery | API create → deliver → email in MailHog, status=SENT |
| 20 | test_worker_consumes_event_sends_email | Outbox event → worker → notification → MailHog |
| 21 | test_failed_delivery_retried_succeeds | Transient failure → retry → eventual success |
| 22 | test_notification_sent_event_published | Successful delivery → NotificationSentEvent on RabbitMQ |
Test Infrastructure¶
- PostgreSQL:
testcontainers[postgres](session-scoped), migrations run once - MailHog: custom
MailHogContainer(session-scoped), API polling for verification - RabbitMQ:
testcontainers[rabbitmq](session-scoped), per-test exclusive queues - Cleanup: function-scoped
cleanup_dbdeletes all rows;_clear_mailhogdeletes messages
CI Pipeline¶
Stage 1 (parallel): ruff format --check . | ruff check . | mypy .
Stage 2: pytest tests/unit/ -v
Stage 3 (Docker): pytest tests/integration/ tests/e2e/ -v --durations=0
8. Acceptance Criteria¶
- All SQL migrations apply cleanly on fresh database
-
ruff check .passes with zero errors -
ruff format .produces no changes -
mypy .passes with zero errors - All 22 test cases pass (
pytest -v) - Service starts cleanly with
fastapi run - Worker starts cleanly with
python -m typer loryth.notifications.worker.cli run - POST
/notificationcreates notification and triggers email delivery - Email arrives in MailHog with correct subject/body/recipient
- Duplicate idempotency key returns existing notification (no re-send)
- Failed delivery is retried (verify via
delivery_attemptrecords) - GET
/notification/{id}shows full lifecycle (status, attempts, timestamps) - Health check returns 200
- Structured logs include
notification_uuidfor traceability
9. Rollout and Migration¶
9.1 Phased Plan¶
Phase 0: Scaffolding (Small)
- Create
loryth-notificationsservice skeleton (pyproject.toml,config/,domain/,api/,worker/) - Set up DI container, infrastructure config (
InfrastructureConfig) - Add SQL migrations (eventlog/outbox + notification + delivery_attempt)
- Add Dockerfile, docker-compose entries, MailHog container
- Verify: service starts, health check passes, migrations run
Phase 1: Core Notification Pipeline — v1.0 (Medium)
- Notification entity, repository, DTO, service
- DeliveryAttempt entity and repository
- Template renderer (file-based,
string.Template) - Email provider interface + SMTP adapter
- Channel dispatcher (routes notification to email adapter)
- Worker: outbox consumer + notification handler + delivery loop
- REST API: POST
/notification, GET/notification, GET/notification/{id} - Idempotency (unique constraint + service-level check)
- Retry logic with exponential backoff
- Recovery sweep for stale PROCESSING notifications
- Structured logging throughout
- Tests: all 8 unit + 10 integration + 4 E2E tests
- Verify: end-to-end flow from API/event to email in MailHog
Phase 1.1: In-App Message Center (Medium)
- InboxMessage entity, repository, service
- Inbox channel adapter (creates InboxMessage on dispatch)
- REST API: GET/PUT/DELETE
/inboxendpoints - SSE endpoint with InboxEventBroker
- Additional tests for inbox functionality
- Verify: notifications appear in inbox, realtime SSE works
Phase 1.2: Cross-Service Integration (Small)
- IAM worker: publish
SendNotificationTaskonAccountCreatedEvent - Wire RabbitMQ bindings:
iam.events→notifications.processqueue - Add welcome email template
- Add email verification template
- E2E test: create account in IAM → notification → email arrives
Phase 2: Production Hardening — v1.1 (Medium, future)
- DLQ monitoring + admin retry API
- Prometheus metrics integration
- SES/SendGrid adapter (when provider is chosen)
- Bounce/complaint webhook handling
- DB-based template management API
- User notification preferences (opt-out per channel/category)
- Rate limiting
9.2 Migration Strategy¶
SQL Migrations¶
Three migration files applied via loryth-cli upgrade:
v00-setup.sql— Standard eventlog/outbox infrastructure (reused from other services)v01-create-notification.sql—notificationanddelivery_attempttables with enums and indexesv02-create-inbox-message.sql—inbox_messagetable with indexes
All tables use the shared loryth database (not iam).
10. Risks and Mitigations¶
| Risk | Impact | Likelihood | Mitigation |
|---|---|---|---|
| Duplicate email sends | High | Low | Idempotency keys + DB unique constraint; all producers must include meaningful keys |
| SMTP provider downtime | Medium | Medium | Exponential retry with backoff; delivery_attempt audit trail; DLQ for exhausted retries |
| Worker crash loses in-flight notifications | Medium | Low | Recovery sweep on startup resets stale PROCESSING → PENDING after 5 min |
| Template injection / XSS in emails | High | Low | string.Template.safe_substitute does not execute code; Pydantic validates context at API boundary |
| MailHog accidentally used in production | Medium | Low | Explicit SMTP_HOST env var required; health check could validate SMTP connectivity |
| SMTP timeout blocking worker thread | Low | Medium | 30s timeout acceptable at <100/hour; upgrade to aiosmtplib if throughput increases |
11. Progress¶
- Phase 0: Scaffolding
- Phase 1: Core Notification Pipeline
- Phase 1.1: In-App Message Center
- Phase 1.2: Cross-Service Integration
- Phase 2: Production Hardening
12. Changelog¶
| Date | Change |
|---|---|
| 2026-02-15 | Initial version |
13. Glossary¶
| Term | Definition |
|---|---|
| Idempotency key | A unique string that prevents the same notification from being created twice. Convention: {service}.{event_type}.{entity_uuid} |
| Channel | A delivery mechanism for notifications (email, in-app inbox) |
| Channel adapter | Implementation that delivers a rendered message through a specific channel |
| Outbox pattern | Events are persisted to a DB outbox table within the same transaction as the business operation, then consumed asynchronously — guarantees at-least-once delivery |
| DLQ (Dead Letter Queue) | RabbitMQ queue where notifications that exceed max retry attempts are routed for manual review |
| SSE (Server-Sent Events) | HTTP-based unidirectional streaming protocol used for realtime inbox updates |
| Recovery sweep | Worker startup routine that resets stale PROCESSING notifications back to PENDING |