Walkthrough: Notification System (push, SMS, email) · System Design

Send notifications over iOS push (APNs), Android push (FCM), SMS (Twilio/SNS), email (SES/SendGrid), in-app.

1. Requirements#

Functional

Send notifications over iOS push (APNs), Android push (FCM), SMS (Twilio/SNS), email (SES/SendGrid), in-app.
Triggered by internal services (order shipped, new message) and by scheduled campaigns.
User preferences and opt-outs per channel; templates with localization; dedupe; rate limit per user (no spamming).

Non-functional

Soft real-time: transactional notifications delivered within seconds; campaigns can take minutes.
At-least-once delivery with dedupe (never lose a "your code is 123456", tolerate a rare duplicate).
Scales to tens of millions/day; third-party outages must not back up the whole system; full delivery tracking.

2. Estimates#

Volume: 10M push + 1M SMS + 5M email per day = 16M/day ÷ 86,400 ≈ 185/s average; campaign bursts of 1M in 10 min ≈ 1,700/s; design for ~5k/s peak.
Payload: ~1 KB per notification event → 16 GB/day of events; keep 30 days ≈ 500 GB (cheap).
Device tokens: 50M users × 1.5 devices × ~200 B ≈ 15 GB (Postgres/DynamoDB).
Third-party limits: APNs/FCM effectively thousands/s per connection pool; Twilio per-number throughput ~1 msg/s long-code, higher via short code/toll-free → SMS needs its own queue and throttling.

3. API sketch#

TEXT

POST /v1/notifications  { idempotencyKey, userId | userIds[] | segmentId, channels:["push","email"], templateId, params:{}, priority:"high"|"normal", scheduleAt? }  -> 202 Accepted { notificationId }          (async; never block the caller on delivery)GET  /v1/notifications/{id}        -> { status per channel: queued|sent|delivered|failed|suppressed }PUT  /v1/users/{id}/preferences    { email:{marketing:false}, push:{...}, quietHours }POST /v1/devices                   { userId, platform, token }   ; DELETE /v1/devices/{token}Webhooks in: /webhooks/ses, /webhooks/twilio, APNs feedback  (bounces, invalid tokens, delivered)POST /v1/notifications
  { idempotencyKey, userId | userIds[] | segmentId, channels:["push","email"], templateId, params:{}, priority:"high"|"normal", scheduleAt? }
  -> 202 Accepted { notificationId }          (async; never block the caller on delivery)
GET  /v1/notifications/{id}        -> { status per channel: queued|sent|delivered|failed|suppressed }
PUT  /v1/users/{id}/preferences    { email:{marketing:false}, push:{...}, quietHours }
POST /v1/devices                   { userId, platform, token }   ; DELETE /v1/devices/{token}
Webhooks in: /webhooks/ses, /webhooks/twilio, APNs feedback  (bounces, invalid tokens, delivered)

4. Data model#

TEXT

notifications(notification_id PK, idempotency_key UNIQUE, user_id, template_id, params JSONB, priority, created_at)notification_deliveries(delivery_id PK, notification_id, channel, status, provider_msg_id, attempts, last_error, updated_at)  INDEX(notification_id), INDEX(status, updated_at)devices(token PK, user_id, platform, last_seen_at, valid BOOLEAN)   INDEX(user_id)user_preferences(user_id PK, prefs JSONB, quiet_hours, timezone)templates(template_id PK, channel, locale, subject, body)   (versioned)Redis:  dedupe:{idempotency_key} TTL 24h ; ratelimit:{user_id}:{channel} ; unsubscribe/suppression setAnalytics: delivery events -> Kafka -> warehousenotifications(notification_id PK, idempotency_key UNIQUE, user_id, template_id, params JSONB, priority, created_at)
notification_deliveries(delivery_id PK, notification_id, channel, status, provider_msg_id, attempts, last_error, updated_at)
  INDEX(notification_id), INDEX(status, updated_at)
devices(token PK, user_id, platform, last_seen_at, valid BOOLEAN)   INDEX(user_id)
user_preferences(user_id PK, prefs JSONB, quiet_hours, timezone)
templates(template_id PK, channel, locale, subject, body)   (versioned)
Redis:  dedupe:{idempotency_key} TTL 24h ; ratelimit:{user_id}:{channel} ; unsubscribe/suppression set
Analytics: delivery events -> Kafka -> warehouse

5. High-level design#

TEXT

[Internal services / scheduler / campaign tool]            |  POST /notifications (idempotency key)            v   [Notification API] --validate, dedupe (Redis), persist--> Postgres            |            v   [Kafka: notification_requests]            |   [Orchestrator workers]: load prefs + devices, check opt-out/quiet hours, per-user rate limit, render template            |                 -> one message per (user, channel)   +--------+---------+-----------+   v        v         v           v [Kafka   [Kafka    [Kafka      [Kafka  push]    sms]      email]      in-app]   |        |         |           | [Push    [SMS      [Email      [In-app workers]  workers]  workers]    writer -> DB/WebSocket]   |        |         | APNs/FCM  Twilio    SES/SendGrid   ---webhooks/feedback---> [Status service] -> deliveries table, Kafka events   each worker: retries w/ backoff, circuit breaker per provider, DLQ after N attempts[Internal services / scheduler / campaign tool]
            |  POST /notifications (idempotency key)
            v
   [Notification API] --validate, dedupe (Redis), persist--> Postgres
            |
            v
   [Kafka: notification_requests]
            |
   [Orchestrator workers]: load prefs + devices, check opt-out/quiet hours, per-user rate limit, render template
            |                 -> one message per (user, channel)
   +--------+---------+-----------+
   v        v         v           v
 [Kafka   [Kafka    [Kafka      [Kafka
  push]    sms]      email]      in-app]
   |        |         |           |
 [Push    [SMS      [Email      [In-app
 workers]  workers]  workers]    writer -> DB/WebSocket]
   |        |         |
 APNs/FCM  Twilio    SES/SendGrid   ---webhooks/feedback---> [Status service] -> deliveries table, Kafka events
   each worker: retries w/ backoff, circuit breaker per provider, DLQ after N attempts

6. Deep dives#

Reliability: at-least-once with dedupe

API dedupes on idempotency_key (unique index + Redis fast path) so a retrying caller doesn't double-send.
Workers commit the Kafka offset only after the provider accepts; crash → redelivery → provider-level dedupe via provider_msg_id/collapse key where supported, otherwise accept rare duplicates for push, never for OTP SMS (check deliveries.status before send).
Retries: exponential backoff with jitter (1 s, 2 s, 4 s … cap 1 h), max 5-10 attempts, then DLQ with alert. Distinguish retryable (5xx, 429) from terminal (invalid token → mark device invalid, bounce → suppress email).

Isolation per channel and provider

Separate topics and worker pools per channel so a Twilio outage doesn't block push. Circuit breaker per provider; fall back to a secondary provider (SES → SendGrid) when open.
Priority lanes: high (OTP, security) and normal (marketing) on separate topics/partitions so campaigns never delay OTPs.

Preferences, throttling, and content

Orchestrator applies: opt-outs, quiet hours (timezone-aware), per-user frequency caps (Redis token bucket, e.g. max 5 marketing pushes/day), channel fallback (push failed → email).
Templates rendered server-side with locale; versioned so in-flight messages don't change. Collapse/aggregate: "5 new comments" instead of 5 pushes (batching window of a few seconds per user).

7. Bottlenecks & scaling#

Campaign bursts: fan a segment of 10M into Kafka in batches; workers scale with partitions; provider rate limits enforced per worker pool (token bucket) so you don't get throttled upstream.
Postgres writes for delivery status: batch updates; or move status to a wide-column store keyed by notification_id; keep 30-90 days hot, archive the rest.
Device token churn: honor APNs/FCM feedback, delete invalid tokens; refresh tokens on app launch.
Observability: per-channel send/success/fail rates, provider latency, queue lag, DLQ depth; alert on delivery success < SLO.
Security/compliance: unsubscribe links (CAN-SPAM), STOP for SMS (TCPA), PII in templates not in logs, encrypted tokens at rest.

8. Words to say#

202 Accepted (async), idempotency key + unique index, at-least-once with dedupe, Kafka topic per channel, priority lanes, consumer groups, exponential backoff with jitter, dead-letter queue, circuit breaker per provider, provider fallback, APNs / FCM / Twilio / SES / SendGrid, device token lifecycle and feedback service, opt-out / suppression list, quiet hours, frequency cap (token bucket), template versioning, collapse key / aggregation, delivery webhooks, bounce handling, CAN-SPAM / TCPA.