Walkthrough: News Feed (Twitter / Facebook timeline)

1. Requirements#

Functional

• Users publish posts (text, media); users follow users.
• Home feed: reverse-chronological (or ranked) posts from followees, paginated, infinite scroll.
• Like/comment counts shown; media served via CDN.

Non-functional

• Read-heavy; feed load p99 < 500 ms; publish-to-visible within ~5 s (eventual consistency acceptable).
• 99.99% availability for reads; 10M DAU scale; handle celebrity accounts with millions of followers.

2. Estimates#

• DAU 10M. Posts: 2/user/day → 20M posts/day ÷ 86,400 ≈ 230 posts/s (peak ~700).
• Feed reads: 10 opens/user/day → 100M/day ≈ 1,160 QPS (peak ~3,500).
• Follow graph: avg 200 followers → fan-out on write = 230 × 200 ≈ 46k feed-cache inserts/s (peak ~140k). Redis handles this across a few shards.
• Feed cache: 500 post IDs × 8 B = 4 KB/user × 10M = 40 GB (plus overhead ~2x → ~80 GB) → Redis cluster of a few nodes.
• Post storage: 20M/day × ~1 KB metadata = 20 GB/day ≈ 7 TB/year (text); media in S3 is separate and larger (say 20% of posts × 500 KB = 2 TB/day).

3. API sketch#

POST /v1/posts                 { text, mediaIds[] }                    -> 201 { postId }GET  /v1/feed?cursor=&limit=20 -> { posts:[...], nextCursor }POST /v1/users/{id}/follow     ; DELETE /v1/users/{id}/followPOST /v1/posts/{id}/like       ; GET /v1/posts/{id}POST /v1/media/upload-url      -> presigned S3 URL

Cursor = (created_at, post_id) of last item; Idempotency-Key on POST /posts.

4. Data model#

users(user_id PK, handle, ...)follows(follower_id, followee_id, created_at)  PK(follower_id, followee_id); INDEX(followee_id)   -- "who follows X" for fan-outposts(post_id PK (Snowflake), author_id, text, media_keys[], created_at)  INDEX(author_id, created_at DESC)likes(post_id, user_id) ; counters in Redis (INCR) with periodic flush Feed cache (Redis):  ZADD feed:{user_id} <post_id as score/time> <post_id>   ZREMRANGEBYRANK to cap at ~500Post cache (Redis):  HSET post:{post_id} ...  ; hot author lists for celebrities

Posts in Postgres (sharded by author_id) or Cassandra (author_id, created_at DESC); the graph in Postgres or a graph DB if traversals grow.

5. High-level design#

client -> CDN (media) client -> LB -> API gateway (auth, rate limit)   POST /posts -> [Post service] -> posts DB + S3 media                        |-> Kafka "post_created" -> [Fan-out workers] -> follows DB (followers of author)                                                            |-> Redis feed:{follower} ZADD  (for normal authors)                                                            |-> skip for celebrities (flag)   GET /feed  -> [Feed service] -> Redis feed:{user} (IDs)                         |-> merge celebrity followees' recent posts (fan-out on read)                        |-> hydrate posts from post cache / posts DB, users from user cache                        |-> optional ranking service -> return page

6. Deep dives#

Fan-out on write vs fan-out on read (the core decision)

• Fan-out on write (push): at publish time, insert post ID into every follower's feed list. Reads are O(1) list fetch. Cost: celebrity with 10M followers = 10M writes per post; wasted work for inactive users.
• Fan-out on read (pull): at read time, fetch recent posts from each followee and merge. Writes are cheap; reads are expensive (N followees × query).
• Hybrid (what Twitter did): push for normal users; pull for users above a follower threshold (e.g. >10k-100k); merge at read. Skip pushing to users inactive >30 days; rebuild their feed lazily on next login.

Feed storage and pagination

• Redis sorted set keyed by user, score = post_id (time-ordered Snowflake) → ZREVRANGEBYSCORE feed:{u} (cursor -inf LIMIT 0 20. Cap length (~500-1000) to bound memory; older pages fall back to DB fan-out on read.
• Hydration: batch MGET post:{id} for the page; cache-aside to posts DB on miss.

Ranking (if asked)

• Candidate generation (the chrono list) → feature fetch (engagement, recency, affinity) → model scoring → re-rank + diversity rules. Keep it a separate service so chrono fallback exists.

7. Bottlenecks & scaling#

• Fan-out spikes: Kafka buffers; workers scale with partitions keyed by author_id; throttle celebrity posts via pull path.
• Hot posts (viral): post cache replicated/local cache per feed node; like counters via Redis INCR, flushed asynchronously; never UPDATE a counter row per like.
• Redis memory: cap feed length, TTL feeds of inactive users, shard by user_id with Redis Cluster.
• Posts DB: shard by author_id so "posts by author" is single-shard; Snowflake IDs keep time order.
• Follow graph for fan-out: INDEX(followee_id); for huge accounts stream the follower list in pages.
• Consistency: user must see their own post immediately → write it into their own feed synchronously (read-your-writes), rest is eventual.
• Media: direct-to-S3 upload with presigned URLs; CDN in front; thumbnails via async workers.

8. Words to say#

fan-out on write (push) vs fan-out on read (pull), hybrid fan-out, celebrity problem, follower threshold, Redis sorted set (ZADD/ZREVRANGEBYSCORE), cursor pagination, Snowflake IDs for time order, hydration, cache-aside, Kafka partitions by author_id, inactive-user skip, read-your-writes for own posts, eventual consistency (publish-to-visible SLA), hot post, counter aggregation, presigned URL, CDN, sharding by author_id, candidate generation vs ranking.