Notification Fanout Architecture

Notification fanout architecture is one of the most-asked system design problems, and it collapses into a single decision: do you do the delivery work when content is written, or when it is read? Fan-out-on-write makes reads cheap and writes expensive; fan-out-on-read does the opposite. The naive version of either breaks on one input, the account with millions of followers, which is why every system at scale ends up hybrid.

This is the Twitter-timeline problem, the push-notification problem, and the activity-feed problem, all the same shape underneath. Get the fan-out model right and the system scales smoothly; get it wrong and one popular user takes the whole thing down.

Why fanout is the core of feed and notification systems

Any system where one action reaches many people, a post hitting followers, an event triggering notifications, a comment alerting subscribers, has a fan-out problem at its heart. The number of deliveries per action can be enormous, and where you spend that work determines whether the system is fast and affordable or slow and expensive.

The reason this shows up in every senior system design interview is that it forces a real tradeoff with no free answer. You are moving cost between write time and read time, and the right move depends on your access patterns and your worst-case input. This post is part of the Distributed systems patterns series.

What is the difference between fan-out-on-write and fan-out-on-read?

Fan-out-on-write does the delivery work at publish time: when a user posts, the system immediately writes that post into every follower’s feed. Reads are then trivial, just read your own pre-built feed. Fan-out-on-read does almost nothing at write time and instead assembles each user’s feed on demand by pulling and merging the posts of everyone they follow when they open the app.

The two models simply move the cost to opposite ends:

	Fan-out-on-write (push)	Fan-out-on-read (pull)
Work at write time	High (write to N feeds)	Minimal (store once)
Work at read time	Minimal (read own feed)	High (gather + merge live)
Best when	Reads >> writes	Writes >> reads, or huge fan-out
Storage	High (feed per user)	Low (store once)
Weakness	The celebrity problem	Slow, expensive reads

Most consumer products are read-heavy (people scroll far more than they post), which biases toward fan-out-on-write so the common operation, reading the feed, is cheap. That bias is correct right up until one account has millions of followers.

What is the celebrity problem in fan-out?

The celebrity problem is what happens to fan-out-on-write when one account has millions of followers. A single post from that account must be written into millions of feeds, turning one write into a massive, slow, expensive operation that can overwhelm the system and delay delivery for everyone. The cost of fan-out-on-write scales with follower count, and celebrities break the model.

This is the specific failure that pushes systems off the naive design. You cannot serve a power-law follower distribution with a single fan-out model, because the model that is efficient for the millions of normal accounts is catastrophic for the handful of huge ones.

How do you scale a notification system? The hybrid model

The answer used by large feed systems is a hybrid: fan-out-on-write for normal accounts, fan-out-on-read for celebrity accounts. Normal users get their content pushed into followers’ feeds so reads stay cheap; high-follower accounts skip the write fan-out, and their posts are pulled in at read time and merged into the requesting user’s feed. You get cheap reads in the common case and avoid the write explosion in the extreme case.

The mechanics of the hybrid:

1. Classify accounts by follower count. Above a threshold, an account is “celebrity” and is excluded from write fan-out.
2. Normal posts fan out on write into follower feeds as usual.
3. Celebrity posts are stored once and not pushed.
4. At read time, a user’s feed is their pre-built feed (from normal accounts) merged with a live pull of the celebrities they follow.
5. Cache aggressively, because a celebrity’s recent posts are read by millions and should be served from cache, not recomputed per reader.

This keeps the expensive operation rare: only the small number of celebrity accounts incur read-time merging, and their content is cacheable precisely because so many people want the same thing.

The infrastructure underneath: queues and idempotency

Fan-out is bursty by nature. A popular post creates a spike of delivery work, and you cannot let that spike hit your databases synchronously. The standard pattern is a queue between the write and the fan-out workers: the post is accepted quickly, a fan-out job is enqueued, and workers drain it at a controlled rate. This is backpressure applied to delivery, the same principle covered in Backpressure Design for Real-Time Systems.

Delivery must also be idempotent. Fan-out workers retry on failure, and without idempotency a retry sends a duplicate notification, which users notice and hate. Key each delivery by (event, recipient) and skip ones already delivered, the same discipline as Idempotency Keys for Distributed Systems. For push notifications specifically, deduplication before the final send is what stops the dreaded double-buzz.

How do you store and trim notification feeds?

Cap each feed and trim it, because under fan-out-on-write a feed grows without bound and most of it is never read. The standard move is to keep a bounded recent window per user (the last few hundred items) in the fast store, and fall back to a query against durable storage for the rare deep scroll. Storing every notification forever in every feed is the cost that quietly sinks fan-out-on-write.

The economics are the point. Fan-out-on-write trades storage for read speed, and if you do not bound that storage, the trade gets worse every day as feeds accumulate items nobody scrolls to. A capped feed keeps the hot path small and cheap: writes append and trim, reads hit a small structure, and the long tail lives in cheaper storage that is queried only when someone actually asks for it.

This also bounds the write amplification. Trimming as you write means a feed never grows past its cap, so the per-recipient cost stays constant rather than creeping upward over a user’s lifetime. Decide the cap from real scroll behavior, keep the recent window fast, and let durable storage hold the history that is rarely requested.

A notification fanout checklist

Before you ship a feed or notification system:

• You have chosen a fan-out model based on your real read/write ratio, not a default.
• High-follower accounts are special-cased (read fan-out) so one post cannot trigger millions of writes.
• A queue sits between the triggering event and fan-out workers to absorb bursts.
• Delivery is idempotent, keyed by (event, recipient), so retries cannot duplicate.
• Celebrity content is cached, since many readers want the same recent posts.
• You have a plan for feed storage growth under fan-out-on-write (it is not free).
• Delivery has a deadline and a dead-letter path, so a stuck recipient does not block the batch.

How do you decide which fan-out model to use?

The write-versus-read choice is usually presented as a binary, and in production it is a per-user decision made at runtime. Getting the criteria explicit is what makes the hybrid implementable rather than aspirational.

Factor	Favours fan-out-on-write	Favours fan-out-on-read
Follower count	Low — cost is bounded	Very high — write amplification is unacceptable
Read frequency	High — precomputed feed is read many times	Low — precomputing is wasted
Read latency requirement	Strict — the feed already exists	Relaxed — assembly at read time is affordable
Write volume	Low — few writes to amplify	High — amplifying every write is prohibitive
Storage cost sensitivity	Tolerant — feeds are duplicated per recipient	Sensitive — one copy of each item

The arithmetic that decides it: a write fans out to N recipients, so write cost is writes × average_followers. At a thousand followers and a thousand writes per second, that is a million feed insertions per second — which is why the model breaks for high-follower accounts specifically, not for the system generally.

The hybrid is the production answer, and the threshold is the design decision. Below a follower count, fan out on write. Above it, do not, and have readers merge those authors’ items at read time. The threshold is tunable and worth setting from measurement rather than intuition — typically somewhere in the thousands to tens of thousands, chosen so the number of accounts above it stays small enough for read-time merging to be cheap.

Two details make the hybrid correct rather than merely clever. The threshold must be evaluated at write time, per author, and accounts crossing it need a migration path — a user who gains followers rapidly should transition without a backfill that itself causes a spike. And readers must merge deterministically, so a feed assembled from precomputed and live-merged sources produces stable ordering. Users notice items reordering between refreshes far more than they notice a slightly stale feed.

What breaks in notification systems at scale?

Four failure modes account for most production incidents in fan-out systems, and none of them is the fan-out itself.

The thundering write on a popular event. A celebrity posts, or a broadcast targets every user, and millions of feed writes are enqueued at once. Without rate limiting on the fan-out workers this saturates the datastore that also serves reads, so a write-path event becomes a read-path outage. Fan-out work should be rate-limited and de-prioritised relative to user-facing traffic.

Duplicate notifications from retries. A fan-out job that partially completes and is retried re-delivers to the recipients it already reached. Users are unusually sensitive to duplicate notifications — it reads as broken in a way a delayed notification does not. Deduplication needs a stable key per recipient-event pair, which is idempotency applied at the fan-out layer.

Unbounded feed growth. Feeds that are never trimmed grow forever, and storage costs scale with users multiplied by retention. Trim on write to a fixed length, since almost nobody reads past the first few hundred items, and an unbounded feed is paying to store data nobody will ever request.

Fan-out to deleted or unsubscribed recipients. Membership changes between when the event was emitted and when the fan-out executes. Checking eligibility at delivery time rather than at enqueue time avoids notifying people who have since opted out — which is a compliance problem as well as an annoyance.

How should the delivery pipeline be structured?

Fan-out produces the list of who should be notified. Actually delivering to a device or an inbox is a separate concern, and conflating the two is the most common structural mistake in these systems.

The stages worth separating:

1. Event — something happened. Emitted once by the owning service, through a transactional outbox so it cannot be lost or duplicated relative to the state change.
2. Fan-out — resolve the event to a recipient list and write feed entries. Bounded work, rate-limited, idempotent per recipient-event pair.
3. Eligibility — filter by preferences, quiet hours, mute settings, and channel opt-ins. Evaluated at delivery time, not at fan-out time, because preferences change.
4. Aggregation — collapse related notifications. “Five people liked your post” rather than five notifications. This is where perceived quality is made or lost.
5. Delivery — push, email, or SMS, each with its own provider, its own rate limits, and its own failure modes.

Keeping these separate buys two things. Each stage can fail and retry independently, so a push provider outage does not lose feed entries or block email. And each scales independently, which matters because fan-out is bursty while delivery is rate-limited by external providers you do not control.

The aggregation stage deserves more attention than it usually gets. Users judge a notification system almost entirely on whether it respects their attention, and unaggregated notifications are the fastest way to get an app’s notifications disabled permanently — after which the channel is gone for good. Batching within a short window, collapsing by type and target, and enforcing a per-user rate cap are worth building early, because retrofitting them means rewriting the delivery path.

One design note on ordering: do not promise strict ordering across channels. A push notification and an email for the same event travel different paths with different latencies and will arrive out of order. Design the content so each is self-contained and order-independent, rather than trying to guarantee sequencing you cannot control.

How do you store and trim a feed cheaply?

Feed storage is where fan-out-on-write systems become expensive, because you are storing one row per recipient per item and that product grows quickly.

Three decisions control the cost.

Store references, not content. A feed entry should hold the item ID, a timestamp, and whatever is needed for ordering and filtering — not a copy of the item body. Duplicating content across every recipient’s feed multiplies storage by follower count and makes edits impossible to propagate. Hydrate the content at read time from a single source.

Trim on write to a fixed length. Cap each feed at a few hundred entries and drop the oldest as new ones arrive. Almost nobody scrolls past the first page, and users who genuinely want history can be served from the source rather than from the precomputed feed. Untrimmed feeds are the single largest avoidable cost in these systems.

Pick a store that matches the access pattern. The access pattern is narrow and predictable: append to a per-user list, read the most recent N, occasionally delete. That suits a sorted-set or wide-column structure keyed by user with a time-ordered range — and it suits a general-purpose relational table poorly once the row count reaches billions.

Two refinements worth adding once the basics work. Do not maintain feeds for inactive users — a user who has not opened the app in months does not need writes; rebuild their feed on next login from the source. In a large user base this removes a substantial share of all fan-out work, and it is invisible to everyone who is actually using the product.

And separate the unread count from the feed itself. The count is read constantly, changes frequently, and is small; keeping it as its own cheap counter avoids scanning a feed to compute a badge number on every app open, which is otherwise one of the highest-volume queries in the entire system.

How do you handle the celebrity problem in practice?

The celebrity case — one account whose write fans out to millions — is the reason the pure write model fails, and the mitigations are more varied than the standard “merge at read time” answer.

Four approaches, which combine:

Read-time merge above a threshold. The standard hybrid: high-follower accounts are excluded from write fan-out, and readers merge their recent items when assembling a feed. Correct, and it makes reads more expensive for everyone who follows such an account.

Asynchronous, rate-limited fan-out with a slower SLA. Rather than excluding them, fan out over minutes instead of milliseconds, deliberately de-prioritised behind normal traffic. Followers get the item slightly later, which is almost always acceptable, and the write amplification is spread rather than concentrated.

Fan out to active users first. Order the fan-out by recency of activity so users likely to open the app in the next minutes get the item promptly, while dormant users are filled in later or on demand. This makes the perceived latency excellent at a fraction of the urgent work.

Do not fan out at all for the extreme tail. For the handful of accounts with tens of millions of followers, the item is effectively public content; serving it from a cached, shared source that everyone reads is far cheaper than any per-user materialisation.

The consistent theme: the celebrity problem is a scheduling problem, not a storage one. The work is not impossible, it is merely unacceptable to do all at once and synchronously — and once framed that way, spreading it over time, prioritising by who will actually look, and sharing one copy where the audience is universal are all straightforward moves.

The operational requirement underneath all four is that fan-out work must be preemptible and rate-limited, so a celebrity event degrades into slower delivery rather than into a datastore saturation event that damages everyone’s reads.

What I’d do differently

The mistake in interviews and in real systems alike is committing to one fan-out model and trying to make it serve every account. Pure fan-out-on-write is elegant until a celebrity joins; pure fan-out-on-read is simple until reads dominate your cost. Neither survives contact with a real follower graph.

If I were designing this from scratch, I would start with fan-out-on-write for the read-heavy common case, identify the high-degree accounts early, and route them to read fan-out before they ever become an incident. The hybrid is not a premature optimization here; it is the known shape of the problem, and building it in from the start is cheaper than retrofitting it after a popular user melts your write path. The closely related real-time delivery layer is covered in WebSocket Capacity Planning for Social Products.

Sources

• The System Design Primer, feeds and fan-out: github.com/donnemartin/system-design-primer
• Redis, solutions and patterns for feeds: redis.io/solutions
• Apache Kafka, use cases (event fan-out): kafka.apache.org/uses