A video streaming platform grew from 1,000 beta users to 10 million subscribers over thirty months. Their recommendation system was rebuilt three times during this period. Each rebuild was triggered not by a feature request or a performance complaint but by an architectural boundary — a point at which the system’s design assumptions stopped holding and no amount of optimization could compensate.
The pattern repeats across scaling journeys. Teams optimize their current architecture until it hits a wall, then rebuild at a higher scale tier. The walls are predictable. The rebuilds do not have to be.
Phase 1: 1,000 to 100,000 users
At 1,000 users, the recommendation system was a single Python service that computed collaborative filtering recommendations on demand. When a user opened the app, the service queried the user’s watch history, computed user-user similarity against the entire user base, and returned the top-N recommendations. Computation took 200ms. The user base fit in memory. Everything worked.
At 10,000 users, computation time increased to 1.8 seconds. The collaborative filtering algorithm computed pairwise similarity between the requesting user and every other user in the system. At 1,000 users, this was one million comparisons. At 10,000 users, it was 100 million comparisons. The algorithm was the same. The input size grew by an order of magnitude, and computation time grew proportionally.
The team optimized: they added approximate nearest neighbor search, pre-computed similarity matrices during off-peak hours, and cached recommendations for active users. These optimizations bought time. At 50,000 users, the pre-computed similarity matrix consumed all available memory on the largest instance the cloud provider offered. At 100,000 users, the nightly pre-computation job took eleven hours to complete, exceeding the available maintenance window.
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
The architectural boundary was the user-user similarity approach. User-user similarity requires comparing the requesting user to every other user. The computation scales quadratically with user count. No optimization can change the asymptotic complexity. The architecture had to change.
Phase 2: 100,000 to 1,000,000 users
The rebuild switched from user-user similarity to item-item similarity. Instead of computing similarity between users, the system pre-computed similarity between items. A user’s recommendations were generated by finding items similar to the ones they had watched. Item-item similarity was computed offline in a batch job and stored in a distributed cache.
This approach scaled better because the item catalog grew more slowly than the user base. The platform had 40,000 titles. The item-item similarity matrix was 40,000 by 40,000 — manageable in memory. User count was irrelevant to the matrix size. The system served 500,000 daily active users with 95th percentile latency under 150ms.
At 750,000 daily active users, a new problem emerged. The item-item similarity matrix was pre-computed nightly. New content was added throughout the day. Users who watched new content within hours of its release received recommendations based on the previous night’s similarity matrix, which did not include the new content. The system could not recommend new content until the next batch run.
The staleness problem was not solvable by increasing batch frequency. Computing the item-item similarity matrix for 40,000 items took four hours. Running it every four hours would reduce the staleness window but consume the entire compute budget in matrix computation, leaving no capacity for serving.
At 1,000,000 users, the team introduced a trending content heuristic that surfaced new releases outside the recommendation system. The heuristic worked but undermined the personalization that the recommendation system was supposed to provide. Users began to notice that the “recommended” section and the “trending” section showed different content, and the recommended section was always stale.
The architectural boundary was the batch pre-computation approach. Batch pre-computation introduces a staleness window that is proportional to the computation time. As the catalog grows, the computation time grows, and the staleness window widens. The architecture had to separate the computation that could be pre-computed from the computation that had to be real-time.
Phase 3: 1,000,000 to 10,000,000 users
The second rebuild introduced a two-stage retrieval architecture: candidate generation followed by ranking.
The candidate generation stage was a fast, approximate retrieval step that selected a few hundred candidate items from the full catalog. It used a lightweight model — a matrix factorization trained on user-item interaction data — that could score all 40,000 items for a given user in under 20ms. The candidate set was intentionally broad: the goal was recall, not precision. The system needed to ensure that relevant items were in the candidate set, even if the set also included irrelevant items.
The ranking stage was a slower, more accurate model that scored the few hundred candidates and selected the top-N to display. The ranking model used richer features: content metadata, user behavioral signals, contextual features like time of day and device type. Because the input was small — a few hundred items rather than 40,000 — the ranking model could be more complex without violating latency constraints.
This two-stage architecture solved the staleness problem. The candidate generation model was retrained daily, but the ranking model was updated continuously. New content was added to the candidate generation index as soon as it was published. The ranking model could incorporate real-time signals — a user’s activity in the current session, trending content, editorial promotions — without waiting for the candidate generation model to retrain.
At 10 million users, the system served recommendations with p50 latency of 60ms and p99 latency of 200ms. New content appeared in recommendations within minutes of publication. The ranking model was retrained every six hours on the most recent interaction data, capturing behavioral shifts within the same day.
The architectural boundaries
Each phase hit a different boundary. Phase 1 hit a computation boundary — the user-user similarity approach could not scale beyond 100,000 users because the computation was quadratic. Phase 2 hit a freshness boundary — the batch pre-computation approach could not serve new content promptly because the matrix computation time grew with the catalog. Phase 3 had not hit a boundary at 10 million users, but the team anticipated a personalization boundary at roughly 50 million users, where the ranking model’s feature space would become too large for the current serving infrastructure.
The boundaries are predictable. User-user similarity fails at roughly 100,000 users. Item-item batch pre-computation fails when the catalog grows large enough that the batch job exceeds the acceptable staleness window. Two-stage retrieval fails when the ranking model’s feature complexity exceeds the serving infrastructure’s latency budget. Each boundary corresponds to a specific architectural constraint, and each constraint has a known solution.
What the team learned
The first lesson: do not optimize past the architectural boundary. The team spent three months optimizing the single-service collaborative filtering approach before rebuilding. Those three months bought a factor-of-two improvement. The rebuild to item-item similarity bought a factor-of-ten improvement. Optimization within an architectural tier has diminishing returns. Crossing an architectural boundary has step-function returns.
The second lesson: build the next architecture before you need it. The team rebuilt reactively each time — they hit the wall, then scrambled to rebuild. A better approach would have been to monitor the scaling metrics that predict each boundary: user count for the computation boundary, catalog size times batch job duration for the freshness boundary, feature count times candidate set size for the personalization boundary. When these metrics approach known thresholds, start the next rebuild.
The third lesson: the two-stage architecture is the right default for recommendation systems at scale. Candidate generation handles the catalog scale problem. Ranking handles the personalization problem. Separating them allows each stage to be optimized independently and updated on different cadences. This architecture is not specific to video streaming. It applies to any recommendation system with a large catalog and a latency budget under 200ms.
The decision heuristic
If your recommendation system is approaching an architectural boundary, measure the specific metric that defines the boundary rather than the symptoms. Slow response times are a symptom. Quadratic computation complexity is the boundary. Stale recommendations are a symptom. Batch computation duration exceeding the acceptable staleness window is the boundary. Fix the boundary by changing the architecture tier, not by optimizing within the current tier. And build the next tier before the current one breaks, not after.
