A media company with a library of twelve million articles, transcripts, and research documents had built a semantic search system on a managed vector database. The system was designed to let journalists and editors find related content across the entire archive using natural language queries. At five million documents, the system performed well. Query latency was under 200ms, relevance was high, and the editorial team adopted it enthusiastically.
By the time the index reached nine million documents, the system was in trouble. Query latency had crept above two seconds. Relevance had degraded — the system was returning semantically similar but editorially irrelevant results. And the monthly infrastructure bill for the vector database had grown to $38,000, roughly triple what the team had projected.
The problem was not the vector database product. The problem was the assumption that a single embedding space could represent twelve million documents across dozens of editorial domains with equal fidelity.
Why the vector space collapsed
Embedding models map text into a high-dimensional space where proximity represents semantic similarity. This works well when the corpus is homogeneous — medical papers, legal contracts, product reviews. The media company’s corpus was not homogeneous. It spanned political analysis, sports reporting, financial markets, cultural criticism, technology reviews, and investigative journalism. A query about “market volatility” could refer to financial markets, real estate, or the market for political advertising.
In a single embedding space, documents from all domains compete for position. As the corpus grew, the embedding space became crowded. Documents from different domains with superficially similar language began clustering together. The vector database’s approximate nearest neighbor search returned results that were close in the embedding space but distant in editorial relevance. The system could not distinguish between “semantically similar” and “editorially related” because the embedding space had no notion of editorial domain.
This is the scaling wall that vector-only search systems hit when the corpus is large and diverse. The embedding model was not the bottleneck. The bottleneck was the single-vector-space assumption.
What they tried
The team first tried re-embedding with a larger model. A model with more dimensions should, in theory, create more separation between unrelated documents. The larger model improved relevance slightly but doubled indexing costs and increased query latency by forty percent. The improvement did not justify the cost.
The second attempt was to add metadata filtering — restrict results by section, date range, or author. This helped for queries where the user knew which section to search. It did not help for cross-domain queries, which were the most common and the most valuable use case. A journalist investigating a story that spanned politics, finance, and technology needed results from all three domains, filtered by relevance rather than section.
The third attempt was to shard the vector database by editorial domain — one index for politics, one for finance, one for sports, and so on. Queries were routed to the relevant shards. This improved precision within each domain but broke cross-domain search. The routing logic also introduced a classification step that was itself error-prone: a story about political fundraising was classified as politics sixty percent of the time and finance forty percent of the time, depending on which paragraphs the classifier weighted most heavily.
The approach: hierarchical retrieval with domain-specific indexes
We replaced the single-index architecture with a hierarchical retrieval system that combined coarse-grained domain routing with fine-grained vector search within each domain.
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
The domain router used a lightweight classifier to determine which domains were relevant to the query. Unlike the previous shard routing attempt, the router was not forced to pick a single domain. It returned a ranked list of relevant domains, and the search fan-out covered the top three. A query about political fundraising would hit both the political analysis and financial markets indexes. A query about a specific technology company would hit the technology index and possibly the financial markets index.
Each domain index contained only documents from its domain. The embedding model was the same across all indexes, but the reduced corpus size per index meant that the vector space was less crowded and the nearest neighbor search was more precise. The same embedding model that produced noisy results on a twelve-million-document corpus produced sharp results on a two-million-document corpus.
The cross-domain re-ranker took the top candidates from each domain index and re-scored them using a lightweight model that considered both semantic similarity and editorial relevance signals. The re-ranker had access to metadata that the embedding model did not: publication date, author reputation, section placement, and citation count within the archive. These signals were not available during embedding but were highly predictive of editorial relevance.
What we gave up
The hierarchical system added latency. The domain routing step, the parallel index queries, and the cross-domain re-ranking each added time. Total query latency was 400ms, compared to 180ms for the original system when it was running on the five-million-document corpus. However, compared to the degraded state at nine million documents, the hierarchical system was five times faster.
The second trade-off was indexing complexity. Instead of writing all documents to a single index, the ingestion pipeline had to classify each document into one or more domain indexes. The domain classifier was accurate about ninety-three percent of the time. The remaining seven percent were indexed in multiple domains, which increased storage costs by roughly fifteen percent.
The third trade-off was maintenance. Six domain indexes meant six index management operations instead of one. Schema changes, re-indexing jobs, and capacity planning all multiplied by the number of domains. The team automated most of this, but the operational surface area was larger.
Results
At twelve million documents, the hierarchical system returned results with median latency of 380ms and p99 latency of 900ms. Editorial relevance, as measured by click-through rate on search results, improved by thirty-four percent compared to the degraded single-index system. The infrastructure cost dropped from $38,000 per month to $14,000 per month, because the domain indexes were smaller and required less powerful instances than the monolithic index.
The most significant improvement was in cross-domain search. Journalists reported that the system surfaced connections between stories in different editorial domains that they would not have found through section-specific search. A query about a specific CEO returned results from financial markets, technology, and investigative journalism — results that had previously been hidden in separate, disconnected parts of the archive.
The decision heuristic
If your vector search relevance degrades as your corpus grows, the problem is not the embedding model or the vector database. The problem is that a single embedding space cannot represent a diverse corpus with equal fidelity at every scale. Split the corpus by domain, build separate indexes, and route queries to the relevant indexes. The embedding model stays the same. The vector space gets room to breathe. And the relevance improves because the nearest neighbor search is no longer competing with unrelated documents for position.
