Most data catalog projects die within six months. The tool gets purchased, a team populates it with metadata for a few hundred tables, enthusiasm fades, and twelve months later the catalog is a stale artifact that nobody checks because it was last updated in March. The pattern is so reliable that data teams have a name for it: catalog shelf-ware.
A mid-market retailer with forty thousand SKUs and eight regional distribution centers had burned through three catalog attempts in five years. Each followed the same arc. A tool vendor promised automated lineage, smart tagging, and search. The data team spent weeks importing metadata. Business users never showed up. The catalog became another internal tool that the data team maintained for itself.
When they engaged us for a fourth attempt, the conversation started with a blunt question: what would make this one different?
Why the First Three Failed
The failure pattern across all three attempts was identical, even though the tools were different. The catalog was treated as a metadata repository — a place to store column descriptions, table ownership, and data lineage. The data team populated it because the data team cared about metadata. Business users did not show up because business users do not search for column descriptions. They search for answers.
The first attempt used a commercial catalog tool. The data team imported schemas from their warehouse, added column descriptions, and configured ownership tags. Business users were given access and training. Attendance at training sessions was high. Usage after training was near zero. The catalog answered questions that business users were not asking.
The second attempt tried a wiki-based approach. Pages for each dataset, manually maintained. The problem was staleness. When a table changed, the wiki page did not update. When a report broke because of a schema change, the wiki still showed the old schema. Within four months, the wiki was actively misleading, which is worse than being empty.
The third attempt was an open-source catalog tool with automated metadata harvesting. The tool ingested schemas automatically, which solved the staleness problem. But the automated metadata was generic — column names, data types, table sizes. It told you that a column existed but not what it meant to the business. The gap between technical metadata and business context remained unfilled.
The Pattern That Matters
The common thread across all three failures was a misunderstanding of what a catalog is for. A catalog is not a metadata repository. A catalog is a decision-making tool. People use it when they need to make a decision about data: which dataset contains the information I need, can I trust this number, who owns this pipeline, what happens if this table changes.
Every question in that list is a decision point. The catalog only gets used if it helps people make those decisions faster than the alternative (asking a colleague on Slack). If the catalog is slower than Slack, Slack wins. Every time.
The fourth attempt started by mapping the actual decisions that analysts, data engineers, and business stakeholders made about data on a daily basis. We spent two weeks interviewing people across four teams. The interviews were structured around one question: what was the last time you needed to find or understand a dataset, and what did you do?
The answers clustered into five decision types:
- Discovery: “I need data about customer returns. Where is it?”
- Trust: “This number looks wrong. Is the pipeline current? When was it last refreshed?”
- Impact: “We are changing the pricing table. What downstream reports will break?”
- Ownership: “This dashboard is broken. Who maintains the pipeline that feeds it?”
- Governance: “Can I use this customer data for the marketing model? Is it approved?”
Each decision type required a different kind of metadata and a different interaction pattern. Discovery needed business glossary mapping — connecting the term “customer returns” to the table that stores them. Trust needed freshness indicators and pipeline status. Impact needed lineage. Ownership needed clear assignment. Governance needed policy tags and approval status.
Building for Decisions, Not Storage
The architecture for the fourth attempt was different in a structural way. Instead of building a catalog and populating it with metadata, we built a thin layer that connected existing systems. The warehouse already had schemas. The orchestration tool already had pipeline status. The access control system already had ownership. The business glossary already existed in a spreadsheet that the finance team maintained.
The catalog did not duplicate any of this information. It indexed it. When a user searched for “customer returns,” the catalog resolved the business term to the technical table using the glossary mapping, displayed the schema from the warehouse, showed the pipeline freshness from the orchestration tool, and listed the owner from the access control system. No metadata was stored in the catalog itself. Everything was pulled live from the source systems.
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
This design had a critical property: the catalog could never go stale because it never stored anything. If a pipeline was reorganized and a table moved, the glossary mapping updated. If ownership changed, the access control system reflected it. The catalog was a view, not a store.
The trade-off was latency. Pulling from source systems on every query was slower than reading from a local cache. For most queries, the latency was under two seconds, which was acceptable. For lineage queries on large schemas, the latency was higher — sometimes ten seconds — because the lineage graph had to be computed from the orchestration tool’s dependency data. We accepted this because lineage queries were infrequent and the user was already in a research mindset, not a browsing mindset.
What Made It Stick
Eighteen months after launch, the catalog was handling three hundred queries per week across four teams. Usage had grown organically because the catalog solved a real problem at the moment the problem occurred. Analysts did not need to schedule time to use the catalog. They used it when they hit a decision point, which happened multiple times per day.
Three factors made the difference.
First, the catalog was embedded in existing workflows. Instead of a separate web application that users had to remember to open, the catalog was integrated into the BI tool. When an analyst opened a report and saw a number that looked wrong, a single click showed the pipeline status and the data freshness for that report’s underlying tables. The catalog appeared at the moment of doubt, which is when trust questions arise.
Second, the catalog had a feedback loop. When a user searched for a term and the glossary mapping was wrong or missing, the catalog prompted the user to suggest a correction. These corrections were routed to the data steward for review. Over eighteen months, the glossary grew from four hundred terms to eleven hundred terms, almost entirely through user contributions. The catalog improved through use rather than through dedicated maintenance effort.
Third, the catalog had a usage-based quality metric. Instead of measuring catalog quality by metadata coverage (percentage of tables with descriptions), we measured it by question answer rate (percentage of searches that returned a useful result). This metric drove investment decisions. When the answer rate for governance questions dropped below sixty percent, the team investigated and found that the policy registry had not been updated after a compliance change. The fix was updating the policy registry, not the catalog.
The Trade-offs
The live-indexing approach had costs that a traditional metadata store would not have imposed.
Dependency on source systems was the primary risk. If the orchestration tool’s API went down, the catalog could not show pipeline status. If the access control system was slow, ownership lookups timed out. The catalog’s reliability was the minimum reliability across all its source systems. We mitigated this with cached fallbacks — if a source system was unreachable, the catalog showed the last-known value with a staleness indicator — but the fallbacks were explicitly marked as stale, which reduced trust in the fallback data.
Custom integrations for each source system required ongoing maintenance. When the orchestration tool upgraded its API, the catalog integration broke. When the access control system changed its data model, the ownership display showed incorrect assignments until the integration was fixed. Each integration was a maintenance contract with an external system that the catalog team did not control.
Search relevance was harder to tune than in a purpose-built catalog tool. Commercial catalog tools have years of investment in search ranking, faceted filtering, and relevance tuning. Our thin layer had basic keyword matching with glossary expansion. It was good enough for the five decision types it was designed for but poor for exploratory browsing. If a user wanted to understand “everything about customers,” the catalog was not the right tool. It was designed for pointed questions, not for discovery browsing.
A Rule of Thumb for Catalog Projects
If you are starting a data catalog project, begin with the decisions, not the metadata. Interview ten people across three roles. Ask them to describe the last time they needed to understand a dataset. Write down what they did, what they searched for, and whether they found what they needed.
If every answer is “I asked someone on Slack,” your catalog needs to be faster than Slack. That is the bar. If your catalog cannot answer the question faster than a colleague can respond in a channel, the catalog will not be used regardless of how complete its metadata is.
Build the minimum integration that clears that bar. Do not import every table in the warehouse. Do not configure every metadata field. Solve the top three decision types for the top five teams. Measure answer rate, not coverage. Expand only when the answer rate for existing decision types stays above eighty percent.
The catalog that sticks is the one that answers the questions people are already asking. Everything else is shelf-ware.
