Consolidating 47 data sources into one knowledge layer

Consolidating 47 data sources into one knowledge layer

Simor Consulting | 01 Jul, 2026 | 05 Mins read

A global professional services firm with 8,000 consultants maintained institutional knowledge across forty-seven separate systems. Project proposals lived in a document management system. Client engagement histories lived in a CRM. Technical methodologies lived in a wiki. Subject matter expertise lived in internal blogs and mailing list archives. Lessons learned lived in post-engagement reviews stored on a shared drive that predated the current document management system.

When a partner prepared a proposal for a new engagement, they needed to answer three questions: have we done similar work before, who in the firm has relevant expertise, and what methodologies and templates are available. Answering these questions required searching across multiple systems with different search interfaces, different metadata schemas, and different access controls. The average time to assemble a proposal was three weeks, and partners estimated that sixty percent of that time was spent searching for existing knowledge rather than creating new content.

The firm had tried to solve this problem three times. Each attempt was a migration project — move everything into a new system that provided unified search. Each migration failed because the scope of consolidation was overwhelming and the disruption to ongoing work was unacceptable.

Why migrations failed

The forty-seven systems were owned by different groups with different workflows. The CRM was owned by the sales organization and was optimized for pipeline management, not knowledge retrieval. The wiki was owned by the methodology group and was optimized for process documentation, not project history. The document management system was owned by IT and was optimized for compliance and retention, not search.

Each system had been configured to serve its primary users. Forcing all forty-seven systems into a single platform required either accepting the lowest common denominator of functionality — which degraded the experience for every primary user — or building a platform sophisticated enough to replace all forty-seven systems, which was a multi-year, multi-million-dollar project that the firm had already attempted and abandoned three times.

The migration approach also failed because of data semantics. A “client” in the CRM was defined by a sales relationship. A “client” in the document management system was defined by an engagement contract. A “client” in the wiki was defined by an industry classification. These were not the same entity, even when they referred to the same organization. Unifying them required a semantic reconciliation layer that understood the different definitions and could map between them.

The approach: knowledge layer without migration

We built a knowledge layer that sat on top of the existing systems without replacing any of them. The knowledge layer indexed content from all forty-seven systems, reconciled the semantic differences, and provided a single search interface. The original systems continued to serve their primary users with their existing workflows. The knowledge layer served the cross-cutting need: find relevant knowledge regardless of which system it lives in.

This diagram requires JavaScript.

Enable JavaScript in your browser to use this feature.

The connector layer was the most critical component. Each of the forty-seven systems had a different interface — REST API, database view, file system export, webhook, or screen scraping for the oldest systems. The connectors were lightweight adapters that extracted content and metadata from each system and normalized it into a common ingestion format. The connectors ran on different schedules depending on the source system’s change frequency. The CRM connector ran every fifteen minutes. The shared drive connector ran nightly. The wiki connector ran on webhook notification.

The entity resolution layer reconciled the semantic differences between systems. A “client” entity in the knowledge graph was constructed by merging the CRM’s sales relationship, the document management system’s engagement contract, and the wiki’s industry classification into a single entity with multiple facets. When a partner searched for “Acme Corp engagements,” the knowledge layer returned results from all three systems, unified under a single client identity.

The knowledge graph connected entities across systems. A person entity linked to project entities they had staffed. A project entity linked to methodology entities that had been used. A methodology entity linked to the wiki pages that documented it and the templates that implemented it. This graph structure enabled queries that were impossible in any individual system: “show me consultants who have used the agile delivery methodology on financial services engagements in the last two years.”

What we gave up

The knowledge layer was eventually consistent, not immediately consistent. Content updates in source systems appeared in the knowledge layer after the next connector run. For the CRM, this was fifteen minutes. For the shared drives, this was up to twenty-four hours. Partners who needed immediate search results for newly created content had to wait for the next sync cycle or search the source system directly.

The second trade-off was search precision. The knowledge layer indexed content across systems with different metadata quality. The CRM had structured, validated metadata. The shared drives had almost no metadata — file names and folder paths were the only organizational structure. The knowledge layer’s search relevance was limited by the lowest-quality metadata in the system. Full-text search mitigated this but did not eliminate it.

The third trade-off was access control. The original systems had their own access control models. The knowledge layer had to enforce the most restrictive policy for any piece of content. If a document was visible to only senior managers in the document management system, the knowledge layer could not show it to associates, even if the CRM data about the same client was visible to everyone. The access control reconciliation was complex and required ongoing maintenance as permissions changed in source systems.

Results

Proposal preparation time dropped from three weeks to one week on average. Sixty percent of the time savings came from the unified search interface — partners could find relevant past engagements, available methodologies, and subject matter experts in a single search rather than querying multiple systems. The remaining forty percent of time savings came from the proposal assistant, which used the knowledge graph to suggest relevant content for new proposals based on client, industry, and engagement type.

Expertise discovery was the most adopted feature. Before the knowledge layer, finding a consultant with specific expertise required asking around — sending emails, calling practice leaders, checking project staffing records. The expertise finder used the knowledge graph to identify consultants based on their engagement history, methodology usage, and published content. Within six months, seventy-two percent of internal staffing requests were initiated through the expertise finder.

The decision heuristic

When your knowledge lives in many systems and your users need to search across all of them, do not migrate. Migrate only when a single system can serve every primary user better than their current system. If no such system exists, build a knowledge layer that indexes and reconciles content from existing systems without displacing them. The migration approach consolidates systems. The knowledge layer approach consolidates search. The second is faster to deliver, less disruptive to adopt, and more tolerant of the semantic differences that make migrations fail.

Shipping a production AI system?

Find the control gaps before they turn into incidents. Take the AI Production Scorecard for a fast baseline across the seven layers, or book an architecture review and we will turn it into a hardening plan.

Similar Articles

How a retailer reduced inference latency 90% with feature store caching
How a retailer reduced inference latency 90% with feature store caching
21 Apr, 2026 | 04 Mins read

A mid-market e-commerce retailer with roughly $200M in annual revenue had invested eighteen months building a product recommendation engine. The models were accurate. Offline evaluation showed meaning

The data pipeline that cost $50K/month — and the audit that found why
The data pipeline that cost $50K/month — and the audit that found why
22 Apr, 2026 | 04 Mins read

A financial services firm running analytics on trade settlement data came to us with a specific complaint: their cloud data platform cost had tripled in eighteen months, and nobody could explain why.

Migrating from batch to streaming: a 6-month journey
Migrating from batch to streaming: a 6-month journey
28 Apr, 2026 | 05 Mins read

A logistics company processing two million shipments per day ran their entire operational reporting stack on nightly batch ETL. Every morning at 6 AM, operations managers reviewed dashboards built on

When RAG failed: a knowledge retrieval project post-mortem
When RAG failed: a knowledge retrieval project post-mortem
29 Apr, 2026 | 05 Mins read

A legal technology company had invested six months building a retrieval-augmented generation system to help contract attorneys find relevant precedent clauses across a corpus of 180,000 executed agree

From 3-hour dashboards to 3-minute insights: a BI modernization story
From 3-hour dashboards to 3-minute insights: a BI modernization story
05 May, 2026 | 05 Mins read

A manufacturing company with facilities in twelve countries ran its operational reporting on a traditional BI stack: a data warehouse, an ETL pipeline, and a dashboard tool that had been deployed six

The vector database that couldn't scale — and what we did instead
The vector database that couldn't scale — and what we did instead
12 May, 2026 | 05 Mins read

A media company with a library of twelve million articles, transcripts, and research documents had built a semantic search system on a managed vector database. The system was designed to let journalis

Building an AI operating system for a 10,000-person company
Building an AI operating system for a 10,000-person company
19 May, 2026 | 05 Mins read

A diversified industrial company with 10,000 employees across manufacturing, logistics, and field services had accumulated forty-seven separate AI projects over three years. Each business unit had bui

How we killed our ETL pipeline (and productivity went up)
How we killed our ETL pipeline (and productivity went up)
26 May, 2026 | 05 Mins read

A B2B SaaS company running a customer success platform had a data pipeline that consumed sixty percent of the data engineering team's time. Not feature work. Not analytics. Pipeline maintenance. The p

A compliance-first AI rollout in financial services
A compliance-first AI rollout in financial services
03 Jun, 2026 | 05 Mins read

A regional bank with $12 billion in assets wanted to use machine learning to improve its commercial loan underwriting process. The existing process was manual, relying on credit analysts who spent fou

The $2M model that never made it to production
The $2M model that never made it to production
09 Jun, 2026 | 05 Mins read

A retail chain with 400 stores spent two years and $2.1 million building an inventory optimization model. The model was technically excellent. It reduced predicted stockouts by thirty-two percent and

Data mesh in practice: year 2 retrospective
Data mesh in practice: year 2 retrospective
16 Jun, 2026 | 05 Mins read

An insurance company with $400 million in premium volume adopted data mesh two years ago. The central data team had become a bottleneck. Every business unit — claims, underwriting, actuarial, and dist

When your AI vendor goes bankrupt — surviving platform lock-in
When your AI vendor goes bankrupt — surviving platform lock-in
23 Jun, 2026 | 05 Mins read

A healthcare analytics company received notice on a Tuesday afternoon that their primary AI infrastructure vendor was filing for Chapter 7 bankruptcy. The platform hosted their patient risk stratifica

Real-time fraud detection: from proof-of-concept to production in 90 days
Real-time fraud detection: from proof-of-concept to production in 90 days
30 Jun, 2026 | 05 Mins read

A payment processor handling twelve million transactions per day had a fraud detection system that was accurate but slow. The system reviewed transactions in batch, four times per day. A fraudulent tr

The GDPR audit that reshaped our entire ML pipeline
The GDPR audit that reshaped our entire ML pipeline
07 Jul, 2026 | 05 Mins read

A European fintech with twelve million customers received a GDPR audit notice from their national data protection authority. The audit focused on the company's machine learning pipeline, which powered

Case Study: End-to-End RAG Platform for Customer Support
Case Study: End-to-End RAG Platform for Customer Support
05 Dec, 2025 | 05 Mins read

A SaaS company with 200 support agents and 10,000+ knowledge base articles had an 18-hour average response time and 23% first-contact resolution. Their largest enterprise client threatened to cancel a

Case Study: Building a Production AI Knowledge Layer for Financial Services
Case Study: Building a Production AI Knowledge Layer for Financial Services
01 Mar, 2026 | 10 Mins read

A regional bank's investment research team spent 60% of their time gathering information and 40% doing analysis. Analysts had to search through regulatory filings, internal research memos, market data

Designing the Enterprise Knowledge Layer: Beyond RAG
Designing the Enterprise Knowledge Layer: Beyond RAG
16 Jan, 2026 | 14 Mins read

Most teams implement retrieval-augmented generation and call it a knowledge layer. Give the model access to a vector database, stuff in some documents, and ship. This approach works for demos. It fall

Case Study: Multi-Agent System for Supply Chain Optimization
Case Study: Multi-Agent System for Supply Chain Optimization
13 Jun, 2026 | 12 Mins read

A mid-size automotive parts manufacturer with operations spanning 15 countries and relationships with over 200 suppliers faced a supply chain coordination problem that was consuming too much of their

Knowledge Graphs and Vector Search: Complementary, Not Competitive
Knowledge Graphs and Vector Search: Complementary, Not Competitive
19 Apr, 2026 | 11 Mins read

The framing of knowledge graphs versus vector databases as competing technologies is a symptom of hype cycles that simplify complex architectural decisions for public discourse. Practitioners argue ab