A city planner designs a metropolitan area. The planner does not assign every building individually. The planner establishes districts: a commercial district, a residential district, an industrial district, a green belt. Buildings go in zones. The zones define what can happen where. The city functions because the layout makes functional sense.
AI architecture does the same job at the system level. You establish districts for different cognitive functions: retrieval here, generation there, validation in this zone, memory in that one. The districts define the boundaries of each component. The overall system functions because the layout makes functional sense.
What the Districts Are
A retrieval district handles finding relevant information. A reasoning district handles drawing conclusions from information. A generation district handles producing output. A validation district checks outputs against rules. A memory district stores and retrieves state across interactions.
These districts are not physical locations. They are functional components that may be implemented as model calls, as databases, as rule engines, or as combinations. The key is that each component has a distinct responsibility and a defined interface.
The boundaries between districts define how information flows. A tightly coupled architecture has information moving freely between components. A loosely coupled architecture has well-defined interfaces between districts. Loose coupling makes components easier to test, replace, and reason about individually. Tight coupling can be more efficient but creates entanglement.
Consider a customer service system. The retrieval district finds relevant policy documents. The reasoning district applies those documents to the customer’s question. The generation district produces the response. The validation district checks the response against tone guidelines and disallowed content. Each district is a distinct step with distinct inputs and outputs.
The Integration Problem
Districts are only useful if the connections between them work. A city with beautiful residential neighborhoods that are unreachable from the commercial district is not a functioning city. An AI system with well-designed components that do not integrate cleanly is not a useful system.
Integration costs are often underestimated. Designing a component is relatively cheap. Designing the interfaces between components, testing the interactions, and debugging when something goes wrong at a boundary: these costs compound.
A well-designed interface is explicit about what it accepts and what it produces. The retrieval district accepts a query and produces a list of documents. The reasoning district accepts documents and a query and produces a conclusion. Each interface is a contract that defines the boundaries.
When interfaces are implicit, debugging flows across boundaries in ways that are hard to trace. When a system produces a wrong answer, you need to know whether the error originated in retrieval (wrong documents found), in reasoning (right documents, wrong conclusion), or in generation (right conclusion, wrong expression). Explicit interfaces make this localisation possible.
Scaling Districts Independently
A city can expand one district without expanding others. A successful industrial zone does not require the residential district to grow. The zones scale somewhat independently.
AI architecture benefits from the same property. If your retrieval load grows faster than your generation load, you can scale the retrieval district independently. If your validation requirements increase, you can add validation capacity without rearchitecting generation.
This independence is only possible if the districts were designed with clear boundaries and well-defined interfaces. Districts that are entangled cannot be scaled independently. If your retrieval system and your reasoning system are one component, you cannot scale retrieval without scaling reasoning.
Scaling independence also affects cost structure. Different model providers may have different cost and performance characteristics for different tasks. If retrieval is a separate component from generation, you can use a retrieval-optimised model for retrieval and a generation-optimised model for generation. If they are entangled, you make a single choice that applies to both.
The Monolithic Temptation
A monolithic architecture, where everything is one component, is simpler to build initially. There are no interface boundaries to design and maintain. There are no network calls between components. There is one system to deploy and monitor.
The monolith is the right starting point when the problem is not well understood. If you are exploring a new domain and do not yet know what the components should be, premature separation creates interfaces that you will later regret. Build the monolith, understand the problem, then refactor into districts.
The risk is that the monolith becomes a habit. Systems that start as exploratory prototypes often grow into production systems without the architectural refactoring they need. The monolith that was appropriate for exploration becomes a constraint on scale, on team structure, and on iteration speed.
The transition from monolith to districts is painful. It requires identifying boundaries that were not explicitly designed, separating components that were entangled, and testing the new interfaces. It is much easier to design the districts upfront than to retrofit them later.
Team Structure and Architecture
AI architecture often mirrors team structure. If one team owns the entire system, a monolithic architecture may be fine. If multiple teams own different parts of the system, the architecture has to reflect the team boundaries or you get coordination overhead that slows everyone down.
Conway’s Law applies to AI systems: the structure of the system will reflect the communication structure of the organisation that builds it. If you want independent scaling districts, you need teams that can work independently on those districts. If one team is responsible for everything, the system will tend towards a monolith.
This is not an argument against good architecture. It is an argument for aligning team structure with architectural goals. If you want districts, build teams around districts.
Real-World Scenario: The E-Commerce Assistant
An e-commerce company builds an AI shopping assistant. The initial monolithic prototype works: one model call takes a user query and produces a product recommendation.
As the product matures, the team identifies distinct districts. Retrieval finds products from the catalogue. Reasoning applies user preferences and contextual constraints. Generation produces the response. Validation checks for disallowed content and brand safety. Memory stores conversation history for continuity.
The team assigns each district to a sub-team. The retrieval team optimises for catalogue search. The reasoning team optimises for preference matching. The generation team optimises for response quality. Each team works independently on their district, communicating through the defined interfaces.
When a catalogue update causes retrieval quality to drop, the retrieval team can debug their component without involving the generation team. When generation quality issues emerge, the generation team can A/B test changes without affecting retrieval.
Real-World Scenario: The Document Intelligence Platform
A legal tech company builds a document intelligence platform. The initial system is a single model that takes a document and produces an analysis.
As the company adds capabilities, the monolith becomes a constraint. Contract review requires different prompting than document classification. Summarisation requires different context management than entity extraction. The team cannot optimise one capability without risking others.
The team refactors into districts. Each capability becomes a separate component with a defined interface. The orchestration layer coordinates between them: given a document and a task, route to the appropriate component, collect results, compose into a final output.
The refactoring takes three months. At the end, each capability can be updated independently. The company can offer contract review as a standalone product and document classification as another. The architecture enables product flexibility that the monolith did not.
The Abstraction Leak Problem
No abstraction is perfect. At some point, the districts will leak across boundaries. A retrieval component will return results in a format slightly different from what the reasoning component expects. A generation component will produce outputs that the validation component cannot properly evaluate.
These leaks are where bugs live. They are also where performance problems hide. A retrieval component that is slightly slower than expected may cause the reasoning component to timeout before it has enough context to produce a good answer.
Leak management is ongoing work. As the system evolves, new leaks emerge. Old leaks that were acceptable become problematic as scale increases. The architecture has to include mechanisms for detecting and fixing leaks before they become production problems.
The Routing Layer Problem
When you have multiple districts, you need a routing layer to direct requests to the right district and compose results from multiple districts into a final output. The routing layer is itself a component that can fail.
Routing failures are architectural failures. If the routing layer does not know which district to send a request to, the system cannot function. If the routing logic has bugs, requests go to wrong districts or with wrong payloads.
The routing layer should be simple, well-tested, and monitored. Complex routing logic that tries to be clever about load balancing or fallback routing is a source of bugs. Start simple. Add complexity only when measurements show it is needed.
The Data Flow Architecture Problem
How data flows through the districts matters. In a pipeline architecture, data flows sequentially through each district: retrieval feeds reasoning, reasoning feeds generation. This is simple and easy to understand, but it is slow: each district waits for the previous one to finish.
In a parallel architecture, data flows through multiple districts simultaneously: retrieval happens in parallel with other work, generation happens while validation happens. This is faster but more complex: you have to manage dependencies and handle partial results.
In a streaming architecture, data flows as it becomes available: the first retrieved document is processed while subsequent documents are still being retrieved. This is the fastest but most complex: you have to handle partial outputs and decide when to proceed with incomplete data.
The right data flow architecture depends on your latency requirements. If latency matters, pipeline is too slow. If throughput matters, streaming may be necessary. If simplicity matters, pipeline may be the right starting point.
The Error Propagation Problem
When districts are connected, errors propagate. A retrieval error means the reasoning district has no documents to reason about. A reasoning error means the generation district has a flawed conclusion to express. A generation error means the validation district has to evaluate text that does not reflect the intended output.
Error propagation is not always bad. A retrieval district that returns “no documents found” is not an error; it is information. The reasoning district can respond to “no documents” by saying “I cannot answer without relevant documents.”
Error propagation becomes a problem when errors cascade: one error triggers another error, which triggers another. A validation error might cause generation to retry, which causes reasoning to recompute, which causes retrieval to run again. The cascade multiplies latency and cost.
Designing districts means designing error handling. Each district should have clear error semantics: what does it return on success, on partial success, on failure? Each calling district should know how to handle each case.
Real-World Scenario: The Healthcare AI System
A hospital deploys a clinical decision support system. The districts include: retrieval of relevant medical literature, reasoning over patient data and literature, generation of treatment recommendations, validation against clinical guidelines.
The system encounters a patient with a rare condition. The retrieval district searches the medical literature and finds limited results. The retrieval district flags this as a low-confidence retrieval. The reasoning district receives the low-confidence flag and adjusts its reasoning accordingly, noting that the evidence base is limited. The generation district produces a recommendation that explicitly notes the limited evidence. The validation district checks that the recommendation does not contradict any hard clinical rules.
The human clinician reviews the recommendation and the confidence flags. They understand that the recommendation is based on limited evidence and apply their own judgment. The system augmented the clinician’s decision-making without replacing it.
Real-World Scenario: The Multi-Modal Customer Service
A customer service platform handles text, image, and voice interactions. The districts include: modality conversion (voice to text, image to description), intent classification, retrieval of relevant responses, generation of responses, validation.
A customer sends a voice message about a defective product and includes a photo. The modality conversion district converts the voice to text and processes the image. The intent classification district identifies that this is a complaint about a defective product with visual evidence. The retrieval district finds relevant policy and previous interactions. The generation district produces a response. The validation district checks the response for tone and policy compliance.
The orchestration layer coordinates the flow. Modality conversion happens first. Intent classification waits for conversion to complete. Retrieval and generation happen in parallel with validation, which runs on the generation output.
The Versioning Problem Across Districts
Districts evolve at different rates. The retrieval district may use a new embedding model while the generation district is still on the old model version. The validation district may get updated guidelines that change what it considers acceptable.
When districts evolve independently, compatibility can break. A new retrieval model may produce embeddings in a different space, but the reasoning district was trained on the old embedding space. A new validation model may have different sensitivity, but the generation district was tuned to the old sensitivity.
Managing cross-district versioning requires coordination. Interface changes have to be backwards compatible or carefully migrated. Co-deployment periods where old and new versions run in parallel are often necessary.
Decision Rules
Design AI architecture around functional districts when:
- The system has multiple distinct responsibilities that can be separated
- Components may need to scale independently based on different load patterns
- Teams will own different components and need clear boundaries for ownership
- You have identified the interfaces between components and can design them explicitly
- You are building for longevity and expect the system to evolve
- You need to optimise different components for different objectives
- Error handling at boundaries is well-understood and designed
- Data flow architecture matches latency requirements
Accept a more monolithic architecture when:
- The system is simple enough that separation creates more overhead than it eliminates
- The interaction patterns between components are so tightly coupled that separation is not practical
- You are still exploring the problem space and the right boundaries are not yet clear
- The team is small and the overhead of multiple components exceeds the benefits
- The system will not need to scale or evolve significantly
- Cross-district compatibility can be maintained without dedicated versioning infrastructure
The city planner who zones everything commercial everywhere has not planned. The AI architect who puts everything in one component has not architected. The architect who designs districts but ignores the roads between them has designed a city that does not function.
