When web applications needed to talk to databases, the industry created ORMs and connection pools. When microservices needed to talk to each other, the industry created API gateways and service meshes. When applications need to talk to AI models, most teams are still making direct API calls from application code to model providers. The middleware layer that should sit between the application and the model is either absent or fragmented across ad-hoc utility functions.

The absence of AI middleware creates coupling. The application code contains provider-specific API calls, model-specific prompt formatting, and hardcoded model selections. Changing a provider requires changing application code. Changing a model requires changing application code. Adding caching requires changing application code. Each change is small in isolation but the cumulative effect is a codebase where AI model interaction is entangled with business logic.

This post defines what AI middleware provides, covers three implementation approaches (library, service, sidecar), and provides decision rules for when to build middleware versus when direct integration is sufficient.

The Problem That Middleware Solves

A startup builds an AI-powered document analysis tool. The prototype calls OpenAI’s API directly from the application code. The code looks like this:

import openai

def analyze_document(text):
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "Analyze this document..."},
            {"role": "user", "content": text}
        ]
    )
    return response.choices[0].message.content

This works. It is simple. It is also a trap.

Six months later, the startup has three applications calling OpenAI: the document analysis tool, a customer support chatbot, and a code review assistant. Each application has its own OpenAI integration. Each has its own prompt templates embedded as string literals. Each has its own error handling. None of them share caching. None of them share observability. None of them can fail over to Anthropic when OpenAI goes down.

When the CTO asks “how much are we spending on OpenAI across all three applications,” no one can answer quickly because each application tracks costs independently. When a prompt needs to be updated, the change must be made in three places. When OpenAI has an outage, all three applications go down because none of them have a fallback.

This is the state that middleware prevents. The middleware centralizes the model interaction so that provider changes, prompt changes, caching, and observability are managed in one place.

What Middleware Provides

AI middleware is the layer that abstracts the model interaction from the application. The application sends a structured request to the middleware. The middleware handles provider selection, prompt assembly, model invocation, response parsing, caching, and observability. The application receives a structured response without knowing which provider fulfilled it, which model generated it, or how many retries were needed.

The middleware provides four capabilities that belong outside application code: routing, prompt management, caching, and observability. Each capability can be implemented independently, and teams should add them in order of immediate need rather than building the complete middleware stack at once.

Routing

Routing selects the provider and model based on request characteristics and configured policies. The application specifies what it needs — a completion, an embedding, a classification — and the middleware decides which provider and model can best fulfill the request. The routing logic is centralized, so changing the routing policy does not require changing application code.

The routing policy can be simple (always use OpenAI for all requests) or complex (use Anthropic for long documents, OpenAI for code generation, Google for multimodal inputs, with automatic failover between providers). The complexity of the routing policy should match the complexity of the application’s needs.

A routing table maps request types to providers:

routing:
  document-analysis:
    primary: anthropic/claude-sonnet
    fallback: openai/gpt-4
  code-generation:
    primary: openai/gpt-4
    fallback: anthropic/claude-sonnet
  classification:
    primary: openai/gpt-4-mini
    fallback: google/gemini-flash

The application sends a request with type document-analysis. The middleware routes to Anthropic’s Claude Sonnet. If Anthropic is unavailable, it routes to OpenAI’s GPT-4. The application does not need to know which provider handled the request.

Prompt Management

Prompt management assembles the final prompt from templates, context, and conversation history. The application provides the user input and any domain-specific context. The middleware retrieves the appropriate prompt template, injects the context, manages conversation history within the context window, and produces the final prompt. Prompt changes are made in the middleware, not in application code.

The prompt template system should support:

Variable substitution. Templates include placeholders like {{user_input}} and {{retrieved_context}}. The middleware replaces these with the actual values from the request.

Conditional sections. Some prompt sections are included only when certain conditions are met. If the request includes retrieved context, the context section is included. If not, it is omitted.

Versioning. Each prompt template has a version. The middleware can serve different versions to different users or traffic segments. Rollback means pointing to the previous version.

Testing. The middleware runs test prompts against new template versions before deploying them. If the test results degrade, the deployment is blocked.

Caching

Caching stores responses for repeated or similar requests. If the same question is asked twice, the middleware returns the cached response instead of calling the model again. Semantic caching extends this to similar questions: if a question is semantically similar to a previously answered question, the cached response is returned. Caching reduces latency and cost, but the cache invalidation strategy must account for the fact that AI responses may legitimately differ for similar inputs.

Exact caching stores request-response pairs keyed by the request hash. If the same request comes in again, the cached response is returned. This is simple and deterministic but has low cache hit rates because exact request matches are rare.

Semantic caching computes an embedding of the incoming request and looks up cached responses for requests with similar embeddings. If the similarity exceeds a threshold (e.g., cosine similarity > 0.95), the cached response is returned. This increases cache hit rates but adds the cost of computing embeddings for every cache lookup.

The cache must account for non-determinism. Two identical requests may produce different responses from the same model (especially at temperature > 0). The cache should either use a deterministic decoding configuration (temperature 0) or accept that cached responses may differ from fresh responses. For most applications, the cost and latency savings of caching outweigh the non-determinism concern.

Cache invalidation is complex for AI responses. Unlike database queries where the same query always produces the same result given the same data, AI responses depend on the model version, the prompt template, and the conversation context. When any of these change, the cache should be invalidated. This requires versioned cache keys that include the model version, prompt template version, and relevant context hash.

Observability

Observability captures request and response details, latency, cost, and quality signals. The middleware instruments every model call with structured tracing that the application can query for debugging and monitoring. The observability is centralized, so adding a new provider or model does not require adding new instrumentation to application code.

The observability data includes:

Request metadata. Which application made the request, which request type, which user or tenant, when.

Model metadata. Which provider, which model, which prompt version, how many tokens in and out.

Performance metadata. Latency (time to first token, time to last token), cache hit or miss, retry count.

Cost metadata. Cost per request, cumulative cost per application, per tenant, per time period.

Quality metadata. Structured output compliance, refusal rate, user feedback signals (if available).

This centralized observability enables cross-application dashboards. The CTO can see total AI spend across all applications. The support team can trace a specific user’s requests across applications. The engineering team can compare latency and quality across providers.

Implementation Approaches

There are three implementation approaches: library-based, service-based, and sidecar-based. Each has trade-offs in complexity, latency, and operational overhead.

Library-Based Middleware

Library-based middleware is a shared library that the application imports. The library provides a client interface for model interactions. The application calls the library, which handles routing, prompt assembly, and invocation. This is the simplest approach and works well for monolithic applications or small teams.

from ai_middleware import ModelClient

client = ModelClient(config="middleware.yaml")
response = client.complete(
    request_type="document-analysis",
    user_input=text,
    context=retrieved_docs
)

The library handles provider selection, prompt assembly, caching, and observability. The application code is clean and provider-agnostic.

The limitation of library-based middleware is that it runs in the application process. Multiple applications need to each import and configure the library independently. Configuration changes require redeploying every application. Cross-application concerns like shared caching and centralized observability are harder to implement.

Library-based middleware is appropriate when: you have one or two applications, you do not need cross-application caching, and your team is small enough that configuration coordination is manageable.

Service-Based Middleware

Service-based middleware is a standalone service that applications call over the network. The application sends a request to the middleware service. The service handles routing, prompt assembly, invocation, and returns the response. This is the gateway pattern applied to model interactions.

┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
│  Application │────▶│  AI Middleware   │────▶│  OpenAI API  │
│              │◀────│  Service         │◀────│              │
└──────────────┘     │                  │     └──────────────┘
                     │  - Routing       │     ┌──────────────┐
                     │  - Prompts       │────▶│  Anthropic   │
                     │  - Cache         │◀────│  API         │
                     │  - Observability │     └──────────────┘
                     └──────────────────┘

Service-based middleware enables centralized configuration, shared caching, and unified observability. Configuration changes are deployed once to the middleware service. All applications benefit immediately. The tradeoff is the added network hop and the operational overhead of running an additional service.

The network hop adds latency. A typical model API call takes 1-10 seconds. The middleware hop adds 5-50 milliseconds. The added latency is usually negligible compared to the model inference time, but it matters for latency-sensitive applications like real-time chat.

Service-based middleware is appropriate when: you have three or more applications using AI models, you need cross-application caching, you need unified observability, or you need centralized prompt management.

Sidecar-Based Middleware

Sidecar-based middleware runs alongside each application instance as a sidecar container or process. The application calls the sidecar over localhost. The sidecar handles routing and invocation. This combines the low latency of library-based middleware with the centralized management of service-based middleware.

┌─────────────────────────────────────────────┐
│  Pod / VM                                   │
│  ┌──────────────┐  ┌──────────────────────┐ │
│  │  Application │◀▶│  AI Middleware       │ │
│  │              │  │  Sidecar             │ │
│  └──────────────┘  │  - Routing           │ │
│                    │  - Prompts           │ │
│                    │  - Local Cache       │ │
│                    └──────────┬───────────┘ │
│                               │             │
└───────────────────────────────┼─────────────┘
                                │
                                ▼
                    ┌──────────────────────┐
                    │  Config Service      │
                    │  Observability       │
                    │  Backend             │
                    └──────────────────────┘

The sidecar approach works well in Kubernetes environments where sidecar containers are a standard pattern. Each application pod includes the middleware sidecar. Configuration is managed centrally and distributed to sidecars through a configuration service. Observability data is collected by the sidecar and forwarded to a centralized backend.

The sidecar has a shared cache across applications running on the same host but not across hosts. For cross-host caching, the sidecar can connect to a shared Redis or Memcached instance.

Sidecar-based middleware is appropriate when: you run on Kubernetes, you want low latency without the operational complexity of a standalone service, and you need centralized configuration with local execution.

Caching Strategies in Detail

Caching is the middleware capability with the highest immediate ROI. Model calls are expensive and slow. Caching eliminates both the cost and the latency for repeated or similar requests.

Cache Key Design

The cache key should include all inputs that affect the model’s response. If the same question produces different answers depending on the system prompt, the cache key must include the system prompt version. If the same question produces different answers depending on conversation history, the cache key must include the conversation context hash.

A cache key structure:

{provider}:{model}:{prompt_version}:{context_hash}:{request_hash}

The context_hash is a hash of the conversation history and any retrieved context. The request_hash is a hash of the user’s input. If any component changes, the cache key changes and the old cached response is not returned.

Cache TTL

Cache TTL (time-to-live) determines how long a cached response is valid. For factual queries (e.g., “what is the capital of France”), the TTL can be long (days or weeks). For dynamic queries (e.g., “what is the current stock price”), the TTL should be short (minutes) or caching should be disabled.

The TTL should be configurable per request type. The routing configuration can specify TTL for each request type:

caching:
  document-analysis:
    ttl: 3600  # 1 hour
    strategy: semantic
  code-generation:
    ttl: 0  # no caching
  classification:
    ttl: 86400  # 24 hours
    strategy: exact

Cache Warming

Cache warming pre-populates the cache with responses to common queries. Run common queries through the model and store the responses. When a user asks a common query, the cached response is returned immediately without calling the model.

Cache warming is useful for applications with predictable query patterns. A customer support chatbot receives many similar questions. Warming the cache with responses to the top 100 questions ensures that most users get cached responses.

When to Build Middleware

Build middleware when you have more than one application using AI models. The shared middleware eliminates duplicated routing, caching, and observability logic across applications.

Build middleware when you need provider failover. The middleware centralizes the failover logic so individual applications do not need to handle it.

Build middleware when prompt changes require application deployments. If changing a prompt means deploying a new application version, the coupling is too tight. Middleware decouples prompt management from application deployment.

Build middleware when you cannot answer “how much are we spending on AI across all applications” within five minutes. If the answer requires checking three different dashboards and adding up numbers manually, you need centralized cost tracking through middleware.

When Not to Build Middleware

Do not build middleware for a single application with a single provider and no plans to add either. The abstraction adds complexity without delivering value until there is something to abstract over.

Do not build middleware if your AI usage is experimental and may be discontinued. The middleware investment only pays off if the AI usage is sustained and growing.

Do not build middleware if you are a team of one or two and the applications are simple. The coordination overhead of middleware exceeds the benefit at small scale.

The decision rule: if your application directly imports an OpenAI SDK and makes API calls, and you have one application and one provider, the direct integration is fine. When you add a second application or a second provider, the middleware investment pays for itself. When you cannot answer the cost question across applications, the middleware is overdue.

Getting Started

Start with library-based middleware. Extract the model interaction code from your application into a shared library. Add routing (even if it is just a single provider). Add caching (exact match is enough to start). Add observability (structured logging of request metadata, latency, and cost).

Once the library works for one application, extend it to a second application. If the second application’s needs are similar, the library approach continues to work. If the second application’s needs force cross-application concerns (shared cache, centralized config), upgrade to service-based or sidecar-based middleware.

The middleware does not need to be complete on day one. Start with routing and caching. Add prompt management when you have more than five prompt templates. Add semantic caching when exact-match cache hit rates are below 20%. Add failover when you have a second provider. Each capability is independently valuable.

Ship it safely

If you’re hardening the middleware control layer for real users, our Model Gateway + MCP Control Plane Build covers it end to end. For a fast baseline across the seven control layers, take the AI Production Scorecard.