The Model Context Protocol gives AI agents a standardized way to discover and invoke external tools. In development, MCP works well with a local server running on localhost and a handful of tools. The developer configures the server, defines the tools, and the agent calls them. Everything runs in a single process or across a local network. Authentication is a shared secret or a localhost connection. Permissions are irrelevant because the developer controls everything.

In production, the operational surface expands dramatically. You need a registry of available MCP servers so agents can discover tools without manual configuration. You need an authentication flow that validates which agents can call which tools. You need a permission model that scopes what each tool call can do. You need an observability layer that tracks what happened across agent-tool interactions. You need deployment patterns that handle scaling, failover, and resource isolation.

Most MCP documentation covers the protocol basics: how to define tools, how to invoke them, how to handle responses. The production operationalization is left as an exercise. This article addresses the operational concerns that determine whether MCP works reliably at scale or becomes a source of incidents and security gaps.

The Registry Problem

In development, the agent knows which MCP servers are available because the developer configured them manually. The server address is hardcoded or stored in a local configuration file. The agent connects to it at startup and discovers its tools through the MCP protocol’s tool listing mechanism.

In production, agents need a way to discover available MCP servers and their capabilities without manual configuration. New MCP servers come online. Existing servers go offline for maintenance. Server endpoints change when deployments happen. Tool sets expand as teams add new capabilities. Hardcoded configurations cannot keep up with this dynamism.

A registry is a service that maintains a catalog of MCP servers, their endpoints, their available tools, and their health status. Agents query the registry to discover which tools are available and where to reach them. The registry handles service discovery in the same way that a service registry like Consul or etcd handles microservice discovery.

Static Registries

The simplest registry is a static configuration file that lists MCP servers and their tools. A YAML file maps server names to endpoints and tool lists. Agents load this file at startup and use it to route tool calls.

This works for a small number of servers where changes are infrequent. If you have five MCP servers that change monthly, a static file updated through your deployment pipeline is sufficient. The file is version-controlled, reviewed in pull requests, and deployed with the application.

The limitation is that static registries cannot handle dynamic changes. When a new MCP server comes online, the configuration file must be updated, reviewed, merged, and deployed before agents can use it. When a server goes offline, agents continue trying to call it until the configuration is updated. The lag between a server change and agent awareness can be minutes to hours, depending on your deployment pipeline.

Dynamic Registries

A dynamic registry provides an API for MCP servers to register and deregister. Servers register when they start by calling a registration endpoint with their address, tool list, and metadata. Servers deregister when they shut down by calling a deregistration endpoint. The registry performs health checks by periodically pinging registered servers and removing unhealthy ones from the catalog.

Agents query the registry at startup or periodically to refresh their view of available tools. The registry returns the current list of servers, their endpoints, and their tool sets. Agents update their routing tables without redeployment.

The trade-off is consistency. An agent’s cached view of available tools may be stale between refresh intervals. It may try to call a tool on a server that went offline thirty seconds ago. It may not know about a server that registered fifteen seconds ago. The registry must balance freshness with query cost.

Aggressive polling (every five seconds) gives fresher data but adds load to both the registry and the agents. Event-driven updates (the registry pushes changes to agents) give immediate freshness but require agents to maintain a persistent connection to the registry, which complicates the architecture. Lazy refresh (every five minutes) is cheap but may miss rapid changes during deployments or incidents.

The right refresh interval depends on your change frequency. If servers change rarely, lazy refresh is fine. If servers change frequently during deployments, aggressive polling or event-driven updates are necessary.

Registry Health Checks

The registry should not just track which servers have registered. It should verify that registered servers are actually healthy and responsive. A server that registered successfully but has since crashed will appear in the registry until its TTL expires or the health check detects the failure.

Health checks can be passive or active. Passive health checks monitor the success rate of tool calls routed through the server. If calls start failing, the registry marks the server as unhealthy. This catches failures only after they affect real traffic. Active health checks periodically ping the server with a lightweight request (like listing tools) and mark it as unhealthy if the ping fails. This catches failures before they affect traffic but adds load to the server.

Use both. Active health checks catch server crashes and network partitions. Passive health checks catch functional degradation where the server is running but producing errors.

Authentication Flows

MCP authentication must answer two questions: is this agent allowed to call this MCP server, and is this agent allowed to invoke this specific tool? The first question is gateway-level authentication. The second is tool-level authorization. Both must be answered before the tool call proceeds.

Gateway-Level Authentication

Gateway-level authentication validates the agent’s identity before allowing any tool calls. The agent presents credentials to the MCP gateway. The gateway validates the credentials against an identity provider and checks whether the agent is authorized to access the MCP server. If validation passes, the gateway forwards the request to the MCP server. If validation fails, the gateway rejects the request without contacting the server.

This is the same pattern as API gateway authentication in microservices. The gateway handles the authentication concern so individual MCP servers do not need to. The MCP server trusts that the gateway has validated the agent and proceeds with the tool call.

Token-based authentication is the most common approach. The agent obtains a token from an identity provider (OAuth2, JWT, or a custom token service) and presents it to the gateway. The gateway validates the token, extracts the agent identity and claims, and forwards the request with an internal identity header. The MCP server receives the identity header and uses it for tool-level authorization and audit logging.

The token lifecycle matters. Short-lived tokens (fifteen minutes to one hour) limit the window of exposure if a token is compromised. Token refresh mechanisms allow agents to obtain new tokens without re-authenticating from scratch. Token revocation allows immediate termination of access when an agent is decommissioned or compromised.

Revocation is the hardest part. JWT tokens are self-validating, which means the gateway cannot revoke them before they expire without maintaining a revocation list. Short-lived tokens reduce the need for revocation because tokens expire quickly. For high-security environments, use opaque tokens that require a lookup against the identity provider on every request, or maintain a revocation list that the gateway checks before accepting a JWT.

Mutual TLS

For high-security environments, mutual TLS (mTLS) provides authentication at the transport layer. Both the agent and the MCP gateway present certificates. The gateway validates the agent’s certificate against a certificate authority. The agent validates the gateway’s certificate. This provides authentication and encryption in a single mechanism.

mTLS is more complex to manage than token-based auth because it requires certificate provisioning, rotation, and revocation. The operational overhead is justified in environments where the network is untrusted (multi-tenant infrastructure, public cloud) or where compliance requires transport-layer authentication.

Service Identity

Agents need identities that are independent of the user interacting with them. A customer-facing chatbot agent should have its own service identity, not the identity of the customer. The service identity determines what the agent can access, separate from what the user can access.

This separation is important because the agent may need access to systems that the user should not access directly. The agent might need to query an internal API that is not exposed to users. The agent’s service identity grants this access. The user’s identity is passed through for authorization decisions that need user context (like “can this user see this record”) but the agent’s identity is used for MCP gateway authentication.

Permission Models

Tool-level authorization determines what an agent can do once it has access to an MCP server. The permission model defines which tools each agent can invoke and what parameters each invocation can use.

Server-Level Permissions

The coarsest permission model is server-level: an agent either has access to all tools on an MCP server or has no access. This is simple to implement. The gateway checks whether the agent is authorized for the server. If yes, all tools are available. If no, none are.

This does not follow the principle of least privilege. An agent that needs read access to a CRM gets write and delete access as well, because the server does not distinguish between tools. If the CRM MCP server has get_customer, update_customer, and delete_customer tools, the agent with server-level access can call all three.

Server-level permissions are adequate when each MCP server has a narrow, homogeneous tool set. A server that only provides read-only search tools can safely use server-level permissions. A server that mixes read and write tools cannot.

Tool-Level Permissions

Tool-level permissions are finer-grained. Each tool declares its required permissions. The agent’s permission set is checked against the tool’s requirements before invocation. An agent with read-only permissions can call the “get customer” tool but not the “update customer” tool.

This requires each tool to declare its permission requirements accurately. The declaration is part of the tool definition: this tool requires “crm:read” permission, that tool requires “crm:write” permission. The gateway checks the agent’s permission set against the declared requirements.

Tool-level permissions are the right default for production MCP deployments. They follow the principle of least privilege, they are auditable (you can list exactly which tools each agent can access), and they catch permission drift when new tools are added (new tools must declare their permissions, and agents without the required permissions cannot access them).

Parameter-Level Permissions

Parameter-level permissions are the finest granularity. Not only can the agent call the “update customer” tool, but it can only update customers in a specific segment, or it can only update specific fields like address but not financial fields.

This requires the permission system to understand the semantic meaning of tool parameters. The gateway must inspect the tool call parameters and evaluate them against resource-level policies. This adds significant complexity to the permission evaluation logic.

Parameter-level permissions are justified for high-sensitivity operations where the blast radius of a wrong tool call is severe. Financial transactions, user data modifications in regulated industries, and infrastructure provisioning are candidates for parameter-level permissions. Most other operations are adequately served by tool-level permissions.

What Breaks at Scale

The operational concerns that are invisible at small scale become critical at large scale.

Connection Management

Each agent maintains connections to multiple MCP servers. At ten agents and five servers, that is fifty connections. Manageable. At a hundred agents and twenty servers, that is two thousand connections. At a thousand agents and fifty servers, that is fifty thousand connections. Without connection pooling and multiplexing, you exhaust file descriptors and network resources.

Connection pooling shares a set of connections across multiple tool calls from the same agent. Instead of opening a new connection for each tool call, the agent reuses an existing connection. This reduces the connection count from (calls x servers) to (agents x servers).

Connection multiplexing sends multiple tool calls over a single connection concurrently. This further reduces the connection count to (agents) if the MCP transport supports multiplexing. Not all MCP transports support this. HTTP/1.1 does not. HTTP/2 and WebSocket do.

Permission Evaluation Latency

Each tool call requires a permission check. At low volume, the check is fast (a few milliseconds for an in-memory policy evaluation). At high volume, if the permission check involves a database lookup or an external policy evaluation service, the latency accumulates. Ten thousand tool calls per second with a ten-millisecond permission check means a hundred seconds of cumulative permission evaluation time per second.

Caching permission decisions with appropriate TTL reduces the latency. Cache the permission result for each (agent, tool, action) tuple for five to fifteen minutes. The cache hit rate should be high because agents typically call the same tools repeatedly. The trade-off is that revoked permissions remain effective for the cache TTL. If a permission is revoked, the cache must be invalidated, not just allowed to expire.

Observability Overhead

At low volume, you can log every tool call with full detail. At high volume, the logging overhead becomes significant. Logging a full tool call (agent identity, tool name, parameters, response, latency) to a structured log system takes one to five milliseconds per call. At ten thousand calls per second, that is ten to fifty milliseconds of logging overhead per second, which may be acceptable. At a hundred thousand calls per second, the overhead becomes a bottleneck.

The solution is selective logging. Log all write operations at full detail because they affect state and need audit trails. Sample read operations at a configurable rate (one percent, ten percent, depending on your needs). Log all permission denials at full detail because they indicate potential security issues.

Configuration Sprawl

When MCP server configurations, tool definitions, and permission policies are stored in files, updating them across a distributed system is error-prone. Each deployment must include the updated configuration. Each agent must reload the configuration. If the configuration is inconsistent across agents, the same agent may have different permissions depending on which instance handles the request.

Moving to a centralized configuration service with versioning and rollback capability becomes necessary before the number of MCP servers exceeds what you can manage manually. The configuration service provides a single source of truth for server registrations, tool definitions, and permission policies. Agents subscribe to configuration changes and update their local state without redeployment.

Deployment Patterns

Three deployment patterns exist for MCP servers. Each has different latency, resource, and operational characteristics.

Sidecar Deployment

Sidecar deployment runs an MCP server alongside each agent instance. The agent and server share a local network, which minimizes network latency. Authentication is simplified because the agent and server are co-located. The server can access local resources that are not available over the network.

The downside is resource duplication. Each agent instance runs its own MCP server, which may be wasteful if the server is lightly used. If you have fifty agent instances and each runs its own CRM MCP server, you are running fifty CRM server processes even if each one handles only a few requests per minute.

Sidecar deployment is appropriate for latency-sensitive tools that benefit from local execution. Code analysis, local file operations, and in-memory data processing are good candidates for sidecar deployment.

Gateway Deployment

Gateway deployment runs MCP servers behind a shared gateway. Agents connect to the gateway, which routes requests to the appropriate server. This enables resource sharing across agents and centralized authentication, authorization, and logging.

The downside is an additional network hop. Every tool call traverses the agent to the gateway to the MCP server and back. This adds latency, typically five to twenty milliseconds per hop depending on network topology. For most tool calls, this latency is acceptable. For latency-sensitive operations, it may not be.

The gateway is also a single point of failure. If the gateway goes down, all MCP tool calls fail. Gateway redundancy (multiple gateway instances behind a load balancer) mitigates this but adds operational complexity.

Gateway deployment is appropriate for shared tools that benefit from centralized management. CRM integrations, external API calls, and database queries are good candidates for gateway deployment.

Hybrid Deployment

Hybrid deployment uses sidecars for latency-sensitive tools and a gateway for shared tools. Code analysis runs as a sidecar. CRM integration runs behind a gateway. Each tool is deployed according to its latency and sharing requirements.

This is the most flexible pattern and the most complex to operate. You manage two deployment models, two authentication flows, and two observability pipelines. The complexity is justified when your tool set has diverse latency and sharing requirements. If all your tools have similar requirements, pick one pattern and use it consistently.

Decision Rules

Use a static registry when you have fewer than ten MCP servers and changes are infrequent. Use a dynamic registry when servers change more than weekly or you need automatic health checking. Use gateway-level authentication for all production deployments without exception. Use tool-level permissions when you have more than one agent accessing the same MCP server. Use parameter-level permissions only when the data sensitivity requires field-level access control.

Start with gateway deployment for all MCP servers. Move latency-sensitive tools to sidecars when you have measured the latency overhead and confirmed it is unacceptable. Use hybrid deployment only when you have both latency-sensitive and shared tools in the same system.

The protocol itself is simple. The production operationalization is where the complexity lives, and that complexity should be driven by actual operational requirements, not anticipated ones. A single MCP server with three tools behind a static configuration is a fine starting point. Scale the operationalization when the scale demands it, not before.

Ship it safely

If you’re hardening MCP registry, auth, and permissions for real users, our Model Gateway + MCP Control Plane Build covers it end to end. For a fast baseline across the seven control layers, take the AI Production Scorecard.