A production AI application calls multiple LLM providers. The primary model is GPT-4o for complex reasoning, but simple classification tasks use Claude Haiku for cost savings, and the fallback for rate-limit errors is Gemini Flash. Each provider has its own API format, its own rate limits, its own pricing, and its own failure modes. Managing this directly in application code means your business logic is tangled with provider-specific retry logic, token counting, and cost tracking.
An LLM gateway sits between your application and the providers. It normalizes API formats, routes requests based on rules you define, tracks costs and latency, handles retries and fallbacks, and gives you a single integration point for observability. Three tools compete in this space: LiteLLM, Portkey, and Martian. They share the gateway concept but differ in scope, pricing model, and operational approach.
TL;DR — which one for your situation
- You want open-source, self-hosted control over routing → LiteLLM (100+ providers, rule-based routing/fallbacks, free).
- You want a managed all-in-one with strong observability + prompt management → Portkey (dashboard, analytics, caching).
- You want automatic, cost-optimized routing → Martian (intelligent per-request routing) — but see the status note below: Martian was acquired by Notion (2024); validate standalone availability or consider OpenRouter / Cloudflare AI Gateway.
- Most production teams: start with LiteLLM as the routing core (ownership + control), add a dedicated observability layer, and only move to a managed gateway when the dashboard matters more than the control.
The rest of this post explains the tradeoffs behind those calls.
LiteLLM: the open-source default
LiteLLM is an open-source proxy that translates between a unified API format and provider-specific formats. You call LiteLLM with a standard OpenAI-compatible request, and LiteLLM routes it to the appropriate provider. It supports over 100 LLM providers and handles the format translation transparently.
The strength of LiteLLM is flexibility. You can self-host it, modify it, and integrate it into any infrastructure. Its configuration-based routing lets you define fallback chains, load balancing across providers, and rate limiting per API key. The OpenAI-compatible interface means any code that works with OpenAI’s API works with LiteLLM against any provider.
LiteLLM’s cost tracking is functional: it logs token counts and estimated costs per request and per API key. It integrates with logging backends (Langfuse, Helicone, custom webhooks) for observability. The tracking is accurate for standard completions but can be imprecise for streaming responses, tool calls, and multi-turn conversations where token counting is provider-dependent.
The limitation is that LiteLLM’s routing is rule-based, not intelligent. You define fallback chains statically: try GPT-4o, if it fails try Claude Sonnet, if it fails try Gemini Flash. There is no automatic routing based on request complexity, latency targets, or cost optimization. For most applications, static routing is sufficient. For applications that want to route a simple classification to a cheap model and a complex reasoning task to an expensive model automatically, LiteLLM requires you to implement that logic in your application before the request reaches the gateway.
Portkey: the managed gateway
Portkey is a managed LLM gateway with a dashboard, analytics, and team management features. It provides the same core gateway functionality as LiteLLM — API normalization, routing, fallbacks, caching — but adds a managed service layer on top.
Portkey’s strongest feature is its observability dashboard. Every request is logged with latency, token counts, cost, model used, cache hit status, and the full request/response payload (configurable). You can filter by API key, model, time range, and status code. For teams that need to understand their LLM usage patterns, Portkey’s dashboard provides this without building a custom analytics pipeline.
Portkey also offers prompt management: versioned prompt templates that can be updated without deploying code. This is useful for teams that iterate on prompts frequently and want non-developers to be able to make prompt changes. The prompt management is basic compared to dedicated prompt management tools, but as a gateway feature it is a convenience that reduces deployment friction.
The limitation is vendor dependency. Portkey is a managed service: your LLM traffic flows through their infrastructure. For most organizations this is acceptable, but for regulated industries or organizations with data residency requirements, routing all LLM traffic through a third-party proxy introduces compliance questions. Portkey offers a self-hosted deployment option, but the self-hosted version has fewer features than the managed version.
Pricing is per-request with a free tier. At moderate volumes (100,000 requests per month), the cost is negligible compared to the LLM costs themselves. At high volumes (10 million requests per month), the per-request fee adds up and you should calculate whether the observability value exceeds the gateway cost.
Martian: the intelligent router
Martian takes a different approach to the gateway concept: instead of rule-based routing, it uses a model router that predicts which provider and model will best handle each request. Given a prompt, Martian estimates the quality, latency, and cost of each available model for that specific prompt and routes accordingly.
The concept is appealing: automatic cost optimization without manual rule configuration. If Martian determines that a simple factual question can be answered equally well by a cheaper model, it routes there. If it determines that a complex reasoning question needs a more capable model, it routes to GPT-4o or Claude Opus. The routing decisions are made per-request based on the prompt content.
The reality is that Martian’s routing accuracy depends on the quality of its prediction model. For clear-cut cases (trivial questions vs. complex analysis), the routing works well. For ambiguous cases (questions that look simple but require deep domain knowledge), the router may under-provision model capability and produce lower-quality results. The risk of quality degradation from incorrect routing is the central trade-off.
Martian also provides cost tracking and observability, though its dashboard is less mature than Portkey’s. The primary value proposition is cost reduction through intelligent routing, not observability. If your main goal is understanding LLM usage patterns, Portkey is stronger. If your main goal is reducing LLM spend without manual routing rules, Martian is worth evaluating.
Status note (2026): Martian was acquired by Notion in 2024. Its standalone model router may no longer be actively marketed as an independent product — verify current availability before committing. For intelligent / cost-optimized routing today, OpenRouter (a broad provider marketplace with pay-as-you-go routing) and Cloudflare AI Gateway (edge caching, rate limiting, logging) are the current alternatives worth evaluating alongside LiteLLM and Portkey.
Caching: the unsexy cost saver
All three gateways support response caching: if the same prompt is sent twice, the cached response is returned without calling the LLM. This is the single most effective cost reduction technique for applications with repeated queries.
LiteLLM supports simple exact-match caching with configurable TTL. Portkey supports both exact-match and semantic caching (similar prompts return cached responses). Martian’s caching is integrated with its routing layer.
Semantic caching is powerful but risky. If two prompts are similar but not identical, the cached response may be subtly wrong for the second prompt. For applications where correctness matters (customer support, legal analysis, medical queries), semantic caching should be disabled or configured with high similarity thresholds. For applications where approximate responses are acceptable (search suggestions, content tagging), semantic caching can reduce costs by 20-40%.
Integration and deployment
LiteLLM: deploy as a Docker container or Python process. Configure via YAML or environment variables. Integrate by changing your OpenAI base URL to point at LiteLLM. Deployment time: one to two hours.
Portkey: sign up, generate an API key, change your OpenAI base URL to Portkey’s endpoint. Add your provider API keys to the Portkey dashboard. Deployment time: fifteen minutes.
Martian: sign up, configure your provider API keys, change your base URL. Deployment time: fifteen minutes.
All three support the OpenAI SDK, which means migration between them (or away from a gateway entirely) is a base URL change. This low switching cost is the best architectural property of the current LLM gateway market.
Decision framework
Use LiteLLM when you want an open-source, self-hosted gateway with full control over routing logic, when your organization has security requirements that prohibit managed proxy services, or when you need to customize the gateway behavior beyond what managed services offer. LiteLLM is the right choice for teams with infrastructure engineering capacity.
Use Portkey when you want the best observability for your LLM usage, when you need prompt management integrated with routing, when you want a managed service with minimal setup, or when your team needs a dashboard that non-engineers can use to monitor LLM costs and performance.
Use Martian when your primary goal is cost reduction through intelligent routing, when you have diverse LLM workloads with varying complexity, and when you are willing to accept the trade-off of automated routing decisions that may occasionally under-provision model capability for ambiguous requests.
The practical starting point: begin with LiteLLM for its flexibility and zero cost. If you find that you need better observability than LiteLLM provides out of the box, evaluate Portkey. If cost optimization becomes your primary concern and your workloads have clear complexity variance, evaluate Martian.
What we’d actually pick for production, multi-user systems
For a customer-facing or multi-tenant AI product, the gateway is the single most important control boundary. The deciding factors are different from a prototype:
- Put the gateway in front of every model call. One chokepoint for keys, policy, fallback, and tracing — never call providers directly from app code.
- Per-tenant secrets and spend limits. Each tenant gets scoped keys and a hard budget ceiling so one tenant’s loop can’t sink the bill (layer L4, budget governance).
- Fallback chains across providers (L1, model control). Survive a provider outage or model deprecation — your primary model will change; plan for it.
- Semantic caching cuts cost and latency on repeated queries — but disable it (or set a high similarity threshold) where correctness matters.
- Emit traces to your observability stack (L6). The gateway is the natural trace boundary; if it isn’t logging, you can’t debug production.
Our default for a team shipping a multi-user AI product: LiteLLM as the routing core (open-source ownership) with a dedicated observability layer, or Portkey if you want routing + guardrails + observability as one managed product. If you want this designed and built for your system, our Model Gateway + MCP Control Plane Build covers model routing, tool auth, permissions, budget limits, and auditability; for a quick self-assess, take the AI Production Scorecard (L1, L4, L5).
