Traditional application observability focuses on three signals: request latency, error rates, and resource utilization. If the request returns a 200 in under two hundred milliseconds, the system is healthy. If it returns a 500, something broke. If CPU is at ninety percent, scale up. These signals work because traditional applications are deterministic. Same input, same output, same behavior.

AI systems break that assumption. A request can succeed with a 200 status code and still produce a hallucination, a refusal, or a response that is technically correct but practically useless. The latency can be within normal bounds and the output can be garbage. The model can return a confident, well-formatted answer that is completely wrong.

Logging the request and response is not enough. You need to understand what happened inside the model call, what context was provided, what tools were invoked, and why the model made the choices it made. This requires a different observability stack than the one you use for traditional web services.

What Standard Logging Misses

Standard request-response logging captures the input prompt and the output response. For a traditional API, this is sufficient. The input is a JSON body. The output is a JSON response. The relationship between them is deterministic and debuggable.

For an AI system, the input to the model is not the user’s message. It is a composite of the user’s message, the system prompt, retrieved context from a vector database, results from previous tool calls, conversation history, and any dynamic prompt augmentation the application performs. The output is not just the model’s text response. It is a sequence of tool calls, intermediate reasoning steps, and the final response.

Consider a RAG system that retrieves three documents, combines them with the user query, sends the augmented prompt to the model, and gets a response. If the response is wrong, you need to answer five questions. Which documents were retrieved? Were they the right documents? Did the model use the retrieved context correctly? Was the error in retrieval, augmentation, or generation? Would a different model have produced a better answer from the same context?

Standard logging shows the user query and the model response. It does not show the retrieval step, the relevance scores of retrieved documents, the augmentation step, the full prompt that reached the model, or the model’s intermediate reasoning. Without this information, debugging a bad response is guesswork. You look at the output, look at the input, and speculate about what went wrong in the middle.

The solution is structured tracing that captures each step in the AI pipeline as a span in a trace. The trace records the user query, the retrieval step with the documents retrieved and their relevance scores, the augmentation step with the combined prompt, the model call with the full prompt and parameters, tool invocations with their inputs and outputs, and the final response. Each span includes timing, cost, and quality signals.

Trace Architecture

A trace is a tree of spans. Each span represents a single operation: a retrieval call, a model inference, a tool invocation, a post-processing step. Parent-child relationships show the dependency chain. The root span is the user request. Child spans are the operations that the request triggers.

Each span captures four categories of data. Timing data records when the operation started, when it ended, and how long it took. This enables latency analysis at each step, not just the total request. Payload data records the inputs and outputs of the operation: the query sent to the retriever, the documents returned, the prompt sent to the model, the response generated. This enables debugging at each step. Cost data records the token counts for model calls, the compute time for retrieval operations, and the monetary cost when pricing information is available. Quality data records validation results: did the response match the expected format, was a refusal detected, did the output pass schema validation.

The trace must capture enough detail to reproduce the request later. This means storing the full prompt, the full model parameters (temperature, top_p, max_tokens), the full retrieval results, and the full tool call sequences. A trace that captures only summaries or truncated payloads is insufficient for replay and forensics.

The storage cost is significant. A single trace might include megabytes of context if it involves large document retrieval or multiple tool calls. A production system handling thousands of requests per day generates gigabytes of trace data daily. You need a storage strategy that balances replay capability with storage cost.

Trace Replay

Trace replay is the ability to re-execute a captured trace to reproduce a result or test a change. If a user reports that a response was wrong, you pull the trace, see exactly what happened, and replay it with a different model, different retrieval parameters, or a different prompt to see if the issue reproduces or resolves.

This is more powerful than traditional log analysis because you can not only see what happened but re-run the exact same scenario with modifications. Did the error come from the retrieval step? Replay the trace with different retrieval parameters and see if the documents improve. Did it come from the model? Replay with a different model and see if the response improves. Did it come from the prompt? Replay with a modified prompt and see if the output changes.

Replay requires a replay engine that can reconstruct the request from the trace data and send it through the pipeline with the specified modifications. The replay engine must support partial replay: replaying only the model call with different parameters while keeping the retrieval results the same, or replaying only the retrieval step with different parameters while keeping the model the same.

Partial replay is important because it isolates variables. If you change both the retrieval parameters and the model simultaneously and the response improves, you do not know which change caused the improvement. If you change only the retrieval parameters and the response improves, you know the retrieval was the problem.

Replay is also useful for regression testing. When you update a prompt template, replay a representative set of traces with the new template and compare the responses. If the new template produces worse responses for any trace, you have caught a regression before it reaches production.

The cost of replay is the cost of the model calls. Replaying a thousand traces against GPT-4 costs real money. Use replay selectively: on reported issues, on a sample of production traffic for ongoing quality monitoring, and on regression test suites before prompt or model changes.

Incident Forensics

When an AI system produces a bad output at scale, the investigation follows a different pattern than traditional application incidents. Traditional incidents are usually caused by infrastructure failures, code bugs, or configuration errors. The root cause is deterministic and reproducible. AI incidents can be caused by model behavior changes, prompt sensitivity, retrieval quality degradation, context window overflow, or temperature-induced variance. The root cause may not be deterministic.

The forensics process starts with the trace. Pull the traces for all affected requests. Compare them against traces for similar requests that produced correct outputs. Look for differences in retrieval results, prompt content, model parameters, or tool call sequences. The difference is usually visible in the trace once you know what to look for.

Retrieval Drift

A common pattern in AI incidents is retrieval drift. The retrieval system starts returning different documents for the same query, either because the document corpus changed (new documents added, old documents updated), the embedding model was updated (a newer embedding model produces different vectors for the same text), or the retrieval parameters were modified (top_k changed from five to three).

The model generates a response based on different context than before, and the response quality changes. Without tracing the retrieval step, this is invisible. The logs show the same query producing a different response, but they do not show why. The retrieval drift is hidden inside the pipeline.

Detecting retrieval drift requires comparing retrieval results across traces. If the same query produces different retrieval results at different times, drift has occurred. Automated drift detection monitors the overlap between retrieval results for repeated queries and alerts when the overlap drops below a threshold.

Prompt Template Drift

Another common pattern is prompt template drift. A prompt template is updated to fix one issue, and the change inadvertently affects a different use case. The template change might add a constraint that conflicts with a specific query pattern, causing the model to refuse or misinterpret those queries.

For example, a team adds a safety instruction to the system prompt: “Do not provide financial advice.” This fixes a compliance issue for one use case. But it causes the model to refuse legitimate queries in another use case that involves financial data analysis. The refusal rate for the second use case spikes, but the root cause is a prompt change intended for the first use case.

Trace comparison reveals the template change as the root cause. Compare traces from before and after the template change. The prompt span shows the difference. The response span shows the impact. The connection is visible in the trace in a way that is not visible in logs.

Model Behavior Changes

Model providers update their models without always announcing changes. A model that performed well yesterday might produce different responses today because the provider deployed a minor version update. These unannounced changes are a known problem in the industry and a significant source of AI incidents.

Detecting model behavior changes requires comparing response quality across time periods. If the refusal rate, hallucination rate, or format compliance rate changes without a corresponding change in your application code, the model behavior has changed. Trace comparison shows whether the change is in the prompt (your code changed), the retrieval (your data changed), or the model (the provider changed).

The defense is version pinning when available (request a specific model version rather than the latest), continuous quality monitoring (track quality signals over time and alert on degradation), and trace comparison (when quality drops, compare recent traces against baseline traces to identify the source of the change).

Cost Attribution

AI costs are harder to attribute than traditional infrastructure costs. A traditional API call has a fixed cost. The compute time is predictable. The database query cost is known. An AI model call has a variable cost based on input tokens, output tokens, and the specific model used. The same application feature might cost ten cents with one query and five dollars with another, depending on the prompt length, the context retrieved, the tools invoked, and the response generation.

Cost attribution requires tracking token counts and model pricing at the individual request level. Each trace should include the input token count, output token count, model used, and calculated cost. This data enables cost attribution by tenant, feature, use case, and time period.

The attribution becomes more complex with multi-step AI pipelines. A single user request might trigger a retrieval step (embedding model call), a first model call (with retrieved context), a tool invocation (which triggers another model call), and a second model call (incorporating tool results). The total cost of the request is the sum of all these steps. The attribution system must aggregate costs across steps and attribute them to the originating request and tenant.

Per-tenant cost attribution is essential for multi-tenant applications. If tenant A generates ten times the AI cost of tenant B, that should be reflected in pricing or usage limits. Without per-tenant attribution, you are either overcharging tenant B (subsidizing tenant A) or undercharging tenant A (losing money).

Per-feature cost attribution identifies which features are expensive and which are cheap. A feature that processes long documents might consume eighty percent of your AI budget. Knowing this lets you optimize the expensive feature (use a cheaper model, reduce context size, add caching) rather than applying cost reduction uniformly across all features.

Cost attribution data enables budget enforcement. Set daily or monthly cost limits per tenant or feature. When a limit is approached, the system can switch to a cheaper model, reduce the number of retrieval results, or refuse requests with a clear message. Without accurate cost attribution, budget enforcement is guesswork. You set a global budget and hope it is enough. When you exceed it, you do not know which tenant or feature caused the overage.

Quality Signals

Beyond cost and latency, AI observability needs quality signals. These are metrics that indicate whether the model output meets your quality bar. Quality signals are harder to define than latency or error rate because quality is subjective and use-case-dependent.

Structured output validation checks whether the model returned data in the expected format. If the model is expected to return JSON in a specific schema, validation checks whether the output conforms. This catches formatting failures but not semantic errors. A response can be valid JSON with wrong values.

Refusal detection checks whether the model declined to answer when it should have answered. Refusals are not always errors. Sometimes the model correctly refuses a harmful request. But an increase in refusal rate for legitimate requests indicates a problem, either in the prompt (the model is being told to refuse too aggressively) or in the model (the provider updated the safety filters).

Hallucination detection uses a secondary model or rule-based system to check whether the output contains claims not supported by the provided context. This is expensive (it requires another model call) and noisy (hallucination detectors produce false positives), but it catches the most dangerous failure mode: confident, plausible, wrong answers.

User feedback integration closes the loop. Users flag bad responses. The flags are correlated with traces to identify patterns. If a specific prompt template produces flagged responses at a higher rate, the template needs revision. If a specific model produces flagged responses more often, consider routing those use cases to a different model.

These signals are noisy individually. Structured output validation catches formatting issues but not semantic errors. Refusal detection catches explicit refusals but not subtle evasions. Hallucination detection catches obvious fabrications but not plausible-sounding inaccuracies. Use them as early warning signals, not as definitive quality assessments. Combine multiple signals to reduce noise. A response that fails structured output validation and is flagged by the user is almost certainly a real problem.

The Observability Stack

The practical AI observability stack has three layers.

The instrumentation layer captures traces from the application, the retrieval system, the model gateway, and the tool layer. OpenTelemetry with custom spans for AI-specific operations is a solid foundation. OpenTelemetry provides the trace structure, span management, and export pipeline. Custom spans capture AI-specific data: token counts, model parameters, retrieval scores, tool call results.

The storage layer stores traces with sufficient detail for replay and forensics. Hot storage (the last seven to thirty days) provides immediate access for debugging recent issues. Cold storage (thirty to ninety days) provides access for trend analysis and compliance. Beyond ninety days, traces are archived or deleted based on retention policy.

The analysis layer provides dashboards, alerting, and cost attribution. Dashboards show quality signals over time, cost breakdowns by tenant and feature, and latency distributions. Alerting fires when quality signals degrade, costs exceed budgets, or latency exceeds thresholds. Cost attribution reports show per-tenant, per-feature, and per-model spending.

Do not build all three layers at once. Start with instrumentation. Capture traces with enough detail to debug issues. Add storage with a retention policy that matches your investigation needs. Add analysis and alerting once you have enough trace data to identify patterns.

The mistake most teams make is building the analysis layer first, creating beautiful dashboards that show metrics they cannot drill into because the underlying trace data is insufficient. Start with the traces. The dashboards will build themselves once the data is available. A raw trace that captures the full request context is more valuable than a polished dashboard that shows aggregate metrics with no drill-down capability.

Start Before You Need It

The worst time to build observability is during an incident. If a model starts producing bad responses at scale and you have no traces, you cannot debug the issue. You see the bad responses in user reports. You check the logs and see normal request-response pairs. You have no visibility into what changed because you were not capturing the intermediate steps.

Build observability before you need it. Instrument your AI pipeline with tracing from the first deployment. Capture the full prompt, the retrieval results, the model parameters, and the response. Store traces with a reasonable retention policy. You do not need the analysis layer immediately. You need the data. The analysis layer is easy to add when you have the data. It is impossible to add when you do not.

The cost of tracing is low. The storage cost is manageable with tiered retention. The engineering cost of adding spans to your pipeline is a few days of work. The cost of not having traces during an incident is measured in hours of manual debugging, wrong root cause theories, and user-visible quality degradation that you cannot explain or fix quickly.

Start with the traces. Add the dashboards when patterns emerge. Add alerting when you know what to alert on. The traces are the foundation. Everything else builds on top of them.

Ship it safely

If you’re hardening traces and incident forensics for real users, our Fractional AI Reliability Partner covers it end to end. For a fast baseline across the seven control layers, take the AI Production Scorecard.