LLM Observability: What to Monitor in Production AI Systems

A production AI system can be fully up — every service healthy, every endpoint responding — while quietly giving wrong answers to half its users. Traditional application monitoring (uptime, error rates, response times) doesn't catch this, because the failure isn't a crash, it's a quality problem inside a response that returned successfully. LLM observability is the discipline of monitoring for that category of failure specifically.

This guide covers what to actually monitor in a production AI system beyond standard infrastructure metrics, how tracing works across a multi-step AI request, the difference between observability and evaluation, and how to build an alerting strategy for problems that don't look like outages.

Why LLM observability is a different discipline

Standard application performance monitoring answers questions like: is the service up, how fast does it respond, what's the error rate. These matter for AI systems too, but they don't answer the question that actually determines whether the system is working: was the answer any good?

An LLM call can return a 200 status code, respond in 400 milliseconds, and still be confidently, fluently wrong — retrieving the wrong context, hallucinating a detail, or answering a question the user didn't actually ask. None of that shows up as an error in traditional monitoring. This is why LLM observability needs its own layer, purpose-built for the specific ways these systems fail.

What to actually monitor

Signal	What it tells you
Token usage and cost	Cost per query, cost trends, and which query patterns are expensive
Latency by stage	Where time is actually spent — retrieval, reranking, generation — not just total response time
Retrieval logs	Exactly what context was retrieved for each query, essential for debugging bad answers
Groundedness / faithfulness	Whether the answer is actually supported by retrieved context or fabricated
Citation accuracy	Whether cited sources actually support the specific claim made
Error and fallback rates	How often the system fails to answer, times out, or falls back to a default response
User feedback	Direct signal — thumbs up/down, corrections, follow-up questions indicating confusion
Drift over time	Whether quality is degrading as underlying data, models or usage patterns change

Tracing a request end to end

A single AI response typically involves multiple steps — query processing, retrieval, reranking, prompt construction, generation, guardrail validation. When an answer is wrong, the question is which step introduced the problem, and answering that requires tracing the full path, not just logging the final input and output.

Query received

The original user question, with a unique trace ID

Query processing

Any rewriting or expansion applied, and why

Retrieval

Exactly which chunks were retrieved, with relevance scores

Reranking

How candidates were re-ordered before reaching the prompt

Prompt construction

The final assembled context sent to the model

Generation

The model's raw response, token count, and latency

Guardrail checks

What validation ran, and whether it passed or modified the output

Response delivered

The final answer, plus the full trace for later debugging

Observability vs. evaluation: different tools for different jobs

These two terms get used interchangeably, but they answer different questions and both are needed.

Evaluation is periodic, structured testing against a maintained set of representative questions — usually run before launch and again whenever the pipeline changes. It answers: "does this system perform acceptably against known cases?" This is covered in depth in our RAG development guide and enterprise RAG architecture guide.

Observability is continuous monitoring of real production traffic. It answers: "how is the system actually performing right now, on real questions we didn't anticipate?" Evaluation catches regressions you can predict and test for. Observability catches the failure modes you didn't think to test — which, in practice, is most of them.

Alerting strategy

Standard alerting (error rate spikes, latency thresholds) still applies, but AI systems need alerts tuned to quality signals too: a sudden drop in average groundedness score, a spike in negative user feedback, an increase in "I don't know" or fallback responses, or a cost spike suggesting the system is retrieving far more context than usual. These thresholds should be based on your own system's baseline behavior, established once you have enough production traffic to know what "normal" looks like — not a generic industry number.

Building this into an agent, not just a RAG pipeline

Everything above applies directly to AI agents as well — in fact, observability matters more for agents, since a multi-step agent has more places a failure can originate, and a wrong tool call can have real-world consequences beyond a bad chat response. See our AI agent development guide for how observability fits into the broader agent architecture, including audit logging for actions taken, not just answers given.