LLM Evaluation Benchmarks: What They Measure and What They Miss

Leaderboard scores answer a single question: how did this base model perform on a fixed task set under controlled conditions? That answer helps narrow your model shortlist. It says very little once you add a retrieval layer, tool calls, a system prompt, user-specific context, and the actual workflow your agent has to complete.

MMLU (Massive Multitask Language Understanding), HumanEval, GSM8K (Grade School Math 8K), HellaSwag, and MT-Bench (Multi-Turn Benchmark) were designed to compare base models in isolation. They score knowledge breadth, code generation, math reasoning, commonsense completion, and dialogue quality. None of them factor in your data, tools, latency constraints, or the failure modes your users will run into. A model can top every benchmark on this list and still fail the task your agent was built to do.

As teams move from model selection to production, benchmarks are useful for narrowing the field. But to know whether the system is ready to ship, you need application-specific evals, golden datasets, and online evaluation that reflect your real users, data, and workflows.

What public LLM benchmarks measure

An LLM evaluation benchmark is a fixed task set with a scoring method that lets researchers and builders compare models on the same prompts. Some are multiple choice, some require code generation or math reasoning, and others use a judge model to grade open-ended answers. The benefit is comparability: when two models are tested on the same questions under the same conditions, their scores reflect meaningful differences in performance.

That comparability has two systemic limits worth understanding before reading any leaderboard.

• Evaluation protocol: The same model can score differently on the same benchmark depending on whether it is tested zero-shot or few-shot. Most leaderboards report one or the other, and not always consistently, which makes direct score comparisons unreliable without checking the methodology.
• Dataset contamination: Models trained on internet-scale data may have encountered benchmark questions during pre-training. Research shows substantial contamination in two large training corpora studied (The Pile and Llama 1's pre-training data), affecting HumanEval, HellaSwag, and MMLU. Contamination is difficult to detect and rarely disclosed, so leaderboard scores read better as upper bounds on capability than clean measurements of it.

Each of the five benchmarks below is affected by both limits, and each introduces additional caveats of its own.

MMLU tests breadth, not your domain

MMLU measures performance across 57 subjects, including mathematics, history, law, medicine, and computer science. The original paper collected 15,908 multiple-choice questions split across few-shot development, validation, and test sets. A strong MMLU score suggests the model can retrieve or reason over many kinds of textbook-style knowledge, which is why it became a convenient shorthand for general model capability.

The scoring is straightforward. The model chooses one answer from four options, and accuracy is the percentage of questions answered correctly.

That simplicity creates three limitations that matter when you are making a production decision:

MMLU works as a broad capability filter, useful for ruling out weak candidates early. The score stops being helpful when the decision involves a specific domain, tool-calling behavior, or workflow that your application actually runs.

HellaSwag tests commonsense completion

Pick the next plausible sentence and you pass HellaSwag. Each example gives a context drawn from a video caption or a WikiHow article along with four candidate endings, and the model selects the most plausible continuation. The original paper used adversarial filtering to make examples hard for models while remaining easy for humans, which helped expose gaps in earlier language models. Scoring is multiple-choice accuracy, measured as the percentage of examples where the model selected the correct ending.

If a model consistently picks implausible continuations, it likely has a commonsense gap that will show up in production. The problem is that this kind of commonsense completion only goes so far once an agent needs to do more than choose the next sentence.

A strong HellaSwag score helps rule out obvious commonsense gaps. But it does not test the parts of an AI agent where production failures usually show up, such as tool calling, retrieval, and multi-turn workflows.

HumanEval tests code generation in a narrow setup

HumanEval evaluates code generation by giving a model a function signature, docstring, and held-out unit tests. The benchmark includes 164 hand-written Python programming problems, and the model's answer passes if the generated code satisfies the tests.

The score depends on whether the generated function works against tests rather than on self-reported correctness. The most common metric is pass@k, which estimates the probability that at least one of k generated samples passes the tests.

HumanEval is useful when you need a rough sense of a model's ability to write small, self-contained Python functions. It is much less useful as a proxy for software engineering work.

The benchmark also inherits the limits of its test setup. Held-out unit tests are better than self-reported correctness, although they can miss edge cases. The tasks are also short, so they do not measure planning across a repo. For coding agents, HumanEval is a starting signal, and repository-level evals and task trajectories are the shipping signal.

GSM8K tests grade-school math reasoning

GSM8K tests multi-step arithmetic reasoning. It is a dataset of 8,500 grade-school math word problems, often evaluated with chain-of-thought prompting. Scoring checks whether the final numeric answer is correct.

A model that scores well on GSM8K can follow multi-step arithmetic reasoning chains reliably. That is a meaningful signal for ruling out weak candidates on any task requiring sequential calculation.

GSM8K also has a documented memorization risk. Scale AI's A Careful Examination paper built a held-out GSM1K set and measured accuracy drops of up to 13% on frontier models, evidence that some GSM8K performance reflects data contamination rather than reasoning. Clean scores on GSM8K are a necessary filter for math-heavy applications, and not a sufficient one.

MT-Bench tests dialogue quality through judge models

MT-Bench spans eight categories: writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge II (humanities and social sciences). It uses a judge model to score open-ended responses across two conversation turns. The second turn tests whether the model can handle follow-up, refinement, and extension of the original task, something single-turn benchmarks miss.

Research on MT-Bench also analyzed judge model biases, including preferences for longer answers, certain response positions in pairwise comparisons, and self-enhancement when a model judges its own outputs. Those biases apply to any LLM-as-a-judge setup, not just MT-Bench.

MT-Bench is the closest of the five benchmarks to real application behavior. For production agents, the score is a useful signal on dialogue quality and two-turn coherence, with the caveat that your tools, users, and workflow are not in the test.

Leaderboards work as filters

Public leaderboards are most useful as a way to narrow the field. When several models are similar on cost and latency, benchmark results can help you pick which ones are worth deeper testing. They can also surface broad strengths and weaknesses, for example a model that excels at coding but struggles with math, or one that performs well on academic knowledge but falls short in dialogue.

Teams run into issues when they use leaderboards as release gates. The leaderboard environment is not your production environment, and public benchmark scores tend to flatten the exact details that matter in an application:

• Dataset fit: The benchmark domain often differs from your users, policies, or edge cases
• Prompt fit: Leaderboard prompts rarely match your system prompt, retrieval context, or tool schema
• Workflow fit: Most benchmarks grade an answer, not the sequence of actions that produced it
• Measurement fit: Accuracy or judgment score can miss latency, cost, safety, escalation behavior, or user satisfaction
• Drift fit: A static benchmark does not tell you whether quality changes after users, docs, tools, or models shift

Two models with similar public scores can behave very differently once they sit inside an AI agent. The application adds state, tools, memory, retrieval, permissions, and product logic. The eval has to cover those layers.

Application-specific evals test the system you ship

Application-specific evals measure whether your LLM application or AI agent achieves its goal in the real conditions it runs in. Instead of asking whether the model produced a plausible answer, the eval asks whether the agent actually resolved the issue and did it the right way by using the correct data sources, following policy, and escalating when appropriate. That reframes evaluation. You are now validating production reliability by testing the full application:

• The model
• The prompts
• The tools
• The retrieval layer
• The routing logic
• The memory
• The product constraints
• The user workflow

The most useful application-specific evals usually combine offline datasets with trajectory checks and online evaluators.

The goal is not to create a giant private benchmark for its own sake. The goal is to turn the cases your application already sees into a tight eval loop that tells you whether the next version is better.

Golden datasets start from traces

A golden dataset is a curated set of examples with expected outputs or grading criteria that define what good looks like for your application. Production or production-like traces are usually the best starting point because they contain actual user intents, failure modes, tool calls, retrieval context, and reasoning paths. That makes traces the raw material for testing the full system.

Synthetic examples can help fill gaps, and they should not be the foundation. Real traces show the messy phrasing, partial context, ambiguous requests, and domain-specific assumptions that public benchmarks smooth away. When a user asks a question in a way your team did not expect, that trace is more valuable than another generic test item.

In LangSmith, our framework-agnostic agent engineering platform, this trace-first workflow connects observability and evaluation. Teams can add runs to datasets, compare experiments against a baseline, and use LangSmith Evaluation to test changes before they ship. A production failure can become a regression test instead of a one-off incident.

A focused dataset of 20 to 50 high-signal traces often teaches more than a broad set of synthetic examples, especially when it covers the intents that matter most to the product. As the application matures, add new user intents, known failures, risky tool paths, and examples that domain experts have corrected.

LLM-as-a-judge needs calibration

LLM-as-a-judge evals are useful for open-ended answers because many production outputs cannot be graded with an exact match. If the agent summarizes a customer conversation, resolves a support ticket, or answers a policy question with citations, a judge model can score dimensions like helpfulness, groundedness, policy compliance, and goal completion.

Judge models are not neutral instruments, though. Research on LLM-as-a-judge methods documents recurring biases, including preferences for longer answers, certain positions in pairwise comparisons, familiar model styles, and answers that look authoritative. A judge can be directionally useful and still wrong.

Calibration separates a useful evaluator from another untrusted model output. Start with a human-labeled seed set, compare the judge against those labels, revise the rubric, and track agreement over time. Align Evals is built for exactly this workflow. Expert feedback shapes the evaluator so automated grading reflects the team's definition of quality.

For quick benchmark-style comparisons, an uncalibrated judge can be a useful directional signal. But if you care about production reliability, the judge must be validated against the humans whose judgment it is meant to scale.

The Agent Development Lifecycle replaces one-off benchmarking

The Agent Development Lifecycle is the production version of eval-driven development. Traces become insights; insights become datasets; datasets become evals; evals guide improvements; and the next production traces tell you whether the system got better. Public benchmarks sit outside that loop. They are useful inputs, and they do not improve your application by themselves.

The loop works because every production interaction can carry a signal:

1. Capture traces from real usage
2. Identify failures, drift, and common patterns
3. Promote representative traces into datasets
4. Run offline evals before model, prompt, or tool changes ship
5. Run online evals on production traffic to monitor live behavior
6. Route ambiguous or high-risk cases to human review
7. Feed corrected examples back into the next dataset

By watching production traces and grouping related failures, LangSmith Engine pushes the loop further, proposing eval coverage and fixes tied directly to the problems it finds. Production behavior becomes the work queue for application improvement.

Human judgment matters here because, for many domains, "correct" is not just a string match. Clinicians, lawyers, support leads, product managers, and other domain experts know what a good answer looks like because they understand the task. Annotation queues give those experts a structured way to review the traces that need human attention, correct outputs, and turn tacit judgment into reusable eval criteria. That is how teams move from a vague sense that the model improved to measured confidence in the cases they care about.

A practical benchmark-to-production workflow

The cleanest way to use public benchmarks is to make them the first step in a longer eval process. Let them help you choose which models to test, then move quickly into your own eval setup.

A practical workflow puts public benchmark scores in the right context and keeps them in their proper place:

1. Use public benchmarks to shortlist models. MMLU, HellaSwag, HumanEval, GSM8K, and MT-Bench can help you avoid obviously weak candidates for your use case.
2. Run a small offline eval on your golden dataset. Compare candidate models against production examples, not generic prompts.
3. Inspect trajectories, not just outputs, by checking whether the agent used the right tools, retrieved the right context, and followed the right path.
4. Calibrate any LLM-as-a-judge rubrics. Use human-labeled examples before trusting automated scores.
5. Deploy behind online evals. Sample production traffic, watch for drift, and route low-confidence cases to review.
6. Promote failures into the next dataset. Every meaningful production failure should become a future regression test.

Public benchmarks still matter, especially for narrowing the field when you are choosing a base model or explaining why a candidate made your shortlist. But they should not be your release gate. The final decision should come from an evaluation suite that mirrors your application, your data, and the way your system behaves in production.

What to do before trusting a leaderboard score

Before a benchmark score influences a production decision, check what the benchmark leaves out. The missing pieces are usually the work your application has to do after the model responds.

• Tools, retrieval, and permissions model coverage
• Trajectory scoring, not only final-answer grading
• Domain-specific edge cases from real users
• Latency, cost, and escalation constraints
• Multi-turn goal completion
• Human review for cases where expert judgment matters
• Regression coverage after production failures

If those pieces are missing, the benchmark is still useful, and it is not enough on its own.

Pull 20 real traces from a workflow you care about. Turn the best and worst examples into a first golden dataset, then run your next model or prompt change against it before it reaches users.

‍

AI observability requires domain experts, not just dashboards

dasdsadsaew

Most AI observability tools for agentic applications answer the question engineers ask:

"Is the system working?"

The harder question is one engineers can't answer alone:

"Is the output actually good?"

As agentic systems move from prototypes to production, observability needs to expand beyond infrastructure health. The question is no longer just whether the LLM responded, but whether the generative AI output met the standard your domain experts expect.

What is AI observability?

AI observability is the practice of collecting, analyzing, and correlating telemetry data (logs, traces, and metrics) across the full AI technology stack to understand how AI systems behave in production. It enables real-time visibility into LLMs, AI agents, orchestration layers, and their downstream impact on applications and infrastructure.

But this definition alone doesn't capture the full picture. Most definitions stop at "visibility into system behavior," which is where traditional APM left off a decade ago. The harder part, and what separates AI observability from a rebranded monitoring dashboard, is understanding whether the system is doing something useful for users. That requires a layer of insight that telemetry alone cannot provide.

The three pillars, and why they're only the starting point

AI observability inherits the three pillars from traditional software observability, each adapted for AI-specific workloads:

• Logs: Time-stamped records of application events used for troubleshooting and debugging. In AI systems, logs capture prompt submissions, model responses, tool invocations, retrieval events, and error states across multi-step agent workflows.
• Traces: End-to-end records of a request's journey through the system, from user input through orchestration, model inference, tool execution, and back. For agentic applications, traces capture the full reasoning chain, including branching decisions, retries, and intermediate outputs that shaped the final response.
• Metrics: Quantitative measures of system health over time such as latency and error rates. AI-specific metrics extend this to token consumption, inference cost, model response times, tool call latency, and feedback scores.

These three pillars give teams the vocabulary for understanding system behavior. But AI systems produce telemetry that stretches well beyond what logs, traces, and metrics were originally designed to capture. The gap between traditional observability and AI-specific observability is where most teams get stuck, and where the most AI-specific failures hide.

AI-specific telemetry: what traditional pillars miss

Token usage and cost attribution

Tokens are the fundamental unit of AI compute. Every prompt and response is broken into tokens that directly determine cost and latency. But token observability goes beyond tracking spend.

Key metrics include token consumption rates across models and inputs, and input and output tokens per trace. Teams tracking token usage at a granular level can attribute costs to specific features, users, or use cases, which makes cost optimization targeted rather than speculative.

However, token optimization solves a cost problem, not a quality problem. A team can reduce token consumption by 40% and still produce outputs that domain experts reject. This is the first hint that engineering metrics, no matter how comprehensive, address only half of the observability challenge.

Model drift

Model drift occurs when AI models gradually change behavior as the data they encounter in production diverges from the conditions they were built for. For teams using managed AI platforms, drift can also occur when providers update models without notice. A response that worked reliably last month may subtly degrade after a provider-side model update.

Key indicators include changes in response patterns over time, output quality degradation on previously reliable prompts, and shifts in latency that suggest the model is handling inputs differently. Drift is hard to catch because it's often gradual. No single response looks broken, but quality erodes over weeks.

This is where AI observability starts to demand more than dashboards. Engineers can detect drift in aggregate metrics as average response length changes, latency distributions shift, and token usage patterns evolve. But domain experts are often the first to notice output quality degrading in the specific contexts that matter most to users. A clinician notices that medication summaries are missing interaction warnings more frequently. A lawyer notices that contract analyses are citing outdated precedent. These signals arrive as anecdotes before they appear as trends, and without a structured mechanism to capture them, they get lost.

Response quality metrics

Response quality is where AI observability diverges most from traditional application monitoring. The metrics that matter include:

• Hallucination frequency across prompt types and models. Not just whether the model hallucinates, but where and how often.
• Factual accuracy, which often requires human validation or checking correctness with AI.
• Output consistency for similar inputs. Does the system produce reliable results, or does the same question yield different answers depending on phrasing?
• Relevance to user intent. Did the output actually address what the user was asking, or did it answer an adjacent question confidently?
• Latency-accuracy tradeoffs. Faster responses that sacrifice quality may look good on dashboards while frustrating users.

Most of these metrics share a common characteristic: they're difficult or impossible to evaluate fully without domain expertise. Automated scoring can approximate quality, but the ground truth for "was this output correct?" frequently lives in the minds of subject matter experts, not in a metrics pipeline.

What AI observability covers across the stack

Effective AI observability gives you end-to-end visibility across the full lifecycle of AI systems, from user input through orchestration, LLM calls, tool usage, and final output. Without this end-to-end trace, teams can troubleshoot isolated components but struggle to understand how the entire AI system behaved.

AI observability spans multiple layers of your technology stack. Each layer captures different signals, and together they form a complete picture of system behavior.

Application layer

Application-level AI observability tracks user interactions and business outcomes. This is where you measure whether AI features are actually being used, and how AI outputs connect to the metrics your business cares about, such as conversion rates, time-to-resolution, and user satisfaction.

Orchestration layer

The orchestration layer manages LLM calls, tool selection, and decision logic, whether you're using LangChain, LangGraph, or custom pipelines. Observability priorities include tracing prompt/response pairs, monitoring retries and fallback behavior, tracking tool execution timing, and understanding branching logic. When an orchestration step fails or produces unexpected results, the trace needs to show exactly which decision led to the downstream problem.

Agentic layer

The agentic layer is distinct from orchestration because it handles multi-step reasoning, autonomous workflows, memory, and goal-directed behavior. Observability priorities include tracing reasoning chains across steps, monitoring memory references and context management, tracking tool usage history and decision logs, and capturing intermediate outputs that influenced the final response.

This distinction matters because agent failures compound across steps. A flawed retrieval in step two shapes the reasoning in step five, which determines the tool call in step eight. Without agentic-layer observability, you see the final bad output but can't trace it back to the decision that caused it. This compounding effect is one of the strongest arguments for treating agents as their own observability concern.

Model and LLM layer

Model-level AI observability captures prompts, completions, token usage, latency, and associated metadata. For teams deploying multiple AI models in parallel, observability must also track which model version generated each output, how different LLM configurations compare, and where performance diverges across environments.

Working with managed AI platforms (OpenAI, Anthropic, Gemini, Bedrock, Azure AI Foundry, Vertex AI) introduces an additional challenge: model execution happens externally and opaquely. You can observe inputs and outputs, but you can't inspect what happens inside the model. This makes external observability even more critical and reinforces why domain expertise is essential. When you can't see inside the model, the quality of its outputs is the primary signal you have.

Semantic search and vector database layer

RAG components and vector databases (Pinecone, Weaviate, FAISS, Chroma, and others) form a critical layer that many observability implementations underinvest in. Observability priorities include embedding quality and consistency, retrieval relevance scores, result set sizes, query latency, and semantic drift, where the meaning of queries evolves while your embeddings remain static.

Retrieval quality is where domain expertise matters most, because only a subject matter expert can judge whether the retrieved context was actually relevant to the query. An engineer can confirm that the vector database returned five results in under 100 milliseconds. A domain expert can tell you that three of those results were from the wrong regulatory jurisdiction and the other two were outdated.

Infrastructure layer

Infrastructure observability for AI workloads extends beyond traditional APM concerns. AI-specific infrastructure considerations for companies running open source models include GPU utilization and memory pressure during inference, cost breakdowns across compute tiers, network bottlenecks for model serving (especially when calling external APIs), availability and rate limiting of managed AI endpoints, and scaling behavior under variable inference loads. Teams running AI workloads need to understand how infrastructure constraints translate to user-facing quality. A GPU memory bottleneck might force smaller context windows, which degrades output quality even though the system technically "works."

Benefits of AI observability

When implemented well, AI observability delivers measurable returns:

• Cost control: Attribute token usage and compute costs to specific requests, models, users, and features, making optimization targeted rather than across-the-board.
• End-to-end tracing: Follow a request from user input through orchestration, model inference, tool execution, and output. This is critical for debugging failures in complex agent workflows.
• Failure mode identification: Catch latency spikes, tool failures, hallucinations, degraded retrieval quality, and cascading errors before they reach users.
• Model quality assurance: Use feedback signals, correctness scores, and grounding metrics to track whether model performance is improving or regressing.
• SLA and compliance tracking: Monitor performance and reliability of AI workflows against contractual and regulatory requirements.

These benefits are real, but they address only the engineering side of the equation. Cost control doesn't tell you whether the output was worth the cost. End-to-end tracing doesn't tell you whether the trace produced a correct result. Failure identification catches system failures but misses the domain failures that erode user trust without triggering any alert.

The role of OpenTelemetry

OpenTelemetry has become the standard open-source framework for collecting and transmitting telemetry data, and its role in AI observability is growing. For teams building AI systems, OpenTelemetry offers vendor independence (avoiding lock-in to any specific observability platform) while providing end-to-end visibility across complex pipelines that span multiple models, services, and infrastructure components.

OpenTelemetry is valuable when working with managed AI platforms where model execution happens externally. It enables metadata standardization across multi-vendor AI solutions, so traces from your orchestration layer, your vector database, and your managed LLM endpoints can be correlated in a single view. For teams running heterogeneous AI stacks (and most production teams do), this interoperability matters.

The adoption gap: tracing without evaluation

As agentic applications mature, observability must account for both system health and output quality. Teams build observability around engineering metrics because that's what engineering teams can act on. Latency, error rates, and token costs are measurable, debuggable, and within the engineer's control.

But engineering metrics alone miss failures that require domain knowledge to detect, where "correct" is defined by expertise that lives outside the codebase. This gap becomes more pronounced as teams deploy AI agents that reason across multiple steps. When AI agents call tools, retrieve context, and chain LLM decisions together, quality failures compound in ways traditional monitoring cannot capture.

The below chart shows agent observability adoption for all organizations surveyed in our State of Agent Engineering survey vs. teams with production agents:

‍

These adoption numbers show a pattern: teams have built this infrastructure. The question becomes what to do with it. High observability adoption with lower evaluation adoption (52% offline, 37% online) suggests the market has largely adopted tracing but hasn't yet linked traces to systematic quality improvements.

This reflects a broader shift across the LangChain ecosystem toward production-grade evaluation. The standard toolkit includes:

• Tracing: Capturing the complete execution tree of an agent, including tool selection, retrieved documents, parameters passed to the model at each step, and the dependencies across those components
• Monitoring dashboards: Tracking cost, latency, and error rates in real time
• Alerting: Notifying teams when metrics cross thresholds
• Usage analytics summaries: Understanding how and why users interact with AI features

These tools answer whether the API call succeeded, how long it took, and what it cost. But they don't reveal whether the output was actually useful to the user.

Why traditional engineering metrics aren't enough for AI observability

AI systems rarely fail in ways that resemble traditional software outages. Instead of crashing, LLMs confidently produce plausible but incomplete answers. Instead of throwing exceptions, AI agents may follow incorrect reasoning paths even when all traditional observability metrics appear healthy.

In these cases, existing dashboards from legacy APM providers fail to reflect true model performance. AI applications also fail in ways that latency dashboards and error rates don't capture. For example:

• A medical summary that returns in 200 milliseconds, costs exactly what you budgeted, and still misses critical clinical context that changes the diagnosis.
• A legal research tool that retrieves documents without errors while citing a precedent that was overturned three years ago.
• A financial risk report that completes successfully and passes all validation checks while using an outdated volatility model that underestimates market exposure.

In each case, every engineering metric looks healthy, yet the output is still wrong.

Domain failures produce outputs that are technically correct by engineering standards but wrong by the standards of the people who actually use them. Engineers have the systems to improve AI applications, but domain experts define what "good" means. Without a shared workspace connecting them, improvement becomes guesswork.

The below table describes a system failure versus a domain failure for agentic systems.

‍

Many organizations have instrumented some form of observability for their agents and AI systems but haven't yet built the human feedback loops that improve agent quality over time. Observability tells you what happened. Evaluation determines whether your generative AI application is improving. For AI product and engineering teams, this shift from monitoring to measurable quality is what separates experimentation from production readiness.

Where domain feedback gets lost

In most organizations, subject-matter expert feedback flows through indirect channels. We hear this pattern repeatedly in customer conversations, e.g.:

A support engineer at a financial services company notices that a customer-facing risk report labeled a transaction as "low risk" even though the underlying analysis flagged three separate compliance concerns: unusual transaction timing, jurisdictional red flags, and a counterparty with recent regulatory scrutiny. The engineering lead files a ticket noting the inconsistency, but without structured feedback, the issue sits in a backlog alongside dozens of other ambiguous bug reports.

Meanwhile, the customer acts on the "low risk" classification, and two weeks later, the transaction triggers a regulatory inquiry. The compliance team traces the issue back to the AI-generated report, but by then, the damage is done: the inquiry costs time, money, and credibility with regulators. The AI system never crashed. Latency was fine. Token costs were within budget. But the output was wrong, and no traditional dashboard caught it.

For teams building complex AI agents, this problem compounds. A single flawed LLM decision early in a chain can propagate through downstream tool calls, yet without structured review, no one captures that failure as a training signal for the broader AI system.

Each of these moments contains a useful signal about what "good" means in that domain. But the feedback doesn't become a test case, an evaluation criterion, or a dataset for regression testing. The insight gets buried in ticket backlogs and competing priorities, replaced by new fires to fight.

The gap exists because of a team’s workflow and tooling, rather than lack of knowledge on the team. In many organizations, engineers don't have a place to send domain experts for structured review and domain experts don't have a way to provide feedback that engineering systems can consume. The result is that AI improvement relies on engineers guessing which problems matter most, and acting on whatever anecdotes happened to reach them that week.

Annotation queues as a collaboration surface

Annotation Queues provide the mechanism that connects domain expertise to engineering workflows. The concept is straightforward: production traces get routed to queues where subject matter experts can review outputs in context, add ratings, leave comments, and flag issues. The expert doesn't need to understand the trace structure or write code. They evaluate whether the output is good for their domain.

This is especially critical for teams deploying AI agents in production. As AI agents orchestrate multiple LLM calls and tools, troubleshooting shifts from fixing a single prompt to understanding how the full system behaved across steps.

Here's how the workflow operates in practice:

An engineering team configures routing rules based on criteria they care about:

• Sample 10% of all traces
• Route all traces with negative user feedback
• Send all traces for a specific use case to a specific queue
• Queues can be configured with granular access control to ensure the right experts review the right traces.

Domain experts receive notifications that traces are ready for review. They open the queue, see the actual prompt and output in context, and provide structured feedback -- a rating on a defined scale, comments explaining their reasoning, and flags for specific issues.

This transforms observability from something engineers browse alone into a shared workspace. The trace view serves as a shared visualization layer for engineers and domain experts.

• The engineer sees the full trace: every tool call, every retrieval, every step in the reasoning chain.
• The domain expert sees what matters to them: the input the user provided and the output the system returned.
• Both perspectives reside in the same place and are connected to the same underlying data.

Traditional observability shows what happened. Annotation Queues add a follow-up and help answer the question, "was it right?" That question invites domain experts, such as Product Managers, into the engineering process without requiring them to become engineers.

From feedback to evaluation criteria

When domain experts review traces and add ratings, comments, or flags, those annotations become inputs that engineering teams can use. The flagged example can flow directly into downstream workflows.

• Dataset entries: A trace marked "incorrect" with an explanation becomes a regression test case. The input-output pair links a dataset against which future model changes are evaluated. Over time, this transforms LLM observability into a structured automation pipeline in which human insight continuously informs model updates and agent behavior. It also enables AI-driven quality improvement rather than reactive debugging.
• Evaluation rubrics: Patterns in annotations show what domain experts check for. If clinicians consistently flag "missed drug interactions," that becomes an evaluation criterion that automated systems can check.
• Prompt improvement signals: Comments explain why something failed in terms that engineers couldn't have known. "This summary omits the contraindication because the source note used non-standard abbreviations," tells engineering exactly what to fix.
• Quality baselines: Aggregated ratings across hundreds of traces establish what "good" looks like quantitatively, enabling teams to measure whether changes improve or regress quality.

We see this as a development flywheel: traces flow to Insights Agent, which provides insights on common usage patterns. These insights inform dataset creation Datasets enable evaluations, evaluations drive improvements, and improvements produce new traces to observe. The flywheel relies on domain knowledge being entered into the system in a structured format. This closes the loop inside the broader AI development lifecycle, connecting production traces to iteration.

Here's what that looks like concretely: A clinician flags an output of a trace as incorrect, noting that the summary missed a drug interaction. An engineer clicks "Add to dataset." That input-output pair now runs against every prompt change. When an engineer modifies the prompt next week, the regression test catches whether the fix worked or whether it introduced new problems. The informal process of "someone mentioned this was broken, so let's try to remember to test for it" becomes a systematic quality gate.

Compliance, audit trails, and regulatory readiness

AI observability plays a direct role in compliance, and this is only becoming more true. The EU AI Act imposes transparency and documentation requirements on high-risk AI systems, and emerging US state regulations are following suit with mandates for explainability and auditability in AI decision-making.

For teams in regulated industries, end-to-end traceability is a compliance requirement, rather than just a debugging convenience. Every LLM decision, tool invocation, and retrieved document needs to be auditable. PII detection and protection is itself an observability concern. Teams need to monitor whether sensitive data is being passed through AI pipelines and whether outputs inadvertently expose protected information.

Compliance in healthcare, legal, and financial services often requires demonstrating that domain-qualified humans reviewed AI outputs: that a clinician validated the medical summary, that an attorney reviewed the contract analysis, that a compliance officer signed off on the risk assessment. Structured annotation workflows both improve quality and create the audit trail that regulators expect. The annotation itself becomes evidence that human oversight occurred, who reviewed it, what they found, and when.

Collaborative workflow in practice

For teams in healthcare, legal, finance, and enterprise B2B, routing production traces to expert-review queues links domain knowledge to product improvements in a repeatable way. The pattern looks similar across domains, even when the specific expertise differs.

If you're building in healthcare, you might route patient-facing summaries to a queue accessible to clinical reviewers. Your reviewers (nurses, physicians, or clinical informaticists) evaluate whether the summaries capture medically relevant information and flag cases where context is missed or language confuses patients. They identify which failure modes occur most frequently and prioritize fixes based on clinical impact rather than engineering convenience.

If you're building legal tools, you might have attorneys review the outputs of contract analysis. They know which clauses matter, which precedents are current, and which jurisdictional nuances the model misses. Their annotations become test cases that catch regressions before they reach clients.

For legal and financial AI agents, end-to-end traceability ensures that every LLM decision and tool invocation can be reviewed in context, which is essential when troubleshooting high-stakes outputs.

If you're building financial applications, you might route forecasting explanations to analysts who validate the reasoning. They know when a model is confusing correlation with causation, when it's extrapolating beyond reasonable bounds, or when its confidence should be lower given market conditions.

This approach requires organizational buy-in. Subject matter experts need time allocated for review, which adds to their workload. Engineering needs to build the routing rules and close the loop by actually using the feedback. There is a tradeoff for this. The payoff is that "the AI isn't good enough" becomes a set of specific issues you can fix and measure.

Tradeoffs and when this fits

While Annotation Queues enable domain experts to review production AI outputs, manual annotation doesn't scale to every trace. With agents producing millions of traces per day, reviewing everything manually is impossible. Teams need sampling strategies: route a percentage of all traces, prioritize traces with negative signals, or focus expert review on high-stakes use cases while letting automated evaluation handle routine cases. The goal is to maximize the signal-to-noise ratio: get domain experts looking at the traces where their judgment matters most, rather than asking them to review outputs that automated systems can already evaluate reliably.

Subject matter expert time is expensive and limited. Clinicians, lawyers, product managers, and analysts have primary responsibilities that don't include reviewing AI outputs. The annotation workflow competes for their attention. Teams that succeed typically start small, demonstrate value quickly, and expand review coverage as they prove the feedback loop works.

The feedback loop is slower than automated evaluation. An automated eval can score every trace in real time. Human annotation takes hours or days, depending on queue depth and reviewer availability. For rapid iteration cycles, human review supplements automated evaluation rather than replacing it.

The decision comes down to the quality signal you need. Automated evaluation works well when you can encode "good" in a prompt or heuristic. Human review becomes essential when "good" requires expertise you can't encode, when the cost of being wrong is high, or when you're still learning what failure modes exist.

Leveraging Annotation Queue works when:

• Domain correctness matters more than speed of iteration
• The cost of being wrong is high (regulated industries, medical decisions, legal advice)
• Automated evaluation can't capture what "good" means because "good" requires expertise you can't encode in a prompt
• You have accessible domain experts who can dedicate some time to reviewing
• You're building for users who will notice and care about domain-specific quality

Annotation Queues aren’t as strong of a fit for situations when:

• You're in a rapid prototyping phase where the product is changing faster than review cycles
• The domain is general enough that automated evaluation handles most quality concerns
• You don't have subject matter experts available, or you can't allocate their time

Annotation Queues bridge the gap between what engineers can measure and what domain experts can judge. Traditional observability tells you the system is running: traces are complete, latency is acceptable, no errors are thrown. But it can't tell you whether the output was right for your domain. That requires human judgment from people who understand regulatory requirements, company policies, clinical standards, or legal precedent.

By routing production traces to subject matter experts for structured review, annotation queues transform observability from a reactive debugging tool into an active quality improvement system. The compliance officer who spots a policy violation, the clinician who catches a missed contraindication, and the analyst who identifies flawed reasoning all contribute expertise that becomes embedded in your evaluation pipeline. This structured feedback replaces ad hoc Slack messages and ticket backlogs with actionable data.

This is the difference between knowing your agent ran and knowing it worked. Engineering instrumentation shows you what happened. Domain expertise tells you whether it should have happened that way. Annotation queues connect the two, turning expert judgment into dataset entries, evaluation criteria, and regression tests that make your AI systematically better at the things that actually matter to your users.

Making AI observability work harder

Most teams treat AI observability as an engineering debugging tool. The real opportunity lies in turning those traces into a quality improvement system. Traces and production data are already flowing through your systems. The missing piece is a mechanism that enables people who know what "good" means to contribute that knowledge in a structured way.

In practice, AI observability becomes the backbone of sustainable improvement across releases. For teams in specialized domains where the hardest failures are domain-level rather than system-level, this workflow turns observability into a feedback loop. The next step is practical. Identify one use case where domain expertise matters more than engineering intuition. Set up a queue. Route a sample of traces. Get one expert reviewing for one week. Watch what they flag.

If the feedback reveals failure modes you didn't know existed, you've found the place where observability becomes collaboration, and where improvement is based on what "good" actually means in your domain.