Introducing LangSmith Tuned Evaluators, starting with Perceived Error

Every conversational agent in production needs evaluation for Perceived Error. It’s one of the clearest signals that your agent is giving users a helpful experience.

Until now, evaluating Perceived Error has been expensive and hard to scale. Teams had to rely on frontier model calls, sample only a small slice of traces, and spend time manually tuning the evaluator.

Today, we’re introducing Tuned Evaluators, which automatically attach quality feedback to production traces and threads. Teams can use that feedback to find behavior that needs attention, understand what went wrong, and take action to improve their agents.

Each Tuned Evaluator is a finished, versioned evaluator for a specific objective. LangChain has already done the work of turning that objective into a production-ready judge. Teams simply select the evaluator they need, and attach it to a tracing project.

High-Quality, Low-Cost Judge that Runs on Every Conversation

Applying high-quality evaluation across production traces has traditionally forced teams to choose between accuracy and coverage.

Frontier models can provide strong judgments, but applying them across many production conversations is expensive. Smaller models cost less, but their lower accuracy makes it harder to trust which traces deserve attention. As a result, many teams evaluate only a small sample of threads, which risks missing the critical signals that an agent is delivering a bad experience to customers.

Inference cost is only part of the work. Building a dependable evaluator also requires defining the behavior, resolving ambiguous cases, producing labels, selecting and benchmarking a judge, and maintaining it as models and production behavior change. Tuned Evaluators package that work into a product that teams can attach and use immediately.

Tuned Evaluators also address the cost/quality tradeoff with specialized models post-trained for narrowly defined evaluation objectives, exceeding frontier performance at a fraction of the cost. When you run Tuned Evaluators, you get more signal to steer agent improvements while saving significant engineering time and inference cost.

How Tuned Evaluators work

1. Add a Tuned Evaluator to a LangSmith tracing project and save it.
2. LangSmith identifies eligible traces or threads based on the selected evaluator’s requirements.
3. A specialized, LangChain-managed judge evaluates each eligible trace or thread.
4. The result and its explanation are attached to the evaluated trace or thread as feedback.
5. Teams use the results to investigate traces, add useful examples to datasets, or build evals for their improvement workflow.

LangChain writes, tests, versions, and maintains the evaluator prompt. We also manage the judge model, benchmarking against frontier models, provider credentials, and inference infrastructure.

Use Tuned Evaluator results in your improvement workflow

Teams can use enriched traces in their own analytics, coding-agent, CI, human-review, and evaluation workflows. Tuned Evaluators help with:

• Finding failures that produced no system error or explicit user rating.
• Filter to conversations that deserve investigation and review the evaluator's explanation alongside the original interaction.
• Compare flagged traces to find repeated failure modes.
• Add tagged traces to evaluation datasets or route ambiguous conversations for human review.
• Use those examples to test and validate changes to the agent.

Starting with Perceived Error

Perceived Error detects conversations containing evidence that an agent made a mistake, misunderstood a request, or took the interaction in the wrong direction.

That evidence may be explicit, such as a user correction, repeated request, or rejected action. The model can also infer it from contradictory responses, acknowledged mistakes, persistent misunderstandings, or unresolved outcomes.

Perceived Error is a useful proxy for whether an agent is meeting a user’s needs. Most users never submit an explicit rating, so the evidence often has to be inferred from the conversation itself.

LangChain post-trained a specialized model on labeled traces of conversational agents. It outperformed every frontier model in our benchmark while reducing evaluation cost by 82%. In some early-partner workloads, savings reached 98%, with cost variance dependent on thread composition.

Vanta’s experience as an early partner demonstrates another benefit: teams can establish quality coverage immediately while developing their own business-specific evaluators.

This out-of-the-box evaluator gave us a safety net to catch failure modes from day one, while we built out our own business-specific evaluators. It’s the fastest way we’ve found to make a team operational on quality.
‍
— Kevin Royer, Staff ML/AI Engineer, Vanta

Eligibility and timing

A thread becomes eligible for the Perceived Error evaluator after it contains at least two human-AI message pairs and reaches the configured idle period. The evaluation completes within 12 hours after the thread becomes eligible.

Availability and pricing

The Perceived Error Tuned Evaluator is available now for all Plus and Cloud Enterprise plans in the US. Each successful evaluation incurs a tuned evaluation charge. Skipped and failed evaluations are not billed. For general usage and billing information, see our pricing page.

Get started

You can try the Perceived Error evaluator today by logging in or signing up for LangSmith, and visit the docs for more detail.

If you have a Tuned Evaluator that you’d like us to build, tell us about your use case.