How to Evaluate AI Agents with LLM-as-Judge and Agentic-Eval Patterns

Mar 10, 2026

Problem

My AI agent was working fine in testing. But in production, it started giving weird answers. No errors, no crashes - just subtle wrong responses that users noticed before I did.

I had unit tests. I had integration tests. But none of them caught the real problem: the agent was reaching correct answers through wrong reasoning paths.

Why Traditional Testing Fails AI Agents

I realized my tests only checked the final output. They missed:

Wrong tool sequences: Agent called search_web then format_response, but should have called validate_input first
Subtle drift: Knowledge base got stale, API schemas changed, responses slowly degraded
Reasoning errors: Right answer, wrong logic - which breaks when edge cases appear

Traditional unit tests are like checking if a car starts. But for AI agents, you need to check if the car takes the right route to the destination.

The Solution: LLM-as-Judge Pattern

I found the LLM-as-Judge pattern. The idea is simple: use a capable LLM to evaluate your agent’s responses.

The mathematical foundation:

R = J(C1, ..., Cn)

Where J is the judge LLM, C_i are candidates to evaluate, and R is the judgment result.

I started with the agentevals library from LangChain:

from agentevals.trajectory.llm import create_trajectory_llm_as_judge

EVALUATION_PROMPT = """
You are evaluating an AI agent's performance. Analyze:

Trajectory: {trajectory}
Question: {question}
Expected: {expected_response}
Actual: {actual_response}

Score each dimension:
1. Completeness (0-1): Did it fully address the question?
2. Accuracy (0-1): Was the information correct?
3. Tool Usage (0-1): Were appropriate tools used correctly?
4. Helpfulness (0-1): Was the response actionable?

Return JSON: {"completeness": X, "accuracy": X, "tool_usage": X, "helpfulness": X, "issues": [...]}
"""

quality_evaluator = create_trajectory_llm_as_judge(
    prompt=EVALUATION_PROMPT,
    model="openai:o3-mini"
)

I chose o3-mini as the judge because it’s capable enough to evaluate nuanced responses but cheaper than GPT-4 for batch evaluation.

Adding Trajectory Matching

LLM-as-Judge evaluates output quality. But I also needed to validate the tool call sequence.

I added trajectory matching:

from agentevals.trajectory.match import create_trajectory_match_evaluator

trajectory_evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="superset"  # Allow extra tool calls
)

# Define expected trajectory
reference_trajectory = [
    {"tool": "validate_input", "input": "user query"},
    {"tool": "search_web", "input": "validated query"},
    {"tool": "format_response", "input": "search results"}
]

# Evaluate
score = trajectory_evaluator(
    outputs=agent_result["messages"],
    reference_outputs=reference_trajectory
)

The superset mode lets the agent call additional tools beyond the expected ones. Other modes:

Mode	Description
`exact`	Every tool call must match exactly
`superset`	Agent can call extra tools
`subset`	Agent may skip some expected tools
`unordered`	Tool calls can be in any order

I use superset because I want to verify minimum required tools are called, but allow flexibility.

Building the Evaluation Framework

I combined both evaluators into a framework:

from agentevals.trajectory.llm import create_trajectory_llm_as_judge
from agentevals.trajectory.match import create_trajectory_match_evaluator
from dataclasses import dataclass
from typing import List, Dict

@dataclass
class QualityThresholds:
    trajectory_accuracy: float = 0.85
    response_quality: float = 0.80
    latency_ms: int = 5000

class AgentEvaluationFramework:
    def __init__(self, thresholds: QualityThresholds):
        self.thresholds = thresholds
        self._setup_evaluators()

    def _setup_evaluators(self):
        self.quality_evaluator = create_trajectory_llm_as_judge(
            prompt="""
            Evaluate the agent response for:
            1. Completeness: Did it fully address the user's request?
            2. Accuracy: Was the information correct?
            3. Helpfulness: Was the response actionable?

            Return JSON: {"score": 0-1, "reasoning": "...", "issues": [...]}
            """,
            model="openai:o3-mini"
        )

        self.trajectory_evaluator = create_trajectory_match_evaluator(
            trajectory_match_mode="superset"
        )

    def evaluate(self, agent, inputs: Dict, expected_trajectory: List) -> Dict:
        result = agent.invoke(inputs)

        trajectory_score = self.trajectory_evaluator(
            outputs=result["messages"],
            reference_outputs=expected_trajectory
        )

        quality_score = self.quality_evaluator(
            outputs=result["messages"],
            reference_outputs=inputs
        )

        metrics = {
            "trajectory_accuracy": trajectory_score.get("score", 0),
            "response_quality": quality_score.get("score", 0),
        }

        alerts = self._check_thresholds(metrics)

        return {
            "metrics": metrics,
            "alerts": alerts,
            "passed": len(alerts) == 0,
        }

    def _check_thresholds(self, metrics: Dict) -> List[str]:
        alerts = []
        if metrics["trajectory_accuracy"] < self.thresholds.trajectory_accuracy:
            alerts.append(f"Trajectory accuracy {metrics['trajectory_accuracy']:.2f} < {self.thresholds.trajectory_accuracy}")
        if metrics["response_quality"] < self.thresholds.response_quality:
            alerts.append(f"Response quality {metrics['response_quality']:.2f} < {self.thresholds.response_quality}")
        return alerts

The Evaluator-Optimizer Loop

The real power comes from self-critique loops. Instead of just measuring, the evaluation drives improvement:

┌─────────────────┐
│ Agent Execution │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ LLM-as-Judge    │
│ Evaluation      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐     Failed
│ Score Analysis  │─────────────┐
└────────┬────────┘             │
         │                      │
         │ Passed               │
         ▼                      ▼
┌─────────────────┐     ┌─────────────────┐
│ Return Result   │     │ Adjust Inputs   │
└─────────────────┘     │ with Critique   │
                        └────────┬────────┘
                                 │
                                 └─────────────┐
                                               │
                                               ▼
                                        ┌─────────────────┐
                                        │ Retry Agent     │
                                        └─────────────────┘

Here’s the implementation:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def run_self_critique_loop(framework, agent, inputs, expected_trajectory, max_iterations=3):
    """Evaluator-optimizer pipeline with self-critique"""

    for iteration in range(max_iterations):
        logger.info(f"Evaluation iteration {iteration + 1}")

        result = framework.evaluate(agent, inputs, expected_trajectory)

        if result["passed"]:
            logger.info(f"Passed on iteration {iteration + 1}")
            return result

        # Self-critique: adjust inputs with identified issues
        issues = result.get("issues", [])
        inputs["critique"] = f"Previous issues: {issues}. Please address these."

        logger.info(f"Issues found: {issues}")

    logger.warning(f"Failed to pass after {max_iterations} iterations")
    return result

Integration with LangSmith

For production, I use LangSmith to run evaluations at scale and track results over time:

from langsmith import Client

client = Client()

# Create evaluation dataset
dataset = [
    {
        "inputs": {"question": "What's the weather in SF?"},
        "outputs": {
            "response": "Currently 65F and sunny in San Francisco",
            "trajectory": ["validate_input", "search_weather", "format_response"]
        }
    },
]

# Run comprehensive evaluation
experiment_results = client.evaluate(
    run_agent,
    data="production_evaluation_dataset",
    evaluators=[
        trajectory_evaluator,
        quality_evaluator
    ],
    max_concurrency=4,
    experiment_prefix="weekly-agent-eval"
)

# Analyze results
for result in experiment_results:
    if result.outputs.get("score", 0) < 0.80:
        trigger_alert(result)

This gives me:

Experiment tracking: Compare performance across versions
Dataset management: Maintain representative test cases
Batch evaluation: Run against hundreds of examples
Historical comparison: Detect drift over time

What I Learned

After implementing this framework, I caught issues that unit tests never would:

An agent that called tools in the wrong order but got lucky with the answer
Subtle response drift after a knowledge base update
A tool that was deprecated but the agent still tried to call it

The key insight: the difference between prototype and production agents is not the model - it’s the evaluation framework.

Summary

In this post, I showed how to evaluate AI agents using LLM-as-Judge and trajectory matching patterns. The framework combines:

LLM-as-Judge: Evaluates response quality against flexible criteria
Trajectory matching: Validates tool call sequences
Self-critique loops: Uses evaluation feedback for automatic improvement
Quality thresholds: Triggers alerts when performance degrades

The key point is that traditional tests only check outputs. For AI agents, you need to evaluate the reasoning process too.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 agentevals - LangChain Evaluation Library
👨‍💻 LangSmith Evaluation Documentation
👨‍💻 LLM-as-a-Judge: A Survey
👨‍💻 Reddit Discussion: agentic-eval for Production Agents

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!