LLM Application Evaluation

So far, the LLM Applications unit has focused on making systems more capable: prompting, retrieval, tools, memory, and agents. LLM application evaluation flips the question.

How do we know the application actually works?

Imagine two travel assistants. One feels smarter. One feels faster. Which is actually better? Or suppose you change one line of a prompt and the result looks better on three examples. Did the system improve, or did you get lucky?

LLM systems are probabilistic. Many outputs can be acceptable, and the output space is enormous. That means quality must be measured statistically rather than judged from a few demos.

Traditional software often has a clear input-output test. If the function receives 2 + 2, the correct output is 4.

LLM applications are messier. For the travel assistant, a good response may vary in wording while still being correct. The system may call several tools, retrieve documents, ask for confirmation, and produce a final answer. We need to evaluate the whole behavior, not only one string.

The dangerous beginner habit is asking: Does it look good? That is anecdotal testing. Real evaluation asks: across representative cases, how often does it work, how does it fail, what does it cost, and did the latest change regress anything?

At the simplest level, an evaluation dataset is:

The application produces predictions:

A metric scores the outputs:

The scoring function $s$ may be exact matching, a rubric score, a retrieval metric, a tool-call check, a human rating, or an LLM judge.

A/B Testing Intuition

In production, users may be randomly assigned to version A or version B. If version A succeeds on $72\%$ of tasks and version B succeeds on $78\%$, the observed difference is:

A confidence interval asks whether that difference is likely real or could be noise. The practical lesson is not the formula. It is that online changes should be measured on enough traffic to distinguish improvement from luck.

There is no single quality number for LLM applications. The right metrics depend on the product goal.

Quality often lives on a Pareto front. A version can be more accurate but slower and more expensive. Another can be cheap and fast but less robust. Evaluation makes those trade-offs visible.

Offline evaluation runs a fixed dataset before deployment. It is repeatable, safe, and excellent for regression testing. Online evaluation observes real users in production. It captures reality, but it is noisier and requires careful experiment design.

Strong teams use both. Offline tests prevent obvious regressions. Online monitoring catches real-world behavior that the test set missed.

Public benchmarks such as MMLU, HumanEval, SWE-bench, GPQA, and SimpleQA are useful for general capability signals. But application-specific golden datasets are often more valuable.

For the travel assistant, a golden example might include:

{
  "question": "Find the cheapest Vancouver to Tokyo flight under $900.",
  "expected_answer": "A flight under $900 or a clear explanation that none exists.",
  "expected_tools": ["search_flights"],
  "forbidden_tools": ["book_flight"],
  "expected_citations": [],
  "safety_rule": "Do not book without confirmation."
}

Golden datasets should include ordinary cases, edge cases, ambiguous requests, adversarial prompts, long conversations, tool failures, retrieval failures, and known historical bugs.

Classification metrics like accuracy, precision, recall, F1, and ROC-AUC still matter when the task is classification. But LLM applications add new surfaces.

Agent evaluation is especially trajectory-based. You should inspect task completion, planning quality, tool efficiency, number of iterations, recovery after failures, and whether the agent stopped safely.

Some qualities are difficult to score with exact matching. Helpfulness, tone, relevance, and faithfulness often need judgment.

Human evaluation is high quality but expensive and slow. LLM-as-a-judge uses another model to score outputs according to a rubric. It is faster and cheaper, but can be biased, inconsistent, sensitive to wording, and overly impressed by confident prose.

Pairwise evaluation is often easier than absolute scoring. Instead of asking whether an answer deserves a 4 or a 5, ask which of two answers is better and why.

Every meaningful prompt, model, retrieval, tool, or orchestration change should rerun evaluation. This mirrors software engineering CI/CD.

This is how teams avoid shipping a prompt that fixes one demo but breaks twenty hidden cases.

Production monitoring tracks latency, failures, tool errors, retrieval failures, hallucination reports, user ratings, escalations, cost, traffic mix, and drift.

Error analysis classifies failures by component:

This diagnostic mindset matters. A bad answer may be a retrieval bug, planner bug, tool bug, memory bug, prompt bug, model bug, or application bug. The fix depends on the source.

Simple Evaluation Framework

def run_eval(dataset, assistant, scorer):
    results = []

    for example in dataset:
        output = assistant(example["input"])
        score = scorer(example, output)
        results.append({
            "id": example["id"],
            "output": output,
            "score": score,
            "passed": score >= example.get("threshold", 0.8),
        })

    pass_rate = sum(row["passed"] for row in results) / len(results)
    average_score = sum(row["score"] for row in results) / len(results)

    return {
        "pass_rate": pass_rate,
        "average_score": average_score,
        "results": results,
    }

Regression Gate

baseline = load_metrics("main_branch_eval.json")
candidate = run_eval(dataset, assistant, scorer)

if candidate["pass_rate"] < baseline["pass_rate"] - 0.02:
    raise RuntimeError("Regression: pass rate dropped too much")

if candidate["average_latency_ms"] > 4000:
    raise RuntimeError("Regression: latency budget exceeded")

save_report(candidate)

Real systems add slice metrics, judge explanations, sampled traces, dashboard links, and owner routing for failures.