Quick Answer: LLM evaluation metrics fall into seven families: reference-based scores, LLM-as-a-judge scores, RAG metrics, hallucination and factuality rates, safety rates, agent task metrics, and operational metrics such as latency and cost per task. No single number covers quality. Pick one or two metrics per failure you actually care about, calibrate any LLM judge against human labels, and run the set on every change.
LLM evaluation metrics are how you turn "the answers seem better" into a number you can track, gate releases on and argue about with evidence. The hard part is that the thing you're measuring won't sit still: the same prompt gives different outputs, there are many correct answers, and a fluent reply can still be wrong.
This guide is the metrics reference. It sorts every metric worth knowing into seven families and says what each one measures, whether it needs a reference answer, what it costs to run and which tools implement it. It includes a master table, DeepEval and RAGAS code checked against the current releases as of November 2026, and a method for building an eval set that reflects real traffic.
If you want the testing workflow around these metrics (unit tests, eval tests, integration tests and production monitoring), our guide to AI agent testing covers that stack. We build cowork.ink, where every AI coworker run is a transcript you can read and correct, so judging agent output is part of our day job.
What are LLM evaluation metrics?
LLM evaluation metrics are scores that measure how well a language model's output meets a defined standard: correct, grounded, safe, useful, fast or cheap. Each one answers a narrow question. "Is the answer supported by the retrieved documents?" is a metric. "Is the model good?" is not.
Three questions place any metric:
- What does it measure? Correctness, grounding, relevance, safety, task completion, speed or cost.
- Does it need a reference? Some metrics compare the output to a known-correct answer. Others score the output on its own, or against the input and context.
- Who scores it? Code (a string match or a regex), a small model (an embedding or a classifier), a large model acting as judge, or a person.
The answers decide where a metric can run. Reference-based metrics only work on a labelled dataset. Reference-free metrics, such as faithfulness or a rubric judge, can also run on live production traffic, where nobody has written the right answer down.
A metric is a scoring rule (faithfulness, pass rate). A benchmark is a fixed dataset scored with a metric (SWE-bench, MMLU). Public benchmarks tell you how models compare in general. Your own eval set, scored with the metrics below, tells you whether your system works on your tasks.
The master table of LLM evaluation metrics
Here is every metric in this guide on one page. "Cost" is per scored example: free means plain code, low means a small local model, medium means one LLM call, high means several LLM calls or human time.
| Metric | What it measures | Needs a reference? | Cost | Tools that implement it |
|---|---|---|---|---|
| Exact match | Output equals the expected answer | Yes | Free | RAGAS, promptfoo, Inspect, Braintrust autoevals |
| Token F1 | Word overlap with the expected answer | Yes | Free | Inspect, custom code |
| BLEU / ROUGE | N-gram overlap with a reference text | Yes | Free | RAGAS, promptfoo |
| BERTScore / embedding similarity | Semantic closeness to a reference | Yes | Low | RAGAS, promptfoo, autoevals |
| Rubric score (LLM judge) | Meets written criteria | Optional | Medium | DeepEval, promptfoo, RAGAS, LangSmith, Braintrust, Inspect |
| G-Eval | Criteria score with judge-written steps | Optional | Medium | DeepEval, promptfoo |
| Pairwise preference | Which of two outputs is better | No | Medium | DeepEval, promptfoo, LangSmith, autoevals |
| Faithfulness / groundedness | Claims supported by the given context | No (needs context) | High | RAGAS, DeepEval, promptfoo |
| Answer relevancy | Answer addresses the question | No | Medium | RAGAS, DeepEval, promptfoo |
| Context precision | Relevant chunks ranked near the top | Usually | Medium | RAGAS, DeepEval |
| Context recall | Retrieval found everything needed | Yes | Medium | RAGAS, DeepEval, promptfoo |
| Factual correctness | Agrees with a reference answer | Yes | Medium | RAGAS, promptfoo, Inspect, autoevals |
| Atomic-fact precision (FActScore) | Share of facts a trusted source supports | Knowledge source | High | FActScore method, custom pipelines |
| Self-consistency (SelfCheckGPT) | Claims stay stable across samples | No | High | SelfCheckGPT method, custom code |
| Toxicity | Harmful or abusive language | No | Low to medium | DeepEval, promptfoo, safety classifiers |
| Attack success rate | Share of jailbreak attempts that work | No | Medium | promptfoo red teaming, Inspect Evals |
| Refusal and over-refusal rate | Refuses harmful asks, answers safe ones | Labelled prompts | Low to medium | promptfoo, Inspect Evals (XSTest) |
| Task success rate | Agent reached the goal state | Goal state | Free to medium | DeepEval, RAGAS, promptfoo, Inspect |
| pass^k | Succeeds on all k repeated tries | Goal state | k × task cost | τ-bench method, custom code |
| Tool-call accuracy / F1 | Right tools with right arguments | Expected calls | Free | DeepEval, RAGAS, promptfoo |
| Trajectory quality | Efficient path, no loops, follows plan | Optional | Medium | DeepEval, promptfoo |
| Latency (TTFT, p95) | Time to first token and to finish | No | Free | promptfoo, LangSmith, Phoenix, Braintrust |
| Cost per successful task | Spend divided by tasks completed | No | Free | promptfoo, LangSmith, Braintrust |
The rest of the guide walks through each family, from the cheapest scores to the most expensive.
Reference-based metrics: exact match, F1, BLEU, ROUGE, BERTScore
Reference-based metrics compare the model's output with a known-correct answer, so they're cheap, deterministic and only as good as that answer. They're the right tool when the space of correct outputs is small.
- Exact match scores 1 if the output equals the reference after normalisation (case, whitespace, punctuation), else 0. Use it for classification labels, extracted IDs, dates, SQL results and multiple-choice answers.
- Token F1 measures word overlap between output and reference as the harmonic mean of precision and recall. It gives partial credit for short factual answers like "Paris, France" against "Paris".
- BLEU counts matching n-grams (word sequences) against the reference, with a penalty for outputs that are too short. It came from machine translation.
- ROUGE is the recall-oriented cousin: how many of the reference's n-grams appear in the output. It came from summarisation.
- BERTScore and embedding similarity compare meaning instead of surface words, by matching token or sentence embeddings. They accept paraphrases that BLEU and ROUGE punish.
Why reference-based metrics fail on open-ended output
They measure resemblance to one answer, and open-ended tasks have many correct answers. A support reply, a research summary or an agent's plan can be phrased a hundred valid ways.
The failures cut both ways:
- Valid answers score low. "Your refund will arrive in five business days" and "Expect the money back within a week" mean nearly the same thing and share almost no n-grams.
- Wrong answers score high. "The meeting moved to Tuesday" against a reference of "The meeting moved to Thursday" differs by one token. BLEU and ROUGE rate it as a near-perfect match. Embedding similarity often does too, because the sentences are about the same thing.
That second failure is the dangerous one. A metric that can't see a flipped fact can't guard against hallucinations.
Keep them for the deterministic slices of your system: classification, extraction, routing decisions, structured output (is it valid JSON, does it match the schema?) and any step with one right answer. They cost nothing, never drift and make perfect CI gates.
LLM-as-a-judge metrics: G-Eval, pairwise and rubric scoring
LLM-as-a-judge means asking a capable model to score an output against written criteria, a reference answer or a competing output. It's the default for open-ended quality because it can read for meaning, follow a rubric and explain its score.
The approach has evidence behind it. In the study that introduced MT-Bench, Zheng et al. (2023) found that strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level at which humans agree with each other. The same paper named the judge's main biases, covered below.
There are three common formats.
Rubric scoring
A rubric judge checks one output against explicit criteria and returns a score with a reason. It's the workhorse format.
Two design choices matter most:
- Prefer binary or short scales. "Does the answer cite the refund policy? yes/no" is more reliable than "rate helpfulness 1–10". A 10-point scale invites the judge to pick numbers that don't mean anything.
- One criterion per judge call. Separate judges for accuracy, tone and completeness are easier to debug than one judge returning a blended score.
Here is a rubric prompt you can adapt. It asks for reasoning first and a verdict last, in a fixed format your code can parse:
You are grading a customer-support reply.
Question from the customer:
{question}
Company policy excerpt the reply must follow:
{policy}
Reply to grade:
{answer}
Check each criterion and answer PASS or FAIL for each:
1. ACCURATE: every factual statement in the reply is supported by the policy excerpt.
2. COMPLETE: the reply answers every question the customer asked.
3. ACTIONABLE: the customer knows exactly what to do next.
Rules:
- Judge only against the policy excerpt, not your own knowledge.
- Length is not a quality signal. A short reply can pass every criterion.
- If a criterion cannot be judged from the inputs, mark it FAIL and say why.
Write one sentence of reasoning per criterion, then finish with a single line:
VERDICT: ACCURATE=<PASS|FAIL> COMPLETE=<PASS|FAIL> ACTIONABLE=<PASS|FAIL>
How you phrase the rubric matters as much as which model runs it. The same principles apply as in prompt engineering for AI agents: explicit instructions, examples of edge cases and a strict output format.
G-Eval
G-Eval is a rubric judge that first writes its own evaluation steps from your criteria, then scores, and weights the score by the probabilities of each possible score token. That last step turns a coarse integer into a smoother number.
In the original paper, Liu et al. (2023) reported that G-Eval with GPT-4 reached a Spearman correlation of 0.514 with human ratings on summarisation, well ahead of earlier automatic metrics. The same paper flagged an early warning: LLM evaluators may prefer text written by LLMs.
DeepEval ships G-Eval as a ready metric. This snippet matches the API in DeepEval 4.3.4, the current release as of November 2026 (note SingleTurnParams, which replaced the older LLMTestCaseParams name):
from deepeval import evaluate
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
correctness = GEval(
name="Correctness",
evaluation_steps=[
"Check whether any fact in the actual output contradicts the expected output.",
"Penalise missing details that the expected output treats as essential.",
"Do not reward extra length or style.",
],
evaluation_params=[
SingleTurnParams.INPUT,
SingleTurnParams.ACTUAL_OUTPUT,
SingleTurnParams.EXPECTED_OUTPUT,
],
threshold=0.7,
# model="<judge model>", # defaults to an OpenAI model; set your own judge here
)
case = LLMTestCase(
input="When does the weekend support desk open?",
actual_output="The desk opens at 9:00 on Saturday and 10:00 on Sunday.",
expected_output="Saturday 9:00, Sunday 10:00.",
)
correctness.measure(case)
print(correctness.score, correctness.reason)
# or score a whole dataset at once
evaluate(test_cases=[case], metrics=[correctness])
Pass either criteria or evaluation_steps, not both. Writing the steps yourself makes the metric more stable between runs.
Pairwise comparison
A pairwise judge sees two outputs for the same input and picks the better one, or a tie. It's the format for A/B decisions: a new prompt against the old one, a cheaper model against the current one.
Picking a winner is easier than assigning an absolute score, for people and models alike. The result is a win rate, which maps directly to "should we ship this change?". DeepEval's ArenaGEval, promptfoo's select-best assertion, LangSmith's pairwise evaluators and the Battle scorer in Braintrust's autoevals all implement it.
Judge biases and how to control them
LLM judges have systematic biases, and each has a known countermeasure.
| Bias | What happens | How to control it |
|---|---|---|
| Position bias | In pairwise tests the judge favours the answer shown first (or second) | Run every pair twice with the order swapped; count a win only if both runs agree, otherwise a tie |
| Verbosity bias | Longer answers win even when they add nothing | State in the rubric that length isn't quality; compare length-matched pairs; check win rate against length difference |
| Self-preference | A judge rates text from its own model family higher | Use a judge from a different model family than the system under test, or a panel of judges from different families |
| Score clustering | Most scores land on 7 or 8 of 10 | Use binary or 3-point scales with anchored descriptions |
| Leniency on hard cases | The judge passes answers it can't verify | Give the judge the reference or source documents, and tell it to fail what it can't check |
The swap-and-agree method is the one Zheng et al. used to control position bias. It doubles the cost of pairwise tests and is worth it.
Our take on LLM-as-a-judge (editorial opinion, not a benchmark score): the most useful metric family for open-ended output, and the easiest one to trust too early. Treat a judge like any other classifier: it needs a labelled test set and an agreement score before its numbers go on a dashboard.
Pros
- Reads for meaning, so it handles paraphrase and open-ended output
- Works without a reference answer, so it can score production traffic
- Explains its scores, which speeds up debugging
- Easy to target at the exact failure you care about
Cons
- Biased toward position, length and its own model family
- Costs one or more model calls per example
- Drifts when the judge model is updated
- Only trustworthy after you measure agreement with human labels
RAG evaluation metrics: faithfulness, relevancy, context precision and recall
RAG evaluation metrics score the retriever and the generator separately, because a bad answer can come from either. Four metrics cover most of it, and the RAGAS framework popularised their definitions.
| Metric | Stage | Question it answers | Low score usually means |
|---|---|---|---|
| Context precision | Retriever | Are the relevant chunks ranked near the top? | Ranking or reranking problem; noise crowds out signal |
| Context recall | Retriever | Did retrieval find everything the right answer needs? | Missing documents, bad chunking or weak queries |
| Faithfulness | Generator | Is every claim in the answer supported by the retrieved context? | The model adds facts from memory or misreads sources |
| Answer relevancy | Generator | Does the answer address the question that was asked? | Off-topic, padded or evasive answers |
How each one is computed:
- Faithfulness splits the answer into individual claims and checks each against the retrieved context. The score is supported claims divided by total claims, from 0 to 1.
- Answer relevancy in RAGAS asks an LLM to generate the questions the answer would fit, then measures how close those are to the original question using embeddings.
- Context precision checks each retrieved chunk for relevance and averages precision at each rank, so a relevant chunk at position 1 counts for more than one at position 8.
- Context recall checks how many claims in the reference answer can be found in the retrieved context. It needs a reference answer.
Read them together. High context recall with low faithfulness means the right documents arrived and the model ignored them. Low recall with high faithfulness means the model stayed honest about incomplete context. That's a retrieval problem, and better prompts won't fix it. Retrieval quality and evaluation are two halves of the same job, which is why context engineering and evaluation go hand-in-hand.
Here is the RAGAS version, written for the collections API in RAGAS 0.4.3, the current release as of November 2026. Older tutorials use evaluate() with a Dataset, which RAGAS calls the legacy API:
import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness, ContextPrecision, ContextRecall
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI()) # any supported judge model
question = "What is the refund window for annual plans?"
contexts = [
"Annual plans can be refunded in full within 15 days of purchase.",
"Monthly plans renew automatically on the billing date.",
]
answer = "Annual plans are fully refundable within 15 days, or 30 days in the EU."
reference = "Annual plans are fully refundable within 15 days of purchase."
async def main():
faith = await Faithfulness(llm=llm).ascore(
user_input=question, response=answer, retrieved_contexts=contexts
)
precision = await ContextPrecision(llm=llm).ascore(
user_input=question, reference=reference, retrieved_contexts=contexts
)
recall = await ContextRecall(llm=llm).ascore(
user_input=question, retrieved_contexts=contexts, reference=reference
)
print(faith.value, precision.value, recall.value)
asyncio.run(main())
In this example the "30 days in the EU" claim has no support in the context, so faithfulness drops below 1 even though retrieval was fine. That's exactly the signal you want.
For agents that decide when and what to retrieve, the same four metrics apply at each retrieval step. Our agentic RAG guide shows where those steps sit.
Hallucination metrics and factuality
A hallucination metric counts claims the model states as fact that no trusted source supports. How you measure it depends on what "trusted source" means for your task.
Grounded tasks: measure faithfulness
When the model should answer only from supplied material (documents, a ticket history, a policy), hallucination is a faithfulness failure. Use the faithfulness metric above, or DeepEval's HallucinationMetric, which checks the output against the context you provide.
Report it two ways:
- Response-level hallucination rate: the share of responses with at least one unsupported claim. This is what users feel.
- Claim-level faithfulness: the share of all claims that are supported. This is what you debug.
Open-world tasks: check atomic facts
When the model answers from its own knowledge, you need an external reference. Three methods are well established:
- Reference-answer judging. A judge compares the output to a known answer and grades agreement. Braintrust's
Factualityscorer, promptfoo'sfactualityassertion and Inspect'smodel_graded_fact()scorer all do this. - Atomic-fact checking (FActScore). Min et al. (2023) split long answers into atomic facts and checked each against a knowledge source. ChatGPT scored 58% on people's biographies, and an automated estimator matched human scoring with under 2% error.
- Self-consistency (SelfCheckGPT). Manakul et al. (2023) sample several answers to the same prompt. Facts the model actually knows tend to recur, while invented details vary between samples. It needs no external database and works on black-box APIs.
Public benchmarks such as SimpleQA and TruthfulQA, both in the Inspect Evals collection, measure the same failure on fixed question sets. They're useful for picking a model and less useful for testing your own system.
A model can drive its hallucination rate down by saying "I don't know" to everything. Always report abstention rate (how often it declines to answer) next to hallucination rate, and score correct abstentions on questions that have no answer in the source as a pass.
Safety metrics: toxicity, jailbreak and refusal rates
Safety metrics measure two opposite failures: the model saying something harmful, and the model refusing something harmless. You need both, or you'll tune one at the expense of the other.
- Toxicity score. A classifier or judge rates outputs for abusive, hateful or harassing language. DeepEval's
ToxicityMetricand promptfoo'smoderationassertion cover it. Note that Google's Perspective API, long a standard toxicity scorer in research, stopped taking new usage requests in February 2026 and shuts down on December 31, 2026. Pipelines that depend on it need a replacement. - Attack success rate (ASR). Run a set of adversarial prompts (jailbreaks, role-play tricks, encoded requests) and count the share that produce a harmful completion. A judge or safety classifier decides what counts as harmful. HarmBench and StrongREJECT define standard sets, and Inspect Evals includes StrongREJECT and AgentHarm, which tests harmful multi-step agent tasks. promptfoo generates adversarial prompts against your own app with its red-teaming mode.
- Refusal rate on harmful prompts. The share of clearly unsafe requests the model declines. Higher is better.
- Over-refusal rate. The share of safe requests the model wrongly declines. The XSTest suite (Röttger et al., 2023) has 250 safe prompts that only sound risky, of the "how do I kill a Python process?" kind, plus 200 unsafe contrast prompts. promptfoo's
is-refusalassertion detects refusals in your own outputs.
For agents, add prompt-injection success rate: plant instructions in a web page, file or tool result the agent will read, and count how often the agent follows them. That failure belongs to agents with tools, not to chat models.
Agent and task metrics: task success, tool calls, trajectory quality
Agent metrics judge what the agent did, not only what it said. An agent can write a perfect summary of a task it never completed.
Task success rate
Task success is the share of tasks where the agent reached the goal state, checked against the world, not against its own claim. Did the ticket get closed? Is the right row in the database? Does the test suite pass? Outcome checks are cheap and hard to fool, so prefer them to an LLM judge reading the agent's final message.
pass^k: reliability over repeated attempts
pass^k asks whether the agent succeeds on all k independent attempts at the same task. It's the reliability counterpart to pass@k from code generation, which asks whether any of k attempts succeeded.
The difference matters in production. In the τ-bench paper, Yao et al. (2024) found that state-of-the-art function-calling agents such as GPT-4o succeeded on under 50% of tasks, and their pass^8 in the retail domain fell under 25%. An agent that works on one try in two isn't something you can hand recurring work to.
Tool-call accuracy
Tool-call accuracy checks whether the agent called the right tools with the right arguments. Score it at two levels:
- Selection: the expected tools were called and no forbidden ones were. Tool-call F1 (in RAGAS and promptfoo) gives partial credit when some calls are right.
- Arguments: the parameters were correct. A search with the wrong date range is a quiet failure that task success alone can miss.
DeepEval's ToolCorrectnessMetric, RAGAS's ToolCallAccuracy and promptfoo's trajectory:tool-args-match assertion implement these.
Trajectory quality
Trajectory metrics score the path: how many steps the agent took, whether it looped, and whether it followed a sensible plan. Two runs can both succeed while one takes 6 steps and the other 40, with 40 times the latency and cost. DeepEval's StepEfficiencyMetric and PlanAdherenceMetric, and promptfoo's trajectory:step-count and trajectory:tool-sequence assertions, cover the common checks.
Public agent benchmarks apply the same ideas at scale; our breakdown of SWE-bench and AI agent benchmarks explains how they score coding agents and where the numbers mislead.
Operational metrics: latency and cost per task
Operational metrics decide whether a system that passes quality checks is usable and affordable. They're cheap to collect from traces and often ignored until the bill arrives.
- Time to first token (TTFT): how long a user waits before anything appears. It drives perceived speed in chat.
- End-to-end latency at p50 and p95: the median hides the slow tail that users complain about. For agents, measure wall-clock time per task, not per call.
- Steps and tokens per task: leading indicators for both latency and cost.
- Cost per successful task: total spend (model tokens, tool and API fees, retries) divided by tasks that actually succeeded. Cost per call flatters a system that retries five times or fails half its tasks.
promptfoo has latency and cost assertions you can set thresholds on, and LangSmith, Phoenix and Braintrust record both from traces. For alert thresholds and dashboards on these numbers, see our guide to AI agent monitoring.
How to build an LLM evaluation framework and eval set
An eval set is the collection of inputs, and where possible expected outputs, that every metric above runs against. The metrics are only as good as this set. Build it from your own traffic, not from what you imagine users ask.
Read real outputs before choosing metrics
Go through 50–100 real outputs or traces and write down every way they go wrong. Group the notes into failure types: wrong fact, missed question, wrong tool, too slow. Each common failure type gets a metric. Rare or harmless ones don't.
Seed a golden set by hand
Write test cases that cover the core tasks, known edge cases and past incidents, with expected outputs where an answer exists. LangSmith's documentation suggests starting with 5–10 hand-curated examples per critical component. Small and correct beats large and noisy.
Sample production traces
Add real inputs: a random sample for coverage, plus every trace a user flagged, every failed task and the long tail of unusual requests. Strip personal data. Labelled failures from production are the most valuable test cases you'll ever get.
Calibrate judges against human labels
Have two people label the same slice independently and measure how often they agree. Then run your LLM judge on the same slice. Iterate on the rubric until the judge agrees with the humans about as well as the humans agree with each other. Report agreement with a statistic like Cohen's kappa or Krippendorff's alpha (Inspect ships the latter as a built-in metric).
Set thresholds and run on every change
Turn each metric into a pass/fail gate in CI: faithfulness at least 0.9, no regressions on the golden set, p95 latency under budget. Run reference-free metrics on a sample of live traffic as well, so drift shows up between releases.
Grow the set with every failure
Every production incident becomes a new test case before the fix ships. Re-check judge agreement whenever you change the judge model or the rubric.
Steps 1, 3 and 6 all depend on having a readable record of what the system did: inputs, retrieved context, every model call, every tool call and the final output. If you don't log that yet, start there. Our guide to AI agent observability covers what to capture and how.
Which LLM evaluation tools should you use?
Pick a tool for how you work, not for its metric list. Most tools implement the same core metrics. They differ in being a code library or a hosted platform, in Python or YAML, and in their focus on RAG, agents or safety. Status below as of November 2026.
| Tool | What it is | Best for | Notes (Nov 2026) |
|---|---|---|---|
| DeepEval | Python library, pytest-style | Broad metric coverage in CI: G-Eval, RAG, agent and safety metrics | Apache-2.0; current release 4.3.4 |
| RAGAS | Python library | RAG metrics; also tool-call and agent-goal metrics | Apache-2.0; 0.4.x moved to a collections API |
| promptfoo | CLI with YAML configs | Prompt and model comparisons, assertions in CI, red teaming | MIT; OpenAI announced in March 2026 it will acquire Promptfoo and keep the open-source project |
| LangSmith | Hosted platform from LangChain | Datasets, offline and online evals, pairwise and human annotation queues | Commercial; Python and TypeScript SDKs |
| Braintrust | Hosted eval and logging platform | Experiment tracking and side-by-side comparisons | Commercial platform; its autoevals scorer library is MIT |
| Arize Phoenix | Tracing and eval app, self-hostable | Evaluating traces from production, built on OpenTelemetry | Elastic License 2.0 (source-available) |
| OpenAI Evals | Eval framework and registry | Teams on the OpenAI platform | Repo README now points to evals in the OpenAI Dashboard |
| Inspect | Python eval framework | Agent and safety evals in sandboxes; 200+ ready-made evals | MIT; built by the UK AI Security Institute and Meridian Labs |
A common split: a code library (DeepEval, RAGAS or promptfoo) for the CI gate, and a tracing platform (LangSmith, Braintrust or Phoenix) for production sampling and human review. Inspect is the strongest choice when you need to run agents with real tools in a sandbox.
Common mistakes with LLM evaluation metrics
Most broken eval setups fail in one of five predictable ways.
- One blended "quality" score. It hides which failure moved. Keep metrics separate and named after the failure they catch.
- Trusting an uncalibrated judge. If you never measured agreement with humans, the judge's score is an opinion with decimals.
- A synthetic-only eval set. Generated test cases miss the strange things real users ask. Mix in production traces.
- Scoring single runs of a non-deterministic system. Run important cases several times and report the spread, or use pass^k.
- Measuring quality without cost. A configuration that's 2% better and three times more expensive per successful task is usually worse.
Make Every Run Something You Can Evaluate
Every metric in this guide starts from the same raw material: a full record of what the model and its tools actually did.
If you run your own agent, GoGogot is open source and self-hosted, and it logs structured events for every LLM call, every tool call and every error. Those events are the traces you sample, label and turn into an eval set.
If you'd rather hand the work to an AI coworker, cowork.ink gives each one its own cloud computer, with a browser, files, an inbox and a code runtime. Every run is a transcript of the steps it took, the pages it opened and the files it produced, which you can read and correct. Corrections stick, so the next run starts from what you fixed. Setup takes about three minutes, and it's $25 a month billed yearly, everything included.