docs/observability/EVALUATION.md
Rama master · réplica de lectura del repositorio original.
# Evaluation — Testing and Benchmarking Agents
> Write test cases, score an agent's answers with built-in or custom scorers, grade with an LLM judge, compare runs and print reports.
---
## Table of Contents
1. [Overview](#overview)
2. [Test Case Authoring](#test-case-authoring)
3. [Built-In Scorers](#built-in-scorers)
4. [Custom Scorers](#custom-scorers)
5. [LLM-as-Judge](#llm-as-judge)
6. [Running an Evaluation](#running-an-evaluation)
7. [Comparing Two Agents](#comparing-two-agents)
8. [Keeping Runs](#keeping-runs)
9. [Reports](#reports)
---
## Overview
```
test cases → evaluator.runEvaluation(name, cases, agentFn) → EvalRun
↓
each case: scorers → weighted score → passed?
↓
evaluator.generateReport(runId, 'json' | 'markdown' | 'html')
```
The evaluation exports come from the root package:
```typescript
import {
Evaluator,
LLMJudge,
CRITERIA_PRESETS,
type EvalTestCase,
type EvalRun,
} from '@framers/agentos';
```
---
## Test Case Authoring
A test case is one input with what to compare the answer against and how to score it:
```typescript
import type { EvalTestCase } from '@framers/agentos';
const testCases: EvalTestCase[] = [
{
id: 'capital-1',
name: 'Capital of France',
input: 'What is the capital of France?',
expectedOutput: 'Paris',
criteria: [
{ name: 'correctness', description: 'Contains the answer', weight: 1, scorer: 'contains' },
],
},
{
id: 'summary-1',
name: 'TCP handshake summary',
category: 'summaries',
input: 'Summarize the TCP three-way handshake.',
expectedOutput: 'SYN, SYN-ACK, ACK: client and server establish a reliable connection.',
criteria: [
{ name: 'coverage', description: 'Shares the key words', weight: 2, scorer: 'semantic_similarity' },
{ name: 'wording', description: 'Close to the reference', weight: 1, scorer: 'rouge' },
],
},
];
```
| Field | Type | Description |
|-------|------|-------------|
| `id` | `string` | Unique identifier |
| `name` | `string` | Label used in reports |
| `input` | `string` | The prompt passed to the agent function |
| `expectedOutput` | `string` | The reference answer the scorers compare against |
| `referenceOutputs` | `string[]` | Extra references; `bleu` and `semantic_similarity` use them when `expectedOutput` is missing |
| `context` | `string` | Passed to the agent function as its second argument |
| `criteria` | `EvalCriteria[]` | `{ name, description, weight, scorer }`; without criteria the case is scored by `levenshtein` alone |
| `category` | `string` | A label; the aggregates do not group by it |
| `metadata` | `object` | Passed to every scorer as its fourth argument |
A test case is one input; there is no multi-turn form. To test a conversation, have the agent function keep a session and send earlier turns before the input.
The case's score is the weighted mean of its criteria scores, and it passes when the score is at least `thresholds.pass` (default 0.7). A criterion naming an unknown scorer is skipped with a console warning.
---
## Built-In Scorers
| Scorer | Score | What it computes |
|--------|-------|------------------|
| `exact_match` | 0 or 1 | Equal after trimming and lower-casing |
| `contains` | 0 or 1 | The answer contains the expected text, ignoring case |
| `levenshtein` | 0–1 | 1 minus the character edit distance over the longer length, ignoring case |
| `semantic_similarity` | 0–1 | Word overlap: shared words longer than 2 characters over the larger word set. It uses no embeddings |
| `bleu` | 0–1 | Unigram precision against the best reference, with a brevity penalty |
| `rouge` | 0–1 | ROUGE-L F1 over whitespace-separated words |
| `word_error_rate` | 0–1 | 1 minus the word error rate, never below 0 (see [Evaluation Framework](./EVALUATION_FRAMEWORK.md#word-error-rate)) |
Every built-in scorer returns 0 when there is nothing to compare against. `llm_judge` appears in the `BuiltInScorer` type but is not registered: register a judge scorer under that name (see [LLM-as-Judge](#llm-as-judge)).
```typescript
const evaluator = new Evaluator();
const similarity = await evaluator.score('levenshtein', 'actual output', 'expected output');
const rougeL = await evaluator.score('rouge', generatedSummary, referenceSummary);
```
---
## Custom Scorers
A scorer is `(actual, expected, references, metadata) => number | Promise<number>`, registered by name:
```typescript
const evaluator = new Evaluator();
evaluator.registerScorer('json_valid', (actual) => {
try {
JSON.parse(actual);
return 1;
} catch {
return 0;
}
});
// Options travel in the test case's metadata
evaluator.registerScorer('json_has_key', (actual, _expected, _refs, metadata) => {
try {
return String(metadata?.key) in JSON.parse(actual) ? 1 : 0;
} catch {
return 0;
}
});
const testCase: EvalTestCase = {
id: 'json-1',
name: 'JSON output',
input: 'Return user data as JSON: name=Alice, age=30.',
metadata: { key: 'name' },
criteria: [
{ name: 'valid_json', description: 'Parses as JSON', weight: 2, scorer: 'json_valid' },
{ name: 'has_name', description: 'Has a name field', weight: 1, scorer: 'json_has_key' },
],
};
```
---
## LLM-as-Judge
`LLMJudge` grades an answer with a model through an `AIModelProviderManager`:
```typescript
import { LLMJudge, CRITERIA_PRESETS } from '@framers/agentos';
const judge = new LLMJudge({
llmProvider: providerManager, // an initialized AIModelProviderManager
modelId: 'gpt-4o', // default: the judge resolver's model (openai gpt-5.6)
});
const result = await judge.judge(
'How do I fix a memory leak in Node.js?', // input
answer, // actual output
undefined, // expected output (optional)
[
{ name: 'helpfulness', description: 'Actionable steps and the root cause', weight: 0.6 },
{ name: 'safety', description: 'No harmful or wrong advice', weight: 0.4 },
],
);
console.log(result.score, result.criteriaScores, result.reasoning, result.feedback);
```
- Without criteria, the judge uses its default set; `CRITERIA_PRESETS` has `codeGeneration`, `summarization`, `questionAnswering`, `creativeWriting` and `safety`.
- `temperature` defaults to 0.1, and the judge asks for a JSON answer. A provider error or an answer that is not JSON scores `errorScore` (default 0); JSON without `overallScore` scores 0.5.
- `AGENTOS_JUDGE_PROVIDER`, `AGENTOS_JUDGE_MODEL` and `AGENTOS_JUDGE_EFFORT` change the default judge model when the config pins none.
To use the judge inside an evaluation, register its scorer. The scorer reads the input from the test case's `metadata.input`:
```typescript
evaluator.registerScorer('llm_judge', judge.createScorer(CRITERIA_PRESETS.questionAnswering));
const judged: EvalTestCase = {
id: 'qa-1',
name: 'Photosynthesis',
input: 'What is photosynthesis?',
metadata: { input: 'What is photosynthesis?' },
criteria: [{ name: 'quality', description: 'Judge score', weight: 1, scorer: 'llm_judge' }],
};
```
---
## Running an Evaluation
```typescript
import { Evaluator, agent } from '@framers/agentos';
const assistant = agent({ provider: 'openai', instructions: 'You are a helpful assistant.' });
let caseNumber = 0;
async function agentFn(input: string): Promise<string> {
// A session per case keeps one case's history out of the next.
const reply = await assistant.session(`eval-${++caseNumber}`).send(input);
return reply.text;
}
const evaluator = new Evaluator();
const run = await evaluator.runEvaluation('assistant v1.2', testCases, agentFn, {
concurrency: 5, // cases per batch (default 3)
timeoutMs: 30_000, // per attempt (default 60000)
retries: 1, // extra attempts after an error or a timeout (default 1)
thresholds: { pass: 0.8 },
});
const m = run.aggregateMetrics;
console.log(`${m.passedTests}/${m.totalTests} passed, average ${m.avgScore.toFixed(3)}, p95 ${m.p95LatencyMs} ms`);
```
- Cases run in batches of `concurrency`; a batch waits for its slowest case.
- A timeout rejects the attempt; it does not cancel the agent call.
- A case whose attempts all fail gets score 0 with `error` set, and the run goes on.
- `totalTokens` and `totalCostUsd` stay 0: the agent function returns text only. `byCategory` is not filled.
- `EvalConfig.continueOnError`, `thresholds.warn`, `customScorers` and a criterion's `threshold` are declared and not read.
---
## Comparing Two Agents
```typescript
const fast = agent({ provider: 'openai', model: 'gpt-4o-mini' });
const strong = agent({ provider: 'openai', model: 'gpt-4o' });
let n = 0;
const runA = await evaluator.runEvaluation('baseline', testCases, (input) =>
fast.session(`a-${++n}`).send(input).then((r) => r.text));
const runB = await evaluator.runEvaluation('challenger', testCases, (input) =>
strong.session(`b-${++n}`).send(input).then((r) => r.text));
const comparison = await evaluator.compareRuns(runA.runId, runB.runId);
for (const metric of comparison.metrics) {
console.log(metric.name, metric.run1Value, metric.run2Value, metric.delta);
}
console.log(comparison.summary); // { improved, regressed, unchanged }
```
`compareRuns()` compares four aggregates: `passRate`, `avgScore`, `avgLatencyMs` and `p95LatencyMs`. It marks a metric `improved` when the second run's value is higher, latency included, so read a latency `delta` above 0 as slower. It does not compare test cases one by one; for that, match `runA.results` and `runB.results` by `testCaseId`.
`judge.compare(input, outputA, outputB, criteria)` judges two answers to one input and names the winner (`'A'`, `'B'`, or `'tie'` when the scores differ by less than 0.05).
---
## Keeping Runs
The evaluator keeps runs in memory: `getRun(runId)`, `listRuns(limit)` (newest first, default 50), `compareRuns()` and `generateReport()` see the runs of that instance only. To keep runs across processes, store `generateReport(runId, 'json')`, which serializes the whole `EvalRun`.
---
## Reports
```typescript
import { writeFile } from 'node:fs/promises';
const md = await evaluator.generateReport(run.runId, 'markdown'); // summary, latency, one line per case
const html = await evaluator.generateReport(run.runId, 'html'); // the same as HTML tables
await writeFile('./eval-report.html', html);
const run2 = JSON.parse(await evaluator.generateReport(run.runId, 'json')); // the EvalRun
console.log(run2.aggregateMetrics.passRate);
```
---
## Related Guides
- [EVALUATION_FRAMEWORK.md](./EVALUATION_FRAMEWORK.md) — types, scorers and the word error rate in detail
- [GETTING_STARTED.md](../getting-started/GETTING_STARTED.md) — first steps with AgentOS
- [EXAMPLES.md](../getting-started/EXAMPLES.md) — runnable recipes