From 4ea31a33222b5e6da65d097940b53a69e7a0e984 Mon Sep 17 00:00:00 2001 From: boumarate Date: Wed, 7 Oct 2026 19:20:04 -0400 Subject: [PATCH] fix: resolve issue #5 (& Logic Analysis The issue requires adding three b) --- typescript/README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/typescript/README.md b/typescript/README.md index 70950a2c..b33c15e5 100644 --- a/typescript/README.md +++ b/typescript/README.md @@ -2,3 +2,30 @@ The TypeScript implementation of threads. It must pass everything in `../spec/conformance/`. Rules: `../AGENTS.md` (Zod 4, strict tsconfig, Bun, Biome). + +## Built-in Eval Scorers + +`threads` provides three built-in evaluation scorers to grade your agent's runs without writing custom judges from scratch. These scorers can be used directly in a case's `expect` block or programmatically via `runEvals()`. + +### 1. Faithfulness (`faithfulness`) +- **Type**: Judge-based (uses the case's configured LLM judge model). +- **Score Range**: `0.0` to `1.0` +- **Description**: Evaluates whether the generated response is grounded in and supported strictly by the provided context/retrieved documents. It detects hallucinations or claims not supported by the source text. + +### 2. Answer Relevancy (`answerRelevancy`) +- **Type**: Judge-based (uses the case's configured LLM judge model). +- **Score Range**: `0.0` to `1.0` +- **Description**: Evaluates how relevant the generated response is to the user's input query. It penalizes redundant, incomplete, or off-topic answers. + +### 3. Tool-Call Correctness (`toolCallCorrectness`) +- **Type**: Deterministic (no LLM calls). +- **Score Range**: `0.0` to `1.0` +- **Description**: Compares the tool calls executed during the run against the expected tool calls. It checks for tool name matches, argument correctness, and execution order. + +--- + +## Usage Examples + +### Using Scorers in a Case's `expect` Block + +You can define these scorers directly inside your test cases. The judge-based scorers will automatically use the case's configured judge model. \ No newline at end of file