Anthropic Adds Plugin Evals to Claude Code With Six Graders
TL;DR
- Anthropic's new `claude plugin eval` command runs each test case twice, with and without the plugin loaded, and reports the delta as the plugin's real contribution.
- Six grader types ship: four free ones (regex, tool_used, tool_order, file_exists) computed from transcripts and files, plus billable llm and baseline judges.
- The workflow requires Claude Code v2.1.269 and exposes CI flags like `--threshold 0.8` and `--max-cost-usd 20` to fail builds below a score or over budget.
Anthropic's new `claude plugin eval` command runs each test case twice by default and calls the difference the plugin's contribution. In the docs' own words: "By default every case runs twice: a with-arm where the plugin is loaded and a without-arm where it is not." That gap, Δ, is the score.
There are six grader types. Four cost nothing because they read the transcript and files on disk: regex, tool_used, tool_order, and file_exists. Two others, llm and baseline, invoke a judge model and incur charges. Test cases live in an `evals/` directory alongside `prompt.md` files and a `graders/` folder; frontmatter sets `max_turns` (default 10) and `timeout_seconds` (default 300).
The recommended CI invocation, per MarkTechPost, chains `--threshold 0.8`, `--max-cost-usd 20`, `--model claude-sonnet-5`, `--judge-model claude-haiku-4-5`, and `--no-publish`, turning the eval into a build gate that fails below the score threshold or over the dollar cap. The feature requires Claude Code v2.1.269.
The article names one recurring failure mode: "a Δ near zero with the `tool_used: Skill` grader failing, which means Claude is not choosing the skill on natural phrasing." A skill can be syntactically valid and still never fire. It lands in a busy week for coding-agent tooling on our tracker, alongside Cursor's Projects beta.
Originally reported by marktechpost.com
Read the original article →Original headline: Anthropic Adds Plugin Evals to Claude Code With Six Grader Types and CI Threshold Gate