Agent Evaluation and Regression Testing

Unit tests prove your tools work. Evaluations test whether the agent uses them correctly to solve realistic tasks.

Build a representative task set

Include normal requests, ambiguous requests, missing information, permission failures, noisy user language, and cases where the correct behavior is not to use a tool.

Measure observable behavior

• Correct final result.
• Correct tool/server selected.
• Required approval used.
• No forbidden tool used.
• Maximum call count.
• Latency and cost budget.

Use model graders carefully

Model-based grading is useful for semantic quality, but deterministic fields and exact policy violations should be checked with code.

Compare before upgrades

Run the same evaluation set when changing models, prompts, tool descriptions, MCP SDK versions, or orchestration logic. A new version is not an improvement unless your target metrics improve.