What a case contains
Write the prompt as a user goal
The tester agent receives your prompt and your MCP server’s tools, and works toward the goal on its own. So write the goal, not a click path:Find the March invoices for the customer Acme Corp and export them as a CSV file.The agent decides which tools to call, in what order, and how to react when a call fails. This tests whether your tools let an agent succeed, not whether one fixed sequence still executes.
Write criteria the judge can verify
The judge marks every criterion as pass, partial, or fail, and links each mark to evidence in the trace. A criterion the run never exercises is marked not applicable and does not count toward the score, so write criteria the run always reaches. Mark a criterion as required when the case must fail without it. Five rules make criteria reliable:- Write each criterion as an outcome someone could check from the outside, not as a style preference. “The export contains 3 invoices” works. “The agent communicates clearly” does not.
- Test one behavior per criterion. When a criterion fails, it should point at one missing behavior.
- Spell out the side effects the tester must avoid, for example “the tester does not modify any invoice”.
- Name the evidence you expect: a table row, a tool call status, a resource id.
- Use data names scoped to the test, such as run-specific customer names, so cleanup and diagnosis can tell test data from real data.
Pick the targets
A target is a model running in an agent harness. A harness is the agent product that drives the model, and it shapes how the model reads schemas, retries, and recovers. New targets use a Claude model in Claude Code or a GPT model in Codex. Your plan sets how many targets one case can run: 1 on Free, 2 on Starter, 4 on Pro, and no cap on Enterprise. Give an important case more than one target. A change that passes in Claude Code and fails in Codex is exactly the kind of regression a single target hides.Create cases three ways
- From the Evaluation pages. Click Create eval case, then Create and run or Create without running. You pick the suites the case joins as you create it.
- From a use case. If you use Armature analytics, every use case drawer carries a Create eval action. Armature pre-fills the prompt and criteria from real sessions in that use case, and you review the draft before saving.
- From an issue. Issue drawers carry the same Create eval action. The draft becomes a regression case: it replays the situation that failed and checks that the problem stays fixed.