Eval suites
A suite is a named set of eval cases for one MCP server. Your first case lands in a suite named Default suite, and Create eval suite adds more, so you can group cases by purpose: the product core, one risky feature, one customer’s setup. A case can belong to several suites at once, and each suite keeps its own run history. Archiving a case removes it from every suite’s next run without deleting the memberships. Restore the case and its suites take it back.Running
Run one case starts an individual run. Run suite starts a suite run, which dispatches one individual run for every case in the suite. The suite run rolls the results up into one answer: how many cases ran, how many passed, and whether the suite passed as a whole. A suite run snapshots the suite’s membership when it starts. If you add or remove cases afterwards, the old result is labeled as historical instead of being presented as a result for the current suite. Runs carry their trigger in the run history: you started it by hand, a schedule fired, or your pipeline called the API and the run is tagged ci.The judge
After the tester finishes, a judge reads the full trace of the run. The judge is a separate model: it never talks to your server, it only reads what the tester did. It marks every criterion in the case as pass, partial, or fail, and links each mark to its evidence, for example the tool call that satisfied the criterion or the error that broke it. When the run gave a criterion nothing to check, the judge marks it not applicable instead. The marks turn into a score through a fixed rule, so the same marks always give the same score:
Scores 4 and 5 read as Passed, 2 and 3 read as Partial, and 0 and 1 read as Failed, everywhere in the product.
Run states
Pass rates count only judged runs: Not judged runs are excluded, and they are never counted as failures. This keeps a problem on Armature’s side from showing up as a regression in your MCP server.