> ## Documentation Index
> Fetch the complete documentation index at: https://docs.armature.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Suites, runs, and scoring

> How eval runs are produced, judged, and read

## Eval suites

A suite is a named set of eval cases for one MCP server. Your first case lands in a suite named **Default suite**, and **Create eval suite** adds more, so you can group cases by purpose: the product core, one risky feature, one customer's setup.

A case can belong to several suites at once, and each suite keeps its own run history. Archiving a case removes it from every suite's next run without deleting the memberships. Restore the case and its suites take it back.

## Running

**Run one case** starts an individual run. **Run suite** starts a suite run, which dispatches one individual run for every case in the suite. The suite run rolls the results up into one answer: how many cases ran, how many passed, and whether the suite passed as a whole.

A suite run snapshots the suite's membership when it starts. If you add or remove cases afterwards, the old result is labeled as historical instead of being presented as a result for the current suite.

Runs carry their trigger in the run history: you started it by hand, a schedule fired, or your pipeline called the API and the run is tagged **ci**.

## The judge

After the tester finishes, a judge reads the full trace of the run. The judge is a separate model: it never talks to your server, it only reads what the tester did. It marks every criterion in the case as pass, partial, or fail, and links each mark to its evidence, for example the tool call that satisfied the criterion or the error that broke it. When the run gave a criterion nothing to check, the judge marks it not applicable instead.

The marks turn into a score through a fixed rule, so the same marks always give the same score:

| Score | Meaning                                                                                            |
| ----- | -------------------------------------------------------------------------------------------------- |
| **5** | Every criterion the judge applied passed.                                                          |
| **4** | Every required criterion the judge applied passed, and at least one optional criterion fell short. |
| **3** | No required criterion failed, and at least one required criterion passed only partially.           |
| **2** | Exactly one required criterion failed.                                                             |
| **1** | Two or more required criteria failed.                                                              |
| **0** | The tester never engaged the MCP server.                                                           |

Scores 4 and 5 read as **Passed**, 2 and 3 read as **Partial**, and 0 and 1 read as **Failed**, everywhere in the product.

<Warning>
  A criterion marked not applicable does not lower the score, even when it is required. **Passed** therefore means no applied criterion failed, not that every criterion was verified. When a release gate must confirm every required behavior, read the per-criterion marks in the run's evaluation instead of the outcome alone, and write criteria the run always exercises.
</Warning>

## Run states

| Label              | Meaning                                                                                                              |
| ------------------ | -------------------------------------------------------------------------------------------------------------------- |
| **Passed**         | The judge scored the run 4 or 5.                                                                                     |
| **Partial**        | The judge scored the run 2 or 3.                                                                                     |
| **Failed**         | The judge scored the run 0 or 1, or the run itself broke: the tester crashed, judging crashed, or the run timed out. |
| **Not judged**     | The run hit a problem on Armature's side or at the model provider, so no verdict exists.                             |
| **Not run**        | Armature declined to start the run, for example when a Free workspace has used its monthly runs.                     |
| **Canceled**       | You stopped the run before it finished.                                                                              |
| **Running**        | The tester or the judge is still working.                                                                            |
| **Waiting to run** | The run sits in the queue.                                                                                           |

Pass rates count only judged runs: **Not judged** runs are excluded, and they are never counted as failures. This keeps a problem on Armature's side from showing up as a regression in your MCP server.

## Included runs

Free workspaces include 100 finished runs per month, pooled across every case, suite, and target. Past that number, new runs show **Not run** until the month resets. A run counts once it finishes, and a run that fails because of Armature does not count. Paid plans have no monthly run cap.