> ## Documentation Index
> Fetch the complete documentation index at: https://docs.armature.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> Run real agent sessions against your MCP server and judge the results

Evaluation answers one question before you ship: does my MCP server still handle the jobs users bring to it? You describe a user goal, a real agent tries to reach that goal through your tools, and a judge model checks the result against criteria you wrote.

<Note>
  Armature enables Evaluation workspace by workspace during the rollout. If your sidebar does not show an **Evaluation** section yet, write to [contact@armature.tech](mailto:contact@armature.tech) and we will turn it on.
</Note>

## The pieces

| Term               | What it means                                                                                                                                |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **Eval case**      | One scripted user goal plus the criteria that decide success. A real agent plays the user and works toward the goal against your MCP server. |
| **Eval suite**     | A named set of eval cases for one MCP server, for example "Product core" or "Billing".                                                       |
| **Suite run**      | One execution of every case in a suite. This is the check you run before shipping a change.                                                  |
| **Individual run** | One execution of one case.                                                                                                                   |

The sidebar's **Evaluation** section has three pages. **Eval suites** lists your suites, **Eval cases** lists every case across suites, and **Eval runs** holds the run history. Every suite, case, and run opens as an overlay with its own link, so you can share a result with a teammate.

## Before you start

Evaluation runs against a deployed MCP server, so Armature must know how to reach it. Register the server with its URL in **MCP Setup**, together with credentials when it needs them. The tester connects to your server over HTTP like any other MCP client, so point it at a deployed environment rather than your laptop.

## How a run works

1. A tester agent receives your case prompt and connects to your MCP server. It works toward the goal the way a real user session would: it reads your tool list, calls tools, and reacts to the results.
2. Armature records the full trace of the run: every tool call, every result, and the agent's reasoning.
3. A judge reads the trace. The judge is a separate model that marks each of your criteria as pass, partial, or fail, links each mark to its evidence in the trace, and marks a criterion the run never exercised as not applicable.
4. The marks produce a score from 0 to 5 and an outcome: **Passed**, **Partial**, or **Failed**. [Suites, runs, and scoring](/evaluation/suites-and-runs) explains the exact rule.

Runs start in three ways: you click run, a schedule fires, or your continuous integration (CI) pipeline calls the API. The run history labels each run with its trigger.

## Evaluation and analytics work alone or together

Evaluation does not require Armature session analytics. You can write every eval case yourself and run suites on their own.

If you do use analytics, Armature drafts eval cases for you from production evidence. Every use case and every issue in the dashboard carries a **Create eval** action: Armature pre-fills the prompt and criteria from real sessions, and you review the draft before saving it. A use case becomes a case that replays what users actually ask for. An issue becomes a regression case that proves the problem stays fixed.

## Plan limits

| Plan       | Targets per case | Scheduled cases | Shortest interval | Runs per month    |
| ---------- | ---------------- | --------------- | ----------------- | ----------------- |
| Free       | 1                | none            | n/a               | 100 finished runs |
| Starter    | 2                | 3               | 5 minutes         | no cap            |
| Pro        | 4                | 25              | 1 minute          | no cap            |
| Enterprise | no cap           | no cap          | 30 seconds        | no cap            |

A target is a model running in an agent harness, and each case can run against several targets. Free workspaces cannot schedule cases, but they can run cases by hand and from CI within the included 100 runs. A run only counts once it finishes, and a run that fails because of Armature does not count.

## Go deeper

<CardGroup cols={2}>
  <Card title="Eval cases" icon="clipboard-check" href="/evaluation/eval-cases">
    Learn how to write prompts and criteria that catch real regressions.
  </Card>

  <Card title="Suites, runs, and scoring" icon="layers" href="/evaluation/suites-and-runs">
    See how the judge scores a run and how to read the results.
  </Card>

  <Card title="Run evals from CI" icon="workflow" href="/evaluation/ci">
    Gate your releases on a suite run from your pipeline.
  </Card>

  <Card title="Eval tools on the Armature MCP" icon="wrench" href="/armature-mcp/tools#evaluation-tools">
    Let your coding agent list suites, start runs, and read verdicts.
  </Card>
</CardGroup>