Evaluation

Evaluation answers one question before you ship: does my MCP server still handle the jobs users bring to it? You describe a user goal, a real agent tries to reach that goal through your tools, and a judge model checks the result against criteria you wrote.

​

The pieces

The sidebar’s Evaluation section has three pages. Eval suites lists your suites, Eval cases lists every case across suites, and Eval runs holds the run history. Every suite, case, and run opens as an overlay with its own link, so you can share a result with a teammate.

​

Before you start

Evaluation runs against a deployed MCP server, so Armature must know how to reach it. Register the server with its URL in MCP Setup, together with credentials when it needs them. The tester connects to your server over HTTP like any other MCP client, so point it at a deployed environment rather than your laptop.

​

How a run works

1. A tester agent receives your case prompt and connects to your MCP server. It works toward the goal the way a real user session would: it reads your tool list, calls tools, and reacts to the results.
2. Armature records the full trace of the run: every tool call, every result, and the agent’s reasoning.
3. A judge reads the trace. The judge is a separate model that marks each of your criteria as pass, partial, or fail, links each mark to its evidence in the trace, and marks a criterion the run never exercised as not applicable.
4. The marks produce a score from 0 to 5 and an outcome: Passed, Partial, or Failed. Suites, runs, and scoring explains the exact rule.

Runs start in three ways: you click run, a schedule fires, or your continuous integration (CI) pipeline calls the API. The run history labels each run with its trigger.

​

Evaluation and analytics work alone or together

Evaluation does not require Armature session analytics. You can write every eval case yourself and run suites on their own. If you do use analytics, Armature drafts eval cases for you from production evidence. Every use case and every issue in the dashboard carries a Create eval action: Armature pre-fills the prompt and criteria from real sessions, and you review the draft before saving it. A use case becomes a case that replays what users actually ask for. An issue becomes a regression case that proves the problem stays fixed.

​

Plan limits

A target is a model running in an agent harness, and each case can run against several targets. Free workspaces cannot schedule cases, but they can run cases by hand and from CI within the included 100 runs. A run only counts once it finishes, and a run that fails because of Armature does not count.

​

Go deeper