> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-5240-1789624976482.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CI / CD

> Run MCP health checks, conformance suites, and evals in GitHub Actions, GitLab CI, and other CI environments

Run `mcpjam` in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.

## GitHub Actions

### Authentication

There are three ways to authenticate in CI, depending on your server setup.

#### Option 1: Headless OAuth login

Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={null}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: OAuth login (headless)
        run: |
          set -euo pipefail
          npx -y @mcpjam/cli@latest oauth login \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --protocol-version 2025-11-25 \
            --registration dcr \
            --auth-mode headless \
            --format json > /tmp/oauth-result.json
          TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
          rm -f /tmp/oauth-result.json
          if [ -z "$TOKEN" ]; then
            echo "::error::OAuth login did not return an access token"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "MCP_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json
```

#### Option 2: Refresh token

Best when you already have a refresh token from a previous `oauth login`. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.

**Secrets needed:**

| Secret              | Description                                         |
| ------------------- | --------------------------------------------------- |
| `MCP_SERVER_URL`    | Your MCP server URL                                 |
| `MCP_REFRESH_TOKEN` | OAuth refresh token from a previous login           |
| `MCP_CLIENT_ID`     | OAuth client ID (required with refresh tokens)      |
| `MCP_CLIENT_SECRET` | OAuth client secret (if the client is confidential) |

```yaml theme={null}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: |
          npx -y @mcpjam/cli@latest server doctor \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --refresh-token ${{ secrets.MCP_REFRESH_TOKEN }} \
            --client-id ${{ secrets.MCP_CLIENT_ID }} \
            --client-secret ${{ secrets.MCP_CLIENT_SECRET }} \
            --format json
```

<Tip>
  To get a refresh token, run `mcpjam oauth login` locally with `--format json` and grab `.credentials.refreshToken` from the output.
</Tip>

#### Option 3: Static API key

Best when your server uses a non-expiring API key instead of OAuth.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |
| `MCP_API_KEY`    | Static API key      |

```yaml theme={null}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token ${{ secrets.MCP_API_KEY }} --format json
```

#### Option 4: No auth

Some servers don't require authentication at all.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={null}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --format json
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).

```yaml theme={null}
      - name: Snapshot before
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > before.json

      # your deploy step here

      - name: Snapshot after
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > after.json

      - name: Diff
        run: diff <(jq -S . before.json) <(jq -S . after.json)
```

### OAuth conformance suite

Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.

```yaml theme={null}
      - name: OAuth conformance
        run: |
          npx -y @mcpjam/cli@latest oauth conformance-suite \
            --config ./oauth-matrix.json \
            --reporter junit-xml > report.xml

      - name: Upload test report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: oauth-conformance
          path: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

### Protocol conformance suite

Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.

```yaml theme={null}
      - name: Protocol conformance
        run: |
          npx -y @mcpjam/cli@latest protocol conformance-suite \
            --config ./protocol-conformance.json \
            --reporter junit-xml > protocol-report.xml

      - name: Upload protocol report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: protocol-conformance
          path: protocol-report.xml
```

### MCP Apps conformance suite

Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.

```yaml theme={null}
      - name: MCP Apps conformance
        run: |
          npx -y @mcpjam/cli@latest apps conformance-suite \
            --config ./apps-conformance.json \
            --reporter junit-xml > apps-report.xml

      - name: Upload apps report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: apps-conformance
          path: apps-report.xml
```

Single-run `protocol conformance`, `oauth conformance`, and `apps conformance` also accept `--reporter junit-xml` when you only need one target/check selection instead of a suite config file.

***

## GitLab CI

The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and `.gitlab-ci.yml` syntax.

### Authentication

#### Headless OAuth login

```yaml theme={null}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
  script:
    - |
      npx -y @mcpjam/cli@latest oauth login \
        --url "$MCP_SERVER_URL" \
        --protocol-version 2025-11-25 \
        --registration dcr \
        --auth-mode headless \
        --format json > /tmp/oauth-result.json
      TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
      rm -f /tmp/oauth-result.json
      if [ -z "$TOKEN" ]; then
        echo "OAuth login did not return an access token"
        exit 1
      fi
      export MCP_TOKEN="$TOKEN"
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Refresh token

```yaml theme={null}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_REFRESH_TOKEN: $MCP_REFRESH_TOKEN
    MCP_CLIENT_ID: $MCP_CLIENT_ID
    MCP_CLIENT_SECRET: $MCP_CLIENT_SECRET
  script:
    - |
      npx -y @mcpjam/cli@latest server doctor \
        --url "$MCP_SERVER_URL" \
        --refresh-token "$MCP_REFRESH_TOKEN" \
        --client-id "$MCP_CLIENT_ID" \
        --client-secret "$MCP_CLIENT_SECRET" \
        --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Static API key

```yaml theme={null}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_API_KEY: $MCP_API_KEY
  script:
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_API_KEY" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes.

```yaml theme={null}
mcp-tool-diff:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_TOKEN: $MCP_TOKEN
  script:
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > before.json
    # your deploy step here
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > after.json
    - jq -S . before.json > /tmp/before-sorted.json
    - jq -S . after.json > /tmp/after-sorted.json
    - diff /tmp/before-sorted.json /tmp/after-sorted.json
    - rm -f /tmp/before-sorted.json /tmp/after-sorted.json
```

### OAuth conformance suite

```yaml theme={null}
mcp-oauth-conformance:
  image: node:20
  script:
    - |
      npx -y @mcpjam/cli@latest oauth conformance-suite \
        --config ./oauth-matrix.json \
        --reporter junit-xml > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

***

## Evals in CI

There are two ways to wire MCPJam evals into a pipeline: trigger a **hosted eval run** with the CLI, or run evals **locally with the SDK** and upload the results. Both authenticate with an MCPJam API key (`sk_…`) from **Settings → API keys**.

### GitHub Action for hosted evals

The MCPJam evals action runs an existing hosted suite, waits for its result, and
uploads reports to GitHub. It sets up Node automatically. It tests the suite's
saved server; it does not build or deploy your PR's source code.

<Note>
  The action's `evals-v1` tag must be published after its live smoke test passes
  before the example below can be used. The direct CLI integration in the next
  section remains available independently.
</Note>

Save this as `.github/workflows/mcpjam.yml`, then replace the project and suite:

```yaml theme={null}
name: MCPJam evals

on:
  workflow_dispatch:
  pull_request:

permissions:
  contents: read

jobs:
  evals:
    # Fork PRs do not receive this repo's secrets.
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: MCPJam/inspector/actions/evals@evals-v1
        with:
          # Add your API key as MCPJAM_API_KEY in GitHub Actions secrets.
          api-key: ${{ secrets.MCPJAM_API_KEY }}
          project: 'My project'
          suite: 'My eval suite'

          # Optional: let the gate decide pass or fail.
          # gate: true
          # min-pass-rate-percent: 95
          # baseline-sha: '<commit with an existing eval run>'
```

Create the `MCPJAM_API_KEY` repository secret under GitHub **Settings → Secrets and
variables → Actions**. The YAML contains only a reference to it, never the key.

Without `gate: true`, the run itself must pass. With gates enabled, each gate
must pass or be waived; a lower threshold or existing waiver can clear an eval
failure. Failed launches, incomplete runs, missing reports, and upload failures
remain blocking. A run waiver does not clear a separate suite-policy failure.

Use `baseline-run` instead of `baseline-sha` to name a baseline by run ID; never
provide both. Optional `wait-timeout-ms` replaces the CLI wait default. The CLI
version is pinned to `5.7.1`, with an exact-version override via `cli-version`.
There are no automatic retries, and the generated idempotency key reuses launched
runs when the same workflow job is retried.

The action exposes `run-ids`, `run-exit-code`, `gate-exit-codes`, `report-path`,
and `artifact-url`. It saves the eval JSON report and optional gate JUnit reports
before reporting failure. See the
[action README](https://github.com/MCPJam/inspector/blob/main/actions/evals/README.md)
for all inputs, outputs, retry details, and release status. GitHub.com with Ubuntu
runners is supported in this first version.

### Trigger a hosted eval suite with the CLI

`mcpjam cloud eval run` starts an asynchronous run of a suite that lives in your MCPJam project. Without `--wait`, it prints a launch receipt and returns immediately. In CI, add `--wait` and `--out` to write a structured JSON report after every launched run reaches a terminal state.

**Secrets needed:**

| Secret           | Description             |
| ---------------- | ----------------------- |
| `MCPJAM_API_KEY` | MCPJam API key (`sk_…`) |

```yaml theme={null}
      - name: Run hosted eval
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          npx -y @mcpjam/cli@latest cloud eval run \
            --suite "Nightly regression" \
            --project "My project" \
            --wait \
            --out eval-report.json \
            --format json > eval-result.json
          echo "Completed run $(jq -r '.runs[0].id' eval-result.json)"

      - name: Gate and write JUnit
        # Always run this step, even if the run above exited 1 on a failed
        # verdict: a default `if:` is `success()`, and skipping this step
        # would lose the gate's configurable policy and its JUnit report
        # exactly when the run failed. The job still ends up failed either
        # way — the run step above already exited nonzero.
        if: always()
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          RUN_ID=$(jq -r '.runs[0].id' eval-result.json)
          npx -y @mcpjam/cli@latest cloud eval gate \
            --run "$RUN_ID" \
            --project "My project" \
            --wait \
            --min-pass-rate-percent 100 \
            --reporter junit-xml \
            --out eval-report.xml

      - name: Upload eval reports
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: hosted-eval
          path: |
            eval-report.json
            eval-report.xml
```

<Note>
  **Runs launched from Actions badge as GitHub, and show GitHub provenance columns.** The CLI
  declares what it is on every launch, so a run started from a workflow shows
  **GitHub** in the Runs table rather than the generic **API** that every hosted
  launch used to show. Filtering the Runs table by the **GitHub** source reveals
  separate **Commit**, **PR**, and **Branch** columns populated from the standard
  `GITHUB_*` environment — nothing to configure. Outside CI the same command
  declares **CLI**.

  That commit is also what `eval compare --baseline-sha` resolves against, so a
  run launched by the CLI in CI can now be compared to a previous commit's run the
  same way an SDK-reported one can.

  This is a *declared* label, deliberately: the platform still stamps the run's
  own `source` server-side and records the verified credential separately, so a
  badge can never claim more than "the client said so". No secret that could prove
  it can live in a public npm package.
</Note>

In human format, `eval run` prints a `View:` line after the payload so you can open the run directly from the terminal:

```text theme={null}
View: https://app.mcpjam.com/evaluate/suite/<suiteId>/runs/<runId>?project=<projectId>
```

This line is only emitted in human format — `--format json` output is unchanged, so scripts that parse the JSON stream are unaffected.

Use `--wait-timeout <ms>` to replace the 35-minute default. `--out` defaults to the structured JSON format; add `--reporter junit-xml` to write JUnit XML instead, or `--reporter html` for a self-contained HTML page (decision summary + failures only — traces, parity, and history are paid tiers not included here). When `--reporter` is present, the same report is also written to stdout.

<Warning>
  **`eval run --wait` sets a verdict-based exit code.** `0` pass, `1` a completed run's verdict failed (the ONLY condition that produces `1`), `2` usage error or an invalid suite file, `3` auth failed (no credential, or the platform rejected it, at launch or mid-wait), `4` a connection/setup failure this CLI itself observed before evaluation ran (or a local `--out` write failure), `5` no valid verdict — `inconclusive`, a null/unrecognized result, a run status of failed/cancelled/timed-out, or a wait that hit its deadline. A multi-target launch merges these worst-of across every waited run, in the order `1 > 3 > 4 > 5 > 0`.

  Retry guidance: `4` and `5` mean infrastructure, or an absence of observation — nothing here says the server is wrong, so retrying the CI job is reasonable. But a bare re-run is not automatically safe: `eval run --suite` only dedupes against an in-flight or already-completed launch when you pass a **stable** `--idempotency-key`, and exit `5` can mean the run is still running (a wait that hit its deadline) — without that key, a retry can start a second paid run alongside the first rather than resuming it. Pass `--idempotency-key` (or poll/resume the run id already in the receipt) before retrying on `4` or `5`. `3` means fix the credential first; it poisons every other observation in the same launch. Never retry blindly on `1` — that code is reserved for a run the platform actually graded as failed.

  Either `eval run --wait` or `eval gate` already fails the job on its own — the `eval gate` step above adds a configurable pass/fail *policy* (thresholds, per-scorer gates) and baseline comparison on top of the same verdict, so keep it when you want more than "did this run's own verdict pass".
</Warning>

<Note>
  `eval gate` sets a verdict-based exit code, and writes its report before doing so: `0` passed **or waived**, `1` an eval verdict failed, `2` usage error, `3` incomplete or non-gateable. Infrastructure conditions never map to `1`, so retrying on `3` is safe. This is a **different, four-code contract from `eval run --wait`** above — `gate`'s `3` means "incomplete", not the six-code scheme's `3` ("auth failed"), and the two are deliberately not unified (see the [CLI reference](/cli/reference#cloud-eval-gate) for why). `eval status` also prints a `View:` line in human format, identical to the one `eval run` prints.
</Note>

#### Gating on cost

Cost is the question people actually have about a change: *did this make the
suite more expensive?* Gate it relatively rather than absolutely — an absolute
ceiling goes stale on every prompt and model change, while a percentage
against the previous run keeps meaning the same thing.

```bash theme={null}
npx -y @mcpjam/cli@latest cloud eval compare --run "$RUN_ID" \
  --project "My project" \
  --max-cost-increase-percent 10
```

With no `--base-run` or `--base-sha`, the baseline is the nearest earlier
completed run in the same suite — which is what "did my change make this more
expensive?" usually means.

An absolute ceiling is also available, for a suite whose budget is fixed:

```bash theme={null}
npx -y @mcpjam/cli@latest cloud eval gate --run "$RUN_ID" \
  --project "My project" \
  --max-cost-usd 0.50
```

Both report **non-gateable** (exit `3`) rather than passing when the cost is
unknown or only partly measured — a run on your own API keys, a harness run,
or a run only some of whose iterations MCPJam priced. Exit `3` is safe to retry
and safe to treat as "no opinion"; what it never does is let an unpriced run
through as if it were cheap.

#### Waiving a gate

A run whose gate failed can be overridden by an authorized user until an expiry
they name, so a release is not blocked while a known regression is being fixed:

```bash theme={null}
mcpjam cloud eval gate waive --run "$RUN_ID" --reason "hotfix ships today; tracked in ENG-4821" --expires-in 3d
```

`eval gate` then exits `0` and reports the outcome as `waived`. It is **not**
reported as a pass: the run keeps its failed result, the failing verdicts stay
in the report, and the waiver — who granted it, why, and until when — is named
in every artifact the command writes, including the JUnit XML your CI job
uploads (as a `<skipped>` element, so it neither fails the build nor renders as
a clean green row).

Only a real verdict failure is waivable. A cancelled run, a `--wait` timeout, or
a network failure still exits `3` with a waiver in place — those established
nothing, and a waiver granted for a regression is not consent to ship on an
infrastructure failure.

Waivers expire, and expiry is enforced on both sides: the platform republishes
the GitHub Check Run when the waiver lapses, and the CLI re-derives the expiry
itself rather than trusting the platform's answer. `mcpjam cloud eval gate
unwaive --run "$RUN_ID"` ends one early.

<Warning>
  The waiver reason is stored **unredacted** and readable by anyone who can see
  the suite, for as long as the suite exists. Never put secrets, tokens, or
  customer data in it.
</Warning>

#### Decision summary

`eval run --wait`, `eval status`, `eval gate` and `eval compare` all read one versioned object — the **run decision summary** — and every output format restates it. Where each command puts it:

| Command           | `--format json`                          | `--format human`                           | `--out` / `--reporter`          |
| ----------------- | ---------------------------------------- | ------------------------------------------ | ------------------------------- |
| `eval run --wait` | `decisionSummary` on the stdout receipt  | block on stdout, after the receipt         | `decisionSummary` on the report |
| `eval status`     | `decisionSummary` on the stdout document | block on stdout, above the `View:` line    | —                               |
| `eval gate`       | `decisionSummary` beside `gate`          | block on **stderr**, under the gate report | `decisionSummary` on the report |
| `eval compare`    | `decisionSummary` beside `compare`       | block on **stderr**, under the gate report | `decisionSummary` on the report |

Two scoping rules that are easy to miss. `eval run --wait` attaches a summary only when the invocation launched **one** run: a fan-out has several, and labelling a receipt about N runs with the decision of one would be a false claim rather than a partial one. `eval compare` reports the **compare side's** decision only — the baseline's failures are a different run's diagnostics, and printing them here would read as this run's.

```text theme={null}
Decision summary: failed (per-case grading) — 2/3 case variants passed, 1 failed
  Why: a case did not meet its pass threshold
  Diagnostics: 2 non-passing of 9 iterations examined (the complete set)
  First break: Tool call — the call arguments did not match what the case expects (1 of 2 measured iterations)
  Fetch order (c_orders, iteration 2) — failed
    First failed stage: Tool call — the call arguments did not match what the case expects
    Failure category: call arguments
    Expected tool calls: fetch_order
    Observed failure: server rejected arguments
    Evidence at Tool call: span ids span-call-2; reasons order_id must be a string
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_2/trace
    Next action: review the authored arguments against the tool input schema
  Setup abort (c_setup, iteration 5) — failed
    First failed stage: none was established — the run never reached the server's stages
    Failure category: setup
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_5/trace
    Next action: check the server connection and environment configuration
```

Four things about it are worth knowing before you script against it.

**The counts carry the population they count.** `measurementUnit` is `caseVariant` under per-case grading — one case under one provider/model, with its configured iterations as *trials inside it* — and `trial` under a suite-wide accuracy threshold. A 3-case suite with 5 iterations is legitimately "3" under one unit and "15" under the other, so a count quoted without its unit is not a fact, and a rate is never converted across that line.

**The summary explains the verdict; it never re-decides it.** Under per-case grading the run's own decision is the authority for the verdict, the rates, the validity phase and the per-case aggregation, and it is carried through on `decision`. The per-iteration diagnostics sit *underneath* that: a case can pass with a failing iteration in it, so tallying the diagnostics gives a different answer than the platform reached.

**`notEstablished` is not a failure.** It is a fourth verdict meaning no verdict exists at all — the run is unfinished, it stopped before finishing, or its decision could not be read. `undecided.reason` says which. It is also not `inconclusive`, which *is* a decision: the validity phase ran and withheld a verdict because the run did not measure the server well enough.

**A page of diagnostics says whether it is the whole story.** `diagnostics.complete` is true only when the listed iterations are the run's entire non-passing set, and `scannedIterations` says how many were examined — so an empty list from a complete page ("nothing failed") is distinguishable from an empty list from a partial one ("we did not look").

Evidence is scoped to the claim it supports: for a measured failure the span ids, prompt indexes and reasons come from the first failed stage's row alone, and a setup abort or evaluator error keeps a stage-less pointer rather than naming a stage nothing established.

`eval status` prints the block only when a terminal run did not pass — a clean pass has nothing to diagnose. `--format json` stays exactly one parseable document in every case: the summary rides *inside* it, never as a second block appended after it. If the summary cannot be fetched, it is omitted rather than failing the command.

**The human block leads with where the chain broke.** Under the diagnostics headline, before any per-iteration detail, a non-passing run gets one line naming the earliest stage at which a readable iteration stopped, why, and how many iterations stopped there:

```text theme={null}
  First break: Tool call — the call arguments did not match what the case expects (2 of 3 measured iterations)
```

"First" means earliest in chain order — `connection → discovery → selection → call → response → userValue` — never "most common", so the count beside it is what tells you whether the run had one problem or several. When the breaks are spread the line says so (`earliest of 3 stages that broke`), and when some chains could not be read it says that too (`1 more had no readable chain`), because otherwise the denominator quietly shrinks to the iterations that happened to validate. A run that reached no stage at all — a setup abort, an evaluator error — names its bucket instead of inventing a location.

The line is **not** a diagnosis. A first failed stage is a location and a failure category is a bucket; neither on its own says what to change.

`eval status --stages` expands each failing iteration to all six chain rows with their states and reasons. It is off by default: six rows per iteration is a lot of terminal on a run with twenty failures, and the first-break line above already carries the answer. Human output is not a stable contract — script against `--format json`, where the enums travel as enums.

Hosted runs execute LLM iterations on the platform and consume your organization's credits or configured provider keys. See the [`cloud eval` command reference](/cli/reference#cloud-eval-commands) for the full surface, including `cloud eval judge` (request LLM-as-judge grading on a finished run), `cloud eval validate` (offline suite-file validation), `cloud eval export` (write a hosted suite to a local file), `cloud eval github list/connect` (GitHub checks integration), and more.

### Upload SDK eval results

If you instead run evals inside your own CI job with [`@mcpjam/sdk`](/sdk/concepts/running-evals) (`EvalTest` / `EvalSuite`), set `MCPJAM_API_KEY` and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):

```yaml theme={null}
      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx vitest run evals/
```

#### Without a provider key

Prefix the model with `mcpjam/` and MCPJam runs it on your organization's credits, so `MCPJAM_API_KEY` is the only secret the job needs:

```yaml theme={null}
      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          EVAL_MODEL: mcpjam/anthropic/claude-sonnet-4.5
        run: npx vitest run evals/
```

See [LLM Providers](/sdk/reference/llm-providers) for which models are hosted and how the spend is capped.

The SDK automatically attaches available CI metadata on GitHub Actions, GitLab CI, CircleCI, Buildkite, Jenkins, Vercel, and Netlify. You do not need to pass `mcpjam.ci`; an explicit object still replaces detection, and `mcpjam: { ci: {} }` disables it. This SDK behavior is separate from the hosted CLI launch detector, which supports GitHub Actions only.

See [Save Results to MCPJam](/sdk/concepts/saving-results) for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON).