> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/mcp-server.md).

# MCP Server

Scorable runs a hosted remote [MCP](https://modelcontextprotocol.io) server at `https://api.scorable.ai/mcp`. Point an MCP client at it with your API key and your agent gets 14 tools for finding, running, and authoring evaluations.

Use it when you want the agent to evaluate as part of its own reasoning loop: score a draft before showing it to the user, check a change against a rubric, or investigate why a score regressed. For scripted and CI use, the [CLI](/concepts-and-examples/cookbooks/cli.md) is usually the better fit.

## Get an API key

Grab one from [app.scorable.ai](https://app.scorable.ai) under **API keys**, or mint a free temporary one from the terminal:

```bash
curl -s -X POST https://api.scorable.ai/create-demo-user/ | jq -r .api_key
```

Keep it in an environment variable so it never lands in a config file you might commit:

```bash
export SCORABLE_API_KEY="your-key"
```

## Claude Code

```bash
claude mcp add --transport http scorable https://api.scorable.ai/mcp \
  --header "Authorization: Bearer $SCORABLE_API_KEY"
```

That registers the server for the current project. Add `--scope user` to make it available in every project, or `--scope project` to write it into `.mcp.json` and share it with your team through version control — in that case reference the variable rather than the key itself, so no secret is committed.

Verify it connected:

```bash
claude mcp list
```

## Codex

Codex configures remote servers in `~/.codex/config.toml` — `codex mcp add` is for stdio servers only:

```toml
[mcp_servers.scorable]
url = "https://api.scorable.ai/mcp"
bearer_token_env_var = "SCORABLE_API_KEY"
```

`bearer_token_env_var` reads the key from your environment at startup and sends it as a bearer token, so the key stays out of the file.

## Cursor, VS Code, and other clients

Anything that speaks streamable HTTP MCP works. Most clients use this shape:

```json
{
  "mcpServers": {
    "scorable": {
      "url": "https://api.scorable.ai/mcp",
      "headers": {
        "Authorization": "Bearer ${SCORABLE_API_KEY}"
      }
    }
  }
}
```

The server also accepts the `Api-Key` scheme used by the rest of the Scorable API, so `Authorization: Api-Key <key>` works if that fits your client better.

## The tools

**Finding what exists**

| Tool              | What it does                                            |
| ----------------- | ------------------------------------------------------- |
| `list_judges`     | Judges available to your organization, newest first     |
| `get_judge`       | One judge in full, including every evaluator it applies |
| `list_evaluators` | Evaluators, including the presets Scorable ships        |
| `get_evaluator`   | One evaluator with its rubric and required inputs       |
| `list_projects`   | Projects, for scoping other calls                       |

**Running evaluations**

| Tool            | What it does                                                 |
| --------------- | ------------------------------------------------------------ |
| `run_judge`     | Score a request/response pair against a judge, by id or name |
| `run_evaluator` | Score against a single evaluator                             |

**Authoring**

| Tool                                    | What it does                                                           |
| --------------------------------------- | ---------------------------------------------------------------------- |
| `generate_judge`                        | Build a judge from a plain-language description of what you care about |
| `create_judge` / `update_judge`         | Create or edit a judge from an explicit evaluator list                 |
| `create_evaluator` / `update_evaluator` | Create or edit a single evaluator and its rubric                       |

**Auditing**

| Tool                  | What it does                                                         |
| --------------------- | -------------------------------------------------------------------- |
| `list_execution_logs` | Past runs, filterable by judge, project, score, cost, tags, and date |
| `get_execution_log`   | One run in full, with per-evaluator scores and justifications        |

There are deliberately no delete tools. Removing a judge or evaluator stays a human action in the UI or CLI.

## Try it

Once connected, prompts like these resolve to tool calls:

```
What Scorable judges do I have?

Generate a judge that checks our support replies are concise, grounded in the
policy documents, and never promise refunds the policy does not allow.

Run that judge against this reply: "Absolutely, I've processed a full refund!"
given the policy "Refunds within 30 days, unopened items only."

Why did last night's evaluation scores drop? Check the execution logs.
```

The generated judge in that third example returns a score per evaluator with a written justification, so the agent can act on *why* something failed rather than just a number.

## Concepts worth knowing

An **evaluator** scores one quality of a response — faithfulness to context, relevance, safety, tone, or a custom rubric — returning a score between 0 and 1 with a justification. A **judge** is a reusable, named set of evaluators applied together, and is the normal unit of work.

The server tells your agent this on connect, so it generally picks the right tool without prompting. See [Concepts](/concepts-and-examples/usage.md) for the full model.

## Troubleshooting

**`401 Unauthorized`** — the key is missing, expired, or malformed. Confirm it works against the REST API first: `curl -H "Authorization: Api-Key $SCORABLE_API_KEY" https://api.scorable.ai/v1/judges/?limit=1`.

**`404 Not Found`** — the endpoint is disabled on that deployment. Self-hosted installations can turn it off; check that `MCP_ENABLED` is not set to `false`.

**A tool reports a missing field** — some evaluators require `contexts` or `expected_output`. Call `get_judge` and check each evaluator's `requires_contexts` and `requires_expected_output` before running.