Coding Agents

Set up and operate Scorable entirely through Claude Code, Cursor, Codex, or any other coding agent, using the CLI, Agent Skills, and the MCP server.

Scorable is designed to be operated by AI coding agents. Everything you can do in the web UI is also available through the scorable CLI and the REST API, so an agent can set up evaluations, run them, and act on the results without a human clicking through the product.

There are four building blocks:

1. The Agent Skill for one-prompt setup

2. Project skills installed via scorable skills-add

3. The CLI as the agent's day-to-day interface

4. The MCP server for direct tool access

One-prompt setup with the Agent Skill

The fastest way to add Scorable evals to an application is to paste this into your coding agent:

Add Scorable evals by following https://scorable.ai/SKILL.md

The Agent Skill instructs the agent to analyze your codebase for LLM interaction points, install the CLI, generate a Judge that matches what your application does, integrate judge execution into your code, and verify the setup. The agent handles all steps itself; you only provide an API key if you do not want to use a temporary demo key.

Install Scorable skills into your project

The CLI can install a set of reusable agent skills into your repository:

scorable skills-add

This runs npx skills add root-signals/scorable-skills and makes the skills available to agents such as Claude Code and Cursor working in that project. Afterwards, prompts like "Integrate scorable evaluators" or "Add OTEL tracing to my agent and auto-evaluate every trace with Scorable" resolve to concrete, tested instructions instead of the agent improvising. See OTEL Trace Evaluation via CLI for a full walkthrough of the tracing skill.

The CLI is agent-friendly by design

Agents and scripts can drive the entire platform through the CLI:

• Non-interactive auth: the API key is read from the SCORABLE_API_KEY environment variable, and scorable auth demo-key creates a free temporary key without leaving the terminal.

• Project scoping: pass --project-id, or set SCORABLE_PROJECT_ID once for the whole session.

• Structured output: every command prints JSON on success and exits non-zero on failure, so results are easy to parse with jq and to gate on in scripts and CI. See Unit Testing in CI/CD for a ready-made GitHub Actions example.

• Full surface: judges, evaluators, models, datasets, annotations, calibration runs, prompt tests, execution logs, and OTEL trace filters are all manageable from the command line.

For example, an agent can create and run a judge end-to-end:

scorable judge generate --intent "Evaluate that support answers are accurate and polite."
scorable judge execute <judge_id> \
  --request "What is the refund policy?" \
  --response "You can return items within 30 days."

Evaluate your agent's own traces

Scorable can also evaluate the coding or production agent itself. Point any OpenTelemetry exporter at Scorable's OTLP endpoint and create a filter that automatically evaluates matching traces:

scorable otel-filter create --name "agent-quality" ...
scorable otel-trace list --since 1h

The CLI ships extractor manifests for common trace shapes, including one for Claude Code traces and one for OpenInference-instrumented agents. See OTEL Trace Evaluation via CLI for details.

MCP server

For agents that speak the Model Context Protocol, Scorable runs a hosted remote MCP server at https://api.scorable.ai/mcp that exposes judges and evaluators as tools. There is nothing to install — point your client at the URL with your API key:

claude mcp add --transport http scorable https://api.scorable.ai/mcp \
  --header "Authorization: Bearer $SCORABLE_API_KEY"

Use it when you want the agent to run evaluations inside its own reasoning loop rather than through shell commands. See MCP Server for the full tool list, Codex and Cursor configuration, and troubleshooting.

Machine-readable docs

These docs are available in agent-readable form at docs.scorable.ai/llms.txt and docs.scorable.ai/llms-full.txt. Point your agent at those URLs when it needs broader context than the Agent Skill provides.