CLI

The scorable CLI is a powerful tool for interacting with the Scorable API, particularly for managing and executing Judges. This guide provides a brief overview of its capabilities.

Installation

To install the CLI, run the following command:

curl -sSL https://scorable.ai/cli/install.sh | sh

Authentication

The CLI requires an API key to be set as an environment variable.

export SCORABLE_API_KEY=$MY_SCORABLE_API_KEY

Judge Management

The judge command is the primary entry point for all judge-related operations.

Generating a Judge

Generate a complete judge from a plain-language description. Scorable picks the right evaluators automatically.

scorable judge generate \
  --intent "I am building a customer support chatbot. Evaluate that responses are helpful, accurate, and follow our refund policy."

Attach a policy document to make the generated evaluators check compliance against it:

# Upload a PDF and generate in one step
scorable judge generate \
  --intent "Evaluate responses against the attached policy." \
  --file ./refund-policy.pdf

# Reuse a previously uploaded file by ID
scorable judge generate \
  --intent "Evaluate responses against the attached policy." \
  --file-id <file_uuid>

Key options: --file (path to PDF/PNG/JPG — uploads and attaches), --file-id (UUID of an already-uploaded file), --stage, --extra-contexts (JSON), --reasoning-effort, --judge-id (regenerate an existing judge), --overwrite

Creating a Judge

You can create a new judge using the create subcommand.

scorable judge create --name "My New Judge" --intent "To evaluate the quality of LLM responses."

Arguments:

• --name: The name of the judge (required).

• --intent: The intent or purpose of the judge (required).

• --stage: The stage of the judge.

• --evaluator-references: A JSON string of evaluator references. Example: [{"id": "<faithfulness_uuid>"}, {"id": "<truthfulness_uuid>"}]

Listing Judges

To see a list of all available judges, use the list subcommand.

scorable judge list

You can filter the list using various options like --search and --name.

Running a Judge

The execute subcommand allows you to run a judge with specific inputs.

scorable judge execute <judge_id> --request "What is the capital of France?" --response "Paris"

Custom Model Management

Register a custom or self-hosted LLM, then reference it from evaluators and judges. See also Connect a model.

Creating a model

# SaaS provider — inline key
scorable model create --name my-gpt --model gpt-5.5 --key sk-...

# Self-hosted endpoint
scorable model create \
  --name azure/gpt-5.5 \
  --model azure/gpt-5.5 \
  --url https://my-azure-openai.openai.azure.com \
  --key sk-...

# Pipe the key from stdin to keep it out of shell history
echo "$MY_PROVIDER_KEY" | scorable model create --name my-gpt --model gpt-5.5 --key -

Key options: --name (required), --model, --url (custom endpoint), --key (provider API key; - reads from stdin), --max-token-count, --max-output-token-count.

Listing, inspecting, updating, deleting

scorable model list                                    # table: ID, Name, Provider, Visibility
scorable model get <model_id>
scorable model update <model_id> --max-output-token-count 4096   # PATCH — only sent fields change
scorable model delete <model_id>                       # prompts; pass --yes to skip