CLI

The scorable CLI is a powerful tool for interacting with the Scorable API, particularly for managing and executing Judges. This guide provides a brief overview of its capabilities.

Installation

To install the CLI, run the following command:

curl -sSL https://scorable.ai/cli/install.sh | sh

Authentication

The CLI requires an API key to be set as an environment variable.

export SCORABLE_API_KEY=$MY_SCORABLE_API_KEY

Judge Management

The judge command is the primary entry point for all judge-related operations.

Creating a Judge

You can create a new judge using the create subcommand.

scorable judge create --name "My New Judge" --intent "To evaluate the quality of LLM responses."

Arguments:

• --name: The name of the judge (required).

• --intent: The intent or purpose of the judge (required).

• --stage: The stage of the judge.

• --evaluator-references: A JSON string of evaluator references. Example: [{"id": "<faithfulness_uuid>"}, {"id": "<truthfulness_uuid>"}]

Listing Judges

To see a list of all available judges, use the list subcommand.

scorable judge list

You can filter the list using various options like --search, --name, and --is-public.

Running a Judge

The execute subcommand allows you to run a judge with specific inputs.

scorable judge execute <judge_id> --request "What is the capital of France?" --response "Paris"