Prompt Testing

Prompt testing compares prompt and model combinations against the same inputs and scores each result with the evaluators you choose, so you can see which combination actually performs best rather than guessing. Variable parametrization is supported.

There are two ways to use it:

• In the app (Beta) — build an experiment under Prompt Testing, run it, and compare results side by side in the browser. Good for exploration.

• From the CLI — define the experiment in a YAML file that lives in your repository. Good for reproducibility and for running in CI.

In the app

Go to Prompt Testing, choose New test, and give it the prompts, models, inputs (inline or from a dataset), and evaluators to score with. Results are listed per experiment and can be compared against each other.

From the CLI

Initialize a config file and run experiments:

scorable pt init
scorable pt run

Use a custom config path:

scorable pt run --config path/to/prompt-tests.yaml

The prompt-test command is an alias for pt.

Config file format

prompts:
  - "Extract info from: {{text}}"

inputs:
  - vars:
      text: "John Doe, [e-mail]"

# Or use a dataset instead of inline inputs:
# dataset_id: "<uuid>"

models:
  - gpt-5.4
  - gemini-3-flash

evaluators:
  - name: Precision
  - name: Confidentiality

# Optional: enforce structured output
# response_schema:
#   type: object
#   properties:
#     name: { type: string }

Using a dataset

Set dataset_id instead of inputs to run every item in a stored dataset.

Dataset columns are matched to your {{variables}} by name. A prompt containing {{text}} needs the dataset's items to carry a variable called text. When you import a CSV, include a header row so the columns are named — a headerless file is read positionally as request, response and expected output instead.

Three column names are special and map onto the item's own fields rather than its variables: request, response and expected_output. They are still addressable in a prompt, so {{request}} works for a dataset of question/answer pairs. Anything else becomes a named variable. A column that must stay a variable despite being named like one of those three can be written as variables.request.

If an item is missing a variable your prompt needs, the run fails for that item and names the missing variable rather than silently substituting the wrong column.

Refer to the CLI documentation for more details https://github.com/root-signals/rs-sdk/tree/main/cli#prompt-testing