> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/concepts-and-examples/usage/prompt-testing.md).

# Prompt Testing

Prompt testing compares prompt and model combinations against the same inputs and scores each result with the evaluators you choose, so you can see which combination actually performs best rather than guessing. Variable parametrization is supported.

There are two ways to use it:

* **In the app** (Beta) — build an experiment under **Prompt Testing**, run it, and compare results side by side in the browser. Good for exploration.
* **From the CLI** — define the experiment in a YAML file that lives in your repository. Good for reproducibility and for running in CI.

## In the app

Go to [Prompt Testing](https://scorable.ai/prompt-testing), choose **New test**, and give it the prompts, models, inputs (inline or from a dataset), and evaluators to score with. Results are listed per experiment and can be compared against each other.

## From the CLI

### Initialize a config file and run experiments:

```
scorable pt init
scorable pt run
```

Use a custom config path:

```
scorable pt run --config path/to/prompt-tests.yaml
```

The `prompt-test` command is an alias for `pt`.

#### Config file format

```
prompts:
  - "Extract info from: {{text}}"

inputs:
  - vars:
      text: "John Doe, john@example.com"

# Or use a dataset instead of inline inputs:
# dataset_id: "<uuid>"

models:
  - gpt-5.4
  - gemini-3-flash

evaluators:
  - name: Precision
  - name: Confidentiality

# Optional: enforce structured output
# response_schema:
#   type: object
#   properties:
#     name: { type: string }
```

#### Using a dataset

Set `dataset_id` instead of `inputs` to run every item in a stored dataset.

Dataset columns are matched to your `{{variables}}` **by name**. A prompt containing `{{text}}` needs the dataset's items to carry a variable called `text`. When you import a CSV, include a header row so the columns are named — a headerless file is read positionally as request, response and expected output instead.

Three column names are special and map onto the item's own fields rather than its variables: `request`, `response` and `expected_output`. They are still addressable in a prompt, so `{{request}}` works for a dataset of question/answer pairs. Anything else becomes a named variable. A column that must stay a variable despite being named like one of those three can be written as `variables.request`.

If an item is missing a variable your prompt needs, the run fails for that item and names the missing variable rather than silently substituting the wrong column.

Refer to the CLI documentation for more details <https://github.com/root-signals/rs-sdk/tree/main/cli#prompt-testing><br>