> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/concepts-and-examples/usage/batch-evaluation.md).

# Batch Evaluation

Batch Evaluation takes a dataset of cases you care about and scores every row with either a judge or a set of evaluators you pick. It is how you decide which evaluator actually fits your data, rather than guessing from a handful of examples.

**Labels are not required.** A dataset of plain request/response pairs is the ordinary case — you read the scores and justifications and judge for yourself. This is what separates it from [calibration](/concepts-and-examples/usage/datasets-and-annotations.md#calibration-runs), which compares an evaluator against human labels and therefore needs them.

## In the app

Go to **Batch Evaluation** and choose **New batch evaluation**:

1. **Pick a dataset** — an existing one, or create one inline by typing request/response rows. Uploading a CSV works too.
2. **Pick what to run** — a judge, or one to many evaluators. Selecting evaluators does not create a judge.
3. **Run.** Optionally limit the run to a range of rows first, to check cost and evaluator fit before committing to the whole set.

Results show one row per dataset item and one column per evaluator, with the score and its justification. Sort by any evaluator's column, or filter to the rows whose lowest score falls below a threshold — with no ground truth to average against, finding where an evaluator disagrees with your own judgement is the point.

## From the API

Submit a run with `dataset_id` and either `judge_id` or `evaluator_ids`:

```bash
curl -X POST "https://api.scorable.ai/v1/batch-executions/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "<uuid>",
    "evaluator_ids": ["<evaluator-uuid>", "<evaluator-uuid>"],
    "tags": ["my-app-v1.2"]
  }'
```

**Request parameters:**

* Exactly one target: `judge_id`, or `evaluator_ids` (an array of evaluator ids)
* Exactly one input source: `dataset_id`, or `inputs` (an array of inline request/response objects)
* `range` (optional): `{"start": 0, "end": 49}`, zero-based and inclusive, to run a subset of the dataset
* `tags` (optional): applied to every execution log in the run
* `judge_version_id` (optional): a specific judge version; defaults to the latest
* `project_id` (optional): defaults to the judge's project, or the organization default

Each entry in `inputs` takes `request`, `response`, and optionally `contexts`, `expected_output` (when the evaluator needs one) and `messages` for evaluating multi-turn agent behaviour.

The call returns immediately:

```json
{
  "batch_execution_id": "123e4567-e89b-12d3-a456-426614174000",
  "status_url": "/v1/judges/batch-executions/123e4567-e89b-12d3-a456-426614174000/"
}
```

Poll the status url until the run reaches a terminal state:

```bash
curl -X GET "https://api.scorable.ai/v1/judges/batch-executions/${BATCH_ID}/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}"
```

A run is `pending`, `processing`, then `completed`, `partial` (some items failed) or `failed`. Individual items carry their own `pending` / `processing` / `completed` / `failed` status, so a `completed` run can still contain failures — check `failed_count`.

`completed_count`, `failed_count` and `total_count` update as the run progresses, so you can show a progress bar rather than waiting blind. Once finished, every item carries its inputs and `evaluator_results`:

```json
{
  "status": "completed",
  "completed_count": 3,
  "failed_count": 0,
  "total_count": 3,
  "items": [
    {
      "index": 0,
      "status": "completed",
      "input": { "request": "What is the capital of France?", "response": "Paris." },
      "evaluator_results": [
        { "score": 0.95, "justification": "The response is relevant...", "evaluator_name": "Relevance" }
      ]
    }
  ]
}
```

Inline `inputs` are capped at 100 per request. Dataset-sourced runs allow up to 5000 items — use `range` to work through a larger set in slices.

## Which to reach for

| You want to                                                   | Use                                                                                      |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Score a dataset and read the results yourself                 | **Batch Evaluation**                                                                     |
| Measure how well an evaluator agrees with your labels         | [Calibration](/concepts-and-examples/usage/datasets-and-annotations.md#calibration-runs) |
| Compare prompt and model combinations against the same inputs | [Prompt Testing](/concepts-and-examples/usage/prompt-testing.md)                         |