> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/concepts-and-examples/usage/batch-evaluation.md).

# Batch Evaluation

Run a whole dataset against a judge or a set of evaluators, and read the results side by side.

Batch Evaluation takes a dataset of cases you care about and scores every row with either a judge or a set of evaluators you pick. It is how you decide which evaluator actually fits your data, rather than guessing from a handful of examples.

**Labels are not required.** A dataset of plain request/response pairs is the ordinary case — you read the scores and justifications and judge for yourself. This is what separates it from [calibration](/concepts-and-examples/usage/datasets-and-annotations.md#calibration-runs), which compares an evaluator against human labels and therefore needs them.

## In the app

Go to **Batch Evaluation** and choose **New batch evaluation**:

1. **Pick a dataset** — an existing one, or create one inline by typing request/response rows. To upload a CSV, choose the delimiter (comma, semicolon, or tab) and click **Upload CSV**. See [CSV import](#csv-import) below.
2. **Pick what to run** — a judge, or one to many evaluators. Selecting evaluators does not create a judge.
3. **Run.** Optionally limit the run to a range of rows first, to check cost and evaluator fit before committing to the whole set.

Results show one row per dataset item and one column per evaluator, with the score and its justification. Sort by any evaluator's column, or filter to the rows whose lowest score falls below a threshold — with no ground truth to average against, finding where an evaluator disagrees with your own judgement is the point.

If a model provider rate-limits the run or is briefly unavailable, the affected items are retried automatically with backoff. Items that still fail are marked failed. Once the run has finished, the results page shows **Retry (N) failed items**, which reruns only the failed items with the same judge or evaluator versions. Organizations with zero data retention cannot retry, because the inputs of failed items are not kept.

### CSV import

A file with a header row is mapped by column name. The editor reads `request`, `response`, `turns`, `expected_output`, `contexts`, and any other column as a variable, and ignores `label` and `justification` columns. A [downloaded dataset](/concepts-and-examples/usage/datasets-and-annotations.md#download-and-re-import) uploads unchanged.

```csv
request,response
"Can I return this?","Yes, within 30 days."
```

A file without a header row is read as `request, response`, and columns after the second are ignored. To import conversations, tick **Conversation (multi-turn)** before uploading. Each row is then read as alternating `request, response, request, response, ...` columns:

```csv
"Can I return this?","Yes, within 30 days.","Do I need a receipt?","Yes, please include it."
```

See [CSV import and export](/concepts-and-examples/usage/datasets-and-annotations.md#csv-import-and-export) for all columns, header detection, and quoting rules.

## From the API

Submit a run with `dataset_id` and either `judge_id` or `evaluator_ids`:

```bash
curl -X POST "https://api.scorable.ai/v1/batch-executions/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "<uuid>",
    "evaluator_ids": ["<evaluator-uuid>", "<evaluator-uuid>"],
    "tags": ["my-app-v1.2"]
  }'
```

**Request parameters:**

* Exactly one target: `judge_id`, or `evaluator_ids` (an array of evaluator ids)
* Exactly one input source: `dataset_id`, or `inputs` (an array of inline request/response objects)
* `range` (optional): `{"start": 0, "end": 49}`, zero-based and inclusive, to run a subset of the dataset
* `tags` (optional): applied to every execution log in the run
* `judge_version_id` (optional): a specific judge version; defaults to the latest
* `project_id` (optional): defaults to the judge's project, or the organization default

Each entry in `inputs` takes `request`, `response`, and optionally `contexts`, `expected_output` (when the evaluator needs one) and `messages` for evaluating multi-turn agent behaviour.

The call returns immediately:

```json
{
  "batch_execution_id": "123e4567-e89b-12d3-a456-426614174000",
  "status_url": "/v1/judges/batch-executions/123e4567-e89b-12d3-a456-426614174000/"
}
```

Poll the status url until the run reaches a terminal state:

```bash
curl -X GET "https://api.scorable.ai/v1/judges/batch-executions/${BATCH_ID}/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}"
```

A run is `pending`, `processing`, then `completed`, `partial` (some items failed) or `failed`. Individual items carry their own `pending` / `processing` / `completed` / `failed` status, so a `completed` run can still contain failures — check `failed_count`.

Items that hit a provider rate limit or outage are retried automatically before they are marked failed. To rerun only the failed items of a finished `partial` or `failed` run, with the original judge or evaluator versions:

```bash
curl -X POST "https://api.scorable.ai/v1/judges/batch-executions/${BATCH_ID}/retry-failed/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}"
```

It returns the same `batch_execution_id` and `status_url`; poll it as before. The final counts include items from both attempts. A run from an organization with zero data retention cannot be retried, because the inputs of its failed items are no longer stored.

`completed_count`, `failed_count` and `total_count` update as the run progresses, so you can show a progress bar rather than waiting blind. Once finished, every item carries its inputs and `evaluator_results`:

```json
{
  "status": "completed",
  "completed_count": 3,
  "failed_count": 0,
  "total_count": 3,
  "items": [
    {
      "index": 0,
      "status": "completed",
      "input": { "request": "What is the capital of France?", "response": "Paris." },
      "evaluator_results": [
        { "score": 0.95, "justification": "The response is relevant...", "evaluator_name": "Relevance" }
      ]
    }
  ]
}
```

Inline `inputs` are capped at 100 per request. Dataset-sourced runs allow up to 5000 items — use `range` to work through a larger set in slices.

## Which to reach for

| You want to                                                   | Use                                                                                      |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Score a dataset and read the results yourself                 | **Batch Evaluation**                                                                     |
| Measure how well an evaluator agrees with your labels         | [Calibration](/concepts-and-examples/usage/datasets-and-annotations.md#calibration-runs) |
| Compare prompt and model combinations against the same inputs | [Prompt Testing](/concepts-and-examples/usage/prompt-testing.md)                         |