Batch Evaluation

Run a whole dataset against a judge or a set of evaluators, and read the results side by side.

Batch Evaluation takes a dataset of cases you care about and scores every row with either a judge or a set of evaluators you pick. It is how you decide which evaluator actually fits your data, rather than guessing from a handful of examples.

Labels are not required. A dataset of plain request/response pairs is the ordinary case — you read the scores and justifications and judge for yourself. This is what separates it from calibration, which compares an evaluator against human labels and therefore needs them.

In the app

Go to Batch Evaluation and choose New batch evaluation:

Pick a dataset — an existing one, or create one inline by typing request/response rows. Uploading a CSV works too.
Pick what to run — a judge, or one to many evaluators. Selecting evaluators does not create a judge.
Run. Optionally limit the run to a range of rows first, to check cost and evaluator fit before committing to the whole set.

Results show one row per dataset item and one column per evaluator, with the score and its justification. Sort by any evaluator's column, or filter to the rows whose lowest score falls below a threshold — with no ground truth to average against, finding where an evaluator disagrees with your own judgement is the point.

From the API

Submit a run with dataset_id and either judge_id or evaluator_ids:

curl -X POST "https://api.scorable.ai/v1/batch-executions/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "<uuid>",
    "evaluator_ids": ["<evaluator-uuid>", "<evaluator-uuid>"],
    "tags": ["my-app-v1.2"]
  }'

Request parameters:

Exactly one target: judge_id, or evaluator_ids (an array of evaluator ids)
Exactly one input source: dataset_id, or inputs (an array of inline request/response objects)
range (optional): {"start": 0, "end": 49}, zero-based and inclusive, to run a subset of the dataset
tags (optional): applied to every execution log in the run
judge_version_id (optional): a specific judge version; defaults to the latest
project_id (optional): defaults to the judge's project, or the organization default

Each entry in inputs takes request, response, and optionally contexts, expected_output (when the evaluator needs one) and messages for evaluating multi-turn agent behaviour.

The call returns immediately:

{
  "batch_execution_id": "123e4567-e89b-12d3-a456-426614174000",
  "status_url": "/v1/judges/batch-executions/123e4567-e89b-12d3-a456-426614174000/"
}

Poll the status url until the run reaches a terminal state:

curl -X GET "https://api.scorable.ai/v1/judges/batch-executions/${BATCH_ID}/" \
  -H "Authorization: Api-Key ${SCORABLE_API_KEY}"

A run is pending, processing, then completed, partial (some items failed) or failed. Individual items carry their own pending / processing / completed / failed status, so a completed run can still contain failures — check failed_count.

completed_count, failed_count and total_count update as the run progresses, so you can show a progress bar rather than waiting blind. Once finished, every item carries its inputs and evaluator_results:

{
  "status": "completed",
  "completed_count": 3,
  "failed_count": 0,
  "total_count": 3,
  "items": [
    {
      "index": 0,
      "status": "completed",
      "input": { "request": "What is the capital of France?", "response": "Paris." },
      "evaluator_results": [
        { "score": 0.95, "justification": "The response is relevant...", "evaluator_name": "Relevance" }
      ]
    }
  ]
}