Datasets & Annotations

Build labeled datasets, annotate them with expected scores, and use them to calibrate evaluators.

Datasets are the ground truth of your evaluation stack. A dataset is a collection of items, where each item is a request and response pair (optionally with contexts and other fields). An annotation attaches a human label, typically an expected score, to an item. Labeled datasets power two things:

• Calibration: measure how well an evaluator agrees with your human labels.

• Demonstrations: attach a labeled dataset to an evaluator as few-shot examples that steer its scoring.

All of this is available in the UI (including CSV import), the SDKs, and the CLI.

Score configs

A score config defines what an annotation value means: a continuous score, a binary pass/fail, or a set of categories mapped to scores. If you do not specify one, the default identity config is used and the annotation value is the expected score directly.

Calibration runs

A calibration run executes an evaluator against every annotated item in a dataset and reports agreement metrics (RMSE and MAE) between the evaluator's scores and your labels. Per-item results are ordered by largest disagreement first, so you can start improving where the evaluator is most wrong. Runs are kept in history, letting you compare before and after each change.

from scorable import Scorable

client = Scorable()

dataset = client.datasets.create(name="Support quality calibration set", type="test")
item = client.datasets.add_item(
    dataset.id,
    request="My internet is not working.",
    response="Check the cable, then run `ping 8.8.8.8` and share the results.",
)
client.annotations.create(dataset_item_id=item.id, value=0.9)

run = client.evaluators.calibrate_run("MY_EVALUATOR_ID", dataset_id=dataset.id)
# Poll client.calibration_runs.get(run.id) until completed, then:
print(run.metrics)
for result in client.calibration_runs.list_items(run.id):
    print(result.human_value, result.evaluator_score, result.disagreement)

If the agreement is poor, you can attach the same dataset as demonstrations and re-run:

client.evaluators.update("MY_EVALUATOR_ID", demonstration_dataset_id=dataset.id)

From the CLI, the same flow uses scorable dataset-item, scorable annotation, and scorable calibration-run commands. See the CLI guide.

Where the data comes from

• Production samples: add real runs from the execution logs to a dataset with one click.

• CSV import: upload existing data in the UI, labeled or not. Include a header row — columns named request, response and expected_output map onto those fields, and any other column becomes a named variable on the item. A headerless file is read positionally as request, response, expected output.

• Ladder generation: synthesize examples that span the full 0.0 to 1.0 score range from your scoring criteria. See Add a calibration set.

For the full walkthrough of calibrating a custom evaluator, see Add a calibration set.