> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/concepts-and-examples/usage/datasets-and-annotations.md).

# Datasets & Annotations

Datasets are the ground truth of your evaluation stack. A **dataset** is a collection of items, where each **item** is a request and response pair (optionally with contexts and other fields). An **annotation** attaches a human label, typically an expected score, to an item. Labeled datasets power two things:

* **Calibration**: measure how well an evaluator agrees with your human labels.
* **Demonstrations**: attach a labeled dataset to an evaluator as few-shot examples that steer its scoring.

All of this is available in the UI (including CSV import), the SDKs, and the CLI.

## Score configs

A **score config** defines what an annotation value means: a continuous score, a binary pass/fail, or a set of categories mapped to scores. If you do not specify one, the default identity config is used and the annotation value is the expected score directly.

## Calibration runs

A **calibration run** executes an evaluator against every annotated item in a dataset and reports agreement metrics (RMSE and MAE) between the evaluator's scores and your labels. Per-item results are ordered by largest disagreement first, so you can start improving where the evaluator is most wrong. Runs are kept in history, letting you compare before and after each change.

```python
from scorable import Scorable

client = Scorable()

dataset = client.datasets.create(name="Support quality calibration set", type="test")
item = client.datasets.add_item(
    dataset.id,
    request="My internet is not working.",
    response="Check the cable, then run `ping 8.8.8.8` and share the results.",
)
client.annotations.create(dataset_item_id=item.id, value=0.9)

run = client.evaluators.calibrate_run("MY_EVALUATOR_ID", dataset_id=dataset.id)
# Poll client.calibration_runs.get(run.id) until completed, then:
print(run.metrics)
for result in client.calibration_runs.list_items(run.id):
    print(result.human_value, result.evaluator_score, result.disagreement)
```

If the agreement is poor, you can attach the same dataset as demonstrations and re-run:

```python
client.evaluators.update("MY_EVALUATOR_ID", demonstration_dataset_id=dataset.id)
```

From the CLI, the same flow uses `scorable dataset-item`, `scorable annotation`, and `scorable calibration-run` commands. See the [CLI guide](/concepts-and-examples/cookbooks/cli.md).

## Where the data comes from

* **Production samples**: add real runs from the [execution logs](https://scorable.ai/monitoring/executions) to a dataset with one click.
* **CSV import**: upload existing labeled data in the UI.
* **Ladder generation**: synthesize examples that span the full 0.0 to 1.0 score range from your scoring criteria. See [Add a calibration set](/concepts-and-examples/cookbooks/add-a-custom-evaluator/add-a-calibration-set.md).

For the full walkthrough of calibrating a custom evaluator, see [Add a calibration set](/concepts-and-examples/cookbooks/add-a-custom-evaluator/add-a-calibration-set.md).