Concepts

Scorable is built on a few core concepts that help you organize your evaluation strategy.

Objective defines what you want to achieve (e.g., "Ensure the bot is polite").

Evaluator is a specific check that scores a piece of text (0-1) based on that objective. Think of it as a unit test for semantics.

Judge is a collection of evaluators bundled together for a specific use case. It represents your definition of quality for a specific task.

Model is the AI model such as an LLM that provides the semantic processing of the inputs. Notably, the list contains both API-based models such as OpenAI and Anthropic models, and open source models such as Llama and Mistral models. Finally, you can add your own locally running models to the list with ease. Any member of the organization can add a model; administrators can additionally restrict the organization to GDPR-compliant providers.

Dataset is a collection of request and response examples. Annotated with expected scores, datasets calibrate your evaluators and steer them via demonstrations.

Project is a workspace inside your organization that groups related judges, evaluators, datasets, and execution logs under a single label. Use projects to keep, for example, a staging app and a production app cleanly separated.