Concepts

Scorable is built on a few core concepts that help you organize your evaluation strategy.

Objective defines what you want to achieve (e.g., "Ensure the bot is polite").

Evaluator is a specific check that scores a piece of text (0-1) based on that objective. Think of it as a unit test for semantics.

An example ready-made evaluator

Judge is a collection of evaluators bundled together for a specific use case. It represents your definition of quality for a specific task.

An example Judge that contains several evaluators

Model is the AI model such as an LLM that provides the semantic processing of the inputs. Notably, the list contains both API-based models such as OpenAI and Anthropic models, and open source models such as Llama and Mistral models. Finally, you can add your own locally running models to the list with ease. The organization Admin controls the availability of models enabled in your organization.