# Frequently Asked Questions

### Terminology

<details>

<summary>What is <em>Intent</em> for?</summary>

Intent is the high-level, human-understandable description of the attribute an Evaluator measures. For example: “To measure how clearly the returns handler explains the 20% discount offer on the next purchase”.

</details>

<details>

<summary>What are <em>Datasets</em>?</summary>

Datasets allow you to bring test data for benchmarking (*Root* & *Custom*) and optimizing (*Custom*) evaluators.

</details>

### Behaviour

<details>

<summary>Does <em>Intent</em> change the behaviour of the evaluator?</summary>

Yes. Evaluator *Intent* does alter the evaluator behaviour.

</details>

<details>

<summary>Does Calibration change the behaviour of the evaluator?</summary>

No. Calibration is for benchmarking (testing) the evaluators to understand whether they are "calibrated" to your expected/desired behaviour or not. Calibration samples do not alter the behaviour of the evaluators.

</details>

<details>

<summary>How do <em>Demonstration</em>s work?</summary>

Demonstrations are used as in-context few-shot samples combined with our well-tuned meta-prompt. They are not utilized for supervised fine-tuning (SFT).

</details>

### Usage

<details>

<summary>Our stack is not in Python, can we still use Scorable?</summary>

Absolutely. We have a [REST API](https://api.docs.scorable.ai/reference/v1_evaluators_execute_by_name_create) that you can run from your favourite tech stack.

<picture><source srcset="/files/38gDXaIzgEm9LmSm3ycW" media="(prefers-color-scheme: dark)"><img src="/files/QSwk0XkvbHk7crnQ2a3w" alt=""></picture>

</details>

<details>

<summary>Do I need to have Calibrations for all Custom Evaluators?</summary>

You do not have to bring *Calibration* samples but we strongly recommend at least a handful of them in order to understand the behaviour of the evaluators.

</details>

<details>

<summary>Can I change the behaviour of the evaluator by bringing labeled data?</summary>

You can change the behaviour of your Custom Evaluators by bringing annotated samples as *Demonstration*s. Behaviour of *Root Evaluators* can not be altered.

</details>

<details>

<summary>Can I run a previous version of a Custom Evaluator?</summary>

Yes.

</details>

<details>

<summary>If we already have a ground truth expected output, can we use your evaluators?</summary>

Yes. Various evaluators from us support reference-based evaluations where you can bring your ground truth expected responses. See our [evaluator catalogue here](https://docs.scorable.ai/quick-start/usage/evaluators#list-of-evaluators-maintained-by-root-signals).

</details>

<details>

<summary>How can I differentiate evaluations and related statistics for different applications (or versions) of mine?</summary>

You can use arbitrary tags for evaluation executions. See the [example here](https://sdk.scorable.ai/examples.html#monitoring-llm-pipelines-with-tags).

</details>

<details>

<summary>Can I integrate Scorable evaluators to experiment tracking tools such as MLflow etc.?</summary>

Yes. Our evaluators return a structured response (e.g. a dictionary) with scores, justifications, tags etc. These results can be logged to any experiment tracking system or database similar to any other metric, metadata, or attribute.

</details>

### Models

<details>

<summary>What is the LLM that powers ready-made Root evaluators? Can I change it?</summary>

Root Evaluators are powered by various LLMs under the hood. This can not be changed except for on-premise deployments.

</details>

<details>

<summary>Can I see which models are GDPR compliant?</summary>

Yes, you can see model metadata under *<mark style="color:purple;">Settings > LLM Accounts</mark>*. More info can be found under [Control & Compliance](https://docs.scorable.ai/usage/usage/models#control-and-compliance) section of our docs.

</details>

<details>

<summary>Are Evaluators/Judges deterministic?</summary>

No. We have tight confidence intervals (for the same input) but small fluctuations are to be expected. Expected standard deviations can be found [in our docs](https://docs.scorable.ai/usage/usage/evaluators#determinism).

</details>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.scorable.ai/frequently-asked-questions.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.