> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/quick-start/getting-started-with-root-signals.md).

# Getting started in 30 seconds

Scorable is an agent that does the job of an LLM evaluation engineer. You describe your AI application in plain language, and it generates the evaluators for it, which you then run against your app's outputs from code, the CLI, or the web app.

You can try it without an account. A free temporary API key is enough to run an evaluator and see a score, which takes about 30 seconds.

## Why Scorable exists

An LLM application that works in the demo will often misbehave in production, and the team has no measurement that tells them when or how often. It ignores an instruction, invents a refund policy, or makes a poor call on a customer case.

The usual answer is LLM-as-a-judge, where a second model reviews and scores the output of the first. We spent two years building an evaluation platform around that idea. What we saw with customers is that most of the effort goes into the work surrounding the judge model: deciding which dimensions to measure, writing an evaluator for each, calibrating it against labeled examples, updating it when policies change, and wiring it into the application. Few teams have an engineer to spare for that.

Scorable takes over those tasks. From a short description of your application or knowledge-work process, it builds a [Judge](/concepts-and-examples/usage/judges.md): a set of evaluators covering the dimensions relevant to your use case, such as hallucinations, policy compliance, or the quality of the decisions your AI makes. The Judge runs in our cloud or [on your own premises](/self-hosting.md), and you keep control of the choices that need human judgment, such as what counts as good enough. For the longer argument, see [Why Anything?](/overview/why-anything.md)

## Choose your path

| Path                                                                      | Best for                                                         | Time          | Account needed |
| ------------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------- | -------------- |
| [A. Let your coding agent do it](#path-a-let-your-coding-agent-do-it)     | You use Cursor, Claude Code, Codex, Antigravity, or similar      | One prompt    | No             |
| [B. Terminal with a temporary key](#path-b-terminal-with-a-temporary-key) | You want to see real scores right now                            | 30 seconds    | No             |
| [C. Web app](#path-c-web-app)                                             | You prefer a UI, or want to attach documents and refine visually | A few minutes | Yes (free)     |

## Path A: Let your coding agent do it

Go to the repository of your AI-powered app and paste this into your coding agent:

```
Add Scorable evals by following https://scorable.ai/SKILL.md
```

The [Agent Skill](/skill.md) tells the agent to find the LLM calls in your code, install the CLI, mint a temporary key, generate a Judge that matches what your application does, integrate it, and verify the setup. See [Coding Agents](/integrations/coding-agents.md) for details.

## Path B: Terminal with a temporary key

### 1. Install the CLI and get a key

```bash
curl -sSL https://scorable.ai/cli/install.sh | sh

# Free temporary key, no registration required
scorable auth demo-key
```

Prefer npm? Use `npm install -g @root-signals/scorable-cli` (Node.js 20 or higher).

{% hint style="info" %}
**Other ways to get a temporary key.** Open [scorable.ai/demo-user](https://scorable.ai/demo-user) in a browser, or mint one over HTTP and use it with the SDKs:

```bash
export SCORABLE_API_KEY=$(curl -s -X POST https://api.scorable.ai/create-demo-user/ | jq -r .api_key)
```

{% endhint %}

### 2. Get your first score

Run one of the ready-made evaluators from the [Evaluator Portfolio](/quick-start/evaluator-portfolio.md):

```bash
scorable evaluator execute-by-name Helpfulness \
  --request "How do I reset my password?" \
  --response "Just figure it out yourself."
```

```json
{
  "evaluator_name": "Helpfulness",
  "score": 0.1,
  "justification": "The user asked: \"How do I reset my password?\" but the response says: \"Just figure it out yourself.\" This does not provide any actionable steps, links, or guidance...",
  "confidence": 0.95
}
```

Every evaluation returns a normalized **score between 0 and 1** and a **justification** explaining it.

The same thing from code:

{% tabs %}
{% tab title="Python" %}

```bash
pip install scorable
```

```python
from scorable import Scorable

client = Scorable()  # reads SCORABLE_API_KEY from the environment

result = client.evaluators.Helpfulness(
    request="How do I reset my password?",
    response="Just figure it out yourself.",
)
print(result.score, result.justification)
```

{% endtab %}

{% tab title="TypeScript" %}

```bash
npm install @root-signals/scorable
```

```typescript
import { Scorable } from "@root-signals/scorable";

const client = new Scorable({ apiKey: process.env.SCORABLE_API_KEY! });

const result = await client.evaluators.executeByName("Helpfulness", {
  request: "How do I reset my password?",
  response: "Just figure it out yourself.",
});
console.log(result.score, result.justification);
```

{% endtab %}
{% endtabs %}

### 3. Generate a Judge for your own application

The ready-made evaluators know nothing about your product. To get evaluators written for your use case, describe what you want to evaluate and let Scorable generate a Judge:

```bash
scorable judge generate \
  --visibility public \
  --intent "Evaluate how well my network troubleshooting assistant diagnoses the problem, explains the fix, and confirms the user has successfully applied it. Users are on Windows workstations in a corporate environment."
```

This takes around ten seconds. The output lists the evaluators in your new Judge, plus any context Scorable thinks is still missing (you can regenerate with `--extra-contexts` to fill it in). Find the Judge ID with `scorable judge list`, then run it:

```bash
scorable judge execute YOUR_JUDGE_ID \
  --request "My laptop can't reach the intranet." \
  --response "Try rebooting."
```

You get one score and justification per evaluator, for example a `Problem Diagnosis` score of `0` explaining that no diagnostic steps were offered.

{% hint style="warning" %}
**Temporary keys are for trying things out.**

* They expire after a limited time.
* Judges created with them are public and visible to everyone. Do not put confidential information in the intent.
* To keep your Judges private and get a key that does not expire, [create a free account](https://scorable.ai/register) and [create an API key](https://scorable.ai/settings/api-keys), then run `scorable auth set-key`.
  {% endhint %}

When you are happy with the Judge, jump to [Integrate into your application](#integrate-into-your-application).

## Path C: Web app

### 1. Generate your first Judge

1. Go to [Scorable](https://scorable.ai/).
2. Write a **plain‑language description** of what you want to evaluate.

   > *Example: “Evaluate how well my network troubleshooting assistant diagnoses the problem, explains the fix, and confirms the user has successfully applied it. Users are on Windows workstations in a corporate environment.”*
3. Optionally:
   * Paste **links** to docs or policies.
   * Paste **example conversations**.
   * Attach **documents** (policies, examples, etc.).
4. Click **Generate**.

Scorable analyzes your intent, picks appropriate evaluators, and builds a Judge with synthetic examples.

### 2. Refine and test

Once generated, you land in the Judge view.

* **Review the evaluator stack.** Each entry shows the name, type, and intent. Remove evaluators you don't need, and edit custom evaluators to adjust their intent and scoring criteria.
* **Add missing context.** The first pass is rarely perfect. Scorable detects missing information (for example, an unspecified refund window) and prompts you for the details.
* **Test in the UI.** Use the **Test** tab to try the example scenario, or write your own, and see how the Judge behaves.

## Integrate into your application

There are three ways to call a Judge from your application, depending on how much control you want. The examples below use `SCORABLE_API_KEY` and `YOUR_JUDGE_ID` as placeholders.

### Option 1: Auto‑refine (Managed Safeguard)

Scorable sits as a proxy between your application and the LLM. You point your OpenAI client at the Judge's `refine` endpoint and leave the rest of your code unchanged. Scorable evaluates each draft response with the Judge. If the draft falls short, the Judge's feedback is used to generate an improved response, and that is what your app receives. Choose this when you want a quality safeguard without touching your application logic.

{% hint style="info" %}
Scorable supports models from multiple providers (OpenAI, Anthropic, Gemini, etc.). See [here](https://scorable.ai/settings/llm-accounts) for more info.
{% endhint %}

{% hint style="warning" %}
The `refine` endpoint proxies the model call through Scorable and requires a customer-managed provider key for the requested model's provider. Add one under **Organization Settings → Providers**, otherwise the call returns `403 byok_required`.
{% endhint %}

{% tabs %}
{% tab title="Python" %}

```python
from openai import OpenAI

client = OpenAI(
    api_key="SCORABLE_API_KEY",
    base_url="https://api.scorable.ai/v1/judges/YOUR_JUDGE_ID/refine/openai",
)

# This call is routed through Scorable.
# The returned response has already been evaluated and improved if necessary.
response = client.responses.create(
    model="gemini-3-pro",
    input="What is the refund policy?"
)
```

{% endtab %}

{% tab title="JavaScript / TypeScript" %}

```javascript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "SCORABLE_API_KEY",
  baseURL: "https://api.scorable.ai/v1/judges/YOUR_JUDGE_ID/refine/openai",
});

const response = await client.chat.completions.create({
  model: "gpt-5.2",
  messages: [
    { role: "user", content: "User question or instruction" },
  ],
});
```

{% endtab %}
{% endtabs %}

### Option 2: Manual Control via SDKs

You send the request and response to the Judge and get back scores (0 to 1) with justifications. What happens next is up to your code: block the response, retry, flag it for review, or only log it. This is the option for CI/CD gating, production monitoring, and offline analysis.

{% tabs %}
{% tab title="Python" %}

```python
from scorable import Scorable

client = Scorable(api_key="SCORABLE_API_KEY")

result = client.judges.run(
    judge_id="YOUR_JUDGE_ID",
    request="What is the refund policy?",
    response="You can return items within 30 days.",
    # (Optional) Tags are free form strings for more powerful filtering and more actionable insights.
    tags=["production", "v0.1"],
    # (Optional) User ID is a unique identifier for your end-user. This allows you to track evaluation results per user.
    user_id="USER_ID",
    # (Optional) Session ID is a unique identifier for the conversation session. This helps in grouping evaluations that belong to the same interaction.
    session_id="SESSION_ID",
)

for evaluator_result in result.evaluator_results:
    print(f"{evaluator_result.evaluator_name}: {evaluator_result.score}")
    # Example logic:
    # if evaluator_result.score < 0.5:
    #     flag_for_review(response)
```

{% endtab %}

{% tab title="TypeScript" %}

```typescript
import { Scorable } from "@root-signals/scorable";

const client = new Scorable({ apiKey: process.env.SCORABLE_API_KEY ?? "" });

const result = await client.judges.execute(
  "YOUR_JUDGE_ID",
  {
    request: "What is the refund policy?",
    response: "You can return items within 30 days.",
    // (Optional) Tags are free form strings for more powerful filtering and more actionable insights.
    tags: ["production", "v0.1"],
    // (Optional) User ID is a unique identifier for your end-user. This allows you to track evaluation results per user.
    user_id: "USER_ID",
    // (Optional) Session ID is a unique identifier for the conversation session. This helps in grouping evaluations that belong to the same interaction.
    session_id: "SESSION_ID",
  },
);

for (const evaluatorResult of result.evaluator_results ?? []) {
  console.log(`${evaluatorResult.evaluator_name}: ${evaluatorResult.score}`);
}
```

{% endtab %}
{% endtabs %}

### Option 3: CLI

Useful for quick checks or shell scripts.

```bash
scorable judge execute YOUR_JUDGE_ID \
  --request="What is the refund policy?" \
  --response="You can return items within 30 days." \
  --tag refund_policy \
  --user-id="USER_ID" \
  --session-id="SESSION_ID"
```

## Core concepts

### What is a Judge?

A **Judge** is a persistent evaluation object in Scorable:

* **Intent**: Describes what it should measure (e.g., “Check that our support bot follows the refund policy”).
* **Evaluators**: A stack of evaluators, each responsible for scoring one aspect of quality.
* **Context**: Optional attached files (e.g., PDFs or policy docs).

When you run a Judge, you send it an LLM **response** (and optionally the **request**). It returns **scores (0-1)** and **justifications**.

### What is an evaluator?

An **evaluator** is a single, reusable rubric with:

* **Intent**: A focused description of what to judge.
* **Model & scoring criteria**: The LLM configuration used to generate scores.
* **Demonstrations**: Example inputs and outputs used to adjust the behavior of the evaluator.
* **Calibration**: Labeled datasets verify that the evaluator scores match your expectations.
* **Score & justification**: A numeric score plus reasoning.

A Judge is essentially a **bundle of evaluators** that together capture your definition of quality.

## Continue learning

* **See it on real use cases:** [Evaluate multi-turn chatbot conversations](/concepts-and-examples/cookbooks/evaluate-chatbot-conversation.md), [RAG evaluation](/concepts-and-examples/cookbooks/rag-evaluation.md), [Batch evaluation](/concepts-and-examples/usage/batch-evaluation.md), [Find the best prompt and model](/concepts-and-examples/cookbooks/find-the-best-prompt-and-model.md), and more in [Examples](/concepts-and-examples/cookbooks.md).
* **Turn scores into decisions:** [Making Sense of Evaluation Results](/overview/making-sense.md).
* **Go deeper on the building blocks:** [Evaluators](/concepts-and-examples/usage/evaluators.md), [Judges](/concepts-and-examples/usage/judges.md), and [Datasets & Annotations](/concepts-and-examples/usage/datasets-and-annotations.md) for calibration.
* **Monitor production automatically:** [Trace your app with OpenTelemetry](/concepts-and-examples/cookbooks/otel-evaluation-via-cli.md) and evaluate live traffic.
* **Plug into your stack:** [Slack digests and bot](/concepts-and-examples/usage/monitoring-and-insights.md#slack-integration), the [MCP server](/mcp-server.md), and other [Integrations](/integrations.md) such as LangChain, LlamaIndex, and Langfuse.

## Need help?

* Check the [Frequently Asked Questions](/frequently-asked-questions.md).
* Browse the [Evaluator Portfolio](/quick-start/evaluator-portfolio.md) to discover what's possible.
* Reach out to our team through the in-app support chat.