Getting started in 30 seconds

Scorable is an automated LLM Evaluation Engineer. You describe your AI application in plain language, and it generates the evaluators for it, which you then run against your app's outputs from code, the CLI, or the web app.

You can try it without an account. A free temporary API key is enough to run an evaluator and see a score, which takes about 30 seconds.

Why Scorable exists

An LLM application that works in the demo will often misbehave in production, and the team has no measurement that tells them when or how often. It ignores an instruction, invents a refund policy, or makes a poor call on a customer case.

The usual answer is LLM-as-a-judge, where a second model reviews and scores the output of the first. We spent two years building an evaluation platform around that idea. What we saw with customers is that most of the effort goes into the work surrounding the judge model: deciding which dimensions to measure, writing an evaluator for each, calibrating it against labeled examples, updating it when policies change, and wiring it into the application. Few teams have an engineer to spare for that.

Scorable does that engineer's work. From a short description of your application or knowledge-work process, it builds a Judge: a set of evaluators covering the dimensions relevant to your use case, such as hallucinations, policy compliance, or the quality of the decisions your AI makes. The Judge runs in our cloud or on your own premises. The goal is to automate the grunt work of AI reliability and leave you only the critical decisions. For the longer argument, see Why Anything?

Choose your path

Path	Best for	Time	Account needed
A. Let your coding agent do it	You use Cursor, Claude Code, Codex, Antigravity, or similar	One prompt	No
B. Terminal with a temporary key	You want to see real scores right now	30 seconds	No
C. Web app	You prefer a UI, or want to attach documents and refine visually	A few minutes	Yes (free)

Path A: Let your coding agent do it

Go to the repository of your AI-powered app and paste this into your coding agent:

Add Scorable evals by following https://scorable.ai/SKILL.md

The Agent Skill tells the agent to find the LLM calls in your code, install the CLI, mint a temporary key, generate a Judge that matches what your application does, integrate it, and verify the setup. See Coding Agents for details.

Path B: Terminal with a temporary key

1. Install the CLI and get a key

curl -sSL https://scorable.ai/cli/install.sh | sh

# Free temporary key, no registration required
scorable auth demo-key

Prefer npm? Use npm install -g @root-signals/scorable-cli (Node.js 20 or higher).

Other ways to get a temporary key. Open scorable.ai/demo-user in a browser, or mint one over HTTP and use it with the SDKs:

export SCORABLE_API_KEY=$(curl -s -X POST https://api.scorable.ai/create-demo-user/ | jq -r .api_key)

2. Get your first score

Run one of the ready-made evaluators from the Evaluator Portfolio:

scorable evaluator execute-by-name Helpfulness \
  --request "How do I reset my password?" \
  --response "Just figure it out yourself."

{
  "evaluator_name": "Helpfulness",
  "score": 0.1,
  "justification": "The user asked: \"How do I reset my password?\" but the response says: \"Just figure it out yourself.\" This does not provide any actionable steps, links, or guidance...",
  "confidence": 0.95
}

Every evaluation returns a normalized score between 0 and 1 and a justification you can read, log, or feed back to your model.

The same thing from code:

Python

pip install scorable

from scorable import Scorable

client = Scorable()  # reads SCORABLE_API_KEY from the environment

result = client.evaluators.Helpfulness(
    request="How do I reset my password?",
    response="Just figure it out yourself.",
)
print(result.score, result.justification)

TypeScript

npm install @root-signals/scorable

import { Scorable } from "@root-signals/scorable";

const client = new Scorable({ apiKey: process.env.SCORABLE_API_KEY! });

const result = await client.evaluators.executeByName("Helpfulness", {
  request: "How do I reset my password?",
  response: "Just figure it out yourself.",
});
console.log(result.score, result.justification);

3. Generate a Judge for your own application

The ready-made evaluators know nothing about your product. To get evaluators written for your use case, describe what you want to evaluate and let Scorable generate a Judge:

scorable judge generate \
  --visibility public \
  --intent "Evaluate how well my network troubleshooting assistant diagnoses the problem, explains the fix, and confirms the user has successfully applied it. Users are on Windows workstations in a corporate environment."

This takes around ten seconds. The output lists the evaluators in your new Judge, plus any context Scorable thinks is still missing (you can regenerate with --extra-contexts to fill it in). Find the Judge ID with scorable judge list, then run it:

scorable judge execute YOUR_JUDGE_ID \
  --request "My laptop can't reach the intranet." \
  --response "Try rebooting."

You get one score and justification per evaluator, for example a Problem Diagnosis score of 0 explaining that no diagnostic steps were offered.

Temporary keys are for trying things out.

• They expire after a limited time.

• Judges created with them are public and visible to everyone. Do not put confidential information in the intent.

• To keep your Judges private and get a key that does not expire, create a free account and create an API key, then run scorable auth set-key.

When you are happy with the Judge, jump to Integrate into your application.

Path C: Web app

1. Generate your first Judge

1. Go to Scorable.

2. Write a plain‑language description of what you want to evaluate.

   Example: “Evaluate how well my network troubleshooting assistant diagnoses the problem, explains the fix, and confirms the user has successfully applied it. Users are on Windows workstations in a corporate environment.”

3. Optionally:

   • Paste links to docs or policies.

   • Paste example conversations.

   • Attach documents (policies, examples, etc.).

4. Click Generate.

Scorable analyzes your intent, picks appropriate evaluators, and builds a Judge with synthetic examples.

2. Refine and test

Once generated, you land in the Judge view.

• Review the evaluator stack. Each entry shows the name, type, and intent. Remove evaluators you don't need, and edit custom evaluators to adjust their intent and scoring criteria.

• Add missing context. The first pass is rarely perfect. Scorable detects missing information (for example, an unspecified refund window) and prompts you for the details.

• Test in the UI. Use the Test tab to try the example scenario, or write your own, and see how the Judge behaves.

Integrate into your application

There are three ways to call a Judge from your application, depending on how much control you want. The examples below use SCORABLE_API_KEY and YOUR_JUDGE_ID as placeholders.

Option 1: Auto‑refine (Managed Safeguard)

In this mode, Scorable acts as a proxy between your application and the LLM. It ensures your AI behaves according to the safeguards set by the Judge.

• How it works: You point your OpenAI client to Scorable's refine endpoint.

• Benefit: Scorable automatically evaluates the draft response. If it doesn't meet your quality standards (as defined by the Judge), Scorable uses the Judge's feedback to generate an improved response before sending it back to your app.

• Use case: "I want to ensure quality without changing my application logic."

Scorable supports models from multiple providers (OpenAI, Anthropic, Gemini, etc.). See here for more info.

The refine endpoint proxies the model call through Scorable and requires a customer-managed provider key for the requested model's provider. Add one under Organization Settings → Providers, otherwise the call returns 403 byok_required.

Python

from openai import OpenAI

client = OpenAI(
    api_key="SCORABLE_API_KEY",
    base_url="https://api.scorable.ai/v1/judges/YOUR_JUDGE_ID/refine/openai",
)

# This call is routed through Scorable.
# The returned response has already been evaluated and improved if necessary.
response = client.responses.create(
    model="gemini-3-pro",
    input="What is the refund policy?"
)

JavaScript / TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "SCORABLE_API_KEY",
  baseURL: "https://api.scorable.ai/v1/judges/YOUR_JUDGE_ID/refine/openai",
});

const response = await client.chat.completions.create({
  model: "gpt-5.2",
  messages: [
    { role: "user", content: "User question or instruction" },
  ],
});

Option 2: Manual Control via SDKs

In this mode, you call the Judge explicitly to get scores and justifications, but you decide how to act on them. This gives you full control over the workflow.

• How it works: You send the request and response to the Judge API.

• Benefit: You get detailed data (scores 0-1, reasoning) to use in your own logic.

• Use case: CI/CD gating, production monitoring, or offline analysis.

Python

from scorable import Scorable

client = Scorable(api_key="SCORABLE_API_KEY")

result = client.judges.run(
    judge_id="YOUR_JUDGE_ID",
    request="What is the refund policy?",
    response="You can return items within 30 days.",
    # (Optional) Tags are free form strings for more powerful filtering and more actionable insights.
    tags=["production", "v0.1"],
    # (Optional) User ID is a unique identifier for your end-user. This allows you to track evaluation results per user.
    user_id="USER_ID",
    # (Optional) Session ID is a unique identifier for the conversation session. This helps in grouping evaluations that belong to the same interaction.
    session_id="SESSION_ID",
)

for evaluator_result in result.evaluator_results:
    print(f"{evaluator_result.evaluator_name}: {evaluator_result.score}")
    # Example logic:
    # if evaluator_result.score < 0.5:
    #     flag_for_review(response)

TypeScript

import { Scorable } from "@root-signals/scorable";

const client = new Scorable({ apiKey: process.env.SCORABLE_API_KEY ?? "" });

const result = await client.judges.execute(
  "YOUR_JUDGE_ID",
  {
    request: "What is the refund policy?",
    response: "You can return items within 30 days.",
    // (Optional) Tags are free form strings for more powerful filtering and more actionable insights.
    tags: ["production", "v0.1"],
    // (Optional) User ID is a unique identifier for your end-user. This allows you to track evaluation results per user.
    user_id: "USER_ID",
    // (Optional) Session ID is a unique identifier for the conversation session. This helps in grouping evaluations that belong to the same interaction.
    session_id: "SESSION_ID",
  },
);

for (const evaluatorResult of result.evaluator_results ?? []) {
  console.log(`${evaluatorResult.evaluator_name}: ${evaluatorResult.score}`);
}

Option 3: CLI

Useful for quick checks or shell scripts.

scorable judge execute YOUR_JUDGE_ID \
  --request="What is the refund policy?" \
  --response="You can return items within 30 days." \
  --tag refund_policy \
  --user-id="USER_ID" \
  --session-id="SESSION_ID"

Core concepts

What is a Judge?

A Judge is a persistent evaluation object in Scorable:

• Intent: Describes what it should measure (e.g., “Check that our support bot follows the refund policy”).

• Evaluators: A stack of evaluators, each responsible for scoring one aspect of quality.

• Context: Optional attached files (e.g., PDFs or policy docs).

When you run a Judge, you send it an LLM response (and optionally the request). It returns scores (0-1) and justifications.

What is an evaluator?

An evaluator is a single, reusable rubric with:

• Intent: A focused description of what to judge.

• Model & scoring criteria: The LLM configuration used to generate scores.

• Demonstrations: Example inputs and outputs used to adjust the behavior of the evaluator.

• Calibration: Labeled datasets verify that the evaluator scores match your expectations.

• Score & justification: A numeric score plus reasoning.

A Judge is essentially a bundle of evaluators that together capture your definition of quality.

Continue learning

• See it on real use cases: Evaluate multi-turn chatbot conversations, RAG evaluation, Batch evaluation, Find the best prompt and model, and more in Examples.

• Turn scores into decisions: Making Sense of Evaluation Results.

• Go deeper on the building blocks: Evaluators, Judges, and Datasets & Annotations for calibration.

• Monitor production automatically: Trace your app with OpenTelemetry and evaluate live traffic.

• Plug into your stack: Slack digests and bot, the MCP server, and other Integrations such as LangChain, LlamaIndex, and Langfuse.

Need help?

• Check the Frequently Asked Questions.

• Browse the Evaluator Portfolio to discover what's possible.

• Reach out to our team through the in-app support chat.