Mastra

This guide shows how to integrate the Scorable SDK as a custom scorer for evaluating your Mastra agents.

Setting Up the Scorable Client

// npm install @root-signals/scorable
import { Scorable } from "@root-signals/scorable";

const scorableClient = new Scorable({
  apiKey: process.env.SCORABLE_API_KEY!,
});

Creating a Scorable Scorer

Use createScorer from mastra to define a custom scorer that calls Scorable's evaluator API:

import { createScorer } from "@mastra/core/scores";
import { Scorable } from "@root-signals/scorable";

const scorableClient = new Scorable({
  apiKey: process.env.SCORABLE_API_KEY!,
});

export const helpfulnessScorer = createScorer({
  name: "Helpfulness",
  description: "Helpfulness of the assistant's response",
})
  .preprocess(({ run }) => {
    const userText = (run.input?.inputMessages?.[0]?.content as string) || "";
    const assistantText = (run.output?.[0]?.content as string) || "";
    return { userText, assistantText };
  })
  .analyze(async ({ results }) => {
    const response = results.preprocessStepResult.assistantText;
    const request = results.preprocessStepResult.userText;
    const scoreResult = await scorableClient.evaluators.executeByName(
      "Helpfulness",
      {
        request: request,
        response: response,
      }
    );
    return scoreResult;
  })
  .generateScore(({ results }) => {
    const scoreResult = results.analyzeStepResult
    return scoreResult.score ?? 0;
  })
  .generateReason(({ results }) => {
    return results.analyzeStepResult?.justification ?? "N/A";
  });

Integrating with an Agent

Attach the scorer to your agent configuration:

import { Agent } from "@mastra/core/agent";
import { weatherTool } from "../tools/weather-tool";
import { helpfulnessScorer } from "../scorers/weather-scorer";

export const weatherAgent = new Agent({
  name: "Weather Agent",
  instructions: `
      You are a helpful weather assistant that provides accurate weather information.
      Your primary function is to help users get weather details for specific locations.
  `,
  model: "openai/gpt-4o-mini",
  tools: { weatherTool },
  scorers: {
    helpfulness: {
      scorer: helpfulnessScorer,
      sampling: {
        type: "ratio",
        rate: 1,
      },
    },
  },
});