Issues

Turn a stream of evaluation results into a short list of recurring, prioritized failure themes

Issues View

Overview

A single evaluation score tells you that one response fell short. The Issues view tells you what keeps going wrong.

Scorable groups the failures it finds across all of your evaluations into recurring, named themes and tracks how often each one happens. Instead of scrolling through hundreds of low scores, you get a short, ranked list of themes like "omits requested information" or "unsupported claims in summaries", each with how frequently it occurs and whether it is trending up or down.

Why it matters

• Patterns, not point failures. Two hundred individual low-faithfulness scores become one issue: "the assistant repeatedly states facts it cannot support." You see the shape of the problem, not just its symptoms.

• Prioritization built in. Issues are ranked by how often they occur, so you spend your time on the failure modes that actually happen most, not the loudest one-off.

• Trends over time. Each issue carries its own frequency history, so you can tell whether a problem is growing, and whether a prompt or model change made a class of failures go away.

• Straight to the evidence. Open any issue to drill from the theme down to the concrete executions that exhibited it, so you can debug from a real example.

• Nothing to set up. Issues are derived automatically from your existing Judge and Evaluator runs. There is no tagging, triage, or configuration step.

How it works

Whenever an Evaluator or Judge run surfaces a problem, Scorable's classification engine reads that result and sorts the failure into a structured set of failure types and categories, for example factual support, instruction-following, relevance, or scope. Failures of the same kind are grouped into a single issue and counted over time.

The effect is a feed of recurring problem themes rather than a raw stream of isolated results: the same underlying evaluation data, organized around what is going wrong and how often.

Reading the view

Sort issues by Frequency (most common first) or Most recent, scope the window to the last 24 hours, 7 days, or 30 days, and narrow by tags or project. Each issue shows its category, a frequency trend, and when it was last seen; opening one reveals its first/last occurrence, total count, and example executions you can inspect.

An empty list is the good outcome: it means no recurring failures were detected in the selected window.