For the complete documentation index, see llms.txt. This page is also available as Markdown.

Exporting evaluation results

Get scores, justifications and costs out of Scorable and into your own reporting, dashboards or data warehouse.

Tracing your AI agent covers traces coming into Scorable. This page covers the opposite direction: getting evaluation results out, so they can live in your own BI tool, dashboard or SIEM alongside the rest of your operational data.

There are three routes, and they suit different jobs:

Route
Best for
Shape

REST API

Ad-hoc queries, scheduled pulls, custom reporting

JSON, filterable and paginated

CSV export

One-off analysis, handing data to someone in a spreadsheet

CSV download

OTLP push

Continuous monitoring next to your existing telemetry

OpenTelemetry spans

REST API

Every execution is available from the execution log endpoint. Filter it the same way you filter the log view in the app — by judge or evaluator, tag, project, user, session or time range — and page through the results.

curl 'https://api.scorable.ai/v1/execution-logs/?page_size=100' \
  -H 'Authorization: Api-Key $MY_API_KEY'

See the REST API reference for the full parameter list.

CSV export

Add export=csv to the same endpoint to get a CSV file instead of JSON. The columns are the execution id, who ran it, when, the executed item and version, the model output, the score, and the per-evaluator results.

curl 'https://api.scorable.ai/v1/execution-logs/?export=csv&page_size=1000' \
  -H 'Authorization: Api-Key $MY_API_KEY' \
  -o logs.csv

The export is paginated like the JSON endpoint, so pass page_size (and page through) when you want more than one page of rows.

OTLP push to your own collector

Scorable can push every evaluation result to an OTLP collector you own — Grafana Tempo, an OpenTelemetry Collector, Honeycomb, Datadog, or anything else that speaks OTLP/HTTP. Scores then show up in your own dashboards without you polling anything.

What gets exported

  • A judge execution produces one parent span per run, plus one child span per evaluator inside it — so the judge and its individual metrics stay linked in your trace view.

  • A standalone evaluator execution produces a single span.

All spans carry the resource attribute service.name = root-signals-evaluation, which is the label to filter on in your collector.

Evaluator spans are named judge.evaluation.<evaluator_name> and carry:

Attribute
Meaning

evaluation.name

Evaluator name, e.g. Faithfulness

evaluation.score

The score, 0.0–1.0

evaluation.duration_ms

How long the evaluation took

evaluation.justification

The written rationale — only with content capture enabled

gen_ai.input.messages / gen_ai.output.messages

The evaluated request and response — only with content capture enabled

evaluation.*

Remaining execution metadata, flattened — evaluator id, version, model used, user id

Judge spans are named judge.execution.<judge_name> and carry judge.name, judge.evaluator_count, judge.duration_seconds, and the run's metadata flattened under judge.metadata.*.

Content capture is off by default

By default the exported spans contain scores and metadata only — no prompts, no model responses, no justifications. This is deliberate: it lets you put evaluation quality on a shared dashboard without pushing the underlying text, which may contain personal or otherwise sensitive data, into a second system.

Turn content capture on only when you want the full text in your collector too, and only when that collector's access controls and retention are appropriate for it. Note that this setting is independent of the execution log retention policy — the retention policy governs what Scorable stores, this governs what Scorable sends you.

Configuration

The export is configured per organization with four values:

  • Enabled — off by default.

  • Endpoint — your collector's OTLP/HTTP trace URL. Include the full path, for example https://collector.example.com:4318/v1/traces.

  • Headers — any headers your collector needs, typically authentication.

  • Capture content — off by default, as described above.

On a self-hosted deployment these live in the organization's settings in the Django admin. On Scorable Cloud, contact support@scorable.ai with the endpoint and headers you want configured.

Last updated