OpenTelemetry

Scorable accepts OpenTelemetry (OTEL) traces from any agent framework. Once traces arrive, Scorable shows a per-trace view of every LLM call, its inputs and outputs, latency, and span count — and can automatically evaluate traces against your configured evaluators and judges.

This page is about traces coming into Scorable. To push evaluation results out to your own collector or dashboard, see Exporting evaluation results.

Prerequisites

You need a Scorable API key. Find it under Settings → API Keys in the dashboard.

---

Example: pydantic-ai

pydantic-ai has built-in OTEL support via InstrumentationSettings. Configure it to export to Scorable:

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from pydantic_ai import Agent, InstrumentationSettings


def _build_tracer_provider() -> TracerProvider:
    exporter = OTLPSpanExporter(
        endpoint="https://api.scorable.ai/otel/v1/traces",
        headers={"Authorization": "Api-Key <your-api-key>"},
    )
    resource = Resource.create({"service.name": "my-agent"})
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(exporter))
    return provider


agent = Agent(
    model="openai:gpt-5.2",
    instrument=InstrumentationSettings(
        tracer_provider=_build_tracer_provider(),
    ),
)

Every agent.run() call now produces a trace visible in Scorable.

---

Example: any other framework

Configure the OTEL SDK to point at Scorable's collector endpoint and set the Authorization header. The example below works with any framework that supports OTEL instrumentation (LangChain, LlamaIndex, raw openai SDK with opentelemetry-instrumentation-openai, etc.).

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = OTLPSpanExporter(
    endpoint="https://api.scorable.ai/otel/v1/traces",
    headers={"Authorization": "Api-Key <your-api-key>"},
)

resource = Resource.create({"service.name": "my-agent"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

Then instrument your framework as usual — Scorable receives whatever spans the framework emits.

Instrumentation libraries

Any OpenTelemetry-compatible instrumentation library works with Scorable. Popular options for AI/LLM workloads:

Library	Frameworks covered
OpenLIT	OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, Cohere, and more
OpenLLMetry	OpenAI, Anthropic, LangChain, LlamaIndex, Haystack, and more
smolagents	Hugging Face smolagents
CrewAI	CrewAI
AutoGen	AutoGen
LlamaIndex	LlamaIndex
Semantic Kernel	Semantic Kernel (Python, .NET, Java)

Environment variable alternative

If you prefer to configure the exporter through env vars rather than code:

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.scorable.ai/otel/v1/traces
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Api-Key <your-api-key>"
OTEL_SERVICE_NAME=my-agent

---

Viewing traces

Traces appear in the Traces tab in the dashboard. Each row represents one agent run (one trace_id), showing the root span name, time, and total span count. Click a trace to see the full span tree.

---

Automatic evaluation

You can configure Scorable to automatically evaluate incoming traces against an evaluator or judge. See Settings → Trace Evaluation Filters to set up filter criteria, sampling rate, and evaluation delay (to allow late-arriving spans before evaluation runs).

Evaluation uses the gen_ai.input.messages and gen_ai.output.messages span attributes, which pydantic-ai and most OTEL-instrumented LLM frameworks emit automatically.