> For the complete documentation index, see [llms.txt](https://docs.scorable.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scorable.ai/ci.md).

# Unit Testing in CI/CD

Integrating Scorable into your CI pipeline allows you to automatically evaluate the quality of your LLM outputs as part of your testing workflow. This ensures that regressions in response quality are caught early, before they reach production.

## How It Works

Scorable integrates with standard test frameworks, allowing you to use Judges and Evaluators as assertions in your unit and integration tests. When tests run in CI, Scorable evaluates your LLM responses and fails the test if quality thresholds aren't met.

## Supported Test Frameworks

Scorable can be integrated into any test framework. Here are guides for popular options:

* [**Evalite guide**](/integrations/evalite.md) - Modern LLM testing framework with built-in support for custom scorers
* [**Pytest guide**](/integrations/pytest.md) - Standard Python testing framework with fixtures for Scorable integration

### CLI Tool

For prompt testing and model comparison, you can use the Scorable CLI tool directly in your CI pipeline:

* [**Prompt Testing CLI**](/concepts-and-examples/cookbooks/find-the-best-prompt-and-model.md) - Compare prompts and models, evaluate outputs, and track metrics

The CLI is particularly useful for:

* Systematically testing multiple prompt variations
* Comparing different model outputs side-by-side
* Running batch evaluations with YAML configuration files
* Generating reports on speed, cost, and quality metrics

Example CI usage:

```bash
# Install CLI
curl -sSL https://scorable.ai/cli/install.sh | sh

# Run prompt tests
scorable prompt-test run
```

The CLI reads its API key from the `SCORABLE_API_KEY` environment variable and an optional project scope from `SCORABLE_PROJECT_ID`, so it works in CI without any interactive login. Every command prints structured JSON on success and exits with a non-zero code on API errors, which makes the output easy to parse and gate on.

### Example: GitHub Actions

```yaml
name: llm-quality
on: [pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Install Scorable CLI
        run: npm install -g @root-signals/scorable-cli
      - name: Run judge against a sample response
        env:
          SCORABLE_API_KEY: ${{ secrets.SCORABLE_API_KEY }}
        run: |
          result=$(scorable judge execute YOUR_JUDGE_ID \
            --request "What is the refund policy?" \
            --response "$(cat sample-response.txt)" \
            --tags "ci,${GITHUB_SHA}")
          echo "$result"
          # Fail the job if any evaluator scores below 0.7
          echo "$result" | jq -e '[.evaluator_results[].score] | min >= 0.7'
```

The same pattern works in GitLab CI, CircleCI, or any runner with Node.js 20+ available.

## Best Practices

### 1. Use Tags for Tracking

Tag your evaluations with metadata like git commit hashes to track results over time:

```typescript
// TypeScript
tags: ["test", process.env.GIT_COMMIT || "local"]
```

```python
# Python
tags=["test", os.getenv("GIT_COMMIT", "local")]
```

### 2. Set Appropriate Thresholds

Start with lower thresholds and gradually increase them as you improve your prompts:

```python
# Start conservative
assert_scorable_quality(..., threshold=0.7)

# Increase as quality improves
assert_scorable_quality(..., threshold=0.85)
```