# Vertex AI Agent Builder

Integrate Scorable evaluations with Google Cloud's Vertex AI Agent Builder to monitor and improve your conversational AI agents in real-time.

## Architecture Overview

```
[Vertex AI Agent Builder]
     |
     |—→ [Webhook call (to Cloud Function / Cloud Run)]
                  |
                  |—→ [Scorable API]
                  |
                  |—→ [Evaluate response]
                  |
           [Log result / augment reply]
                  |
     ←——————— Reply to Agent Builder user
```

***

## 🔧 Step-by-Step Integration

### 1. **Set up a webhook in Vertex AI Agent Builder**

* Go to **"Manage Fulfillment"** in the Agent Builder UI.
* Create a webhook (can be a **Cloud Function**, **Cloud Run**, or any HTTP endpoint).
* This webhook will receive `request` and `response` pairs from user interactions.

***

### 2. **Create a middleware endpoint (Cloud Function or Cloud Run)**

This endpoint will:

* Receive user input and the LLM response.
* Construct an evaluator call to Scorable API.
* Send the result back as part of the webhook response (optional).

**Option 1: Using Built-in Evaluators**

```js
app.post('/evaluate', async (req, res) => {
  const userInput = req.body.sessionInfo.parameters.input;
  const modelResponse = req.body.fulfillmentResponse.messages[0].text.text[0];

  // Use a built-in evaluator (e.g., Relevance)
  const evaluatorPayload = {
    request: userInput,
    response: modelResponse,
  };

  const evaluatorResult = await fetch('https://api.scorable.ai/v1/skills/evaluator/execute/YOUR_EVALUATOR_ID/', {
    method: 'POST',
    headers: {
      'Authorization': 'Api-Key YOUR_API_KEY',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(evaluatorPayload),
  });

  const result = await evaluatorResult.json();
  console.log('Evaluator Score:', result.score);

  // Return modified response (if needed)
  res.json({
    fulfillment_response: {
      messages: [
        {
          text: {
            text: [
              `${modelResponse} (Quality score: ${result.score.toFixed(2)})`
            ]
          }
        }
      ]
    }
  });
});
```

**Option 2: Using Custom Judges**

```js
app.post('/evaluate', async (req, res) => {
  const userInput = req.body.sessionInfo.parameters.input;
  const modelResponse = req.body.fulfillmentResponse.messages[0].text.text[0];

  // Use a custom judge
  const judgePayload = {
    request: userInput,
    response: modelResponse,
  };

  const judgeResult = await fetch('https://api.scorable.ai/v1/judges/YOUR_JUDGE_ID/execute/', {
    method: 'POST',
    headers: {
      'Authorization': 'Api-Key YOUR_API_KEY',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(judgePayload),
  });

  const result = await judgeResult.json();
  console.log('Judge Score:', result.evaluator_results);

  // Return modified response (if needed)
  res.json({
    fulfillment_response: {
      messages: [
        {
          text: {
            text: [
              `${modelResponse} (Judge results: ${JSON.stringify(result.evaluator_results)})`
            ]
          }
        }
      ]
    }
  });
});
```

***

### 3. **Configure evaluators and judges**

**Built-in Evaluators:**

* Use evaluators like `Relevance`, `Precision`, `Completeness`, `Clarity`, etc.
* Get available evaluators by logging in to <https://scorable.ai/>
* Examples: Relevance, Truthfulness, Safety, Professional Writing

**Custom Judges:**

* Create custom judges that combine multiple evaluators - use <https://scorable.ai/> to generate a judge.
* Judges provide aggregated scoring across multiple criteria