# Vertex AI Agent Builder

Integrate Scorable evaluations with Google Cloud's Vertex AI Agent Builder to monitor and improve your conversational AI agents in real-time.

## Architecture Overview

```
[Vertex AI Agent Builder]
     |
     |—→ [Webhook call (to Cloud Function / Cloud Run)]
                  |
                  |—→ [Scorable API]
                  |
                  |—→ [Evaluate response]
                  |
           [Log result / augment reply]
                  |
     ←——————— Reply to Agent Builder user
```

***

## 🔧 Step-by-Step Integration

### 1. **Set up a webhook in Vertex AI Agent Builder**

* Go to **"Manage Fulfillment"** in the Agent Builder UI.
* Create a webhook (can be a **Cloud Function**, **Cloud Run**, or any HTTP endpoint).
* This webhook will receive `request` and `response` pairs from user interactions.

***

### 2. **Create a middleware endpoint (Cloud Function or Cloud Run)**

This endpoint will:

* Receive user input and the LLM response.
* Construct an evaluator call to Scorable API.
* Send the result back as part of the webhook response (optional).

**Option 1: Using Built-in Evaluators**

```js
app.post('/evaluate', async (req, res) => {
  const userInput = req.body.sessionInfo.parameters.input;
  const modelResponse = req.body.fulfillmentResponse.messages[0].text.text[0];

  // Use a built-in evaluator (e.g., Relevance)
  const evaluatorPayload = {
    request: userInput,
    response: modelResponse,
  };

  const evaluatorResult = await fetch('https://api.scorable.ai/v1/skills/evaluator/execute/YOUR_EVALUATOR_ID/', {
    method: 'POST',
    headers: {
      'Authorization': 'Api-Key YOUR_API_KEY',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(evaluatorPayload),
  });

  const result = await evaluatorResult.json();
  console.log('Evaluator Score:', result.score);

  // Return modified response (if needed)
  res.json({
    fulfillment_response: {
      messages: [
        {
          text: {
            text: [
              `${modelResponse} (Quality score: ${result.score.toFixed(2)})`
            ]
          }
        }
      ]
    }
  });
});
```

**Option 2: Using Custom Judges**

```js
app.post('/evaluate', async (req, res) => {
  const userInput = req.body.sessionInfo.parameters.input;
  const modelResponse = req.body.fulfillmentResponse.messages[0].text.text[0];

  // Use a custom judge
  const judgePayload = {
    request: userInput,
    response: modelResponse,
  };

  const judgeResult = await fetch('https://api.scorable.ai/v1/judges/YOUR_JUDGE_ID/execute/', {
    method: 'POST',
    headers: {
      'Authorization': 'Api-Key YOUR_API_KEY',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(judgePayload),
  });

  const result = await judgeResult.json();
  console.log('Judge Score:', result.evaluator_results);

  // Return modified response (if needed)
  res.json({
    fulfillment_response: {
      messages: [
        {
          text: {
            text: [
              `${modelResponse} (Judge results: ${JSON.stringify(result.evaluator_results)})`
            ]
          }
        }
      ]
    }
  });
});
```

***

### 3. **Configure evaluators and judges**

**Built-in Evaluators:**

* Use evaluators like `Relevance`, `Precision`, `Completeness`, `Clarity`, etc.
* Get available evaluators by logging in to <https://scorable.ai/>
* Examples: Relevance, Truthfulness, Safety, Professional Writing

**Custom Judges:**

* Create custom judges that combine multiple evaluators - use <https://scorable.ai/> to generate a judge.
* Judges provide aggregated scoring across multiple criteria


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.scorable.ai/integrations/vertex-ai-agent-builder.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.