gemini/evaluation/evaluate_rag_gen_ai_evaluation_service_sdk.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2eec5cc39a59"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "307d01b6f573"
      },
      "source": [
        "# Evaluate Generated Answers from Retrieval-Augmented Generation (RAG) for Question Answering with Gen AI Evaluation Service SDK\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_rag_gen_ai_evaluation_service_sdk.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaluate_rag_gen_ai_evaluation_service_sdk.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_rag_gen_ai_evaluation_service_sdk.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaluate_rag_gen_ai_evaluation_service_sdk.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>    \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07f66b85fabc"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Jason Dai](https://github.com/jsondai), [Kelsi Lakey](https://github.com/lakeyk) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ce61c0013e2c"
      },
      "source": [
        "**_NOTE_**: This notebook has been tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ebb820243a69"
      },
      "source": [
        "## Overview\n",
        "\n",
        "\n",
        "In this tutorial, you will learn how to use the use the *Vertex AI Python SDK for Gen AI Evaluation* to evaluate **Retrieval-Augmented Generation** (RAG) generated answers for **Question Answering** (QA) task.\n",
        "\n",
        "RAG is a technique to improve groundness, relevancy and factuality of large language models (LLMs) by finding relevant information from the model's knowledge base. RAG is done by converting a query into a vector representation (embeddings), and then finding the most similar vectors in the knowledge base. The most similar vectors are then used to help generate the response.\n",
        "\n",
        "This tutorial demonstrates how to use Gen AI Evaluation for a Bring-Your-Own-Response scenario:\n",
        "\n",
        "* The context was retrieved and the answers were generated based on the retrieved context using RAG.\n",
        "\n",
        "* Evaluate the quality of the RAG generated answers for QA task programmatically using the SDK.\n",
        "\n",
        "The examples used in this notebook is from Stanford Question Answering Dataset [SQuAD 2.0](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/reports/default/15785042.pdf).\n",
        "\n",
        "Learn more about [Vertex AI Rapid Evaluation SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/models/online-pipeline-services).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8925ff9e165e"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b07851728f7b"
      },
      "source": [
        "### Install Vertex AI SDK for Gen AI Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3286d0a38129"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a68700272183"
      },
      "source": [
        "### Restart runtime\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6425258a0ef9"
      },
      "outputs": [],
      "source": [
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ffcde4d56c00"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "791c9835eceb"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ff2655451691"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "baa90bc5077a"
      },
      "source": [
        "### Increase quota (optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQiXNXaUN42j"
      },
      "source": [
        "Increasing the quota may lead to better performance and user experience. Read more about this at [online evaluation quotas](https://cloud.google.com/vertex-ai/generative-ai/docs/quotas#eval-quotas)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "384d31063617"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "416bbaf5a2a4"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "EXPERIMENT = \"rag-eval-01\"  # @param {type:\"string\"}\n",
        "\n",
        "\n",
        "import vertexai\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    raise ValueError(\"Please set your PROJECT_ID\")\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26bcfc24e720"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "35c5cf226809"
      },
      "outputs": [],
      "source": [
        "import inspect\n",
        "import logging\n",
        "import warnings\n",
        "\n",
        "# General\n",
        "from IPython.display import HTML, Markdown, display\n",
        "import pandas as pd\n",
        "import plotly.graph_objects as go\n",
        "\n",
        "# Main\n",
        "from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples, PointwiseMetric"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4658fb61af1"
      },
      "source": [
        "### Library settings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "69d490ad8329"
      },
      "outputs": [],
      "source": [
        "logging.getLogger(\"urllib3.connectionpool\").setLevel(logging.ERROR)\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eb47cbc98ac0"
      },
      "source": [
        "### Helper functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d435a8f422ac"
      },
      "outputs": [],
      "source": [
        "def print_doc(function):\n",
        "    print(f\"{function.__name__}:\\n{inspect.getdoc(function)}\\n\")\n",
        "\n",
        "\n",
        "def display_eval_report(eval_result, metrics=None):\n",
        "    \"\"\"Display the evaluation results.\"\"\"\n",
        "\n",
        "    title, summary_metrics, report_df = eval_result\n",
        "    metrics_df = pd.DataFrame.from_dict(summary_metrics, orient=\"index\").T\n",
        "    if metrics:\n",
        "        metrics_df = metrics_df.filter(\n",
        "            [\n",
        "                metric\n",
        "                for metric in metrics_df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "        report_df = report_df.filter(\n",
        "            [\n",
        "                metric\n",
        "                for metric in report_df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "\n",
        "    # Display the title with Markdown for emphasis\n",
        "    display(Markdown(f\"## {title}\"))\n",
        "\n",
        "    # Display the metrics DataFrame\n",
        "    display(Markdown(\"### Summary Metrics\"))\n",
        "    display(metrics_df)\n",
        "\n",
        "    # Display the detailed report DataFrame\n",
        "    display(Markdown(\"### Report Metrics\"))\n",
        "    display(report_df)\n",
        "\n",
        "\n",
        "def display_explanations(df, metrics=None, n=1):\n",
        "    style = \"white-space: pre-wrap; width: 800px; overflow-x: auto;\"\n",
        "    df = df.sample(n=n)\n",
        "    if metrics:\n",
        "        df = df.filter(\n",
        "            [\"instruction\", \"context\", \"reference\", \"completed_prompt\", \"response\"]\n",
        "            + [\n",
        "                metric\n",
        "                for metric in df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "\n",
        "    for index, row in df.iterrows():\n",
        "        for col in df.columns:\n",
        "            display(HTML(f\"<h2>{col}:</h2> <div style='{style}'>{row[col]}</div>\"))\n",
        "        display(HTML(\"<hr>\"))\n",
        "\n",
        "\n",
        "def plot_radar_plot(eval_results, max_score=5, metrics=None):\n",
        "    fig = go.Figure()\n",
        "\n",
        "    for eval_result in eval_results:\n",
        "        title, summary_metrics, report_df = eval_result\n",
        "\n",
        "        if metrics:\n",
        "            summary_metrics = {\n",
        "                k: summary_metrics[k]\n",
        "                for k, v in summary_metrics.items()\n",
        "                if any(selected_metric in k for selected_metric in metrics)\n",
        "            }\n",
        "\n",
        "        fig.add_trace(\n",
        "            go.Scatterpolar(\n",
        "                r=list(summary_metrics.values()),\n",
        "                theta=list(summary_metrics.keys()),\n",
        "                fill=\"toself\",\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "\n",
        "    fig.update_layout(\n",
        "        polar=dict(radialaxis=dict(visible=True, range=[0, max_score])), showlegend=True\n",
        "    )\n",
        "\n",
        "    fig.show()\n",
        "\n",
        "\n",
        "def plot_bar_plot(eval_results, metrics=None):\n",
        "    fig = go.Figure()\n",
        "    data = []\n",
        "\n",
        "    for eval_result in eval_results:\n",
        "        title, summary_metrics, _ = eval_result\n",
        "        if metrics:\n",
        "            summary_metrics = {\n",
        "                k: summary_metrics[k]\n",
        "                for k, v in summary_metrics.items()\n",
        "                if any(selected_metric in k for selected_metric in metrics)\n",
        "            }\n",
        "\n",
        "        data.append(\n",
        "            go.Bar(\n",
        "                x=list(summary_metrics.keys()),\n",
        "                y=list(summary_metrics.values()),\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "\n",
        "    fig = go.Figure(data=data)\n",
        "\n",
        "    # Change the bar mode\n",
        "    fig.update_layout(barmode=\"group\")\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a7cbf0f80994"
      },
      "source": [
        "# Bring-Your-Own-Response Evaluation for RAG: Reference-Free (without Golden Answer)\n",
        "\n",
        "Perform bring-your-own-response evaluation by assessing the generated answer's quality based on the context retrieved. It does not compare with golden answer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "65f3c127e2a4"
      },
      "source": [
        "### Prepare your dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6faab1bac7e9"
      },
      "source": [
        "To evaluate the RAG generated answers, the evaluation dataset is required to contain the following fields:\n",
        "\n",
        "* Prompt: The user supplied prompt consisting of the User Question and the RAG Retrieved Context\n",
        "* Response: The RAG Generated Answer\n",
        "\n",
        "Your dataset must include a minimum of one evaluation example. We recommend around 100 examples to ensure high-quality aggregated metrics and statistically significant results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "45357974c82f"
      },
      "outputs": [],
      "source": [
        "questions = [\n",
        "    \"Which part of the brain does short-term memory seem to rely on?\",\n",
        "    \"What provided the Roman senate with exuberance?\",\n",
        "    \"What area did the Hasan-jalalians command?\",\n",
        "]\n",
        "\n",
        "retrieved_contexts = [\n",
        "    \"Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning.\",\n",
        "    \"In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship.\",\n",
        "    \"The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh.\",\n",
        "]\n",
        "\n",
        "generated_answers_by_rag_a = [\n",
        "    \"frontal lobe and the parietal lobe\",\n",
        "    \"The Roman Senate was filled with exuberance due to successes against Catiline.\",\n",
        "    \"The Hasan-Jalalians commanded the area of Syunik and Vayots Dzor.\",\n",
        "]\n",
        "\n",
        "generated_answers_by_rag_b = [\n",
        "    \"Occipital lobe\",\n",
        "    \"The Roman Senate was subdued because they had food poisoning.\",\n",
        "    \"The Galactic Empire commanded the state of Utah.\",\n",
        "]\n",
        "\n",
        "eval_dataset_rag_a = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [\n",
        "            \"Answer the question: \" + question + \" Context: \" + item\n",
        "            for question, item in zip(questions, retrieved_contexts)\n",
        "        ],\n",
        "        \"response\": generated_answers_by_rag_a,\n",
        "    }\n",
        ")\n",
        "\n",
        "eval_dataset_rag_b = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [\n",
        "            \"Answer the question: \" + question + \" Context: \" + item\n",
        "            for question, item in zip(questions, retrieved_contexts)\n",
        "        ],\n",
        "        \"response\": generated_answers_by_rag_b,\n",
        "    }\n",
        ")\n",
        "\n",
        "eval_dataset_rag_a"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6109ce3b080c"
      },
      "source": [
        "### Select and create metrics\n",
        "\n",
        "\n",
        "You can run evaluation for just one metric, or a combination of metrics. For this example, we select a few RAG-related predefined metrics, and create a few of our own custom metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41860924a9da"
      },
      "source": [
        "#### Explore predefined metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f4bd293770e8"
      },
      "outputs": [],
      "source": [
        "# See all the available metric examples\n",
        "MetricPromptTemplateExamples.list_example_metric_names()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "24e423f8781a"
      },
      "outputs": [],
      "source": [
        "# See the prompt example for one of the pointwise metrics\n",
        "print(MetricPromptTemplateExamples.get_prompt_template(\"question_answering_quality\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2e30cfbf9f75"
      },
      "source": [
        "#### Create custom metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2c5330ad250a"
      },
      "outputs": [],
      "source": [
        "relevance_prompt_template = \"\"\"\n",
        "You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.\n",
        "\n",
        "You will be assessing relevance, which measures the ability to respond with relevant information when given a prompt.\n",
        "\n",
        "You will assign the writing response a score from 5, 4, 3, 2, 1, following the rating rubric and evaluation steps.\n",
        "\n",
        "## Criteria\n",
        "Relevance: The response should be relevant to the instruction and directly address the instruction.\n",
        "\n",
        "## Rating Rubric\n",
        "5 (completely relevant): Response is entirely relevant to the instruction and provides clearly defined information that addresses the instruction's core needs directly.\n",
        "4 (mostly relevant): Response is mostly relevant to the instruction and addresses the instruction mostly directly.\n",
        "3 (somewhat relevant): Response is somewhat relevant to the instruction and may address the instruction indirectly, but could be more relevant and more direct.\n",
        "2 (somewhat irrelevant): Response is minimally relevant to the instruction and does not address the instruction directly.\n",
        "1 (irrelevant): Response is completely irrelevant to the instruction.\n",
        "\n",
        "## Evaluation Steps\n",
        "STEP 1: Assess relevance: is response relevant to the instruction and directly address the instruction?\n",
        "STEP 2: Score based on the criteria and rubrics.\n",
        "\n",
        "Give step by step explanations for your scoring, and only choose scores from 5, 4, 3, 2, 1.\n",
        "\n",
        "\n",
        "# User Inputs and AI-generated Response\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "## AI-generated Response\n",
        "{response}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f2da7b722d0e"
      },
      "outputs": [],
      "source": [
        "helpfulness_prompt_template = \"\"\"\n",
        "You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.\n",
        "\n",
        "You will be assessing helpfulness, which measures the ability to provide important details when answering a prompt.\n",
        "\n",
        "You will assign the writing response a score from 5, 4, 3, 2, 1, following the rating rubric and evaluation steps.\n",
        "\n",
        "## Criteria\n",
        "Helpfulness: The response is comprehensive with well-defined key details. The user would feel very satisfied with the content in a good response.\n",
        "\n",
        "## Rating Rubric\n",
        "5 (completely helpful): Response is useful and very comprehensive with well-defined key details to address the needs in the instruction and usually beyond what explicitly asked. The user would feel very satisfied with the content in the response.\n",
        "4 (mostly helpful): Response is very relevant to the instruction, providing clearly defined information that addresses the instruction's core needs.  It may include additional insights that go slightly beyond the immediate instruction.  The user would feel quite satisfied with the content in the response.\n",
        "3 (somewhat helpful): Response is relevant to the instruction and provides some useful content, but could be more relevant, well-defined, comprehensive, and/or detailed. The user would feel somewhat satisfied with the content in the response.\n",
        "2 (somewhat unhelpful): Response is minimally relevant to the instruction and may provide some vaguely useful information, but it lacks clarity and detail. It might contain minor inaccuracies. The user would feel only slightly satisfied with the content in the response.\n",
        "1 (unhelpful): Response is useless/irrelevant, contains inaccurate/deceptive/misleading information, and/or contains harmful/offensive content. The user would feel not at all satisfied with the content in the response.\n",
        "\n",
        "## Evaluation Steps\n",
        "STEP 1: Assess comprehensiveness: does the response provide specific, comprehensive, and clearly defined information for the user needs expressed in the instruction?\n",
        "STEP 2: Assess relevance: When appropriate for the instruction, does the response exceed the instruction by providing relevant details and related information to contextualize content and help the user better understand the response.\n",
        "STEP 3: Assess accuracy: Is the response free of inaccurate, deceptive, or misleading information?\n",
        "STEP 4: Assess safety: Is the response free of harmful or offensive content?\n",
        "\n",
        "Give step by step explanations for your scoring, and only choose scores from 5, 4, 3, 2, 1.\n",
        "\n",
        "\n",
        "# User Inputs and AI-generated Response\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "## AI-generated Response\n",
        "{response}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "185c98ea135e"
      },
      "outputs": [],
      "source": [
        "relevance = PointwiseMetric(\n",
        "    metric=\"relevance\",\n",
        "    metric_prompt_template=relevance_prompt_template,\n",
        ")\n",
        "\n",
        "helpfulness = PointwiseMetric(\n",
        "    metric=\"helpfulness\",\n",
        "    metric_prompt_template=helpfulness_prompt_template,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "421a1e185be3"
      },
      "source": [
        "### Run evaluation with your dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c02813283dff"
      },
      "outputs": [],
      "source": [
        "rag_eval_task_rag_a = EvalTask(\n",
        "    dataset=eval_dataset_rag_a,\n",
        "    metrics=[\n",
        "        \"question_answering_quality\",\n",
        "        relevance,\n",
        "        helpfulness,\n",
        "        \"groundedness\",\n",
        "        \"safety\",\n",
        "        \"instruction_following\",\n",
        "    ],\n",
        "    experiment=EXPERIMENT,\n",
        ")\n",
        "\n",
        "rag_eval_task_rag_b = EvalTask(\n",
        "    dataset=eval_dataset_rag_b,\n",
        "    metrics=[\n",
        "        \"question_answering_quality\",\n",
        "        relevance,\n",
        "        helpfulness,\n",
        "        \"groundedness\",\n",
        "        \"safety\",\n",
        "        \"instruction_following\",\n",
        "    ],\n",
        "    experiment=EXPERIMENT,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cacdc1fce57e"
      },
      "outputs": [],
      "source": [
        "result_rag_a = rag_eval_task_rag_a.evaluate()\n",
        "result_rag_b = rag_eval_task_rag_b.evaluate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "66dee7f2c766"
      },
      "source": [
        "### Display evaluation results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b74f297d44bb"
      },
      "source": [
        "#### View summary results\n",
        "\n",
        "If you want to have an overall view of all the metrics from individual model's evaluation result in one table, you can use the `display_eval_report()` helper function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eb8cdb489f5c"
      },
      "outputs": [],
      "source": [
        "display_eval_report(\n",
        "    (\n",
        "        \"Model A Eval Result\",\n",
        "        result_rag_a.summary_metrics,\n",
        "        result_rag_a.metrics_table,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3a6b252645eb"
      },
      "outputs": [],
      "source": [
        "display_eval_report(\n",
        "    (\n",
        "        \"Model B Eval Result\",\n",
        "        result_rag_b.summary_metrics,\n",
        "        result_rag_b.metrics_table,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a9fd83c386c9"
      },
      "source": [
        "#### Visualize evaluation results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "42d1a52893d5"
      },
      "outputs": [],
      "source": [
        "eval_results = []\n",
        "eval_results.append(\n",
        "    (\"Model A\", result_rag_a.summary_metrics, result_rag_a.metrics_table)\n",
        ")\n",
        "eval_results.append(\n",
        "    (\"Model B\", result_rag_b.summary_metrics, result_rag_b.metrics_table)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fd2f741ab03c"
      },
      "outputs": [],
      "source": [
        "plot_radar_plot(\n",
        "    eval_results,\n",
        "    metrics=[\n",
        "        f\"{metric}/mean\"\n",
        "        # Edit your list of metrics here if you used other metrics in evaluation.\n",
        "        for metric in [\n",
        "            \"question_answering_quality\",\n",
        "            \"safety\",\n",
        "            \"groundedness\",\n",
        "            \"instruction_following\",\n",
        "            \"relevance\",\n",
        "            \"helpfulness\",\n",
        "        ]\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "83bed587a29d"
      },
      "outputs": [],
      "source": [
        "plot_bar_plot(\n",
        "    eval_results,\n",
        "    metrics=[\n",
        "        f\"{metric}/mean\"\n",
        "        for metric in [\n",
        "            \"question_answering_quality\",\n",
        "            \"safety\",\n",
        "            \"groundedness\",\n",
        "            \"instruction_following\",\n",
        "            \"relevance\",\n",
        "            \"helpfulness\",\n",
        "        ]\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0dd4e086c3ea"
      },
      "source": [
        "#### View detailed explanation for an individual instance\n",
        "\n",
        "If you need to delve into the individual result's detailed explanations on why a score is assigned and how confident the model is for each model-based metric, you can use the `display_explanations()` helper function. For example, you can set `n=2` to display explanation of the 2nd instance result as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7ce3ac58057b"
      },
      "outputs": [],
      "source": [
        "display_explanations(result_rag_a.metrics_table, n=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3285d336a616"
      },
      "source": [
        "You can also focus on one or a few metrics as follows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "401840759309"
      },
      "outputs": [],
      "source": [
        "display_explanations(result_rag_b.metrics_table, metrics=[\"groundedness\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "54a0c8e939dd"
      },
      "source": [
        "# Bring-Your-Own-Response Evaluation for RAG: Referenced (with Golden Answer)\n",
        "\n",
        "Perform bring-your-own-response evaluation by assessing the generated answer's quality based on the context retrieved and the golden answer provided in the reference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "15200a0e9613"
      },
      "source": [
        "### Prepare your dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a05fe0ea0a7b"
      },
      "source": [
        "To evaluate the RAG generated answers, the evaluation dataset is required to contain the following fields:\n",
        "\n",
        "* Prompt: The user supplied prompt consisting of the User Question and the RAG Retrieved Context\n",
        "* Response: The RAG Generated Answer\n",
        "* Reference: The Golden Answer groundtruth to compare model response to"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "608be2b08512"
      },
      "outputs": [],
      "source": [
        "questions = [\n",
        "    \"Which part of the brain does short-term memory seem to rely on?\",\n",
        "    \"What provided the Roman senate with exuberance?\",\n",
        "    \"What area did the Hasan-jalalians command?\",\n",
        "]\n",
        "\n",
        "retrieved_contexts = [\n",
        "    \"Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning.\",\n",
        "    \"In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship.\",\n",
        "    \"The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh.\",\n",
        "]\n",
        "\n",
        "generated_answers_by_rag_a = [\n",
        "    \"frontal lobe and the parietal lobe\",\n",
        "    \"The Roman Senate was filled with exuberance due to successes against Catiline.\",\n",
        "    \"The Hasan-Jalalians commanded the area of Syunik and Vayots Dzor.\",\n",
        "]\n",
        "\n",
        "generated_answers_by_rag_b = [\n",
        "    \"Occipital lobe\",\n",
        "    \"The Roman Senate was subdued because they had food poisoning.\",\n",
        "    \"The Galactic Empire commanded the state of Utah.\",\n",
        "]\n",
        "\n",
        "golden_answers = [\n",
        "    \"frontal lobe and the parietal lobe\",\n",
        "    \"Due to successes against Catiline.\",\n",
        "    \"The Hasan-Jalalians commanded the area of Artsakh and Utik.\",\n",
        "]\n",
        "\n",
        "referenced_eval_dataset_rag_a = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [\n",
        "            \"Answer the question: \" + question + \" Context: \" + item\n",
        "            for question, item in zip(questions, retrieved_contexts)\n",
        "        ],\n",
        "        \"response\": generated_answers_by_rag_a,\n",
        "        \"reference\": golden_answers,\n",
        "    }\n",
        ")\n",
        "\n",
        "referenced_eval_dataset_rag_b = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [\n",
        "            \"Answer the question: \" + question + \" Context: \" + item\n",
        "            for question, item in zip(questions, retrieved_contexts)\n",
        "        ],\n",
        "        \"response\": generated_answers_by_rag_b,\n",
        "        \"reference\": golden_answers,\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9d4a23df4c01"
      },
      "source": [
        "### Create custom metrics\n",
        "\n",
        "\n",
        "Create a custom metric to compare model response to the golden answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1f42729fca3f"
      },
      "outputs": [],
      "source": [
        "question_answering_correctness_prompt_template = \"\"\"\n",
        "You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.\n",
        "\n",
        "You will be assessing question answering correctness, which measures the ability to correctly answer a question.\n",
        "\n",
        "You will assign the writing response a score from 1, 0, following the rating rubric and evaluation steps.\n",
        "\n",
        "### Criteria:\n",
        "Reference claim alignment: The response should contain all claims from the reference and should not contain claims that are not present in the reference.\n",
        "\n",
        "### Rating Rubric:\n",
        "1 (correct): The response contains all claims from the reference and does not contain claims that are not present in the reference.\n",
        "0 (incorrect): The response does not contain all claims from the reference, or the response contains claims that are not present in the reference.\n",
        "\n",
        "### Evaluation Steps:\n",
        "STEP 1: Assess the response' correctness by comparing with the reference according to the criteria.\n",
        "STEP 2: Score based on the rubrics.\n",
        "\n",
        "Give step by step explanations for your scoring, and only choose scores from 1, 0.\n",
        "\n",
        "\n",
        "# User Inputs and AI-generated Response\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "## Reference\n",
        "{reference}\n",
        "\n",
        "## AI-generated Response\n",
        "{response}\n",
        "\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "adb261d8f699"
      },
      "outputs": [],
      "source": [
        "question_answering_correctness = PointwiseMetric(\n",
        "    metric=\"question_answering_correctness\",\n",
        "    metric_prompt_template=question_answering_correctness_prompt_template,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1715f919dee7"
      },
      "source": [
        "### Run evaluation with your dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f5ad22f52ac4"
      },
      "outputs": [],
      "source": [
        "referenced_answer_eval_task_rag_a = EvalTask(\n",
        "    dataset=referenced_eval_dataset_rag_a,\n",
        "    metrics=[\n",
        "        question_answering_correctness,\n",
        "        \"rouge\",\n",
        "        \"bleu\",\n",
        "        \"exact_match\",\n",
        "    ],\n",
        "    experiment=EXPERIMENT,\n",
        ")\n",
        "\n",
        "referenced_answer_eval_task_rag_b = EvalTask(\n",
        "    dataset=referenced_eval_dataset_rag_b,\n",
        "    metrics=[\n",
        "        question_answering_correctness,\n",
        "        \"rouge\",\n",
        "        \"bleu\",\n",
        "        \"exact_match\",\n",
        "    ],\n",
        "    experiment=EXPERIMENT,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5a143d35377f"
      },
      "outputs": [],
      "source": [
        "referenced_result_rag_a = referenced_answer_eval_task_rag_a.evaluate()\n",
        "referenced_result_rag_b = referenced_answer_eval_task_rag_b.evaluate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "901e4a490473"
      },
      "source": [
        "### Display evaluation results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7671dad25944"
      },
      "source": [
        "#### View summary results\n",
        "\n",
        "If you want to have an overall view of all the metrics evaluation result in one table, you can use the `display_eval_report()` helper function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "baedc54b897a"
      },
      "outputs": [],
      "source": [
        "display_eval_report(\n",
        "    (\n",
        "        \"Model A Eval Result\",\n",
        "        referenced_result_rag_a.summary_metrics,\n",
        "        referenced_result_rag_a.metrics_table,\n",
        "    )\n",
        ")\n",
        "display_eval_report(\n",
        "    (\n",
        "        \"Model B Eval Result\",\n",
        "        referenced_result_rag_b.summary_metrics,\n",
        "        referenced_result_rag_b.metrics_table,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eab913a13b1"
      },
      "source": [
        "#### Visualize evaluation results\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "29740e79e37a"
      },
      "outputs": [],
      "source": [
        "referenced_eval_results = []\n",
        "referenced_eval_results.append(\n",
        "    (\n",
        "        \"Model A\",\n",
        "        referenced_result_rag_a.summary_metrics,\n",
        "        referenced_result_rag_a.metrics_table,\n",
        "    )\n",
        ")\n",
        "referenced_eval_results.append(\n",
        "    (\n",
        "        \"Model B\",\n",
        "        referenced_result_rag_b.summary_metrics,\n",
        "        referenced_result_rag_b.metrics_table,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "89f606f0a235"
      },
      "outputs": [],
      "source": [
        "plot_radar_plot(\n",
        "    referenced_eval_results,\n",
        "    max_score=1,\n",
        "    metrics=[\n",
        "        f\"{metric}/mean\"\n",
        "        # Edit your list of metrics here if you used other metrics in evaluation.\n",
        "        for metric in [\n",
        "            \"question_answering_correctness\",\n",
        "            \"rouge\",\n",
        "            \"bleu\",\n",
        "            \"exact_match\",\n",
        "        ]\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ff21c08d7214"
      },
      "outputs": [],
      "source": [
        "plot_bar_plot(\n",
        "    referenced_eval_results,\n",
        "    metrics=[\n",
        "        f\"{metric}/mean\"\n",
        "        for metric in [\"question_answering_correctness\", \"rouge\", \"bleu\", \"exact_match\"]\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c6ea0658bdef"
      },
      "source": [
        "#### View detailed explanation for an individual instance\n",
        "\n",
        "If you need to delve into the individual result's detailed explanations on why a score is assigned and how confident the model is for each model-based metric, you can use the `display_explanations()` helper function. For example, you can set `n=2` to display explanation of the 2nd instance result as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8ef40d011c7d"
      },
      "outputs": [],
      "source": [
        "display_explanations(referenced_result_rag_a.metrics_table, n=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f720dc29f04e"
      },
      "outputs": [],
      "source": [
        "display_explanations(\n",
        "    referenced_result_rag_a.metrics_table, metrics=[\"question_answering_correctness\"]\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "evaluate_rag_gen_ai_evaluation_service_sdk.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}