Copy_of_PyTorch_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "A100",
      "authorship_tag": "ABX9TyP6e1s/MhrPdYVPldtruR/L",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/christophergaughan/PyTorch/blob/main/Copy_of_PyTorch_Classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "| Hyperparameter             | Binary Classification                                                                                              | Multiclass Classification                                                                                  |\n",
        "|----------------------------|-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|\n",
        "| **Input layer shape (in_features)** | Same as number of features (e.g. 5 for age, sex, height, weight, smoking status in heart disease prediction) | Same as binary classification                                                                             |\n",
        "| **Hidden layer(s)**        | Problem specific, minimum = 1, maximum = unlimited                                                               | Same as binary classification                                                                             |\n",
        "| **Neurons per hidden layer** | Problem specific, generally 10 to 512                                                                            | Same as binary classification                                                                             |\n",
        "| **Output layer shape (out_features)** | 1 (one class or the other)                                                                                  | 1 per class (e.g. 3 for food, person, or dog photo)                                                       |\n",
        "| **Hidden layer activation** | Usually ReLU (rectified linear unit) but can be many others                                                      | Same as binary classification                                                                             |\n",
        "| **Output activation**      | Sigmoid (`torch.sigmoid` in PyTorch)                                                                             | Softmax (`torch.softmax` in PyTorch)                                                                      |\n",
        "| **Loss function**          | Binary crossentropy (`torch.nn.BCELoss` in PyTorch)                                                              | Cross entropy (`torch.nn.CrossEntropyLoss` in PyTorch)                                                    |\n",
        "| **Optimizer**              | SGD (stochastic gradient descent), Adam (see `torch.optim` for more options)                                    | Same as binary classification                                                                             |\n"
      ],
      "metadata": {
        "id": "uVKHBLYJLUx0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Classification is a problem connecting to whether one thing is identified with another"
      ],
      "metadata": {
        "id": "U-7nftIzMrSp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Make classification data and get it ready\n",
        "\n",
        "- This is a dataset already made in scikitlearn"
      ],
      "metadata": {
        "id": "HT05ni0-NBQn"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dYoqfWpsI-lt"
      },
      "outputs": [],
      "source": [
        "import sklearn\n",
        "from sklearn.datasets import make_circles\n",
        "\n",
        "# Make 1000 circles\n",
        "n_samples = 1000\n",
        "\n",
        "# Create circles\n",
        "X, y = make_circles(n_samples,\n",
        "                    noise = 0.03,\n",
        "                    random_state=42)\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(X), len(y)"
      ],
      "metadata": {
        "id": "kWrKs39mNtnJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f'First 5 samples of X:\\n {X[:5]}')\n",
        "print(f'First 5 samples of y:\\n {y[:5]}')"
      ],
      "metadata": {
        "id": "EVNk5PF3NzqH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y"
      ],
      "metadata": {
        "id": "SV_1vU-2OB_d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Clearly, we have a binary classification problem here as we have only 0's and 1's in the predictor column $(y)$"
      ],
      "metadata": {
        "id": "eyw30mjFOWbh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Make a dataframe\n",
        "import pandas as pd\n",
        "circles = pd.DataFrame({'X1': X[:, 0],\n",
        "                        'X2': X[:, 1],\n",
        "                        'label': y})\n",
        "circles.head()"
      ],
      "metadata": {
        "id": "VMtSojY0OSy0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize data\n",
        "import matplotlib.pyplot as plt\n",
        "plt.scatter(x=X[:, 0],\n",
        "            y=X[:, 1],\n",
        "            c=y,\n",
        "            cmap=plt.cm.RdYlBu);"
      ],
      "metadata": {
        "id": "CoY2gV6LO0pe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### This is a *toy dataset*: small enough to experiment with, but it gives us a platform to employ PyTorch code\n",
        "\n",
        "**Our goal: separate the blue dots from the red dots**"
      ],
      "metadata": {
        "id": "ch88W8SePoQB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Check input and output shapes\n",
        "X.shape, y.shape"
      ],
      "metadata": {
        "id": "tSRUPJPaPO3J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# The data is in numpy arrays, we need to turn into pytorch tensors\n",
        "import torch\n",
        "X = torch.from_numpy(X).type(torch.float)\n",
        "y = torch.from_numpy(y).type(torch.float)"
      ],
      "metadata": {
        "id": "CVcr6WPlQc2e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X[:5], y[:5]"
      ],
      "metadata": {
        "id": "2hofD8v1QrIn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f'Shape of X: {X.shape}')\n",
        "print(f'Shape of y: {y.shape}')"
      ],
      "metadata": {
        "id": "KodNZ4lCQvL2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(f'Values for one sample of X: {X[0]} with shape: {X[0].shape}')\n",
        "print(f'Values for one sample of y: {y[0]} with shape: {y[0].shape}')"
      ],
      "metadata": {
        "id": "_7d3YFuYQ9N6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create train and test splits"
      ],
      "metadata": {
        "id": "XMGHfSv2SKM7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "torch.__version__"
      ],
      "metadata": {
        "id": "Om660CplRMPN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X.dtype, y.dtype"
      ],
      "metadata": {
        "id": "-d7tVr_oSYlx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Split data randomly\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,\n",
        "                                                    y,\n",
        "                                                    test_size=0.2,\n",
        "                                                    random_state=42)\n"
      ],
      "metadata": {
        "id": "x0EvBfaYSvPf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(X_train), len(X_test), len(y_train), len(y_test)"
      ],
      "metadata": {
        "id": "xGAgoCn0Ttyn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Build Model\n",
        "\n",
        "1. <b> *Device agnostoc code* </b>\n",
        "2. Construct a model by subclassing `nn.Module`\n",
        "3. loss function and optimizer\n",
        "4. Create training and test loop"
      ],
      "metadata": {
        "id": "A96B4o3oUP1E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from torch import nn\n",
        "\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "device\n"
      ],
      "metadata": {
        "id": "YakUu5hPTwYq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. Subclass `nn.Module`\n",
        "2. Create 2 `nn.Linear()` layers capable of handling the shapes in our data\n",
        "3. Define `forward()` method that outlines the forward pass\n",
        "4. Instantiate an instance of our model class and sen to target `device`"
      ],
      "metadata": {
        "id": "h28PnnBZVuZ4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Subclass nn.Module\n",
        "class CircleModelV0(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        # # Create nn.Linear layers capable of handling the shapes of our data\n",
        "        # self.layer_1 = nn.Linear(in_features=2,\n",
        "        #                          out_features=5) # upscales to 5 features (hidden layers\n",
        "\n",
        "        # self.layer_2 = nn.Linear(in_features=5,\n",
        "        #                          out_features=1) # we're predicting a 0 or 1\n",
        "        self.two_linear = nn.Sequential(\n",
        "            nn.Linear(in_features=2,\n",
        "                      out_features=5),\n",
        "            nn.Linear(in_features=5,\n",
        "                      out_features=1)\n",
        "        )\n",
        "    # define the forward pass\n",
        "    def forward(self, x):\n",
        "        return self.two_linear(x)\n",
        "    #   return self.layer_2(self.layer_1(x)) # x-> layer_1 -> layer_2 -> output\n",
        "\n",
        "# Instantiate instance of model class and send to target device\n",
        "model_0 = CircleModelV0().to(device)\n",
        "model_0\n"
      ],
      "metadata": {
        "id": "kI0BAKhWVEfc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Note in the code above:\n",
        "\n",
        "The forward pass in the provided code may seem \"backwards\" because the sequence in which the operations are written in code starts with the last layer and progresses to the input layer, but this is simply a reflection of the computation flow in neural networks. Let's break it down:\n",
        "\n",
        "#### Understanding the forward Pass\n",
        "Order of Operations:\n",
        "\n",
        "* When you call self.layer_1(x), the input x is passed through layer_1. * This produces the intermediate output of the first layer.\n",
        "* The intermediate output is then passed to self.layer_2, which produces the final output.\n",
        "\n",
        "In functional terms:\n",
        "`x -> layer_1 -> layer_2 -> output\n",
        "`\n",
        "However, the Python code is written as:\n",
        "`return self.layer_2(self.layer_1(x))\n",
        "`\n",
        "\n",
        "This is standard practice in programming because you apply the innermost function (layer 1) first and then the outermost function (layer 2).\n",
        "\n",
        "#### Why It Feels \"Backwards\":\n",
        "\n",
        "* Neural network layers are typically thought of as a forward progression from input to output.\n",
        "* In the `forward` method, the \"nesting\" structure can feel reversed because you start with the input, apply transformations in order, but write it with the innermost function first.\n",
        "\n",
        "#### It's Just Function Composition:\n",
        "\n",
        "* The code uses function composition, where one function's output is the input to the next. This is conceptually similar to:\n",
        "`f(g(x))\n",
        "`\n"
      ],
      "metadata": {
        "id": "QXo2mjsHbzDG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "next(model_0.parameters()).device"
      ],
      "metadata": {
        "id": "9b3YRciCYrJb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's replicate the model above using nn.Sequential\n",
        "model_0 = nn.Sequential(\n",
        "    nn.Linear(in_features=2,\n",
        "              out_features=5),\n",
        "    nn.Linear(in_features=5,\n",
        "              out_features=1)).to(device)\n",
        "\n",
        "model_0\n"
      ],
      "metadata": {
        "id": "okPtwW4TY_wJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_0.state_dict()"
      ],
      "metadata": {
        "id": "NKZMXPcmfAMy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Make preds *rmbr to use the inference mode\n",
        "with torch.inference_mode():\n",
        "    untrained_preds = model_0(X_test.to(device))\n",
        "print(f'Length of preds: {len(untrained_preds)}')\n",
        "print(f'Shape of preds: {untrained_preds.shape}')\n",
        "print(f'First 10 preds: {untrained_preds[:10]}')\n",
        "print(f'First 10 y_test: {y_test[:10]}')"
      ],
      "metadata": {
        "id": "CTSdV1G9hYRT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_test[:10], y_test[:10]"
      ],
      "metadata": {
        "id": "3ioaiiElitSA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Set-up loss function and optimizer\n",
        "\n",
        "Which loss and optimizer should we use?\n",
        "\n",
        "- Depends on the problem\n",
        "    - regression: MAE, MSE\n",
        "    - Classification: binary cross entropy or categorical cross entropy\n",
        "\n",
        "# Optimizer and Loss Functions in PyTorch\n",
        "\n",
        "However, the same optimizer function can often be used across different problem spaces.\n",
        "\n",
        "For example, the stochastic gradient descent optimizer (SGD, `torch.optim.SGD()`) can be used for a range of problems, and the same applies to the Adam optimizer (`torch.optim.Adam()`).\n",
        "\n",
        "| **Loss Function/Optimizer**               | **Problem Type**                   | **PyTorch Code**                        |\n",
        "|-------------------------------------------|-------------------------------------|-----------------------------------------|\n",
        "| **Stochastic Gradient Descent (SGD)**     | Classification, regression, many others. | `torch.optim.SGD()`                     |\n",
        "| **Adam Optimizer**                         | Classification, regression, many others. | `torch.optim.Adam()`                    |\n",
        "| **Binary Cross Entropy Loss**             | Binary classification               | `torch.nn.BCELossWithLogits` or `torch.nn.BCELoss` |\n",
        "| **Cross Entropy Loss**                    | Multi-class classification          | `torch.nn.CrossEntropyLoss`             |\n",
        "| **Mean Absolute Error (MAE) or L1 Loss**  | Regression                          | `torch.nn.L1Loss`                       |\n",
        "| **Mean Squared Error (MSE) or L2 Loss**   | Regression                          | `torch.nn.MSELoss`                      |\n"
      ],
      "metadata": {
        "id": "mglQZykoTc1f"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Make `loss_fn` and `optimizer`\n"
      ],
      "metadata": {
        "id": "3fX6kqtx_imD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Setup loss function\n",
        "\n",
        "loss_fn = nn.BCEWithLogitsLoss()\n",
        "\n",
        "# Setup optimizer\n",
        "optimizer = torch.optim.SGD(params=model_0.parameters(),\n",
        "                            lr=0.1)\n",
        "\n"
      ],
      "metadata": {
        "id": "tgWsGe7PjXaY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_0.state_dict()"
      ],
      "metadata": {
        "id": "g3gb0_JjaVoi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate accuracy- out of 100 examples what percentage does our model get right?\n",
        "def accuracy_fn(y_true, y_pred):\n",
        "    correct = torch.eq(y_true, y_pred).sum().item()\n",
        "    acc = (correct / len(y_pred)) * 100\n",
        "    return acc"
      ],
      "metadata": {
        "id": "5k0jeebHaaNx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Train Model\n",
        "\n",
        "1. Forward pass\n",
        "2. Calculate the loss\n",
        "3. Optimizer zero grad\n",
        "4. Loss backward (backpropagation)\n",
        "5. Optimizer step (gradient descent)"
      ],
      "metadata": {
        "id": "zqOSAp-NbgWR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Aslo we are going to perform the folowing:\n",
        "\n",
        "`going from raw logits -> prediction probabilities -> prediction labels`\n",
        "\n",
        "Our raw outputs from our model are logits. Convert into prediction probabilities  by passing them to some kind of activation function (e.g. sigmoid for binary classification or softmax for multiclass classificsation)\n",
        "\n",
        "Then we convert our models prediction probabilities to **prediction labels** by either rounding them or taking `argmax()`"
      ],
      "metadata": {
        "id": "cvzMb_jrcNay"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_0"
      ],
      "metadata": {
        "id": "JqgIrDKWbCj-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# View the first 5 outputs of the forward pass on the test data\n",
        "model_0.eval()\n",
        "with torch.inference_mode():\n",
        "    y_logits = model_0(X_test.to(device))[:5]\n",
        "y_logits"
      ],
      "metadata": {
        "id": "KkKHrPogdcTp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_test[:5]"
      ],
      "metadata": {
        "id": "ecRPonR4d3VS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Since we are performing a binary classification- use sigmoid activation function\n",
        "y_probs = torch.sigmoid(y_logits)\n",
        "y_probs"
      ],
      "metadata": {
        "id": "kUyUw95Ye_ze"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "For our predicition probability values, we need to perform a range-style rounding on them:\n",
        "* `y_pred_probs` >= 0.5 y = 1 (class 1)\n",
        "* `y_pred+probs` < 0.5 y=0 (class 0)"
      ],
      "metadata": {
        "id": "nAkWDKorfi8A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Find predicition probabilities\n",
        "y_preds = torch.round(y_probs)\n",
        "\n",
        "# In full (logits->pred_probs->pred_labels)\n",
        "y_pred_labels = torch.round(torch.sigmoid(model_0(X_test.to(device))[:5]))\n",
        "y_pred_labels\n",
        "\n",
        "# Check for equality\n",
        "print(torch.eq(y_preds.squeeze(), y_pred_labels.squeeze()))\n",
        "\n",
        "# Get rid of extra dimension\n",
        "y_preds.squeeze()"
      ],
      "metadata": {
        "id": "qlcVJBJqfRVL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_test[:5]"
      ],
      "metadata": {
        "id": "zV8KE3PLfZbS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Building a training and test loop"
      ],
      "metadata": {
        "id": "J6Bx3QO4iJym"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "device"
      ],
      "metadata": {
        "id": "omqwSLw-iHW0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "id": "SarnNGAdy6QO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "can also use a cuda manual seed"
      ],
      "metadata": {
        "id": "n8h9GXMuzPVf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "torch.manual_seed(42)"
      ],
      "metadata": {
        "id": "-AQHSxajy-Kz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "torch.cuda.manual_seed(42)"
      ],
      "metadata": {
        "id": "qyheycIOzMGt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Remember we are using BCEWITHLOGITSLOSS\n",
        "MORE NUMERICALLY STABLE (as per docs)\n",
        "\n",
        "## PyTorch BCEWithLogitsLoss\n",
        "\n",
        "In **PyTorch**, `BCEWithLogitsLoss` combines a Sigmoid layer and the Binary Cross-Entropy (BCE) loss in one single class. Mathematically, for each scalar input $( x_i)$ (the **logit**) and corresponding label $( y_i \\in \\{0,1\\} )$, the loss for one sample is given by:\n",
        "\n",
        "$$[\n",
        "\\ell_i = -\\Bigl[y_i \\cdot \\log\\bigl(\\sigma(x_i)\\bigr) \\;+\\; \\bigl(1 - y_i\\bigr)\\cdot \\log\\bigl(1 - \\sigma(x_i)\\bigr)\\Bigr],\n",
        "]$$\n",
        "\n",
        "where $( \\sigma(x_i) )$ is the Sigmoid function:\n",
        "\n",
        "$$[\n",
        "\\sigma(x_i) = \\frac{1}{1 + e^{-x_i}}.\n",
        "]$$\n",
        "\n",
        "If we have a mini-batch of \\( N \\) samples, the **mean** (or **sum**, depending on the `reduction` parameter) of all individual losses $( \\ell_i )$ is typically taken as the final scalar loss value:\n",
        "\n",
        "$$[\n",
        "\\text{BCEWithLogitsLoss} = \\frac{1}{N} \\sum_{i=1}^{N} \\ell_i.\n",
        "]$$\n",
        "\n",
        "### Optional Weights\n",
        "\n",
        "- **Weight:** In PyTorch, you can assign a per-sample weight $( w_i )$ to handle unbalanced data. This modifies the loss term to:\n",
        "\n",
        "  $$[\n",
        "  \\ell_i = -\\, w_i\\, \\Bigl[y_i \\cdot \\log\\bigl(\\sigma(x_i)\\bigr) + (1 - y_i)\\cdot \\log\\bigl(1 - \\sigma(x_i)\\bigr)\\Bigr].\n",
        "  ]$$\n",
        "\n",
        "- **Positional Weight (`pos_weight`)**: This is an additional multiplier for the positive targets, useful when you have *many* more negatives than positives. It modifies the loss term for $( y_i=1 )$. Specifically,\n",
        "\n",
        "$$\n",
        "\\ell_i = -\\Bigl[\\mathrm{pos\\_weight} \\cdot y_i \\cdot \\log(\\sigma(x_i)) + (1 - y_i) \\cdot \\log\\bigl(1 - \\sigma(x_i)\\bigr)\\Bigr].\n",
        "$$\n",
        "\n",
        "\n",
        "By accepting raw logits $( x_i )$ (i.e., values **before** the Sigmoid), `BCEWithLogitsLoss` is more numerically stable than applying a Sigmoid followed by a separate `BCELoss`.\n"
      ],
      "metadata": {
        "id": "Ne56_8d01pnz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Set number of epochs = 100\n",
        "epochs = 100\n",
        "\n",
        "# Put split data to target device\n",
        "X_train, y_train, X_test, y_test = X_train.to(device), y_train.to(device), X_test.to(device), y_test.to(device)\n",
        "\n",
        "# Build our training and avaluation loop\n",
        "for eopch in range(epochs):\n",
        "    #training\n",
        "    model_0.train()\n",
        "\n",
        "    # Forward pass- remember that squeeze removes an extra 1-dimension from a tensor\n",
        "    y_logits = model_0(X_train).squeeze()\n",
        "    y_pred = torch.round(torch.sigmoid(y_logits)) # turn logits -> pred probs -> pred labels\n",
        "\n",
        "    # Calculate loss/accuracy- rememeber function above\n",
        "    # loss = loss_fn(torch.sigmoid(y_logits), # nn.BCELoss expects prediction probabilities as input\n",
        "    #                y_train)\n",
        "    loss = loss_fn(y_logits, # nn.BCEWithLogitsLoss expects raw logits as input\n",
        "                   y_train)  # Note the order of the arguments here\n",
        "    acc = accuracy_fn(y_true=y_train,\n",
        "                      y_pred=y_pred)\n",
        "\n",
        "\n",
        "    # remember our accuracy function\n",
        "    acc = accuracy_fn(y_true=y_train,\n",
        "                      y_pred=y_pred)\n",
        "\n",
        "    # Optimizer zero grad\n",
        "    optimizer.zero_grad()\n",
        "\n",
        "    # Loss backward- backpropagation\n",
        "    loss.backward()\n",
        "\n",
        "    # Optimizer step (gradient descent)\n",
        "    optimizer.step()\n",
        "\n",
        "    # Testing\n",
        "    model_0.eval()\n",
        "    with torch.inference_mode():\n",
        "        #forward pass\n",
        "        test_logits = model_0(X_test).squeeze()\n",
        "        test_pred = torch.round(torch.sigmoid(test_logits))\n",
        "        # Calculate test loss/accuracy\n",
        "        test_loss = loss_fn(test_logits,\n",
        "                            y_test)\n",
        "        test_acc = accuracy_fn(y_true=y_test,\n",
        "                               y_pred=test_pred)\n",
        "\n",
        "    # Print out whats happening\n",
        "    if eopch % 10 == 0:\n",
        "        print(f\"Epoch: {eopch} | Loss: {loss:.5f}, Acc: {acc:.2f}% | Test Loss: {test_loss:.5f}, Test Acc: {test_acc:.2f}%\")\n",
        "\n"
      ],
      "metadata": {
        "id": "zUljnLzlzbKS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Our results are akin to flipping a coin. Not ideal"
      ],
      "metadata": {
        "id": "Ir3zuXLf--ol"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "circles.label.value_counts()"
      ],
      "metadata": {
        "id": "EGJyIhfYzgk7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Why is our model not learning?\n",
        "\n",
        "Let's visualize\n",
        "\n",
        "to do so, we'll import a function called `plot_decision_boundary`\n",
        "\n",
        "we note a very important website for our endevors in ML/DL: https://madewithml.com/\n",
        "\n",
        "Specifically a repo by `Goku Mohandas`:\n",
        "https://madewithml.com/courses/mlops/evaluation/\n",
        "\n",
        "here we will use `mrdbourke's` helper function for visualizing our results\n",
        "\n"
      ],
      "metadata": {
        "id": "xzmSx3Q0_Pyn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from pathlib import Path\n",
        "\n",
        "# 1. (Optional) Remove the existing (likely invalid) helper_functions.py\n",
        "# !rm helper_functions.py\n",
        "\n",
        "# 2. Use the *raw* GitHub URL\n",
        "url_to_download = \"https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/helper_functions.py\"\n",
        "\n",
        "if Path(\"helper_functions.py\").is_file():\n",
        "    print(\"helper_functions.py already exists, skipping download\")\n",
        "else:\n",
        "    print(\"Downloading helper_functions.py\")\n",
        "    request = requests.get(url_to_download)\n",
        "    with open(\"helper_functions.py\", \"wb\") as f:\n",
        "        f.write(request.content)\n"
      ],
      "metadata": {
        "id": "au6u9LaiI_P4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from helper_functions import plot_predictions, plot_decision_boundary\n"
      ],
      "metadata": {
        "id": "V5HMvV4uH7j0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot the decision boundary of the model\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plot_decision_boundary(model_0, X_train, y_train)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plot_decision_boundary(model_0, X_test, y_test)"
      ],
      "metadata": {
        "id": "DbeTxupyIJy2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### This shows us why our model is getting such poor results."
      ],
      "metadata": {
        "id": "qrpRYQLrKWdb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Improve our model\n",
        "* add more layers- give the model more chances to learn about the patterns in the data\n",
        "* Add more hidden units - go from 5 hidden units to 10 hidden units\n",
        "* fit for longer\n",
        "* Change activation function - we're using sigmoid at the moment (good for binary data)\n",
        "* Change the learning rate (warning vanishing/exploding gradients)\n",
        "* Change the loss function\n",
        "* Change the optimization function\n",
        "\n",
        "These options are all from our model's perspective b/c they relate to the form of our model, rather than the data\n",
        "\n",
        "Because these options are all values we can change within the model itself- they are called **hyperparameters**\n",
        "\n",
        "##### Below we will\n",
        "* Add more hidden units\n",
        "* Increase the number of layers 2 -> 3\n",
        "* Increase the number of epochs 100 -> 1000\n",
        "*Ideally, we would only change 1 at a time b/c we will not know which of these improved/degraded our model. We do this just to save time"
      ],
      "metadata": {
        "id": "TzlryuqQLQno"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CircleModelV1(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.layer_1 = nn.Linear(in_features=2,\n",
        "                                 out_features=10)\n",
        "        self.layer_2 = nn.Linear(in_features=10,\n",
        "                                 out_features=10)\n",
        "        self.layer_3 = nn.Linear(in_features=10,\n",
        "                                 out_features=1) # out just has 1 layer as it is binary choice\n",
        "    def forward(self, x):\n",
        "        # z = self.layer_1(x)\n",
        "        # z = self.layer_2(z)\n",
        "        # z = self.layer_3(z)\n",
        "        return self.layer_3(self.layer_2(self.layer_1(x))) # think f(g(x)) speeds up everything behind the scenes\n",
        "\n",
        "model_1 = CircleModelV1().to(device)\n",
        "model_1\n",
        "\n"
      ],
      "metadata": {
        "id": "3TNNVugFKOxU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the loss function\n",
        "loss_fn = nn.BCEWithLogitsLoss()\n",
        "\n",
        "# Create an optimizer\n",
        "optimizer = torch.optim.SGD(params=model_1.parameters(),\n",
        "                            lr=0.1)"
      ],
      "metadata": {
        "id": "UDH1VNFxQ2rz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Write a training and evaluation loop\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "epochs = 1000 #training for longer\n",
        "\n",
        "# Put data on target device\n",
        "X_train, y_train, X_test, y_test = X_train.to(device), y_train.to(device), X_test.to(device), y_test.to(device)\n",
        "\n",
        "for epoch in range(epochs):\n",
        "    model_1.train()\n",
        "    # forward pass\n",
        "    y_logits = model_1(X_train).squeeze()\n",
        "    y_pred = torch.round(torch.sigmoid(y_logits))\n",
        "\n",
        "    # Calculate loss/accuracy\n",
        "    loss = loss_fn(y_logits,\n",
        "                   y_train)\n",
        "    acc = accuracy_fn(y_true=y_train,\n",
        "                      y_pred=y_pred)\n",
        "\n",
        "    # Zero the gradients\n",
        "    optimizer.zero_grad()\n",
        "\n",
        "    # Loss backward\n",
        "    loss.backward()\n",
        "\n",
        "    # Optimizer step (gradient descent)\n",
        "    optimizer.step()\n",
        "\n",
        "    #Testing\n",
        "    model_1.eval()\n",
        "    with torch.inference_mode():\n",
        "        #Frward pass\n",
        "        test_logits = model_1(X_test).squeeze()\n",
        "        test_pred = torch.round(torch.sigmoid(test_logits))\n",
        "\n",
        "        # Calculate loss/accuracy\n",
        "        test_loss = loss_fn(test_logits,\n",
        "                            y_test)\n",
        "        test_acc = accuracy_fn(y_true=y_test,\n",
        "                               y_pred=test_pred)\n",
        "    # Print out whats happening\n",
        "    if epoch % 100 == 0:\n",
        "        print(f\"Epoch: {epoch} | Loss: {loss:.5f}, Acc: {acc:.2f}% | Test Loss: {test_loss:.5f}, Test Acc: {test_acc:.2f}%\")"
      ],
      "metadata": {
        "id": "uQZmx2HlRMwF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot decision boundary\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plot_decision_boundary(model_1, X_train, y_train)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plot_decision_boundary(model_1, X_test, y_test)"
      ],
      "metadata": {
        "id": "YJ4ylX9VULoq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Nope- coin flip"
      ],
      "metadata": {
        "id": "aV7rwv6uUdp7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "One ay to trouble-shoot a larger problem is to test out a smaller problem"
      ],
      "metadata": {
        "id": "JX9X3TsWVZ_p"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create some data (same as previous notebook)\n",
        "weight = 0.7\n",
        "bias = 0.3\n",
        "start = 0\n",
        "end = 1\n",
        "step = 0.01\n",
        "X_regression = torch.arange(start, end, step).unsqueeze(dim=1)\n",
        "y_regression = weight * X_regression + bias\n",
        "\n",
        "# Check the data\n",
        "print(X_regression[:5], y_regression[:5])"
      ],
      "metadata": {
        "id": "HpIiWc9dUbLe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_split = int(0.8 * len(X_regression))\n",
        "X_train_regression, y_train_regression = X_regression[:train_split], y_regression[:train_split]\n",
        "X_test_regression, y_test_regression = X_regression[train_split:], y_regression[train_split:]\n",
        "# Check lengths of each\n",
        "len(X_train_regression), len(X_test_regression), len(y_train_regression), len(y_test_regression)"
      ],
      "metadata": {
        "id": "YpRMP0iFWbfm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plot_predictions(train_data=X_train_regression,\n",
        "                 train_labels = y_train_regression,\n",
        "                 test_data = X_test_regression,\n",
        "                 test_labels = y_test_regression);"
      ],
      "metadata": {
        "id": "xGldHOQsXxTC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train_regression.shape, y_train_regression.shape"
      ],
      "metadata": {
        "id": "aah_praVY8dH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Our present model is set up to recieve 2 x-features, here we are only feeding in one feature."
      ],
      "metadata": {
        "id": "8z_F5tsZ03Jp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_train_regression[:10]"
      ],
      "metadata": {
        "id": "5AWnpc3VZmIX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "only one value as shown above"
      ],
      "metadata": {
        "id": "XN2r_T521N7O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Adjust `model_1` to fit a straight line- use `nn.Sequential`"
      ],
      "metadata": {
        "id": "m8m4MJhtaLih"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# same architecture as model_1 (but using nn.Sequential())\n",
        "model_2 = nn.Sequential(\n",
        "    nn.Linear(in_features=1,\n",
        "              out_features=10),\n",
        "    nn.Linear(in_features=10,\n",
        "              out_features=10),\n",
        "    nn.Linear(in_features=10,\n",
        "              out_features=1)\n",
        ").to(device)\n",
        "model_2"
      ],
      "metadata": {
        "id": "lRnhh6RVZ30I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Loss and optimizer\n",
        "loss_fn = nn.L1Loss() # MAE loss with regression data\n",
        "optimizer = torch.optim.SGD(model_2.parameters(),\n",
        "                            lr=0.01)"
      ],
      "metadata": {
        "id": "KC3R4VWPa_fT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the model\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "# Set the number of epochs\n",
        "epochs = 1000\n",
        "\n",
        "# Put the data on the target device\n",
        "X_train_regression, y_train_regression, X_test_regression, y_test_regression = X_train_regression.to(device), y_train_regression.to(device), X_test_regression.to(device), y_test_regression.to(device)\n",
        "\n",
        "# Training\n",
        "for epoch in range(epochs):\n",
        "    y_pred = model_2(X_train_regression)\n",
        "    loss = loss_fn(y_pred, y_train_regression)\n",
        "\n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    # Testing\n",
        "    model_2.eval()\n",
        "    with torch.inference_mode():\n",
        "        test_pred = model_2(X_test_regression)\n",
        "        test_loss = loss_fn(test_pred, y_test_regression)\n",
        "\n",
        "    # Print out what is happening- Note, there is no accuracy here because we\n",
        "    # are doing regression\n",
        "    if epoch % 100 == 0:\n",
        "        print(f\"Epoch: {epoch} | Loss: {loss:.5f} | Test Loss: {test_loss:.5f}\")"
      ],
      "metadata": {
        "id": "4-rbOtt72R5Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**OK, we see that model_2 can indeed learn as we see such small losses, particularly after we lowered the hyperparameter lr=0.1. -> lr = 0.01. This would lead us to think that model_2 is a valid approach**"
      ],
      "metadata": {
        "id": "H6B5aK2t5ELq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Turn on evaluation mode\n",
        "model_2.eval()\n",
        "\n",
        "# Create predictions (inference) with the model\n",
        "with torch.inference_mode():\n",
        "    y_preds = model_2(X_test_regression)\n",
        "\n",
        "# Plot data and predictions\n",
        "plot_predictions(train_data=X_train_regression,\n",
        "                 train_labels=y_train_regression,\n",
        "                 test_data=X_test_regression,\n",
        "                 test_labels=y_test_regression,\n",
        "                 predictions=y_preds)"
      ],
      "metadata": {
        "id": "FNUx1-aR4F0M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Wait, why is the the plot blank. We get the error `can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.`\n",
        "\n",
        "*explanation*: matplot lib references NumPy- which uses the cpu rather than the gpu!\n"
      ],
      "metadata": {
        "id": "N6TC-jnc7zSN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# write the correct code!\n",
        "model_2.eval()\n",
        "\n",
        "# Create predictions (inference) with the model\n",
        "with torch.inference_mode():\n",
        "    y_preds = model_2(X_test_regression)\n",
        "\n",
        "# Plot data and predictions\n",
        "plot_predictions(train_data=X_train_regression.cpu(),\n",
        "                 train_labels=y_train_regression.cpu(),\n",
        "                 test_data=X_test_regression.cpu(),\n",
        "                 test_labels=y_test_regression.cpu(),\n",
        "                 predictions=y_preds.cpu())"
      ],
      "metadata": {
        "id": "MJxH4QAI7sdo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "*Now the question becomes: is it the data our model can't learn on? Is it the circular nature of the data? Our model only comprises linear functions- which are all related to straight lines- is it the fact that our data has some non-linear characteristics?*\n",
        "\n",
        "The big reveal here is that we will have to employ non linear activation functions!\n",
        "\n",
        "Neural networks have the benifit of combining straight lines with non-linear lines. Thus, our model was hamstrung by giving it the power to only use straight lines in its calculations. Our data is is non-linear, thus the introduction of non-linear components will be the key to our success."
      ],
      "metadata": {
        "id": "rRolwcTo9H0x"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Recreating non-linear data (red and blue circles)"
      ],
      "metadata": {
        "id": "R-0enaaEAe2y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Make and plot data\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from sklearn.datasets import make_circles\n",
        "\n",
        "n_samples = 1000\n",
        "X, y = make_circles(n_samples,\n",
        "                    noise=0.03,\n",
        "                    random_state=42)\n",
        "\n",
        "plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu)\n",
        "\n"
      ],
      "metadata": {
        "id": "m3ucv8V9857Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert data to tensors and then to train and test splits\n",
        "import torch\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Turn data into tensors\n",
        "X = torch.from_numpy(X).type(torch.float)\n",
        "y = torch.from_numpy(y).type(torch.float)\n",
        "\n",
        "# Split data\n",
        "X_train, X_test, y_train, y_test = train_test_split(X,\n",
        "                                                    y,\n",
        "                                                    test_size=0.2,\n",
        "                                                    random_state=42)\n",
        "\n",
        "# Check the data\n",
        "X_train[:5], y_train[:5]"
      ],
      "metadata": {
        "id": "LJutR0agBdOF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Building a model with non-linearity\n",
        "\n",
        "### Non-Linear Activation Functions in PyTorch\n",
        "\n",
        "Activation functions introduce non-linearity into a neural network, allowing it to learn complex patterns in data. Below are some commonly used non-linear activation functions in PyTorch:\n",
        "\n",
        "1. **ReLU (Rectified Linear Unit)**:\n",
        "   - Formula: $( f(x) = \\max(0, x))$\n",
        "   - Introduces sparsity and avoids the vanishing gradient problem.\n",
        "   - Commonly used in hidden layers.\n",
        "\n",
        "2. **Sigmoid**:\n",
        "   - Formula: $( f(x) = \\frac{1}{1 + e^{-x}} )$\n",
        "   - Squashes input to a range between 0 and 1.\n",
        "   - Suitable for binary classification problems but may suffer from the vanishing gradient issue.\n",
        "\n",
        "3. **Tanh (Hyperbolic Tangent)**:\n",
        "   - Formula: $( f(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}} )$\n",
        "   - Squashes input to a range between -1 and 1.\n",
        "   - Zero-centered, which can help during optimization compared to Sigmoid.\n",
        "\n",
        "4. **Leaky ReLU**:\n",
        "   - Formula: $( f(x) = x ) if ( x > 0 ), else ( f(x) = \\alpha x )$ where $( \\alpha )$ is a small positive constant.\n",
        "   - Addresses the \"dying ReLU\" problem by allowing a small gradient for negative inputs.\n",
        "\n",
        "5. **Softmax**:\n",
        "   - Formula: $( f(x_i) = \\frac{e^{x_i}}{\\sum_{j} e^{x_j}} )$\n",
        "   - Converts outputs into probabilities, typically used in the final layer for multi-class classification.\n",
        "\n",
        "6. **ELU (Exponential Linear Unit)**:\n",
        "   - Formula:\n",
        "$$\n",
        "f(x) =\n",
        "\\begin{cases}\n",
        "x & \\text{if } x > 0 \\\\\n",
        "\\alpha (e^x - 1) & \\text{if } x \\leq 0\n",
        "\\end{cases}\n",
        "$$\n",
        "\n",
        "\n",
        "   - Similar to ReLU but smoothens the transition for negative inputs.\n",
        "\n",
        "7. **Swish**:\n",
        "   - Formula: $( f(x) = x \\cdot \\text{sigmoid}(x) )$\n",
        "   - A smooth, self-gated activation function known to improve performance on some deep learning tasks.\n",
        "\n",
        "#### Key Considerations:\n",
        "- **Choice of Activation Function**: Depends on the task and architecture. ReLU and its variants are widely used in hidden layers, while Softmax and Sigmoid are popular in output layers for classification tasks.\n",
        "- **Non-Linearity**: Activation functions enable neural networks to approximate complex, non-linear mappings.\n",
        "\n",
        "We could experiment with these activation functions to see their impact on the **make_circles** dataset, which is inherently non-linear!\n"
      ],
      "metadata": {
        "id": "y1LRFqYyFEMo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch import nn\n",
        "class CircleModeV2(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.layer_1 = nn.Linear(in_features=2, out_features=10)\n",
        "        self.layer_2 = nn.Linear(in_features=10, out_features=10)\n",
        "        self.layer_3 = nn.Linear(in_features=10, out_features=1)\n",
        "        self.relu = nn.ReLU() # This function is applied elelment-wise and is a non-linear act func\n",
        "    def forward(self, x):\n",
        "        # Where should we put our non-linear activation functions\n",
        "        return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x))))) # Remember chain rule\n",
        "\n",
        "model_3 = CircleModeV2().to(device)\n",
        "model_3"
      ],
      "metadata": {
        "id": "voIMGEl5CqL_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Why I think of the Chain Rule in Calculus\n",
        "\n",
        "the code snippet on the line 11 has a form that had confused me at first. One thing to remember is that it is conceptually similar to the **chain rule** in calculus! Here's why:\n",
        "\n",
        "---\n",
        "\n",
        "### The Chain Rule in Calculus\n",
        "The chain rule is used to compute the derivative of a composition of functions. If $( f(x) = g(h(x)) )$, then the derivative $( f'(x) )$ is:\n",
        "\n",
        "$$[\n",
        "f'(x) = g'(h(x)) \\cdot h'(x)\n",
        "]$$\n",
        "\n",
        "It works by applying the derivative of the outer function $( g )$, evaluated at the inner function $( h(x) )$, and multiplying it by the derivative of the inner function $( h )$.\n",
        "\n",
        "---\n",
        "\n",
        "### Thus when I see the code snippet below I think of that chain rule\n",
        "The code:\n",
        "```python\n",
        "return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))\n"
      ],
      "metadata": {
        "id": "Zzex6QgjLfZc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the loss function\n",
        "loss_fn = nn.BCEWithLogitsLoss()\n",
        "\n",
        "# Create an optimizer\n",
        "optimizer = torch.optim.SGD(params=model_3.parameters(),\n",
        "                            lr=0.01) # this hyperparameter will affect how long it takes for the model to complete"
      ],
      "metadata": {
        "id": "Jt0j8ukqL27I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Write a training and evaluation loop\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "epochs = 1000\n",
        "\n",
        "# Put data on target device\n",
        "X_train, y_train, X_test, y_test = X_train.to(device), y_train.to(device), X_test.to(device), y_test.to(device)\n",
        "\n",
        "# Loop tthroughh the data\n",
        "for epoch in range(epochs):\n",
        "    #Training\n",
        "    model_3.train()\n",
        "    # forward pass\n",
        "    y_logits = model_3(X_train).squeeze()\n",
        "    y_pred = torch.round(torch.sigmoid(y_logits))\n",
        "\n",
        "    # Calculate loss/accuracy\n",
        "    loss = loss_fn(y_logits,\n",
        "                   y_train)\n",
        "    acc = accuracy_fn(y_true=y_train,\n",
        "                      y_pred=y_pred)\n",
        "\n",
        "    # Zero the gradients\n",
        "    optimizer.zero_grad()\n",
        "\n",
        "    # Loss backward\n",
        "    loss.backward()\n",
        "\n",
        "    # Optimizer step (gradient descent)\n",
        "    optimizer.step()\n",
        "\n",
        "    #Testing\n",
        "    model_3.eval()\n",
        "    with torch.inference_mode():\n",
        "        #Frward pass\n",
        "        test_logits = model_3(X_test).squeeze()\n",
        "        test_pred = torch.round(torch.sigmoid(test_logits)) #logits -> prediction probabilities -> prediction labels\n",
        "\n",
        "        # Calculate loss/accuracy\n",
        "        test_loss = loss_fn(test_logits,\n",
        "                            y_test)\n",
        "        test_acc = accuracy_fn(y_true=y_test,\n",
        "                               y_pred=test_pred)\n",
        "    # Print out whats happening\n",
        "    if epoch % 100 == 0:\n",
        "        print(f\"Epoch: {epoch} | Loss: {loss:.4f}, Acc: {acc:.2f}% | Test Loss: {test_loss:.4f}, Test Acc: {test_acc:.2f}%\")"
      ],
      "metadata": {
        "id": "cV--5FZPNwgc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Visualize the model\n",
        "* non linear activation function\n",
        "* lr = 0.01 (my change)"
      ],
      "metadata": {
        "id": "RTFcGQUldtJl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_3.eval()\n",
        "with torch.inference_mode():\n",
        "    y_preds = torch.round(torch.sigmoid(model_3(X_test))).squeeze()\n",
        "y_preds[:10], y_test[:10]"
      ],
      "metadata": {
        "id": "L_ugxzV9b_2J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# plot decision Boundaries\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plot_decision_boundary(model_3, X_train, y_train)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plot_decision_boundary(model_3, X_test, y_test)"
      ],
      "metadata": {
        "id": "2Igj419CfczD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Test setups that may work\n",
        "\n",
        "### Increase Model Capacity\n",
        "* To handle the non-linear separability of the `make_circles` dataset and potential outlier effects from unscaled data, add more hidden units and layers:"
      ],
      "metadata": {
        "id": "7091z7U5yD0h"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Approach Overview**\n",
        "In this part of ther project, we're trying to get better results than we have previously. We train a binary classification neural network on the `make_circles` dataset, without scaling the features. The network is designed to handle the dataset's non-linear nature by including multiple hidden layers with ReLU activations and sufficient hidden units. The key components of the approach include:\n",
        "\n",
        "1. **Model Architecture**:\n",
        "   - A 3-layer neural network with ReLU activations is used.\n",
        "   - The output layer produces logits, which are raw predictions used by `BCEWithLogitsLoss` for numerical stability.\n",
        "\n",
        "2. **Loss Function**:\n",
        "   - `BCEWithLogitsLoss` is chosen as it combines sigmoid activation with binary cross-entropy loss, making it efficient for binary classification tasks.\n",
        "\n",
        "3. **Optimizer**:\n",
        "   - The Adam optimizer is used with a learning rate of `0.01` to enable efficient training and faster convergence.\n",
        "\n",
        "4. **Weight Initialization**:\n",
        "   - Xavier uniform initialization is applied to improve weight scaling, particularly with ReLU activations.\n",
        "\n",
        "5. **Training and Evaluation**:\n",
        "   - During training, the model computes logits, calculates loss, and updates weights using backpropagation.\n",
        "   - In evaluation mode, test predictions are generated, and the test loss and accuracy are reported.\n",
        "\n",
        "6. **Key Hyperparameters**:\n",
        "   - The model is trained for `1500` epochs, and results are printed every `100` epochs.\n",
        "\n",
        "This approach ensures robust training and testing, resulting in high accuracy for binary classification tasks on the `make_circles` dataset.\n"
      ],
      "metadata": {
        "id": "9t-W3W0GDNMk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the neural network model for binary classification\n",
        "class CircleModeV2(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        # Input layer to the first hidden layer (2 input features, 64 hidden units)\n",
        "        self.layer_1 = nn.Linear(2, 64)\n",
        "        self.relu_1 = nn.ReLU()  # Apply ReLU activation for non-linearity\n",
        "\n",
        "        # Second hidden layer (64 hidden units)\n",
        "        self.layer_2 = nn.Linear(64, 64)\n",
        "        self.relu_2 = nn.ReLU()  # Apply ReLU activation for non-linearity\n",
        "\n",
        "        # Output layer (1 unit for binary classification logits)\n",
        "        self.layer_3 = nn.Linear(64, 1)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # Pass the input through the layers with activations\n",
        "        x = self.relu_1(self.layer_1(x))\n",
        "        x = self.relu_2(self.layer_2(x))\n",
        "        return self.layer_3(x)  # Return raw logits for use with BCEWithLogitsLoss\n"
      ],
      "metadata": {
        "id": "Uty2DLZpCXfX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Instantiate the model and move it to the appropriate device (GPU)\n",
        "model_3a = CircleModeV2().to(device)\n",
        "\n",
        "# Ensure training and testing data are also on the same device\n",
        "X_train, y_train = X_train.to(device), y_train.to(device)\n",
        "X_test, y_test = X_test.to(device), y_test.to(device)\n"
      ],
      "metadata": {
        "id": "WSh6ijBWCf_w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(torch.unique(y_train))  # Should output: tensor([0, 1])\n"
      ],
      "metadata": {
        "id": "SqZli1FuACwT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training Setup"
      ],
      "metadata": {
        "id": "sP8mlLI3yOnN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ☠ Beware!\n",
        "\n",
        "of a a mismatch in the shapes of `y_logits` and `y_train`. The `BCEWithLogitsLoss` function requires the shapes of both the logits (`y_logits`) and targets (`y_train`) to match exactly.\n",
        "\n",
        "In our code, `y_logits` has the shape [800], while `y_train` has the shape [800, 1]."
      ],
      "metadata": {
        "id": "p8Wpk_JN1_9W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Reinitialize Weights"
      ],
      "metadata": {
        "id": "XZqSr_JRA2M6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to initialize weights for the linear layers\n",
        "def initialize_weights(m):\n",
        "    if isinstance(m, nn.Linear):\n",
        "        # Xavier initialization for weights (good for layers with ReLU activations)\n",
        "        nn.init.xavier_uniform_(m.weight)\n",
        "        # Set biases to zero\n",
        "        nn.init.zeros_(m.bias)\n",
        "\n",
        "# Apply the weight initialization to all layers of the model\n",
        "model_3a.apply(initialize_weights)\n"
      ],
      "metadata": {
        "id": "KKbfNfa0C38i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the Binary Cross-Entropy Loss with Logits\n",
        "# This loss function is designed for binary classification and expects raw logits\n",
        "loss_fn = nn.BCEWithLogitsLoss()\n",
        "\n",
        "# Use the Adam optimizer with a learning rate of 0.01 for efficient training\n",
        "# Adam dynamically adjusts learning rates for each parameter\n",
        "optimizer = torch.optim.Adam(params=model_3a.parameters(), lr=0.01)\n"
      ],
      "metadata": {
        "id": "b5bc1X0WCpcT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Set manual seeds for reproducibility\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "# Define the number of epochs for training\n",
        "epochs = 1500\n",
        "\n",
        "for epoch in range(epochs):\n",
        "    # Set the model to training mode\n",
        "    model_3a.train()\n",
        "\n",
        "    # Perform a forward pass to calculate logits\n",
        "    y_logits = model_3a(X_train).squeeze()  # Squeeze to ensure dimensions match\n",
        "\n",
        "    # Calculate the loss using BCEWithLogitsLoss\n",
        "    loss = loss_fn(y_logits, y_train.squeeze())\n",
        "\n",
        "    # Zero gradients to prevent accumulation\n",
        "    optimizer.zero_grad()\n",
        "\n",
        "    # Backpropagate the loss to compute gradients\n",
        "    loss.backward()\n",
        "\n",
        "    # Update model weights using the optimizer\n",
        "    optimizer.step()\n",
        "\n",
        "    # Set the model to evaluation mode for testing\n",
        "    model_3a.eval()\n",
        "    with torch.no_grad():  # Disable gradient computation for efficiency\n",
        "        # Forward pass for the test data\n",
        "        test_logits = model_3a(X_test).squeeze()  # Logits for test data\n",
        "\n",
        "        # Calculate test loss\n",
        "        test_loss = loss_fn(test_logits, y_test.squeeze())\n",
        "\n",
        "        # Convert logits to probabilities and round to binary predictions\n",
        "        test_pred = torch.round(torch.sigmoid(test_logits))\n",
        "\n",
        "        # Calculate test accuracy\n",
        "        test_acc = (test_pred == y_test.squeeze()).float().mean().item() * 100\n",
        "\n",
        "    # Print results every 100 epochs\n",
        "    if epoch % 100 == 0:\n",
        "        print(f\"Epoch {epoch}: Loss = {loss:.4f}, Test Loss = {test_loss:.4f}, Test Acc = {test_acc:.2f}%\")\n"
      ],
      "metadata": {
        "id": "ebXr4foLDCkc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_3a.eval()\n",
        "with torch.inference_mode():\n",
        "    y_preds = torch.round(torch.sigmoid(model_3a(X_test))).squeeze()\n",
        "y_preds[:10], y_test[:10]"
      ],
      "metadata": {
        "id": "f1IPQfm-vIrb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# plot decision Boundaries\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plot_decision_boundary(model_3a, X_train, y_train)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plot_decision_boundary(model_3a, X_test, y_test)"
      ],
      "metadata": {
        "id": "_2_NK5mT3Ur7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Here we have successfully isolated the red from the blue circles"
      ],
      "metadata": {
        "id": "5tF9z0osEU0A"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Replicate non-linear activation functions\n",
        "\n",
        "Neural networks, rather than us telling the model what to learn, we give it the tools to discover patterns in the data and it tries to figure out those patterns on its own\n",
        "\n",
        "These tools are linear and non-linear functions."
      ],
      "metadata": {
        "id": "S3MUBCMegwVP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor\n",
        "A = torch.arange(-10, 10, 1, dtype = torch.float32)\n",
        "A.dtype\n"
      ],
      "metadata": {
        "id": "L0-V2kHyfxm1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize the tensor\n",
        "plt.plot(A);"
      ],
      "metadata": {
        "id": "MUfxmG05h3F7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(torch.relu(A));"
      ],
      "metadata": {
        "id": "u8R5LxqpiKOz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def relu(x: torch.tensor) -> torch.Tensor:\n",
        "    return torch.maximum(torch.tensor(0), x) # inputs must be tensors\n",
        "relu(A)"
      ],
      "metadata": {
        "id": "it_GmHf4ih-o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(relu(A));"
      ],
      "metadata": {
        "id": "KUiUkb_WjWg6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Now lets do the same for sigmoid\n",
        "def sigmoid(x: torch.tensor) -> torch.Tensor:\n",
        "    return 1 / (1 + torch.exp(-x))\n",
        "sigmoid(A)"
      ],
      "metadata": {
        "id": "lxLlIYHUjnMf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(sigmoid(A));"
      ],
      "metadata": {
        "id": "B_ulMWYqj84R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Challenge** Improve this model"
      ],
      "metadata": {
        "id": "oKRb39IaevOW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Building a multi-class classification model using PyTorch"
      ],
      "metadata": {
        "id": "CNU_U4CmaWNJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import dependencies\n",
        "import torch\n",
        "from torch import nn\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.datasets import make_blobs\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Set Hyperparamaeters\n",
        "NUM_CLASSES = 4\n",
        "NUM_FEATURES = 2\n",
        "RANDOM_SEED = 42\n",
        "HIDDEN_UNITS = 8\n",
        "\n",
        "X_blob, y_blob = make_blobs(n_samples=1000,\n",
        "                            n_features=NUM_FEATURES,\n",
        "                            centers=NUM_CLASSES,\n",
        "                            cluster_std=1.5, # Creates noise in the data\n",
        "                            random_state=RANDOM_SEED)\n",
        "\n",
        "# Turn data into tensors\n",
        "X_blob = torch.from_numpy(X_blob).type(torch.float)\n",
        "y_blob = torch.from_numpy(y_blob).type(torch.float)\n",
        "\n",
        "# Split into training and test\n",
        "X_blob_train, X_blob_test, y_blob_train, y_blob_test = train_test_split(\n",
        "                                                                        X_blob,\n",
        "                                                                        y_blob,\n",
        "                                                                        test_size=0.2,\n",
        "                                                                        random_state=RANDOM_SEED\n",
        ")\n",
        "\n",
        "# Plot the data\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.scatter(X_blob[:, 0], X_blob[:, 1], c=y_blob, cmap=plt.cm.RdYlBu);"
      ],
      "metadata": {
        "id": "x5uQJazUc0GD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create device agnostic code\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "device"
      ],
      "metadata": {
        "id": "SGpCe2T4kBUz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class BlobModel(nn.Module):\n",
        "    def __init__(self, input_features, output_features, hidden_units=8):\n",
        "        \"\"\"\n",
        "        Initializes a multi-class classification model.\n",
        "        Args:\n",
        "        input_features (int): Number of input features to the model.\n",
        "        output_features (int): Number of output features of the model.\n",
        "        hidden_units (int): Number of hidden units between layers, default 8.\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "        self.linear_layer_stack = nn.Sequential(\n",
        "            nn.Linear(in_features=input_features, out_features=hidden_units),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(in_features=hidden_units, out_features=hidden_units),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(in_features=hidden_units, out_features=output_features)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):  # Correctly indented at the class level- WATCH INDENTATION\n",
        "        return self.linear_layer_stack(x)\n",
        "\n",
        "# Create instance of the BlobModel and send to target device\n",
        "model_4 = BlobModel(\n",
        "    input_features=NUM_FEATURES,\n",
        "    output_features=NUM_CLASSES,\n",
        "    hidden_units=HIDDEN_UNITS\n",
        ").to(device)\n",
        "\n",
        "model_4  # Display the model architecture\n"
      ],
      "metadata": {
        "id": "oxKMBAL7pDZ6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_blob_train.shape, y_blob_train.shape, X_blob_test.shape, y_blob_test.shape"
      ],
      "metadata": {
        "id": "NRqrbQjVlxxV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "torch.unique(y_blob_train)"
      ],
      "metadata": {
        "id": "FlJqJUxgl8K_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create loss and optimizer- with multiclass we use cross entropy loss\n",
        "# Note: we have a balanced training set\n",
        "loss_fn = nn.CrossEntropyLoss() # loss function measures how wrong our model our model's predictions are\n",
        "optimizer = torch.optim.SGD(params=model_4.parameters(), # optimizer updates our model parameter's to try to reduce the loss\n",
        "                            lr=0.1)\n"
      ],
      "metadata": {
        "id": "joY-GkuymQUM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Getting raw ourputs of our model (i.e. logits)\n",
        "# Note: our data is on the cpu\n",
        "model_4.eval()\n",
        "with torch.inference_mode():\n",
        "    y_logits = model_4(X_blob_test.to(device))\n",
        "\n",
        "y_logits[:10]\n"
      ],
      "metadata": {
        "id": "y-4U0Z0ykqo8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_blob_test[:10]"
      ],
      "metadata": {
        "id": "SpQEJ2YvlDZS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### NOTE: we need to get y_pred shown above into the form of y_blob_test shown directly above this cell\n",
        "\n",
        "* each member of y_preds has 4 features - these are the logits\n",
        "* **In order to evaluate and train and test our model, we need to convert our models's output (logits) to prediction probabilities and then to prediction labels.**\n",
        "\n",
        "`logits` (raw outputs of model) -> `prediction probabilities` (use `torch.softmax()`) -> `prediction labels` (take the `argmax` of the prediction probabilities)  "
      ],
      "metadata": {
        "id": "l5ZC1dEKplVp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert our models logit outputs to prediction probabilities\n",
        "y_pred_probs = torch.softmax(y_logits, dim=1) # we want them accross the first dimension\n",
        "print(y_logits[:5])\n",
        "print(y_pred_probs[:5])"
      ],
      "metadata": {
        "id": "7w5lMNJhpfjf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_pred_probs[0]"
      ],
      "metadata": {
        "id": "WW9ERSB_sV3v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "torch.sum(y_pred_probs[0])"
      ],
      "metadata": {
        "id": "5nCVwRnFs0DZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**note, using the Softmax function, the sum of y_pred_probs sums up to 1**"
      ],
      "metadata": {
        "id": "i-gCEp9is_lj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "torch.max(y_pred_probs[0])"
      ],
      "metadata": {
        "id": "74VdmzWQs5_0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### So we we look at our data:\n",
        "\n",
        "[-0.3817,  0.2051,  0.1333, -0.9696]\n",
        "    \n",
        "\n",
        "`-0.3817`-> prob that this is class 0\n",
        "\n",
        "`0.2051` -> prob that is class 1\n",
        "\n",
        "`0.1333` -> prob that this is class 2 etc.."
      ],
      "metadata": {
        "id": "nbRTg5m4tbOz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### convert our models prediction probabilities to prediction labels - done using `argmax()`--> finds the index of this argmax"
      ],
      "metadata": {
        "id": "ie2RF61kvucN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_preds = torch.argmax(y_pred_probs, dim=1)\n",
        "y_preds"
      ],
      "metadata": {
        "id": "-8kYINM7tROq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_blob_test"
      ],
      "metadata": {
        "id": "vXjlTAuowD2V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Ideally these two blocks of numbers would match up but since we are using random numbers, it is of little surprise that they don't match since no training has taken place**"
      ],
      "metadata": {
        "id": "dkq8VEmUwabu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Now we will create our training and testing loop\n",
        "**BE CAREFUL of data types here or else errors will occur**\n",
        "\n",
        "An error is easily generated because `y_blob_train` is of type Float, but the CrossEntropyLoss function in PyTorch expects the target (labels) to be of type Long. The issue is likely that your labels (y_blob_train) are floating-point numbers, but they should be integers representing class indices.\n",
        "\n",
        "```\n",
        "# Ensure labels are of type Long\n",
        "loss = loss_fn(y_logits, y_blob_train.long())\n",
        "\n",
        "```\n",
        "**ALSO**\n",
        "`y_blob_test` is still of type Float when passed to the `CrossEntropyLoss` function. As mentioned earlier, the `CrossEntropyLoss` function requires the target labels (`y_blob_test`) to be of type `Long`. Additionally, the `test_logits` tensor might need to have the correct shape for the loss function to work.\n",
        "\n",
        "#### Solution:\n",
        "\n",
        "1. Ensure `y_blob_test` is of type `torch.long`: Convert y_blob_test to torch.long before passing it to the loss function.\n",
        "`test_loss = loss_fn(test_logits, y_blob_test.long())`\n",
        "2. Check the shape of `test_logits`: The shape of `test_logits` must be (`batch_size`, `num_classes`) for `CrossEntropyLoss`. If `test_logits` is squeezed improperly or has an incorrect shape, it can cause errors.\n",
        "\n",
        "Remove `.squeeze()` from `test_logits` unless you are absolutely sure the dimensions are correct after squeezing.\n",
        "`test_logits = model_4(X_blob_test)  # Don't squeeze\n",
        "`\n",
        "\n",
        "#### here is the correct validation loop:\n",
        "```\n",
        "model_4.eval()\n",
        "with torch.inference_mode():\n",
        "    test_logits = model_4(X_blob_test)  # Raw logits without squeezing\n",
        "    test_pred = torch.softmax(test_logits, dim=1).argmax(dim=1)  # Get predictions\n",
        "    # Ensure y_blob_test is of type Long\n",
        "    test_loss = loss_fn(test_logits, y_blob_test.long())  \n",
        "    test_acc = accuracy_fn(y_true=y_blob_test, y_pred=test_pred)\n",
        "\n",
        "```\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "HlCLJr39xk-B"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Principles to Avoid Runtime Errors in PyTorch Training Loops\n",
        "\n",
        "When building and training models in PyTorch, it’s important to follow these principles to prevent common runtime errors:\n",
        "\n",
        "---\n",
        "\n",
        "## 1. **Ensure Data Type Compatibility**\n",
        "- **Input Data (`X`)**:\n",
        "  - Must be of type `torch.float32` (default for `torch.Tensor`).\n",
        "  - Ensure that the device matches your model's device (e.g., `X.to(device)`).\n",
        "\n",
        "- **Target Data (`y`)**:\n",
        "  - Must be of type `torch.long` when using classification loss functions like `CrossEntropyLoss`.\n",
        "  - Convert using `.long()`: `y.long()`.\n",
        "\n",
        "---\n",
        "\n",
        "## 2. **Understand `CrossEntropyLoss` Requirements**\n",
        "- The `CrossEntropyLoss` function expects:\n",
        "  - **Logits**: Raw model outputs of shape `(batch_size, num_classes)` with no activation applied (e.g., no `softmax`).\n",
        "  - **Targets**: Ground truth labels of shape `(batch_size,)` with integer class indices (type `torch.long`).\n",
        "\n",
        "- **Avoid Applying `softmax`**:\n",
        "  - `CrossEntropyLoss` includes `log_softmax` internally, so applying `softmax` beforehand is unnecessary and may cause incorrect behavior.\n",
        "\n",
        "---\n",
        "\n",
        "## 3. **Shape Consistency**\n",
        "- Ensure the model's output (`logits`) and target labels (`y`) are compatible:\n",
        "  - **Logits**: `(batch_size, num_classes)`\n",
        "  - **Targets**: `(batch_size,)`\n",
        "- **Avoid Unnecessary Squeezing**:\n",
        "  - Do not use `.squeeze()` unless you are certain of its effect on tensor dimensions.\n",
        "  - Print tensor shapes using `.shape` for debugging.\n",
        "\n",
        "---\n",
        "\n",
        "## 4. **Device Management**\n",
        "- Always ensure tensors are on the same device as the model (`cpu` or `cuda`):\n",
        "  ```python\n",
        "  X = X.to(device)\n",
        "  y = y.to(device)\n"
      ],
      "metadata": {
        "id": "29b2tcmS9Q0G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "# Set the number of epochs\n",
        "epochs = 100\n",
        "\n",
        "# Put data to target device\n",
        "X_blob_train, y_blob_train, X_blob_test, y_blob_test  = X_blob_train.to(device), y_blob_train.to(device), X_blob_test.to(device), y_blob_test.to(device)\n",
        "\n",
        "# Loop through data\n",
        "for epoch in range(epochs):\n",
        "    model_4.train()\n",
        "    # Forward pass\n",
        "    y_logits = model_4(X_blob_train).squeeze()\n",
        "    y_pred = torch.softmax(y_logits, dim=1).argmax(dim=1)\n",
        "    # Calculate loss\n",
        "    loss = loss_fn(y_logits, y_blob_train.long())\n",
        "    acc = accuracy_fn(y_true=y_blob_train,\n",
        "                      y_pred=y_pred)\n",
        "\n",
        "    optimizer.zero_grad()\n",
        "\n",
        "    # Loss backward\n",
        "    loss.backward()\n",
        "\n",
        "    # Optimizer step\n",
        "    optimizer.step() # update paramters in our model\n",
        "\n",
        "    # testing\n",
        "    model_4.eval()\n",
        "    with torch.inference_mode():\n",
        "        test_logits = model_4(X_blob_test)  # Raw logits without squeezing\n",
        "        test_pred = torch.softmax(test_logits, dim=1).argmax(dim=1)  # Get predictions\n",
        "        # Ensure y_blob_test is of type Long\n",
        "        test_loss = loss_fn(test_logits, y_blob_test.long())\n",
        "        test_acc = accuracy_fn(y_true=y_blob_test, y_pred=test_pred)\n",
        "\n",
        "    # Print out what is happening\n",
        "    if epoch % 10 == 0:\n",
        "        print(f\"Epoch: {epoch} | Loss: {loss:.4f}, Acc: {acc:.2f}% | Test Loss: {test_loss:.4f}, Test Acc: {test_acc:.2f}%\")\n",
        "\n"
      ],
      "metadata": {
        "id": "ahMQEyDjwUjl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Wow! Excellent results!!!!"
      ],
      "metadata": {
        "id": "JPHV3DOC_bj5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate - Make predictions\n",
        "model_4.eval()\n",
        "with torch.inference_mode():\n",
        "    y_logits = model_4(X_blob_test).squeeze() # remember - logits are the raw output of our model\n",
        "y_logits[:10]"
      ],
      "metadata": {
        "id": "3RefjgAr2HMI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Go from logits -> prediction probabilities\n",
        "y_pred_probs = torch.softmax(y_logits, dim=1)\n",
        "y_pred_probs[:10]"
      ],
      "metadata": {
        "id": "cqvNR9_1AHPw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "y_blob_test"
      ],
      "metadata": {
        "id": "6ZRRNrk6Ae5c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**So we are not apples to apples yet wrt our data-types**"
      ],
      "metadata": {
        "id": "eAjFQSP_Ar0D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Go from pred_probs -> pred_labels\n",
        "y_preds = torch.argmax(y_pred_probs, dim=1)\n",
        "y_preds"
      ],
      "metadata": {
        "id": "1Fe2VcuvAoKk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plot_decision_boundary(model_4, X_blob_train, y_blob_train)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plot_decision_boundary(model_4, X_blob_test, y_blob_test)\n"
      ],
      "metadata": {
        "id": "gwi5nrLlBMlF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### We have successfully separated our data almost as best we could\n",
        "**Question**\n",
        "* Could we have analyzed our data without the help of non-linear activation functions?\n",
        "* let's comment out our ReLU activation functions"
      ],
      "metadata": {
        "id": "mRya3FnvB441"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class BlobModel(nn.Module):\n",
        "    def __init__(self, input_features, output_features, hidden_units=8):\n",
        "        \"\"\"\n",
        "        Initializes a multi-class classification model.\n",
        "        Args:\n",
        "        input_features (int): Number of input features to the model.\n",
        "        output_features (int): Number of output features of the model.\n",
        "        hidden_units (int): Number of hidden units between layers, default 8.\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "        self.linear_layer_stack = nn.Sequential(\n",
        "            nn.Linear(in_features=input_features, out_features=hidden_units),\n",
        "            # nn.ReLU(),\n",
        "            nn.Linear(in_features=hidden_units, out_features=hidden_units),\n",
        "            # nn.ReLU(),\n",
        "            nn.Linear(in_features=hidden_units, out_features=output_features)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):  # Correctly indented at the class level- WATCH INDENTATION\n",
        "        return self.linear_layer_stack(x)\n",
        "\n",
        "# Create instance of the BlobModel and send to target device\n",
        "model_4a = BlobModel(\n",
        "    input_features=NUM_FEATURES,\n",
        "    output_features=NUM_CLASSES,\n",
        "    hidden_units=HIDDEN_UNITS\n",
        ").to(device)\n",
        "\n",
        "model_4a  # Display the model architecture\n"
      ],
      "metadata": {
        "id": "Se_cdOQhB0g1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create loss and optimizer- with multiclass we use cross entropy loss\n",
        "# Note: we have a balanced training set\n",
        "loss_fn = nn.CrossEntropyLoss() # loss function measures how wrong our model our model's predictions are\n",
        "optimizer = torch.optim.SGD(params=model_4a.parameters(), # optimizer updates our model parameter's to try to reduce the loss\n",
        "                            lr=0.1)\n"
      ],
      "metadata": {
        "id": "HKo7jONODGvb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Getting raw ourputs of our model (i.e. logits)\n",
        "# Note: our data is on the cpu\n",
        "model_4a.eval()\n",
        "with torch.inference_mode():\n",
        "    y_logits = model_4a(X_blob_test.to(device))\n",
        "\n",
        "y_logits[:10]"
      ],
      "metadata": {
        "id": "GlTJZiWiDIZz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert our models logit outputs to prediction probabilities\n",
        "y_pred_probs = torch.softmax(y_logits, dim=1) # we want them accross the first dimension\n",
        "print(y_logits[:5])\n",
        "print(y_pred_probs[:5])"
      ],
      "metadata": {
        "id": "W0Yy603HDQXm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "# Set the number of epochs\n",
        "epochs = 100\n",
        "\n",
        "# Put data to target device\n",
        "X_blob_train, y_blob_train, X_blob_test, y_blob_test  = X_blob_train.to(device), y_blob_train.to(device), X_blob_test.to(device), y_blob_test.to(device)\n",
        "\n",
        "# Loop through data\n",
        "for epoch in range(epochs):\n",
        "    model_4a.train()\n",
        "    # Forward pass\n",
        "    y_logits = model_4a(X_blob_train).squeeze()\n",
        "    y_pred = torch.softmax(y_logits, dim=1).argmax(dim=1)\n",
        "    # Calculate loss\n",
        "    loss = loss_fn(y_logits, y_blob_train.long())\n",
        "    acc = accuracy_fn(y_true=y_blob_train,\n",
        "                      y_pred=y_pred)\n",
        "\n",
        "    optimizer.zero_grad()\n",
        "\n",
        "    # Loss backward\n",
        "    loss.backward()\n",
        "\n",
        "    # Optimizer step\n",
        "    optimizer.step() # update paramters in our model\n",
        "\n",
        "    # testing\n",
        "    model_4a.eval()\n",
        "    with torch.inference_mode():\n",
        "        test_logits = model_4a(X_blob_test)  # Raw logits without squeezing\n",
        "        test_pred = torch.softmax(test_logits, dim=1).argmax(dim=1)  # Get predictions\n",
        "        # Ensure y_blob_test is of type Long\n",
        "        test_loss = loss_fn(test_logits, y_blob_test.long())\n",
        "        test_acc = accuracy_fn(y_true=y_blob_test, y_pred=test_pred)\n",
        "\n",
        "    # Print out what is happening\n",
        "    if epoch % 10 == 0:\n",
        "        print(f\"Epoch: {epoch} | Loss: {loss:.4f}, Acc: {acc:.2f}% | Test Loss: {test_loss:.4f}, Test Acc: {test_acc:.2f}%\")\n"
      ],
      "metadata": {
        "id": "zc_FC5h7DtNE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate - Make predictions\n",
        "model_4a.eval()\n",
        "with torch.inference_mode():\n",
        "    y_logits = model_4a(X_blob_test).squeeze() # remember - logits are the raw output of our model\n",
        "y_logits[:10]"
      ],
      "metadata": {
        "id": "1RT_G3P7EYbN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Go from logits -> prediction probabilities\n",
        "y_pred_probs = torch.softmax(y_logits, dim=1)\n",
        "y_pred_probs[:10]"
      ],
      "metadata": {
        "id": "pUQENKYpEmCF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Go from pred_probs -> pred_labels\n",
        "y_preds = torch.argmax(y_pred_probs, dim=1)\n",
        "y_preds"
      ],
      "metadata": {
        "id": "4rCjVyfdEwFi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plot_decision_boundary(model_4a, X_blob_train, y_blob_train)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plot_decision_boundary(model_4a, X_blob_test, y_blob_test)\n"
      ],
      "metadata": {
        "id": "PYL8EnzUE3bJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### The answer is YES!!!!"
      ],
      "metadata": {
        "id": "El2SPGfrFC2N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_4a"
      ],
      "metadata": {
        "id": "-vzslpgpFBXy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## A few more classification metrics..(to evaluate classification model)\n",
        "* Accuracy- out of 100 samples, how many does our model get correct? (Good to use when you have balanced classes\n",
        "* Precision\n",
        "* Recall\n",
        "* F1-score\n",
        "* Confusion matrix\n",
        "* Classification report"
      ],
      "metadata": {
        "id": "t9s7LPcJFu4I"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Metrics Table\n",
        "\n",
        "| **Metric Name**         | **Metric Formula**                                                         | **PyTorch Code**                                                                                                                                          | **When to Use**                                                                 |\n",
        "|--------------------------|---------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|\n",
        "| **Precision**            | $ \\text{Precision} = \\frac{\\text{TP}}{\\text{TP} + \\text{FP}} $          | `precision = tp / (tp + fp + 1e-10)`                                                                                | Use when minimizing false positives is important (e.g., spam detection).        |\n",
        "| **Recall**               | $ \\text{Recall} = \\frac{\\text{TP}}{\\text{TP} + \\text{FN}} $             | `recall = tp / (tp + fn + 1e-10)`                                                                                  | Use when minimizing false negatives is important (e.g., disease diagnosis).     |\n",
        "| **F1-Score**             | $ \\text{F1} = 2 \\cdot \\frac{\\text{Precision} \\cdot \\text{Recall}}{\\text{Precision} + \\text{Recall}} $ | `f1 = 2 * (precision * recall) / (precision + recall + 1e-10)`                                                     | Use when balancing precision and recall is important.                           |\n",
        "| **Accuracy**             | $ \\text{Accuracy} = \\frac{\\text{TP} + \\text{TN}}{\\text{TP} + \\text{TN} + \\text{FP} + \\text{FN}} $ | `accuracy = torchmetrics.Accuracy()(predictions, targets)` <br>`# OR` <br>`accuracy = (tp + tn) / (tp + tn + fp + fn + 1e-10)`                           | Use when evaluating overall correctness of predictions.                         |\n",
        "| **Confusion Matrix**     | N/A                                                                       | `conf_matrix = torch.zeros(num_classes, num_classes)` <br>`for t, p in zip(targets, predictions): conf_matrix[t.long(), p.long()] += 1`                  | Use for a detailed view of model performance for each class.                    |\n",
        "| **Classification Report**| N/A                                                                       | `from sklearn.metrics import classification_report` <br>`report = classification_report(y_true, y_pred, target_names=class_names)`                      | Use for a comprehensive summary of precision, recall, F1-score, and support.    |\n",
        "\n",
        "---\n",
        "\n",
        "### Notes:\n",
        "1. **TP (True Positives)**: Correctly predicted positive samples.\n",
        "2. **FP (False Positives)**: Incorrectly predicted positive samples.\n",
        "3. **FN (False Negatives)**: Positive samples incorrectly predicted as negative.\n",
        "4. **TN (True Negatives)**: Correctly predicted negative samples.\n",
        "5. **num_classes**: Total number of classes in the classification task.\n",
        "6. **1e-10**: Small value added *to avoid division by zero.*\n",
        "\n",
        "---\n",
        "\n",
        "## Precision-Recall Relationship\n",
        "\n",
        "### Precision and Recall:\n",
        "- **Precision** focuses on the proportion of correct positive predictions out of all positive predictions made by the model.\n",
        "- **Recall** focuses on the proportion of actual positive samples that were correctly identified by the model.\n",
        "\n",
        "### When to Use:\n",
        "- **High Precision, Low Recall**: Use when false positives are costly (e.g., spam detection).\n",
        "- **High Recall, Low Precision**: Use when false negatives are costly (e.g., disease diagnosis).\n",
        "- **F1-Score**: Use to balance precision and recall when both are critical.\n",
        "\n",
        "---\n",
        "\n",
        "### Example Usage in PyTorch\n",
        "\n",
        "```python\n",
        "import torch\n",
        "import torchmetrics\n",
        "\n",
        "# Example inputs\n",
        "targets = torch.tensor([1, 0, 1, 1, 0, 1])\n",
        "predictions = torch.tensor([1, 0, 1, 0, 0, 1])\n",
        "\n",
        "# True positives, false positives, false negatives, true negatives\n",
        "tp = torch.sum((predictions == 1) & (targets == 1))\n",
        "fp = torch.sum((predictions == 1) & (targets == 0))\n",
        "fn = torch.sum((predictions == 0) & (targets == 1))\n",
        "tn = torch.sum((predictions == 0) & (targets == 0))\n",
        "\n",
        "# Precision\n",
        "precision = tp / (tp + fp + 1e-10)\n",
        "\n",
        "# Recall\n",
        "recall = tp / (tp + fn + 1e-10)\n",
        "\n",
        "# F1-Score\n",
        "f1 = 2 * (precision * recall) / (precision + recall + 1e-10)\n",
        "\n",
        "# Accuracy (TorchMetrics)\n",
        "accuracy_metric = torchmetrics.Accuracy()\n",
        "accuracy = accuracy_metric(predictions, targets)\n",
        "\n",
        "# Confusion Matrix\n",
        "num_classes = 2\n",
        "conf_matrix = torch.zeros(num_classes, num_classes)\n",
        "for t, p in zip(targets, predictions):\n",
        "    conf_matrix[t.long(), p.long()] += 1\n",
        "\n",
        "# Print results\n",
        "print(f\"Precision: {precision.item():.4f}\")\n",
        "print(f\"Recall: {recall.item():.4f}\")\n",
        "print(f\"F1-Score: {f1.item():.4f}\")\n",
        "print(f\"Accuracy: {accuracy.item():.4f}\")\n",
        "print(f\"Confusion Matrix:\\n{conf_matrix}\")\n",
        "\n",
        "\n",
        "**If you want access to a lot of PyTorch metrics, see pages like (i.e. for accuracy: https://pytorch.org/ignite/generated/ignite.metrics.Accuracy.html"
      ],
      "metadata": {
        "id": "K4K-B_ImNlAT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![Alt Text](https://bit.ly/42h5KMI)\n"
      ],
      "metadata": {
        "id": "RE0lcqUcJLd-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install torchmetrics"
      ],
      "metadata": {
        "id": "4Uzhw_uTFeq8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from torchmetrics import Accuracy\n",
        "\n",
        "# Setup metric, specifying the task as 'multiclass' and num_classes\n",
        "torchmetric_accuracy = Accuracy(task=\"multiclass\", num_classes=4).to(device) # NUM_CLASSES should be defined previously\n",
        "\n",
        "# Calculate accuracy\n",
        "torchmetric_accuracy(y_preds, y_blob_test)"
      ],
      "metadata": {
        "id": "ebW7sqUQWk3t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## From the Py Torch Documentation"
      ],
      "metadata": {
        "id": "qWEOSYa0nNEH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    from torchmetrics import Accuracy\n",
        "except:\n",
        "    !pip install torchmetrics==0.9.3 # this is the version we're using in this notebook (later versions exist here: https://torchmetrics.readthedocs.io/en/stable/generated/CHANGELOG.html#changelog)\n",
        "    from torchmetrics import Accuracy\n",
        "\n",
        "# Setup metric and make sure it's on the target device\n",
        "torchmetrics_accuracy = Accuracy(task='multiclass', num_classes=4).to(device)\n",
        "\n",
        "# Calculate accuracy\n",
        "torchmetrics_accuracy(y_preds, y_blob_test)"
      ],
      "metadata": {
        "id": "CkIZYKdBnMtW"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}