Lecture 03_Regression Recap and Asset Pricing Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center><font color=navy>Data Science for Quantitative Finance</font></center>\n",
    "## <center><font color=navy>Regression Recap and Asset Pricing Models</font> </center>\n",
    "### <center> Ali Habibnia</center>\n",
    "\n",
    "    \n",
    "<center> Assistant Professor, Department of Economics, </center>\n",
    "<center> and Division of Computational Modeling & Data Analytics at Virginia Tech</center>\n",
    "<center> habibnia@vt.edu </center> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Readings:\n",
    "\n",
    "1. ***Chapter 3.2***, [The Elements of Statistical Learning Data Mining, Inference, and Prediction](https://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12.pdf)\n",
    "\n",
    "2. For a quick review see: ***Chapter 9.2***, [Understanding Machine Learning From Theory to Algorithms](https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf)\n",
    "\n",
    "3. For students with no background in Econometrics: [Econometric Data Science](https://www.sas.upenn.edu/~fdiebold/Textbooks.html)\n",
    "\n",
    "4. For more advanced topics: ***Chapters 1-2 and 3.1-3.2***, J.D. Angrist and J.S. Pischke, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press, 2009\n",
    "\n",
    "5. Notations for Econometrics http://www.principlesofeconometrics.com/poe5/writing/abadir_magnus.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python for Econometrics\n",
    "\n",
    "1.  Kevin Sheppard, Oxford University https://www.kevinsheppard.com , https://www.kevinsheppard.com/files/teaching/python/notes/python_introduction_2020.pdf\n",
    "\n",
    "2. Wooldridge's INTRODUCTORY ECONOMETRICS: A MODERN APPROACH https://www.cengage.com/c/introductory-econometrics-a-modern-approach-7e-wooldridge/9781337558860PF/, http://www.upfie.net, https://pypi.org/project/wooldridge/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Motivation\n",
    "\n",
    "Why are we learning linear regression?\n",
    "- widely used\n",
    "- runs fast\n",
    "- easy to use (not a lot of tuning required)\n",
    "- highly interpretable\n",
    "- basis for many other methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\"[N]othing at all takes place in the universe in which some rule of maximum or minimum does not appear.\" \n",
    "\n",
    "-- Leonhard Euler\n",
    "\n",
    "<img src=\"images/Leonhard_Euler.jpg\"  width=\"220\">\n",
    "</div>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: right;\" src=\"images/error.png\"  width=\"350\">\n",
    "\n",
    "Optimization problems arise in a large variety of contexts, including **econometrics**, and **machine learning**. The underlying mathematical problem always amounts to finding parameters that **minimize** (cost/loss) or **maximize** (utility/profit) an objective function in the presence or absence of a set of constraints.\n",
    "\n",
    "Together, $\\beta_0$ and $\\beta_1$ are called the **model coefficients/parameters**. To create your model, you must \"learn/estimate\" the values of these coefficients. And once we've learned these coefficients, we can use the model to predict!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression\n",
    "\n",
    "\n",
    "* Linear regression is an old method from statistics for describing the relationships between variables.\n",
    "\n",
    "* LR is an algorithm for learning to predict a real-valued **target/dependent/response** variable as a linear function of one or more real-valued **input/explanatory/regressor/predictor/attribute/covariate/features...** variables. \n",
    "\n",
    "* LR is one of the most widely used statistical learning algorithms, and with care it can be made to work very well in practice. \n",
    "\n",
    "* There are different ways to describe and solve the linear regression problem. The most common way of solving linear regression is via a least squares optimization that is solved using matrix factorization methods from linear regression.\n",
    "\n",
    "* Linear regression can be conveniently represented in terms of matrix-vector multiplication.\n",
    "\n",
    "* Linear regression has a closed-form (analytical) solution in terms of basic linear algebra operations. This makes it a useful starting point for understanding many other statistical learning algorithms.\n",
    "\n",
    "* In contrast to the models with a closed-form solution, sometimes we need an Iterative Optimization Algorithms to minimize or maximize an objective function. \n",
    "\n",
    "* The two basic types of regression are simple linear regression and multiple linear regression which are used for continuos variable, although there are non-linear regression methods like logistic regression which is used for classification problems among a few discrete values.\n",
    "\n",
    "> Goals:\n",
    "\n",
    "- Know the cost function for linear regression (e.g. least squares fit)\n",
    "- Know of at least one method for fitting the model (e.g. gradient descent, closed-form solution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear regression via the least squares method is the simplest approach to performing a regression analysis of a dependent and a explanatory variable. The objective is to find (mathematically) the best-fitting straight **line/plane/hyperplane** through a set of points that minimizes the sum of the squared **residuals/errors/offsets** from the line. \n",
    "\n",
    "The offsets come in 2 different flavors: perpendicular and vertical - with respect to the line.  \n",
    "![](https://raw.githubusercontent.com/rasbt/python_reference/master/Images/least_squares_vertical.png)  \n",
    "![](https://raw.githubusercontent.com/rasbt/python_reference/master/Images/least_squares_perpendicular.png)  \n",
    "\n",
    "The perpendicular offset method delivers a more precise result but is are more complicated to handle. Therefore normally the vertical offsets are used.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\newcommand{\\vct}[1]{\\mathbf{#1}}$\n",
    "$\\newcommand{\\mtx}[1]{\\mathbf{#1}}$\n",
    "$\\newcommand{\\e}{\\varepsilon}$\n",
    "$\\newcommand{\\norm}[1]{\\|#1\\|}$\n",
    "$\\newcommand{\\minimize}{\\text{minimize}\\quad}$\n",
    "$\\newcommand{\\maximize}{\\text{maximize}\\quad}$\n",
    "$\\newcommand{\\subjto}{\\quad\\text{subject to}\\quad}$\n",
    "$\\newcommand{\\R}{\\mathbb{R}}$\n",
    "$\\newcommand{\\trans}{T}$\n",
    "$\\newcommand{\\ip}[2]{\\langle {#1}, {#2} \\rangle}$\n",
    "Suppose we want to understand the relationship of a quantity $Y$ (for example, sales data) to a series of **predictors** $X_1,\\dots,X_p$ (for example, advertising budget in different media). We can often assume the relationship to be **approximately linear**,\n",
    "\n",
    "[1]\\begin{equation*}\n",
    " Y = \\beta_0+\\beta_1 X_1 + \\cdots + \\beta_p X_p + \\varepsilon, \n",
    "\\end{equation*}\n",
    "\n",
    "where $\\varepsilon$ is some error or noise term. The goal is to determine the **model parameters** $\\beta_0,\\dots,\\beta_p$.\n",
    "To determine these, we can collect $n\\geq p$ sample realizations (from observations or experiments),\n",
    "\n",
    "\\begin{equation*}\n",
    " Y=y_i, \\quad X_1=x_{i1},\\dots,X_p=x_{ip}, \\quad 1\\leq i\\leq n,\n",
    "\\end{equation*}\n",
    "\n",
    "and assume that the data is related according to [1], \n",
    "\n",
    "\\begin{equation*}\n",
    " y_i = \\beta_0+\\beta_1x_{i1}+\\cdots +\\beta_p x_{ip}+\\varepsilon_i, \\quad 1\\leq i\\leq n.\n",
    "\\end{equation*}\n",
    "\n",
    "Collecting the data in matrices and vectors,\n",
    "\n",
    "\\begin{equation*}\n",
    " \\vct{y} = \\begin{pmatrix}\n",
    "            y_1\\\\ \\vdots \\\\ y_n\n",
    "           \\end{pmatrix},\n",
    "\\quad \\mtx{X} = \\begin{pmatrix} \n",
    "           1 & x_{11} & \\cdots & x_{1p}\\\\\n",
    "           \\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "           1 & x_{n1} & \\cdots & x_{np}\n",
    "          \\end{pmatrix},\n",
    "\\quad \\vct{\\beta} = \\begin{pmatrix}\n",
    "                     \\beta_0\\\\\n",
    "                     \\beta_1\\\\\n",
    "                     \\vdots\\\\\n",
    "                     \\beta_p\n",
    "                    \\end{pmatrix},\n",
    "\\quad \\vct{\\varepsilon} = \\begin{pmatrix}\n",
    "                  \\e_1\\\\\n",
    "                  \\vdots\\\\\n",
    "                  \\e_n\n",
    "                 \\end{pmatrix},\n",
    "\\end{equation*}\n",
    "\n",
    "we can write the relationship concisely as \n",
    "\n",
    "\\begin{equation*}\n",
    " \\vct{y} = \\mtx{X}\\vct{\\beta}+\\vct{\\e}.\n",
    "\\end{equation*}\n",
    "\n",
    "We would then like to find $\\vct{\\beta}$ in such a way that the difference $\\vct{\\e}=\\vct{y}-\\mtx{X}\\vct{\\beta}$ is as *small* as possible. One way of measuring the size of a vector $\\vct{\\e}\\in \\R^n$ is the square of its **$2$-norm**, or Euclidean norm, \n",
    "\n",
    "\\begin{equation*}\n",
    " \\norm{\\vct{\\e}}_2^2=\\vct{\\e}^{T}\\vct{\\e}=\\sum_{i=1}^n\\e_i^2.\n",
    "\\end{equation*}\n",
    "\n",
    "The best $\\vct{\\beta}$ is then the vector that solves the unconstrained optimization problem\n",
    "\n",
    "\\begin{equation*}\n",
    " \\minimize \\norm{\\mtx{X}\\vct{\\beta}-\\vct{y}}_2^2.\n",
    "\\end{equation*}\n",
    "\n",
    "This is an example of an optimization problem, with variables $\\vct{\\beta}$, no constraints (*all* $\\beta$ are valid candidates and the constraint set is $\\Omega=\\R^{p+1}$), and a **quadratic** objective function \n",
    "\n",
    "\\begin{equation*}\n",
    "f(\\vct{\\beta})=\\norm{\\mtx{X}\\vct{\\beta}-\\vct{y}}_2^2 = (\\mtx{X}\\vct{\\beta}-\\vct{y})^{T}(\\mtx{X}\\vct{\\beta}-\\vct{y}) = \\vct{\\beta}^{T}\\mtx{X}^{T}\\mtx{X}\\vct{\\beta}-2\\vct{y}^{T}\\mtx{X}\\vct{\\beta}+\\vct{y}^{T}\\vct{y},\n",
    "\\end{equation*}\n",
    "\n",
    "where $\\mtx{X}^{T}$ is the matrix transpose.\n",
    "Quadratic functions are convex, so this is a convex optimization problem.\n",
    "This simple optimization problem has a **unique closed-form solution**,\n",
    "\n",
    "This can be re-arranged in order to specify the solution for $\\vct{\\beta}$ as:\n",
    "\n",
    "\\begin{equation*}\n",
    " \\vct{\\beta}^* = (\\vct{X}^{\\trans}\\vct{X})^{-1}\\vct{X}^{\\trans}\\vct{y}.\n",
    "\\end{equation*}\n",
    "\n",
    "In practice one wouldn't compute $\\vct{\\beta}^*$ by evaluating [1], as there are more efficient methods available. \n",
    "\n",
    "Bear in mind that the regression has six key assumptions:\n",
    "\n",
    "1. Linear relationship between target and features\n",
    "2. Data is normally distributed or contains Multivariate normality (but doesn't have to be)\n",
    "3. No or little multicollinearity\n",
    "4. No auto-correlation\n",
    "5. Homoscedasticity\n",
    "6. Independent features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python libraries for Regression Analysis\n",
    "\n",
    "In this section, first, we will see how to solve linear regression via closed-form solution and the least-squares solution from scrach by using numpy.\n",
    "\n",
    "Then we will be using [Statsmodels](http://statsmodels.sourceforge.net/) for linear modeling. However, I recommend that you spend most of your energy on [scikit-learn](http://scikit-learn.org/stable/) since it provides significantly more useful functionality for machine learning in general.\n",
    "\n",
    "#### Requirements for Working With Data in scikit-learn\n",
    "\n",
    "1. Features and response should be separate objects.\n",
    "2. Features and response should be entirely numeric.\n",
    "3. Features and response should be NumPy arrays (or easily converted to NumPy arrays).\n",
    "4. Features and response should have specific shapes (outlined below).\n",
    "5. Note that **p-values** and **confidence intervals** are not (easily) accessible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "\n",
    "# %matplotlib (percent sign) is a magic function in IPython (Jupyter Notebook). \n",
    "# this allows plots to appear directly in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.05 0.12]\n",
      " [0.18 0.22]\n",
      " [0.31 0.35]\n",
      " [0.42 0.38]\n",
      " [0.5  0.49]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# linear regression \n",
    "\n",
    "'''\n",
    "    NumPy is a library for multi-dimensional arrays and matrices along with a large collection\n",
    "    of mathematical functions to operate on these arrays.\n",
    "    \n",
    "    Matplotlib is a plotting library for the Python programming language  \n",
    "'''\n",
    "\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "# define dataset\n",
    "\n",
    "'''\n",
    "X = [x_i*random.randrange(8,12)/10 for x_i in range(500)]\n",
    "y = [y_i*random.randrange(8,12)/10 for y_i in range(100,600)]\n",
    "'''\n",
    "\n",
    "data = np.array([\n",
    "  [0.05, 0.12],\n",
    "  [0.18, 0.22],\n",
    "  [0.31, 0.35],\n",
    "  [0.42, 0.38],\n",
    "  [0.5, 0.49]])\n",
    "print(data)\n",
    "\n",
    "# split into inputs and outputs\n",
    "X, y = data[:,0], data[:,1]\n",
    "X = X.reshape((len(X), 1))\n",
    "\n",
    "# scatter plot\n",
    "plt.scatter(X, y)\n",
    "plt.ylabel('y')\n",
    "plt.xlabel('X')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beta = [1.00233226]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcFOW1//HPcQQZQUUCMYAoxgVlmThKcAk3atTgEpG45EJirqhE/SkXjYgRogZXVCRoFFFwQ1FRFBENOCIGkYjKCAgiokhAFqOg4oqs5/fHU8xtJjPMDNPV1d3zfb9e86Jr6arTRXefPk9VPY+5OyIiIgA7JB2AiIhkDyUFEREpo6QgIiJllBRERKSMkoKIiJRRUhARkTJKClnAzP7LzBYmHUeuMrPfmdmLScdRGTPraWbTM7zPqWbWaxvLf2ZmH5jZN2bWLZOxbS8zO9rMltfi+QPM7L50xpSPlBQyyMyWmNlx5ee7+6vu3iaJmPKBuz/q7r9MOo4ccx1wl7s3cvfxSQeTbhUlEHe/yd0rTZQSKCnUYWa2YzrWSfc+JSP2BuZvzxP1f5jflBSyQPlfNVFFcbmZzTWzL83sCTNrkLL8V2Y2x8zWmNlrZlaUsuxKM/vQzL42s3fN7Ncpy3qa2T/NbKiZfQ4MrCCWgWb2lJmNNrOvgJ5mtkPKdj8zsyfNrEnKc/7HzJZGy65OrYhquj0zaxCt+1n0+maa2R4p8S+OXtu/zOx3KfOnp8RzZPS8L6N/j0xZNtXMro+Ow9dm9qKZNa3k/2V3M3vezFaZ2RfR4z238f/YyszGRet/ZmZ3VbLetuKr8DVGy841swVRLCVmtnfKsuPN7L1om3cBto04PwR+DDwXNR/tZGYtzGyCmX1uZovM7A8p61f0fzjQzMZG8742s3lmdoCZ9TezT81smZlVWr2Z2Z/MbEX03IVmdmw0fyczu93MVkZ/t5vZTpVsw81sv5Tph8zsBjNrCEwCWkSv75vo9Q00s9Ep63c1s/nR+2yqmR2Usmybn8G85u76y9AfsAQ4roL5RwPLy633JtACaAIsAC6Mlh0CfAocBhQAZ0fr7xQtPzN63g7AfwPfAs2jZT2BjcD/AjsChRXEMhDYAHSLtlEIXAq8DuwJ7ATcCzwerd8W+AboDNQHbouef9x2bu8C4Dlg5+j1HQrsCjQEvgLaROs1B9qlvK7p0eMmwBfA76PX2COa/kG0fCrwIXBAFMtU4OZK/r9+AJwexbILMBYYX8m6BcDbwNAo1gZA55rEV8Vr7AYsAg6KnncV8Fq0rGn0vDOAesAfo//nXtV9LwKvAHdHcR8MrAKO3cb/4UDge6BLFM/DwL+AP0cx/AH4VyX7bgMsA1pE062BfaPH10XvjR8CzYDXgOsr+Zw4sF/K9EPADRWtm/I6RkePDyB8No6P4r0iOr71q/oM5vtf4gHUpb/yH8SU+eXf7EuAs1KmbwXuiR4P3/IhSVm+EDiqkn3OAU6NHvcEPqoixoHAtHLzFmz5goimm0dfEjsC1xB9oUfLdgbWs3VSqMn2zo2+CIrKPachsIbwJV1YbllP/u9L9/fAm+WWzwB6Ro+nAlelLLsIeKGa/38HA19UsuwIwhfpjhUsq1Z8VbzGScB5KdM7AN8RmoH+B3g9ZZkBy6lmUgBaAZuAXVKWDwIe2sb/4UBgcsr0KYQfBwXR9C6EL+3GFex7P8IPm+OAeuWWfQiclDLdBVhSyeekNknhauDJcsdzBXB0VZ/BfP9T81H2+nfK4++ARtHjvYG+Ucm7xszWED7ULaCsKWdOyrL2hF+SWyyrxr7Lr7M38EzKNhcQvkT2iPZbtr67fwd8VovtPQKUAGOi5oNbzayeu39LqHwuBD42s7+b2YEVxN4CWFpu3lKgZcp0Zcd2K2a2s5nda6Fp7CtgGtDYzAoqWL0VsNTdN1a0rerEV8Vr3Bu4I+WYfU748m/Jf/4feOp01ESypRnlvyqJ6XN3/7p8TCnTFb1vPkl5vBZY7e6bUqahgmPr7osI1eJA4FMzG2NmLVJiST0+S6N56bbVftx9M+E11vh9km+UFHLPMuBGd2+c8rezuz8etTGPBHoTmksaA++wdftydbrFLb/OMuDEcvts4O4rgI8JzUAAmFkhoSlku7bn7hvc/Vp3bwscCfyK8EsYdy9x9+MJlcV70WstbyXhCzTVXoRfgTXVl9DUcZi77wr8fMvLrGDdZcBeVvVJ2G3Gt43XuAy4oNwxK3T31wj/B622bMzMLHXa3dt5uMqokbu/WklMTcxsl4pi2rKZKl5Xjbj7Y+7emXAsHLglJZbU47NXNK8i3xEq0y1+lLqLKkLYaj8px2x73id5RUkh8+pZOJm65a+mV3KMBC40s8MsaGhmJ0cf6IaED8MqADM7h1Ap1NY9wI1bTmyaWTMzOzVa9hRwioWTp/WBa9nGSc6qtmdmx5hZh+jX+FeEZqVNZrZHdGKwIbCO0FSxqYJtTwQOMLPfmtmOZvbfhPMez2/H696F8It3jYUT4X/ZxrpvEr6cb47+TxqY2c9qEl8Vr/EeoL+ZtQMws93M7Mxo2d+BdmZ2WvR+6sPWX5Db5O7LCE12g6K4i4DzgEeru42aMLM2ZvaL6ATy94RjvOV1Pg5cFb0nmhKaJ0dXsqk5wG/NrMDMTgCOSln2CfADM9utkuc+CZxsZseaWT3CD4B1hONQpykpZN5Ewodgy9/AmjzZ3UsJJ/HuIpygXERoj8bd3wWGENqoPwE6AP9MQ8x3ABOAF83sa8KJwMOifc4nnLgeQ/hS/JrQXrxue7ZH+DJ7ipAQFhBOgI4mvFf7En7hfU74Ario/Ibd/TNCddGX0Ix1BfArd1+9Ha/7dsJJ1dVRjC9UtmLUbHIKob38I0Kb/n/XML5KX6O7P0P4NT0masp6BzgxWraacIHBzdE296fm/+89CCd8VwLPAH9x98k13EZ17USIdTWhieaHwIBo2Q1AKTAXmAfMiuZV5BLCMV8D/A4ou9/C3d8jJJjFUZPbVk1Q7r4QOAu4M4rjFOAUd1+fhteX0yw6iSKSFmbWiPAh3d/d/5V0PCJSM6oUpNbM7JTopGxDwiWp8whXb4hIjlFSkHQ4ldDssJLQdNHdVYKK5CQ1H4mISBlVCiIiUibnOrZq2rSpt27dOukwRERyyltvvbXa3ZtVtV7OJYXWrVtTWlqadBgiIjnFzMrfSV8hNR+JiEgZJQURESmjpCAiImWUFEREpIySgoiIlFFSEBGRMkoKIiJSJtakYGYnWBiUe5GZXVnB8p4WBjmfE/31ijMeERHZttiSQjRIyjBCn+9tgR5m1raCVZ9w94Ojv/viikdEJCc89xy8+25iu4+zUugELHL3xdHAFWMIvWmKiEh5a9ZAz57QtSvcckuVq8clzqTQkq0H+17O1oNib3G6mc01s6fMrFUFyzGz882s1MxKV61aFUesIiLJmTgR2rWD0aPh6qthZEXDj2dGnEmhonF6y/fT/RzQ2t2LgJeAURVtyN1HuHtHd+/YrFmV/TmJiOSGL7+E886Dk0+G3XfnH6Oe42c7H80+10zmZze/zPjZKzIeUpxJYTmQ+st/T8IgLGXc/TN33zKW70jg0BjjERHJHi++CO3bw0MPQf/+TLh/Ahe9Z6xYsxYHVqxZS/9x8zKeGOJMCjOB/c1sHzOrD3QnDNZexsyap0x2JQzULiKSv77+Gi64ALp0gUaNYMYMuOkmbvnHEtZu2LTVqms3bGJwycKMhhdb19nuvtHMegMlQAHwgLvPN7PrgFJ3nwD0MbOuwEbgc6BnXPGIiCRuypTQXLRsGfTrB9ddBw0aALByzdoKn1LZ/LjEOp6Cu08EJpabd03K4/5A/zhjEBFJ3DffwBVXwPDhcMABMH06HHHEVqu0aFzIigoSQIvGhZmKEtAdzSIi8Zo6FYqK4J574LLLYM6c/0gIAP26tKGwXsFW8wrrFdCvS5sMBRooKYiIxOHbb6FPHzjmGCgogGnTYMgQKKz4l3+34pYMOq0DLRsXYkDLxoUMOq0D3YorupI/Pjk3HKeISNabPj3ciPbhhyEx3HQTNGxY5dO6FbfMeBIoT5WCiEi6fPddaCL6+c9h8+bQdHTHHdVKCNlClYKISDrMmBGqg/ffh4suCl1VNGqUdFQ1pkpBRKQ2vv8+XFnUuTOsWxcuOx02LCcTAqhSEBHZfm++CWefDe+9F25IGzwYdtkl6ahqRZWCiEhNrVsH/fuHS0u//RZKSsIlpzmeEECVgohIzZSWhnMH8+eHu5OHDIHddks6qrRRpSAiUh3r14durQ8/HL74InR3fd99eZUQQJWCiEjVZs8O1cHcueHfoUOhceOko4qFKgURkcps2ADXXgudOsGnn4ahMh98MG8TAqhSEBGp2Ny54cqiOXPgrLPCTWhNmiQdVexUKYiIpNq4EW68ETp2hJUr4Zln4JFH6kRCAFUKIiL/Z/78UB289RZ07w533glNmyYdVUapUhAR2bgxdEtxyCGwdCmMHQuPP17nEgKoUhCRum7BgnBF0ZtvwhlnhC4qfvjDpKNKjCoFEambNm2C226D4uLQxfUTT4QKoQ4nBFClICJ10fvvh+pgxgzo1i10UbHHHklHlRVUKYhI3bF5M9x+O/zkJ6ETu0cfhXHjlBBSqFIQkbph0SI491x49VU45RS4915o3jzpqLKOKgURyW+bN4dLS4uKwg1po0bBs88qIVRClYKI5K/Fi0N18MorcOKJMHIktEx2DORsp0pBRPLP5s0wfHioDmbPhvvvh7//XQmhGlQpiEh+Wbo0jHMwZQr88pehe+tWrZKOKmeoUhCR/OAemofat4c33oARI+CFF5QQakiVgojkvmXLoFcvePFF+MUv4IEHYO+9k44qJ6lSEJHc5R7GN2jfHv75T7j7bpg8WQmhFlQpiEhuWrECzj8/DIt51FGhOvjxj5OOKuepUhCR3OIODz8M7drB1KnhHoSXX1ZCSBNVCiKSOz7+GC64IAyL2blzaDrab7+ko8orqhREJPu5w2OPhepg8mQYOjRUCUoIaaekICLZ7ZNP4PTT4Xe/gwMPhLffhksvhYKCpCPLS0oKIpK9nngiVAcTJ8LgwaEzuwMOSDqqvBZrUjCzE8xsoZktMrMrt7HeGWbmZtYxznhEJEesWgW/+U0YJ3nffUNXFZdfruogA2JLCmZWAAwDTgTaAj3MrG0F6+0C9AHeiCsWEckhTz8dqoNnn4VBg8L9BwcdlHRUdUaclUInYJG7L3b39cAY4NQK1rseuBX4PsZYRCTbffYZ9OgRxkneay+YNQuuvBJ21EWSmRRnUmgJLEuZXh7NK2NmxUArd39+Wxsys/PNrNTMSletWpX+SEUkWc8+G6qDp5+GG24Iw2S2a5d0VHVSnEnBKpjnZQvNdgCGAn2r2pC7j3D3ju7esVmzZmkMUUQS9fnn8Pvfh3GSmzeH0lL485+hXr2kI6uz4kwKy4HU7gn3BFamTO8CtAemmtkS4HBggk42i9QRzz8f+iwaMwYGDoQ33wzjH0ii4kwKM4H9zWwfM6sPdAcmbFno7l+6e1N3b+3urYHXga7uXhpjTCKStDVroGfPME5y06YhGfzlL6oOskRsScHdNwK9gRJgAfCku883s+vMrGtc+xWRLDZpUqgORo+Gq64KzUXFxUlHJSliPa3v7hOBieXmXVPJukfHGYuIJOjLL6Fv3zAsZrt2MH48dFRLcTbStV4iEq/Jk8PwmCtWQP/+oalop53Ssunxs1cwuGQhK9espUXjQvp1aUO3Yo3DXBtKCiISj6+/hn794N57Q59FM2ZAp05p2/z42SvoP24eazdsAmDFmrX0HzcPQImhFtT3kYik38svQ4cOYZzkfv1CNxVpTAgAg0sWliWELdZu2MTgkoVp3U9do6QgIunzzTdw8cVw7LGhiWj6dLj1VmjQIO27WrlmbY3mS/UoKYhIerzySrjPYPhwuOwymDMHjjwytt21aFxYo/lSPUoKIlI7334Ll1wCRx8NO+wA06bBkCFQGO+Xc78ubSist3WvqYX1CujXpU2s+813OtEsIttv+nQ45xxYtAj69IGbboKGDTOy6y0nk3X1UXopKYhIza1dG/oouv12aN0a/vGPUClkWLfilkoCaaakICI1M2NG6Kbi/ffhoovgllugUaOko5I00TkFEame77+HK66Azp3D45degmHDlBDyjCoFEanam2/C2WfDe+/B+eeH8ZJ33TXpqCQGqhREpHLr1sGAAXDEEeEqo5KScIeyEkLeUqUgIhV7661QHcyfH/ouGjIEdtst6agkZkoKIlkskQ7f1q+H66+HQYNgjz1g4kQ48cR49ylZQ0lBJEsl0uHb7NnhyqK5c0OVMHQo7L57PPuSrKRzCiJZKqMdvm3YANdeGzqt+/RTmDABHnpICaEOUqUgkqUy1uHb3LmhOpg9G846C+64A5o0Se8+JGeoUhDJUrF3+LZxI9x4YxgBbcUKeOYZeOQRJYQ6TklBJEvF2uHb/Plw+OFhnOTTTw/T3brVfruS85QURLJUt+KWDDqtAy0bF2JAy8aFDDqtQ+1OMm/cGLqlOOQQWLoUxo6Fxx+Hpk3TFrfkNp1TEMliae3wbcGCcO7gzTdDdXD33fDDH6Zn25I3VCmI5LtNm+C226C4OHRxPWZMqBCUEKQCqhRE8tn774fqYMaMcM5g+HD40Y+SjkqymCoFkXy0aVO48ewnPwmd2D36KIwbp4QgVVKlIJJvFi0Ko6FNnw6/+hWMGAHNmycdleQIVQoi+WLzZrjzTigqgnnzYNSocGeyEoLUgCoFkXyweDGcey688krovG7kSGipYSql5lQpiOSyzZvDyeOiotBNxf33w9//roQg202VgkiuWro0jHMwZQocfzzcdx/stVfSUUmOU6UgkmvcQ/NQ+/bwxhthJLSSEiUESQtVCiK5ZNky6NULXnwRfvGL0FzUunXSUUkeUaUgkgvc4YEHQnXwz3+GLiomT1ZCkLRTpSCS7VasgD/8ASZNgqOOCsnhxz9OOirJU6oURLKVe7jXoF07mDoV/vY3ePllJQSJlSoFkWz08cdw/vnw/PPQuTM8+CDst1/SUUkdEGulYGYnmNlCM1tkZldWsPxCM5tnZnPMbLqZtY0zHpGs5x76KWrXDl56KfRfNHWqEoJkTGxJwcwKgGHAiUBboEcFX/qPuXsHdz8YuBX4a1zxiGS9Tz6B004L4yQfeCDMmQOXXgoFBVU/VyRN4qwUOgGL3H2xu68HxgCnpq7g7l+lTDYEPMZ4RLKTOzzxRKgOJk2CwYPh1VehTRqG3RSpoTjPKbQElqVMLwcOK7+SmV0MXAbUB35R0YbM7HzgfIC9dIOO5JNVq+Cii+Cpp6BTJ3joITjooKSjkjoszkrBKpj3H5WAuw9z932BPwFXVbQhdx/h7h3dvWOzZs3SHKZIQp5+OlQHEybAoEHh/gMlBElYnElhOdAqZXpPYOU21h8DdIsxHpHssHo1dO8OZ5wRuqZ46y248krYURcDSvLiTAozgf3NbB8zqw90ByakrmBm+6dMngx8EGM8IskbPz5UB+PGwQ03hGEy27dPOiqRMrH9NHH3jWbWGygBCoAH3H2+mV0HlLr7BKC3mR0HbAC+AM6OKx6RRH3+OfTpEy43Pfjg0EVFUVHSUYn8h1jrVXefCEwsN++alMeXxLl/kazw3HPhRrTVq2HgQBgwAOrVSzoqkQqpEVMkLmvWhPsMRo2CDh1g4kQoLk46KpFtUt9HInGYNCmcOxg9Gq66CkpLlRAkJygpiKTTl1+G0dBOOgkaN4bXX4frr4f69ZOOTKRaqkwKZtbbzHbPRDAiOe3FF8OVRA89BP37w6xZ0LFj0lGJ1Eh1KoUfATPN7Mmog7uKbkoTqbu+/houuAC6dIFGjcJlpjfdBDvtlHRkIjVWZVJw96uA/YH7gZ7AB2Z2k5ntG3NsItlvypRwEnnkSOjXD2bPDt1ViOSoap1TcHcH/h39bQR2B54ys1tjjE0ke33zTeiz6LjjQkUwfTrceis0aJB0ZCK1UuUlqWbWh3BT2WrgPqCfu28wsx0IdyBfEW+IIllm6lQ491xYsgQuuyzcmVxYmHRUImlRnfsUmgKnufvS1JnuvtnMfhVPWCJZ6NtvwwnkO++EffeFadPCqGgieaTKpJB6B3IFyxakNxyRLPXqq3DOOfDhh6G7iptugoYNk45KJO10n4LItnz3XWgiOuoo2Lw5NB3dcYcSguQtdXMhUpkZM6BnT3j//XBS+ZZbwiWnInlMlYJIeWvXhstLO3eG77+Hl16CYcOUEKROUKUgkuqNN0J18N57oWfTwYNh112TjkokY1QpiACsWxeuLDryyHCVUUkJ3HuvEoLUOaoUREpLQ3Uwf37ozG7IENhtt6SjEkmEKgWpu9atC91aH344fPFFGO/gvvuUEKROU6UgddPs2XD22TBvXqgShg4NXV2L1HGqFKRu2bABrr02dFq3alUYKvPBB5UQRCKqFKTuePvtUBXMmQNnnRVuQmvSJOmoRLKKKgXJfxs2hE7rfvpTWLkSnnkGHnlECUGkAqoUJL+9806oDt56C7p3D53ZNW2adFQiWUuVguSnjRth0CA49FD46CN46il4/HElBJEqqFKQ/LNgQbiyaOZMOOMMuPtuaNYs6ahEcoIqBckfmzaFbimKi2HxYnjiCRg7VglBpAZUKUh+WLgwjHcwYwZ06wb33AN77JF0VCI5R0lB0mr87BUMLlnIyjVradG4kH5d2tCtuGV8O9y0Cf72NxgwIAyJ+eij0KMHmMW3T5E8pqQgaTN+9gr6j5vH2g2bAFixZi39x80DiCcxLFoUqoPp0+GUU0IHds2bp38/InWIzilI2gwuWViWELZYu2ETg0sWpndHmzeH6qCoKHRTMWoUPPusEoJIGqhSkLRZuWZtjeZvl8WLQ3UwbRqcdBKMGAEtY2yeEqljVClI2rRoXFij+TWyeXO4tLSoKHRT8cAD8PzzSggiaaakIGnTr0sbCusVbDWvsF4B/bq0qd2GlyyB44+Hiy+Gn/0s3KV8zjk6mSwSAzUfSdpsOZmctquP3EPz0OWXhwQwYgT06qVkIBIjJQVJq27FLdNzpdFHH4UEMHkyHHss3H8/7L137bcrItsUa/ORmZ1gZgvNbJGZXVnB8svM7F0zm2tmU8xMn/q6zj0kgPbt4bXXYPjwkBiUEEQyIrakYGYFwDDgRKAt0MPM2pZbbTbQ0d2LgKeAW+OKR3LA8uXhiqJevUJHdvPmwYUXqrlIJIPirBQ6AYvcfbG7rwfGAKemruDu/3D376LJ14E9Y4xHspV7uNegfftwqemdd8KUKbDPPklHJlLnxJkUWgLLUqaXR/Mqcx4wKcZ4JButXAldu4YxDzp0CKOj9e4NO+jCOJEkxPnJq6jm9wpXNDsL6AgMrmT5+WZWamalq1atSmOIkhh3GD06VAcvvQRDh8Irr8B++yUdmUidFmdSWA60SpneE1hZfiUzOw74M9DV3ddVtCF3H+HuHd29YzN1g5z7/v1v+PWv4fe/h4MOCtXBpZeqOhDJAnF+CmcC+5vZPmZWH+gOTEhdwcyKgXsJCeHTGGORbOAOY8ZAu3bwwgtw223hHMIBByQdmYhEYksK7r4R6A2UAAuAJ919vpldZ2Zdo9UGA42AsWY2x8wmVLI5yXWffgpnnhm6td5vv9BVRd++UFBQ9XNFJGNivXnN3ScCE8vNuybl8XFx7l+yxNixcNFF8NVXcPPNIRnsqPsmRbKRGnElPqtXQ/fu8JvfQOvWMGsW/OlPSggiWUxJQeLxzDPh3MG4cXDjjWGYzHbtko5KRKqgn2ySXp99Bn36wGOPQXFx6KKiqCjpqESkmlQpSPpMmBDuO3jySRg4EN54QwlBJMeoUpDa++KLcJ/Bww+HJDBpEhx8cNJRich2UKUgtTNxYqgOHn0Urr4aZs5UQhDJYUoKsn2+/BLOPRdOPhl23z00FV13HdSvn3RkIlILSgpScyUloToYNQoGDIC33gpdXYtIztM5Bam+r74KQ2OOHBn6LJoxAzp1SjoqEUkjVQpSPS+9FLq2vv9+uOKKcCOaEoJI3lFSkG375pvQRcXxx0ODBjB9OtxyS3gsInlHSUEqN3VqqA7uuQcuuyx0YnfEEUlHJSIxUlKQ//Ttt/C//wvHHBP6KZo2DYYMgcLCpCMTkZgpKcjWXn013IB2111wySVhAJzOnZOOSkQyRElBgu++gz/+EY46KkxPnQq33w4775xoWCKSWbokVeC116BnT/jgA7j44jDmQaNGSUclIglQpVCXrV0L/fqF5qH162HKlNBspIQgUmepUqir3ngDzj4bFi6ECy+EW2+FXXZJOioRSZgqhbrm++/hyivhyCPDeYQXX4Thw5UQRARQpVC3lJaG6uDdd6FXL7jtNthtt6SjEpEsokqhLli3Dq66Cg4/PPRuOnFi6L9ICUFEylGlkO9mzQpXFs2bF/4dOhQaN046KhHJUqoU8tX69WFIzMMOg9Wr4bnn4MEHlRBEZJtUKeSjt98OVcGcOXDWWXDHHdCkSdJRiUgOUKWQTzZsgOuvh44d4eOPYfx4eOQRJQQRqTZVCvninXfClUWzZkGPHnDnnfCDHyQdlYjkmDqdFMbPXsHgkoWsXLOWFo0L6delDd2KWyYdVs1s3AiDB4fzB7vtBk89BaefnnRUIpKj6mxSGD97Bf3HzWPthk0ArFizlv7j5gHkTmJ4991w7mDmTDjzTBg2DJo1SzoqEclhdfacwuCShWUJYYu1GzYxuGRhQhHVwKZNoTo45BBYvBieeAKefFIJQURqrc5WCivXrK3R/KyxcGGoDl5/HX7969BFxR57JB2ViOSJOlsptGhc8Shilc1P3KZN8Ne/wsEHh8Tw6KPw9NNKCCKSVnU2KfTr0obCegVbzSusV0C/Lm0SimgbPvggDH7Tty8cfzzMnw+//S2YJR2ZiOSZOpsUuhW3ZNBpHWjZuBADWjYuZNBpHbLrJPPmzfC3v8FPfhISwcMPw7PPQvPmSUcmInmqzp5TgJAYsioJpFq8GM45B6ZNg5NOghEjoGWWxioieaPOVgpZa/PjPi0eAAAHgElEQVTmcGlpUVHopuKBB+D555UQRCQjYk0KZnaCmS00s0VmdmUFy39uZrPMbKOZnRFnLDlhyRI47jjo3TsMkfnOO6Fa0LkDEcmQ2JKCmRUAw4ATgbZADzNrW261j4CewGNxxZET3OHee6FDhzAQzsiRMGkStGqVdGQiUsfEeU6hE7DI3RcDmNkY4FTg3S0ruPuSaNnmGOPIbh99FEZBmzwZjj0W7r8f9t476ahEpI6Ks/moJbAsZXp5NK/GzOx8Mys1s9JVq1alJbjEuYcE0L49vPZauAlt8mQlBBFJVJxJoaKGcN+eDbn7CHfv6O4dm+VDVw7Ll4crinr1gkMPDaOiXXihzh2ISOLiTArLgdRG8T2BlTHuL/u5w6hRoTqYNi10bz1lCuyzT9KRiYgA8SaFmcD+ZraPmdUHugMTYtxfdlu5Ek45JfRbVFQEc+eGq4x20FXBIpI9YvtGcveNQG+gBFgAPOnu883sOjPrCmBmPzWz5cCZwL1mNj+ueBLjDqNHQ7t28PLLcPvtMHUq7Ltv0pGJiPyHWO9odveJwMRy865JeTyT0KyUn/7973Cu4Nln4cgj4cEH4YADko5KRKRSaruIgzuMGROqgxdegNtuC+cQlBBEJMspKaTbp5+GUdB69ID99w9dVfTtCwUFVT9XRCRhSgrpNHZsqA6eew5uuQWmT4cDD0w6KhGRalNSSIfVq6F7d/jNb6B1a5g1C664Anas053QikgOUlKorWeeCdXBuHFw440wY0aYFhHJQfopu70++wz69IHHHoPiYnjppdChnYhIDlOlsD0mTAh3JT/5JFx7LbzxhhKCiOQFVQo18cUXcOmlYVjMoqLQvfXBBycdlYhI2qhSqK6JE0N18OijcPXVMHOmEoKI5B0lhap8+SWcey6cfDI0aRKaiq67DurXTzoyEZG0U1LYlpKSUB2MGgUDBoRR0Q49NOmoRERio3MKFfnqK7j88jAs5kEHhctMO3VKOioRkdipUihvy6Wl998fbkCbNUsJQUTqDCWFLb75Bi66CI4/Hho0CF1U3HJLeCwiUkcoKUAY36BDB7jnHrjsstCJ3RFHJB2ViEjGKSkMGADHHBP6KZo2DYYMgcLCpKMSEUmEksK++8Ill8Dbb0PnzklHIyKSKF19dN55SUcgIpI1VCmIiEgZJQURESmjpCAiImWUFEREpIySgoiIlFFSEBGRMkoKIiJSRklBRETKmLsnHUONmNkqYGnSccSgKbA66SASpmOgYwA6BhDPMdjb3ZtVtVLOJYV8ZWal7t4x6TiSpGOgYwA6BpDsMVDzkYiIlFFSEBGRMkoK2WNE0gFkAR0DHQPQMYAEj4HOKYiISBlVCiIiUkZJQUREyigpZJCZnWBmC81skZldWcHyn5vZLDPbaGZnJBFj3KpxDC4zs3fNbK6ZTTGzvZOIM07VOAYXmtk8M5tjZtPNrG0SccatquOQst4ZZuZmlneXqVbjvdDTzFZF74U5ZtYr9qDcXX8Z+AMKgA+BHwP1gbeBtuXWaQ0UAQ8DZyQdc0LH4Bhg5+jx/wOeSDruBI7BrimPuwIvJB13EschWm8XYBrwOtAx6bgTeC/0BO7KZFyqFDKnE7DI3Re7+3pgDHBq6gruvsTd5wKbkwgwA6pzDP7h7t9Fk68De2Y4xrhV5xh8lTLZEMjHq0GqPA6R64Fbge8zGVyGVPcYZJSSQua0BJalTC+P5tUlNT0G5wGTYo0o86p1DMzsYjP7kPCF2CdDsWVSlcfBzIqBVu7+fCYDy6Dqfh5Oj5pTnzKzVnEHpaSQOVbBvHz8Bbgt1T4GZnYW0BEYHGtEmVetY+Duw9x9X+BPwFWxR5V52zwOZrYDMBTom7GIMq8674XngNbuXgS8BIyKOyglhcxZDqRm+T2BlQnFkpRqHQMzOw74M9DV3ddlKLZMqen7YAzQLdaIklHVcdgFaA9MNbMlwOHAhDw72Vzle8HdP0v5DIwEDo07KCWFzJkJ7G9m+5hZfaA7MCHhmDKtymMQNRncS0gInyYQY9yqcwz2T5k8Gfggg/FlyjaPg7t/6e5N3b21u7cmnF/q6u6lyYQbi+q8F5qnTHYFFsQd1I5x70ACd99oZr2BEsJVBw+4+3wzuw4odfcJZvZT4Blgd+AUM7vW3dslGHZaVecYEJqLGgFjzQzgI3fvmljQaVbNY9A7qpY2AF8AZycXcTyqeRzyWjWPQR8z6wpsBD4nXI0UK3VzISIiZdR8JCIiZZQURESkjJKCiIiUUVIQEZEySgoiIlJGSUGkFsyslZn9y8yaRNO7R9N517ur1A1KCiK14O7LgOHAzdGsm4ER7r40uahEtp/uUxCpJTOrB7wFPAD8ASiOer0UyTm6o1mkltx9g5n1A14AfqmEILlMzUci6XEi8DGhEzeRnKWkIFJLZnYwcDyhJ88/luvETCSnKCmI1IKFXvuGA5e6+0eEDv1uSzYqke2npCBSO38g9OQ6OZq+GzjQzI5KMCaR7aarj0REpIwqBRERKaOkICIiZZQURESkjJKCiIiUUVIQEZEySgoiIlJGSUFERMr8fyMX2ZIhBQ/WAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# direct solution to linear least squares\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "# define dataset\n",
    "data = np.array([\n",
    "  [0.05, 0.12],\n",
    "  [0.18, 0.22],\n",
    "  [0.31, 0.35],\n",
    "  [0.42, 0.38],\n",
    "  [0.5, 0.49]])\n",
    "# print(data)\n",
    "# split into inputs and outputs\n",
    "X, y = data[:,0], data[:,1]\n",
    "X = X.reshape((len(X), 1))\n",
    "# linear least squares\n",
    "\n",
    "b = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)\n",
    "\n",
    "print(\"Beta =\", b)\n",
    "# predict using coefficients\n",
    "yhat = X.dot(b)\n",
    "# plot data and predictions\n",
    "plt.scatter(X, y) \n",
    "plt.plot(X, yhat, color='red') \n",
    "plt.ylabel('y')\n",
    "plt.xlabel('X')\n",
    "plt.title('Linear regression a closed-form solution')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Build a function to compute the closed-form solution to a linear matrix equation. \"\"\"\n",
    "import numpy as np\n",
    "\n",
    "def OLS(x, y):\n",
    "    X = np.vstack([x, np.ones(len(x))]).T\n",
    "    return (np.linalg.inv(X.T.dot(X)).dot(X.T)).dot(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Least Squares Estimation for Simple Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we mentioned in the previous sectoin, the parameters $\\beta_0$ and $\\beta_1$ are unknown, and we have to estimate them by using sample data points. Suppose we have N paires of $x$ and $y$. There are infinite number of lines that we can use, but we are looking for selecting the best line through the data. <br>\n",
    "We have to estimate the parameters in way that we minimize the difference between the data points which is $y_i$ and the value of straight line in the certain $x$ point. From now, we called this diffrences as **residual** and we show it by $e_i$. Residual play an important role in evaluating the accuracy of the model. We will talk more about it the next section. We use $\\hat{y}$ to show the regression value to distinguish the observed value of $y$ from the value that calculated from the regression model.\n",
    "So <br>\n",
    "$$SS_{res}=\\sum{e_i}^2=\\sum\\limits_{i=1}^n{(y_i-\\hat{y_i})^2}=\\sum\\limits_{i=1}^n(y_i-(\\hat{\\beta_0}+\\hat{\\beta_1}x_i))^2$$ <br>\n",
    "We call the above equition the **sum of squares for the residuals ($SS_{res}$)**. Our best estimated line, then, is the one which minimizes the $SS_{res}$. <br> <br>\n",
    "Minimizing $e$ amounts to solving the so-called normal equations<br>\n",
    "$$\\frac{\\partial SS_{res}}{\\partial \\hat{\\beta_0}}=-2\\sum\\limits_{i=1}^n[y_i-\\hat{\\beta_0}-\\hat{\\beta_1}x_i]=0$$ <br>\n",
    "$$\\frac{\\partial SS_{res}}{\\partial \\hat{\\beta_1}}=-2\\sum\\limits_{i=1}^n[x_iy_i-\\hat{\\beta_0}x_i-\\hat{\\beta_1}x_i^2]=0$$ <br>\n",
    "That is, <br>\n",
    "$$n\\hat{\\beta_0}+\\hat{\\beta_1}\\sum\\limits_{i=1}^n{x_i}=\\sum\\limits_{i=1}^n{y_i}$$ <br> \n",
    "and <br>\n",
    "$$\\hat{\\beta_0}\\sum\\limits_{i=1}^n{x_i}+\\hat{\\beta_1}\\sum\\limits_{i=1}^n{x_i^2}=\\sum\\limits_{i=1}^n{x_iy_i}$$ <br>\n",
    "Let $$\\bar{x}=\\frac{1}{n}\\sum\\limits_{i=1}^n{x_i}$$ and $$\\bar{y}=\\frac{1}{n}\\sum\\limits_{i=1}^n{y_i}$$ be the sample means of predictor values\n",
    "and the responses. If <br>\n",
    "$$S_{xy}=\\sum\\limits_{i=1}^n{(x_i-\\bar{x})(y_i-\\bar{y})}=\\sum\\limits_{i=1}^ny_i(x_i-\\bar{x})=\\sum\\limits_{i=1}^nx_iy_i-n\\bar{x}\\bar{y}$$ <br>\n",
    "$$S_{xx} = \\sum\\limits_{i=1}^n{(x_i-\\bar{x})}^2=\\sum\\limits_{i=1}^n{x_i}^2-n\\bar{x}^2$$ <br>\n",
    "$SS_{xx}$ is the sum of the squares of the difference between each 𝑥 and the mean 𝑥 value, and <br>\n",
    "$SS_{xy}$ is sum of the product of the difference between 𝑥 its means and the difference between 𝑦 and its mean.\n",
    "$$S_{yy} = \\sum\\limits_{i=1}^n{(y_i-\\bar{y})}^2=\\sum\\limits_{i=1}^n{y_i}-n\\bar{y}^2$$ <br>\n",
    "then the values for $\\hat{\\beta_0}$ and $\\hat{\\beta_1}$ minimizing $e$ or, equivalently, solving the\n",
    "normal equations are <br>\n",
    "$$\\hat{\\beta_0}=\\bar{y}-\\hat{\\beta_1}\\bar{x}$$ <br>\n",
    "$$\\hat{\\beta_1}=\\frac{S_{xy}}{S_{xx}}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The classic approach in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\beta_1 = \\frac{S_{xy}}{\\sigma_{x}^{2}}\\quad$  (slope/coefficient)\n",
    "\n",
    "\n",
    "$\\beta_0 = \\bar{y} - \\beta_1\\bar{x}\\quad$ (y-axis intercept)\n",
    "\n",
    "where \n",
    "\n",
    "\n",
    "$S_{xy} = \\sum_{i=1}^{n} (x_i - \\bar{x})(y_i - \\bar{y})\\quad$ (covariance)\n",
    "\n",
    "\n",
    "$\\sigma{_x}^{2} = \\sum_{i=1}^{n} (x_i - \\bar{x})^2\\quad$ (variance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Build a function to compute the least-squares solution to a linear matrix equation. \"\"\"\n",
    "\n",
    "def classic_lstsqr(x_list, y_list):\n",
    "    N = len(x_list)\n",
    "    x_avg = sum(x_list)/N\n",
    "    y_avg = sum(y_list)/N\n",
    "    var_x, cov_xy = 0, 0\n",
    "    for x,y in zip(x_list, y_list):\n",
    "        temp = x - x_avg\n",
    "        var_x += temp**2\n",
    "        cov_xy += temp * (y - y_avg)\n",
    "    slope = cov_xy / var_x\n",
    "    y_interc = y_avg - slope*x_avg\n",
    "    return (slope, y_interc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capital Asset Pricing Model (CAPM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/index.html\n",
    "\n",
    "We briefly explore the mathematical and explanatory description of key asset pricing models (i.e., CAPM, Fama-French 3 Factor, Fama-French 5 factor), and how to run these models in Python.\n",
    "\n",
    "All finance students should have some knowledge of one of the most important topics in finance, that is the Capital Asset Pricing Model (CAPM). The Capital Asset Pricing Model (CAPM) describes the relationship between systematic risk and expected return for assets, particularly stocks. CAPM is widely used throughout finance for pricing risky securities and generating expected returns for assets given the risk of those assets and cost of capital.\n",
    "\n",
    "The formula for calculating the expected return of an asset given its risk is as follows:\n",
    "\n",
    "### $ r_i = r_f + \\beta_i (r_m − r_f)$\n",
    "\n",
    "where:\n",
    "\n",
    "\n",
    "$r_i$ : The expected return of the investment in asset i. This can be the return of the any stock (i.e., Apple, Google, Tesla) or investment portfolio (i.e., any mutual/hedge fund portfolio).\n",
    "\n",
    "$r_f$: The return of the risk-free asset. The risk-free asset is usually given by the US 3-month Treasury bill. It is assumed that the US government will not default on a short-term government security, thus the US 3-month Treasury bill is widely assumed in finance to be risk-free. \n",
    "\n",
    "$r_m$ : The return of the market. This is usually given by the S&P500 return as it is the largest market index in the world.\n",
    "\n",
    "$(r_m − r_f)$ : Market risk premium\n",
    "\n",
    "\n",
    "Investors expect to be compensated for risk and the time value of money. The risk-free rate in the CAPM formula accounts for the time value of money. The other components of the CAPM formula account for the investor taking on additional risk.\n",
    "\n",
    "The beta of a potential investment is a measure of how much risk the investment will add to a portfolio that looks like the market. If a stock is riskier than the market, it will have a beta greater than one. If a stock has a beta of less than one, the formula assumes it will reduce the risk of a portfolio.\n",
    "\n",
    "A beta coefficient can measure the volatility of an individual stock compared to the systematic risk of the entire market. In statistical terms, beta represents the slope of the line through a regression of data points. In finance, each of these data points represents an individual stock's returns against those of the market as a whole.\n",
    "\n",
    "A stock’s beta is then multiplied by the market risk premium, which is the return expected from the market above the risk-free rate. The risk-free rate is then added to the product of the stock’s beta and the market risk premium. The result should give an investor the required return or discount rate they can use to find the value of an asset.\n",
    "\n",
    "The goal of the CAPM formula is to evaluate whether a stock is fairly valued when its risk and the time value of money are compared to its expected return.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Factor Investing and Fama-French model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FF3 Factor Asset Pricing Model\n",
    "\n",
    "The Fama and French Three-Factor Model (or the Fama French Model for short) is an asset pricing model developed in 1992 that expands on the capital asset pricing model (CAPM) by adding size risk and value risk factors to the market risk factor in CAPM. This model considers the fact that value and small-cap stocks outperform markets on a regular basis. By including these two additional factors, the model adjusts for this outperforming tendency, which is thought to make it a better tool for evaluating manager performance. The model is essentially the result of an econometric regression of historical stock prices.\n",
    "\n",
    "The Formula for the Fama French Model Is:\n",
    "\n",
    "### $ r_{it} - r_{ft} = \\alpha_{it} + \\beta_1 (r_{mt} − r_{ft}) + \\beta_2 SMB_t + \\beta_3 HML_t + \\epsilon_{it}$\n",
    "\n",
    "$SMB$ : Size premium (small minus big)\n",
    "\n",
    "$HML$ : Value premium (High minus low)\n",
    "\n",
    "\n",
    "SMB accounts for traded companies with small market caps that generate higher returns, while HML accounts for value stocks with high book-to-market ratios that generate higher returns in comparison to the market.\n",
    "\n",
    "\n",
    "#### FF5 Factor Asset Pricing Model\n",
    "\n",
    "The theoretical starting point for the five-factor model is the dividend discount model as the model states that the value of a stock today is dependent upon future dividends. Fama and French use the dividend discount model to get two new factors from it, investment and profitability (Fama and French, 2014).\n",
    "\n",
    "### $ r_{it} - r_{ft} = \\alpha_{it} + \\beta_1 (r_{mt} − r_{ft}) + \\beta_2 SMB_t + \\beta_3 HML_t + \\beta_4 RMW_t + \\beta_5 CMA_t + \\epsilon_{it}$\n",
    "\n",
    "$RMW$ : Return spread of the most profitable firms minus the least profitables\n",
    "\n",
    "$CMA$ : Return spread of firms that invest conservatively minus aggressively\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Analysis with statsmodels\n",
    "\n",
    "https://www.statsmodels.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "statsmodels provides a large range of cross-sectional models as well as some time-series models. Basic linear regression is provided by OLS. Estimating a model requires specifying the model and then calling a method to estimate the parameters (named fit in most models). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: yfinance in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (0.1.64)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Requirement already satisfied: pandas>=0.24 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from yfinance) (1.3.4)\n",
      "Requirement already satisfied: numpy>=1.15 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from yfinance) (1.22.0)\n",
      "Requirement already satisfied: requests>=2.20 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from yfinance) (2.26.0)\n",
      "Requirement already satisfied: multitasking>=0.0.7 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from yfinance) (0.0.9)\n",
      "Requirement already satisfied: lxml>=4.5.1 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from yfinance) (4.6.4)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas>=0.24->yfinance) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas>=0.24->yfinance) (2021.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.20->yfinance) (1.26.7)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.20->yfinance) (2.0.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.20->yfinance) (3.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.20->yfinance) (2021.10.8)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from python-dateutil>=2.7.3->pandas>=0.24->yfinance) (1.16.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "You should consider upgrading via the 'C:\\Users\\habibnia\\AppData\\Local\\Programs\\Python\\Python310\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas-datareader in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (0.10.0)\n",
      "Requirement already satisfied: lxml in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas-datareader) (4.6.4)\n",
      "Requirement already satisfied: requests>=2.19.0 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas-datareader) (2.26.0)\n",
      "Requirement already satisfied: pandas>=0.23 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas-datareader) (1.3.4)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas>=0.23->pandas-datareader) (2.8.2)\n",
      "Requirement already satisfied: numpy>=1.21.0 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas>=0.23->pandas-datareader) (1.22.0)\n",
      "Requirement already satisfied: pytz>=2017.3 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from pandas>=0.23->pandas-datareader) (2021.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.19.0->pandas-datareader) (2021.10.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.19.0->pandas-datareader) (3.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.19.0->pandas-datareader) (1.26.7)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests>=2.19.0->pandas-datareader) (2.0.7)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\habibnia\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from python-dateutil>=2.7.3->pandas>=0.23->pandas-datareader) (1.16.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.\n",
      "You should consider upgrading via the 'C:\\Users\\habibnia\\AppData\\Local\\Programs\\Python\\Python310\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "! pip install yfinance\n",
    "! pip install pandas-datareader --upgrade\n",
    "import yfinance as yf\n",
    "import bs4 as bs\n",
    "import numpy as np\n",
    "import os\n",
    "import requests\n",
    "import pandas as pd\n",
    "import datetime\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import statsmodels.formula.api as sm # module for stats models\n",
    "from statsmodels.iolib.summary2 import summary_col # module for presenting stats models outputs nicely\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[*******               14%                       ]  70 of 506 completed"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_45780/1222149664.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[0mtickers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtickers\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"SPY\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m;\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0myf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdownload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtickers\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstart\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'Adj Close'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     17\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     18\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\yfinance\\multi.py\u001b[0m in \u001b[0;36mdownload\u001b[1;34m(tickers, start, end, actions, threads, group_by, auto_adjust, back_adjust, progress, period, show_errors, interval, prepost, proxy, rounding, **kwargs)\u001b[0m\n\u001b[0;32m     95\u001b[0m                                    rounding=rounding)\n\u001b[0;32m     96\u001b[0m         \u001b[1;32mwhile\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mshared\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_DFS\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m<\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtickers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 97\u001b[1;33m             \u001b[0m_time\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m0.01\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     98\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     99\u001b[0m     \u001b[1;31m# download synchronously\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[*******               14%                       ]  71 of 506 completed"
     ]
    }
   ],
   "source": [
    "### Historical Stock Price Data\n",
    "\n",
    "resp = requests.get('http://en.wikipedia.org/wiki/List_of_S%26P_500_companies')\n",
    "soup = bs.BeautifulSoup(resp.text, 'lxml')\n",
    "table = soup.find('table', {'class': 'wikitable sortable'})\n",
    "tickers = []\n",
    "for row in table.findAll('tr')[1:]:\n",
    "    ticker = row.findAll('td')[0].text\n",
    "    tickers.append(ticker)\n",
    "\n",
    "tickers = [s.replace('\\n', '') for s in tickers]\n",
    "start = datetime.datetime(2006,1,1)\n",
    "end = datetime.datetime(2020,6,30)\n",
    "tickers=tickers+[\"SPY\"];\n",
    "\n",
    "data = yf.download(tickers, start=start, end=end)['Adj Close']\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('Stock_Data'):\n",
    "    os.makedirs('Stock_Data')\n",
    "data.to_csv('Stock_Data/SP500Prices.csv')\n",
    "data.to_hdf('Stock_Data/SP500Prices.h5', 'fixed', mode='w', complib='blosc', complevel=9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>AAL</th>\n",
       "      <th>AAP</th>\n",
       "      <th>AAPL</th>\n",
       "      <th>ABC</th>\n",
       "      <th>ABMD</th>\n",
       "      <th>ABT</th>\n",
       "      <th>ACN</th>\n",
       "      <th>ADBE</th>\n",
       "      <th>ADI</th>\n",
       "      <th>...</th>\n",
       "      <th>WYNN</th>\n",
       "      <th>XEL</th>\n",
       "      <th>XLNX</th>\n",
       "      <th>XOM</th>\n",
       "      <th>XRAY</th>\n",
       "      <th>XRX</th>\n",
       "      <th>YUM</th>\n",
       "      <th>ZBH</th>\n",
       "      <th>ZBRA</th>\n",
       "      <th>ZION</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006-01-03</th>\n",
       "      <td>20.752956</td>\n",
       "      <td>35.305672</td>\n",
       "      <td>41.007545</td>\n",
       "      <td>2.302969</td>\n",
       "      <td>16.858438</td>\n",
       "      <td>9.350000</td>\n",
       "      <td>13.123610</td>\n",
       "      <td>22.026203</td>\n",
       "      <td>38.520000</td>\n",
       "      <td>24.942032</td>\n",
       "      <td>...</td>\n",
       "      <td>29.771486</td>\n",
       "      <td>10.370291</td>\n",
       "      <td>18.501081</td>\n",
       "      <td>36.244297</td>\n",
       "      <td>24.628942</td>\n",
       "      <td>27.373835</td>\n",
       "      <td>12.616449</td>\n",
       "      <td>63.187962</td>\n",
       "      <td>42.830002</td>\n",
       "      <td>61.661987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-04</th>\n",
       "      <td>20.808710</td>\n",
       "      <td>36.955460</td>\n",
       "      <td>41.299648</td>\n",
       "      <td>2.309747</td>\n",
       "      <td>16.746733</td>\n",
       "      <td>9.620000</td>\n",
       "      <td>13.143523</td>\n",
       "      <td>22.146482</td>\n",
       "      <td>38.419998</td>\n",
       "      <td>25.173103</td>\n",
       "      <td>...</td>\n",
       "      <td>29.716314</td>\n",
       "      <td>10.420551</td>\n",
       "      <td>19.333887</td>\n",
       "      <td>36.306290</td>\n",
       "      <td>24.856865</td>\n",
       "      <td>27.337107</td>\n",
       "      <td>12.724398</td>\n",
       "      <td>63.787724</td>\n",
       "      <td>42.410000</td>\n",
       "      <td>62.097343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-05</th>\n",
       "      <td>21.353859</td>\n",
       "      <td>37.436264</td>\n",
       "      <td>41.497532</td>\n",
       "      <td>2.291571</td>\n",
       "      <td>16.539299</td>\n",
       "      <td>9.550000</td>\n",
       "      <td>13.312750</td>\n",
       "      <td>22.311872</td>\n",
       "      <td>38.070000</td>\n",
       "      <td>25.975050</td>\n",
       "      <td>...</td>\n",
       "      <td>29.539719</td>\n",
       "      <td>10.414965</td>\n",
       "      <td>20.496935</td>\n",
       "      <td>36.126522</td>\n",
       "      <td>24.574240</td>\n",
       "      <td>27.043371</td>\n",
       "      <td>13.266838</td>\n",
       "      <td>63.114136</td>\n",
       "      <td>42.529999</td>\n",
       "      <td>62.661716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-06</th>\n",
       "      <td>21.465374</td>\n",
       "      <td>36.766911</td>\n",
       "      <td>41.450405</td>\n",
       "      <td>2.350724</td>\n",
       "      <td>16.355791</td>\n",
       "      <td>9.750000</td>\n",
       "      <td>13.568257</td>\n",
       "      <td>23.416941</td>\n",
       "      <td>39.000000</td>\n",
       "      <td>26.076996</td>\n",
       "      <td>...</td>\n",
       "      <td>29.821154</td>\n",
       "      <td>10.454056</td>\n",
       "      <td>20.906160</td>\n",
       "      <td>36.839401</td>\n",
       "      <td>24.957150</td>\n",
       "      <td>26.878132</td>\n",
       "      <td>13.229057</td>\n",
       "      <td>62.791172</td>\n",
       "      <td>44.119999</td>\n",
       "      <td>63.314800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-09</th>\n",
       "      <td>21.403425</td>\n",
       "      <td>36.399246</td>\n",
       "      <td>41.987503</td>\n",
       "      <td>2.343021</td>\n",
       "      <td>16.355791</td>\n",
       "      <td>10.150000</td>\n",
       "      <td>14.072625</td>\n",
       "      <td>23.349283</td>\n",
       "      <td>38.380001</td>\n",
       "      <td>26.443987</td>\n",
       "      <td>...</td>\n",
       "      <td>31.228340</td>\n",
       "      <td>10.426137</td>\n",
       "      <td>20.884611</td>\n",
       "      <td>36.820793</td>\n",
       "      <td>25.203304</td>\n",
       "      <td>27.245319</td>\n",
       "      <td>13.331607</td>\n",
       "      <td>65.014938</td>\n",
       "      <td>44.790001</td>\n",
       "      <td>62.790722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-06-23</th>\n",
       "      <td>88.932564</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>149.495514</td>\n",
       "      <td>91.310051</td>\n",
       "      <td>100.665817</td>\n",
       "      <td>252.580002</td>\n",
       "      <td>90.644806</td>\n",
       "      <td>205.836899</td>\n",
       "      <td>440.549988</td>\n",
       "      <td>122.075447</td>\n",
       "      <td>...</td>\n",
       "      <td>84.510002</td>\n",
       "      <td>63.369690</td>\n",
       "      <td>94.635109</td>\n",
       "      <td>44.717655</td>\n",
       "      <td>44.475010</td>\n",
       "      <td>15.669960</td>\n",
       "      <td>87.100418</td>\n",
       "      <td>125.274361</td>\n",
       "      <td>260.769989</td>\n",
       "      <td>34.837280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-06-24</th>\n",
       "      <td>86.223152</td>\n",
       "      <td>13.040000</td>\n",
       "      <td>142.547577</td>\n",
       "      <td>89.698242</td>\n",
       "      <td>98.027679</td>\n",
       "      <td>241.660004</td>\n",
       "      <td>86.673508</td>\n",
       "      <td>200.318817</td>\n",
       "      <td>431.679993</td>\n",
       "      <td>118.331467</td>\n",
       "      <td>...</td>\n",
       "      <td>75.209999</td>\n",
       "      <td>62.866207</td>\n",
       "      <td>92.290344</td>\n",
       "      <td>42.611038</td>\n",
       "      <td>42.577591</td>\n",
       "      <td>14.844720</td>\n",
       "      <td>85.753098</td>\n",
       "      <td>117.576492</td>\n",
       "      <td>250.500000</td>\n",
       "      <td>32.618225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-06-25</th>\n",
       "      <td>86.920433</td>\n",
       "      <td>13.170000</td>\n",
       "      <td>142.457855</td>\n",
       "      <td>90.889038</td>\n",
       "      <td>99.842644</td>\n",
       "      <td>241.750000</td>\n",
       "      <td>88.976860</td>\n",
       "      <td>215.682159</td>\n",
       "      <td>436.950012</td>\n",
       "      <td>118.955460</td>\n",
       "      <td>...</td>\n",
       "      <td>73.559998</td>\n",
       "      <td>62.096176</td>\n",
       "      <td>92.687759</td>\n",
       "      <td>43.252594</td>\n",
       "      <td>42.637337</td>\n",
       "      <td>15.017443</td>\n",
       "      <td>85.198318</td>\n",
       "      <td>114.672386</td>\n",
       "      <td>253.639999</td>\n",
       "      <td>33.629562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-06-26</th>\n",
       "      <td>85.565720</td>\n",
       "      <td>12.380000</td>\n",
       "      <td>139.547089</td>\n",
       "      <td>88.096405</td>\n",
       "      <td>98.364883</td>\n",
       "      <td>239.270004</td>\n",
       "      <td>88.073387</td>\n",
       "      <td>210.868698</td>\n",
       "      <td>426.920013</td>\n",
       "      <td>116.746719</td>\n",
       "      <td>...</td>\n",
       "      <td>69.050003</td>\n",
       "      <td>61.513714</td>\n",
       "      <td>90.432411</td>\n",
       "      <td>41.768391</td>\n",
       "      <td>42.936058</td>\n",
       "      <td>14.451291</td>\n",
       "      <td>83.999603</td>\n",
       "      <td>115.041138</td>\n",
       "      <td>246.419998</td>\n",
       "      <td>31.705069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-06-29</th>\n",
       "      <td>87.132896</td>\n",
       "      <td>13.320000</td>\n",
       "      <td>142.298355</td>\n",
       "      <td>90.126732</td>\n",
       "      <td>98.176453</td>\n",
       "      <td>240.220001</td>\n",
       "      <td>88.371239</td>\n",
       "      <td>211.116806</td>\n",
       "      <td>424.200012</td>\n",
       "      <td>117.865944</td>\n",
       "      <td>...</td>\n",
       "      <td>74.220001</td>\n",
       "      <td>61.661797</td>\n",
       "      <td>91.366348</td>\n",
       "      <td>42.438679</td>\n",
       "      <td>43.722687</td>\n",
       "      <td>14.929422</td>\n",
       "      <td>86.169189</td>\n",
       "      <td>116.506187</td>\n",
       "      <td>252.050003</td>\n",
       "      <td>32.451302</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3647 rows × 423 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    A        AAL         AAP       AAPL         ABC  \\\n",
       "Date                                                                  \n",
       "2006-01-03  20.752956  35.305672   41.007545   2.302969   16.858438   \n",
       "2006-01-04  20.808710  36.955460   41.299648   2.309747   16.746733   \n",
       "2006-01-05  21.353859  37.436264   41.497532   2.291571   16.539299   \n",
       "2006-01-06  21.465374  36.766911   41.450405   2.350724   16.355791   \n",
       "2006-01-09  21.403425  36.399246   41.987503   2.343021   16.355791   \n",
       "...               ...        ...         ...        ...         ...   \n",
       "2020-06-23  88.932564  14.000000  149.495514  91.310051  100.665817   \n",
       "2020-06-24  86.223152  13.040000  142.547577  89.698242   98.027679   \n",
       "2020-06-25  86.920433  13.170000  142.457855  90.889038   99.842644   \n",
       "2020-06-26  85.565720  12.380000  139.547089  88.096405   98.364883   \n",
       "2020-06-29  87.132896  13.320000  142.298355  90.126732   98.176453   \n",
       "\n",
       "                  ABMD        ABT         ACN        ADBE         ADI  ...  \\\n",
       "Date                                                                   ...   \n",
       "2006-01-03    9.350000  13.123610   22.026203   38.520000   24.942032  ...   \n",
       "2006-01-04    9.620000  13.143523   22.146482   38.419998   25.173103  ...   \n",
       "2006-01-05    9.550000  13.312750   22.311872   38.070000   25.975050  ...   \n",
       "2006-01-06    9.750000  13.568257   23.416941   39.000000   26.076996  ...   \n",
       "2006-01-09   10.150000  14.072625   23.349283   38.380001   26.443987  ...   \n",
       "...                ...        ...         ...         ...         ...  ...   \n",
       "2020-06-23  252.580002  90.644806  205.836899  440.549988  122.075447  ...   \n",
       "2020-06-24  241.660004  86.673508  200.318817  431.679993  118.331467  ...   \n",
       "2020-06-25  241.750000  88.976860  215.682159  436.950012  118.955460  ...   \n",
       "2020-06-26  239.270004  88.073387  210.868698  426.920013  116.746719  ...   \n",
       "2020-06-29  240.220001  88.371239  211.116806  424.200012  117.865944  ...   \n",
       "\n",
       "                 WYNN        XEL       XLNX        XOM       XRAY        XRX  \\\n",
       "Date                                                                           \n",
       "2006-01-03  29.771486  10.370291  18.501081  36.244297  24.628942  27.373835   \n",
       "2006-01-04  29.716314  10.420551  19.333887  36.306290  24.856865  27.337107   \n",
       "2006-01-05  29.539719  10.414965  20.496935  36.126522  24.574240  27.043371   \n",
       "2006-01-06  29.821154  10.454056  20.906160  36.839401  24.957150  26.878132   \n",
       "2006-01-09  31.228340  10.426137  20.884611  36.820793  25.203304  27.245319   \n",
       "...               ...        ...        ...        ...        ...        ...   \n",
       "2020-06-23  84.510002  63.369690  94.635109  44.717655  44.475010  15.669960   \n",
       "2020-06-24  75.209999  62.866207  92.290344  42.611038  42.577591  14.844720   \n",
       "2020-06-25  73.559998  62.096176  92.687759  43.252594  42.637337  15.017443   \n",
       "2020-06-26  69.050003  61.513714  90.432411  41.768391  42.936058  14.451291   \n",
       "2020-06-29  74.220001  61.661797  91.366348  42.438679  43.722687  14.929422   \n",
       "\n",
       "                  YUM         ZBH        ZBRA       ZION  \n",
       "Date                                                      \n",
       "2006-01-03  12.616449   63.187962   42.830002  61.661987  \n",
       "2006-01-04  12.724398   63.787724   42.410000  62.097343  \n",
       "2006-01-05  13.266838   63.114136   42.529999  62.661716  \n",
       "2006-01-06  13.229057   62.791172   44.119999  63.314800  \n",
       "2006-01-09  13.331607   65.014938   44.790001  62.790722  \n",
       "...               ...         ...         ...        ...  \n",
       "2020-06-23  87.100418  125.274361  260.769989  34.837280  \n",
       "2020-06-24  85.753098  117.576492  250.500000  32.618225  \n",
       "2020-06-25  85.198318  114.672386  253.639999  33.629562  \n",
       "2020-06-26  83.999603  115.041138  246.419998  31.705069  \n",
       "2020-06-29  86.169189  116.506187  252.050003  32.451302  \n",
       "\n",
       "[3647 rows x 423 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaned_data = data.copy()\n",
    "\n",
    "# Remove companies (columns) with all missing values for whole time range\n",
    "cleaned_data.dropna(axis='columns', how='all', inplace=True)\n",
    "\n",
    "# Remove days (rows) with missing values for all of companies\n",
    "cleaned_data.dropna(axis='index', how='all', inplace=True)\n",
    "\n",
    "# Finally, remove the columns with at least one Nan (missing value)\n",
    "cleaned_data.dropna(axis='columns', how='any', inplace=True)\n",
    "\n",
    "cleaned_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3646, 423)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>AAL</th>\n",
       "      <th>AAP</th>\n",
       "      <th>AAPL</th>\n",
       "      <th>ABC</th>\n",
       "      <th>ABMD</th>\n",
       "      <th>ABT</th>\n",
       "      <th>ACN</th>\n",
       "      <th>ADBE</th>\n",
       "      <th>ADI</th>\n",
       "      <th>...</th>\n",
       "      <th>WYNN</th>\n",
       "      <th>XEL</th>\n",
       "      <th>XLNX</th>\n",
       "      <th>XOM</th>\n",
       "      <th>XRAY</th>\n",
       "      <th>XRX</th>\n",
       "      <th>YUM</th>\n",
       "      <th>ZBH</th>\n",
       "      <th>ZBRA</th>\n",
       "      <th>ZION</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006-01-04</th>\n",
       "      <td>0.002683</td>\n",
       "      <td>0.045670</td>\n",
       "      <td>0.007098</td>\n",
       "      <td>0.002939</td>\n",
       "      <td>-0.006648</td>\n",
       "      <td>0.028468</td>\n",
       "      <td>0.001516</td>\n",
       "      <td>0.005446</td>\n",
       "      <td>-0.002599</td>\n",
       "      <td>0.009222</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.001855</td>\n",
       "      <td>0.004835</td>\n",
       "      <td>0.044030</td>\n",
       "      <td>0.001709</td>\n",
       "      <td>0.009212</td>\n",
       "      <td>-0.001343</td>\n",
       "      <td>0.008520</td>\n",
       "      <td>0.009447</td>\n",
       "      <td>-0.009855</td>\n",
       "      <td>0.007036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-05</th>\n",
       "      <td>0.025861</td>\n",
       "      <td>0.012926</td>\n",
       "      <td>0.004780</td>\n",
       "      <td>-0.007901</td>\n",
       "      <td>-0.012464</td>\n",
       "      <td>-0.007303</td>\n",
       "      <td>0.012793</td>\n",
       "      <td>0.007440</td>\n",
       "      <td>-0.009152</td>\n",
       "      <td>0.031360</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.005960</td>\n",
       "      <td>-0.000536</td>\n",
       "      <td>0.058416</td>\n",
       "      <td>-0.004964</td>\n",
       "      <td>-0.011435</td>\n",
       "      <td>-0.010803</td>\n",
       "      <td>0.041746</td>\n",
       "      <td>-0.010616</td>\n",
       "      <td>0.002826</td>\n",
       "      <td>0.009047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-06</th>\n",
       "      <td>0.005209</td>\n",
       "      <td>-0.018042</td>\n",
       "      <td>-0.001136</td>\n",
       "      <td>0.025486</td>\n",
       "      <td>-0.011157</td>\n",
       "      <td>0.020726</td>\n",
       "      <td>0.019011</td>\n",
       "      <td>0.048341</td>\n",
       "      <td>0.024135</td>\n",
       "      <td>0.003917</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009482</td>\n",
       "      <td>0.003746</td>\n",
       "      <td>0.019769</td>\n",
       "      <td>0.019541</td>\n",
       "      <td>0.015462</td>\n",
       "      <td>-0.006129</td>\n",
       "      <td>-0.002852</td>\n",
       "      <td>-0.005130</td>\n",
       "      <td>0.036703</td>\n",
       "      <td>0.010368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-09</th>\n",
       "      <td>-0.002890</td>\n",
       "      <td>-0.010050</td>\n",
       "      <td>0.012874</td>\n",
       "      <td>-0.003282</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.040206</td>\n",
       "      <td>0.036498</td>\n",
       "      <td>-0.002893</td>\n",
       "      <td>-0.016025</td>\n",
       "      <td>0.013975</td>\n",
       "      <td>...</td>\n",
       "      <td>0.046108</td>\n",
       "      <td>-0.002674</td>\n",
       "      <td>-0.001031</td>\n",
       "      <td>-0.000505</td>\n",
       "      <td>0.009815</td>\n",
       "      <td>0.013569</td>\n",
       "      <td>0.007722</td>\n",
       "      <td>0.034803</td>\n",
       "      <td>0.015072</td>\n",
       "      <td>-0.008312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-10</th>\n",
       "      <td>0.012941</td>\n",
       "      <td>-0.021202</td>\n",
       "      <td>0.001121</td>\n",
       "      <td>0.061328</td>\n",
       "      <td>0.000244</td>\n",
       "      <td>0.005894</td>\n",
       "      <td>-0.009715</td>\n",
       "      <td>-0.021807</td>\n",
       "      <td>0.005975</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001236</td>\n",
       "      <td>0.004809</td>\n",
       "      <td>0.005485</td>\n",
       "      <td>0.007714</td>\n",
       "      <td>-0.008537</td>\n",
       "      <td>-0.005406</td>\n",
       "      <td>-0.000810</td>\n",
       "      <td>-0.004266</td>\n",
       "      <td>0.036607</td>\n",
       "      <td>-0.011753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-11</th>\n",
       "      <td>-0.000286</td>\n",
       "      <td>-0.082691</td>\n",
       "      <td>-0.008554</td>\n",
       "      <td>0.036906</td>\n",
       "      <td>-0.000244</td>\n",
       "      <td>0.032758</td>\n",
       "      <td>-0.007095</td>\n",
       "      <td>0.008193</td>\n",
       "      <td>0.018476</td>\n",
       "      <td>0.004871</td>\n",
       "      <td>...</td>\n",
       "      <td>0.029562</td>\n",
       "      <td>-0.007491</td>\n",
       "      <td>-0.006517</td>\n",
       "      <td>0.006826</td>\n",
       "      <td>0.022724</td>\n",
       "      <td>0.024098</td>\n",
       "      <td>-0.008053</td>\n",
       "      <td>0.025748</td>\n",
       "      <td>-0.009516</td>\n",
       "      <td>0.005829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-12</th>\n",
       "      <td>-0.022252</td>\n",
       "      <td>0.005445</td>\n",
       "      <td>0.002935</td>\n",
       "      <td>0.004637</td>\n",
       "      <td>0.007774</td>\n",
       "      <td>0.015052</td>\n",
       "      <td>-0.002174</td>\n",
       "      <td>-0.001633</td>\n",
       "      <td>0.003807</td>\n",
       "      <td>-0.008476</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.009469</td>\n",
       "      <td>0.010152</td>\n",
       "      <td>-0.008640</td>\n",
       "      <td>-0.010508</td>\n",
       "      <td>-0.001606</td>\n",
       "      <td>0.014445</td>\n",
       "      <td>-0.016304</td>\n",
       "      <td>0.005956</td>\n",
       "      <td>0.010591</td>\n",
       "      <td>-0.015227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-13</th>\n",
       "      <td>-0.007627</td>\n",
       "      <td>-0.027233</td>\n",
       "      <td>0.000224</td>\n",
       "      <td>0.015305</td>\n",
       "      <td>0.002417</td>\n",
       "      <td>-0.041951</td>\n",
       "      <td>-0.003635</td>\n",
       "      <td>-0.012500</td>\n",
       "      <td>-0.016344</td>\n",
       "      <td>0.006428</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009298</td>\n",
       "      <td>0.000531</td>\n",
       "      <td>-0.000694</td>\n",
       "      <td>0.022056</td>\n",
       "      <td>0.004988</td>\n",
       "      <td>-0.022414</td>\n",
       "      <td>-0.000208</td>\n",
       "      <td>-0.025453</td>\n",
       "      <td>-0.007554</td>\n",
       "      <td>0.010308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-17</th>\n",
       "      <td>-0.005907</td>\n",
       "      <td>-0.126633</td>\n",
       "      <td>-0.019572</td>\n",
       "      <td>-0.010335</td>\n",
       "      <td>-0.012390</td>\n",
       "      <td>-0.036692</td>\n",
       "      <td>-0.002918</td>\n",
       "      <td>0.020639</td>\n",
       "      <td>0.015330</td>\n",
       "      <td>-0.031236</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.025164</td>\n",
       "      <td>0.011097</td>\n",
       "      <td>-0.013639</td>\n",
       "      <td>0.009306</td>\n",
       "      <td>-0.004096</td>\n",
       "      <td>-0.024976</td>\n",
       "      <td>-0.012355</td>\n",
       "      <td>0.001416</td>\n",
       "      <td>-0.020132</td>\n",
       "      <td>-0.004815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-18</th>\n",
       "      <td>0.020523</td>\n",
       "      <td>0.032790</td>\n",
       "      <td>-0.008308</td>\n",
       "      <td>-0.026556</td>\n",
       "      <td>0.003904</td>\n",
       "      <td>0.045417</td>\n",
       "      <td>0.003646</td>\n",
       "      <td>0.004852</td>\n",
       "      <td>0.007326</td>\n",
       "      <td>-0.004505</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.022932</td>\n",
       "      <td>0.002624</td>\n",
       "      <td>0.013986</td>\n",
       "      <td>-0.014073</td>\n",
       "      <td>0.002139</td>\n",
       "      <td>0.001366</td>\n",
       "      <td>0.005673</td>\n",
       "      <td>0.009993</td>\n",
       "      <td>-0.006876</td>\n",
       "      <td>0.002345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 423 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   A       AAL       AAP      AAPL       ABC      ABMD  \\\n",
       "Date                                                                     \n",
       "2006-01-04  0.002683  0.045670  0.007098  0.002939 -0.006648  0.028468   \n",
       "2006-01-05  0.025861  0.012926  0.004780 -0.007901 -0.012464 -0.007303   \n",
       "2006-01-06  0.005209 -0.018042 -0.001136  0.025486 -0.011157  0.020726   \n",
       "2006-01-09 -0.002890 -0.010050  0.012874 -0.003282  0.000000  0.040206   \n",
       "2006-01-10  0.012941 -0.021202  0.001121  0.061328  0.000244  0.005894   \n",
       "2006-01-11 -0.000286 -0.082691 -0.008554  0.036906 -0.000244  0.032758   \n",
       "2006-01-12 -0.022252  0.005445  0.002935  0.004637  0.007774  0.015052   \n",
       "2006-01-13 -0.007627 -0.027233  0.000224  0.015305  0.002417 -0.041951   \n",
       "2006-01-17 -0.005907 -0.126633 -0.019572 -0.010335 -0.012390 -0.036692   \n",
       "2006-01-18  0.020523  0.032790 -0.008308 -0.026556  0.003904  0.045417   \n",
       "\n",
       "                 ABT       ACN      ADBE       ADI  ...      WYNN       XEL  \\\n",
       "Date                                                ...                       \n",
       "2006-01-04  0.001516  0.005446 -0.002599  0.009222  ... -0.001855  0.004835   \n",
       "2006-01-05  0.012793  0.007440 -0.009152  0.031360  ... -0.005960 -0.000536   \n",
       "2006-01-06  0.019011  0.048341  0.024135  0.003917  ...  0.009482  0.003746   \n",
       "2006-01-09  0.036498 -0.002893 -0.016025  0.013975  ...  0.046108 -0.002674   \n",
       "2006-01-10 -0.009715 -0.021807  0.005975  0.000000  ...  0.001236  0.004809   \n",
       "2006-01-11 -0.007095  0.008193  0.018476  0.004871  ...  0.029562 -0.007491   \n",
       "2006-01-12 -0.002174 -0.001633  0.003807 -0.008476  ... -0.009469  0.010152   \n",
       "2006-01-13 -0.003635 -0.012500 -0.016344  0.006428  ...  0.009298  0.000531   \n",
       "2006-01-17 -0.002918  0.020639  0.015330 -0.031236  ... -0.025164  0.011097   \n",
       "2006-01-18  0.003646  0.004852  0.007326 -0.004505  ... -0.022932  0.002624   \n",
       "\n",
       "                XLNX       XOM      XRAY       XRX       YUM       ZBH  \\\n",
       "Date                                                                     \n",
       "2006-01-04  0.044030  0.001709  0.009212 -0.001343  0.008520  0.009447   \n",
       "2006-01-05  0.058416 -0.004964 -0.011435 -0.010803  0.041746 -0.010616   \n",
       "2006-01-06  0.019769  0.019541  0.015462 -0.006129 -0.002852 -0.005130   \n",
       "2006-01-09 -0.001031 -0.000505  0.009815  0.013569  0.007722  0.034803   \n",
       "2006-01-10  0.005485  0.007714 -0.008537 -0.005406 -0.000810 -0.004266   \n",
       "2006-01-11 -0.006517  0.006826  0.022724  0.024098 -0.008053  0.025748   \n",
       "2006-01-12 -0.008640 -0.010508 -0.001606  0.014445 -0.016304  0.005956   \n",
       "2006-01-13 -0.000694  0.022056  0.004988 -0.022414 -0.000208 -0.025453   \n",
       "2006-01-17 -0.013639  0.009306 -0.004096 -0.024976 -0.012355  0.001416   \n",
       "2006-01-18  0.013986 -0.014073  0.002139  0.001366  0.005673  0.009993   \n",
       "\n",
       "                ZBRA      ZION  \n",
       "Date                            \n",
       "2006-01-04 -0.009855  0.007036  \n",
       "2006-01-05  0.002826  0.009047  \n",
       "2006-01-06  0.036703  0.010368  \n",
       "2006-01-09  0.015072 -0.008312  \n",
       "2006-01-10  0.036607 -0.011753  \n",
       "2006-01-11 -0.009516  0.005829  \n",
       "2006-01-12  0.010591 -0.015227  \n",
       "2006-01-13 -0.007554  0.010308  \n",
       "2006-01-17 -0.020132 -0.004815  \n",
       "2006-01-18 -0.006876  0.002345  \n",
       "\n",
       "[10 rows x 423 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This converts prices to arithmetic or log returns.\n",
    "\n",
    "prices = cleaned_data\n",
    "return_values = np.log(prices).diff()\n",
    "return_values = return_values.iloc[1:]  # removes first row which is NaN after diff()\n",
    "print(return_values.shape)\n",
    "return_values.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def assetPriceReg(df_stk):\n",
    "    import pandas_datareader.data as web  # module for reading datasets directly from the web\n",
    "    \n",
    "    # Reading in factor data\n",
    "    df_factors = web.DataReader('F-F_Research_Data_5_Factors_2x3_daily', 'famafrench')[0]\n",
    "    df_factors.rename(columns={'Mkt-RF': 'MKT'}, inplace=True)\n",
    "    df_factors['MKT'] = df_factors['MKT']/100\n",
    "    df_factors['SMB'] = df_factors['SMB']/100\n",
    "    df_factors['HML'] = df_factors['HML']/100\n",
    "    df_factors['RMW'] = df_factors['RMW']/100\n",
    "    df_factors['CMA'] = df_factors['CMA']/100\n",
    "    \n",
    "    df_stock_factor = pd.merge(df_stk,df_factors,left_index=True,right_index=True) # Merging the stock and factor returns dataframes together\n",
    "    df_stock_factor['XsRet'] = df_stock_factor['Returns'] - df_stock_factor['RF'] # Calculating excess returns\n",
    "\n",
    "    # Running CAPM, FF3, and FF5 models.\n",
    "    CAPM = sm.ols(formula = 'XsRet ~ MKT', data=df_stock_factor).fit(cov_type='HAC',cov_kwds={'maxlags':1})\n",
    "    FF3 = sm.ols( formula = 'XsRet ~ MKT + SMB + HML', data=df_stock_factor).fit(cov_type='HAC',cov_kwds={'maxlags':1})\n",
    "    FF5 = sm.ols( formula = 'XsRet ~ MKT + SMB + HML + RMW + CMA', data=df_stock_factor).fit(cov_type='HAC',cov_kwds={'maxlags':1})\n",
    "\n",
    "    CAPMtstat = CAPM.tvalues\n",
    "    FF3tstat = FF3.tvalues\n",
    "    FF5tstat = FF5.tvalues\n",
    "\n",
    "    CAPMcoeff = CAPM.params\n",
    "    FF3coeff = FF3.params\n",
    "    FF5coeff = FF5.params\n",
    "\n",
    "    # DataFrame with coefficients and t-stats\n",
    "    results_df = pd.DataFrame({'CAPMcoeff':CAPMcoeff,'CAPMtstat':CAPMtstat,\n",
    "                               'FF3coeff':FF3coeff, 'FF3tstat':FF3tstat,\n",
    "                               'FF5coeff':FF5coeff, 'FF5tstat':FF5tstat},\n",
    "    index = ['Intercept', 'MKT', 'SMB', 'HML', 'RMW', 'CMA'])\n",
    "\n",
    "\n",
    "    dfoutput = summary_col([CAPM,FF3, FF5],stars=True,float_format='%0.4f',\n",
    "                  model_names=['CAPM','FF3','FF5'],\n",
    "                  info_dict={'N':lambda x: \"{0:d}\".format(int(x.nobs)),\n",
    "                             'Adjusted R2':lambda x: \"{:.4f}\".format(x.rsquared_adj)}, \n",
    "                             regressor_order = ['Intercept', 'MKT', 'SMB', 'HML', 'RMW', 'CMA'])\n",
    "\n",
    "    print(dfoutput)\n",
    "    \n",
    "    return results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[*********************100%***********************]  1 of 1 completed\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'AAPL'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_45780/2288097120.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mdf_stk\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0myf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdownload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'AAPL'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstart\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mdf_stk\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\yfinance\\multi.py\u001b[0m in \u001b[0;36mdownload\u001b[1;34m(tickers, start, end, actions, threads, group_by, auto_adjust, back_adjust, progress, period, show_errors, interval, prepost, proxy, rounding, **kwargs)\u001b[0m\n\u001b[0;32m    120\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    121\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtickers\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 122\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mshared\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_DFS\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mtickers\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    123\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    124\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: 'AAPL'"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************1900%***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************]  19 of 1 completed"
     ]
    }
   ],
   "source": [
    "df_stk = yf.download('AAPL', start=start, end=end)\n",
    "df_stk.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a2331e080>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEACAYAAACj0I2EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd8VFX6+PHPmcmk90JCSAgtAQKhhl6MIMiKvYEFRF3L2lgrsvqzd3dta1vUVVcRWHHFgn5dVkCKjSLSAqFDSCAhhfQ2Ob8/7jDJmIQEMpmZxOf9evHizrnn3vtMYJ6cOffcc5TWGiGEEO2fyd0BCCGEcA5J6EII0UFIQhdCiA5CEroQQnQQktCFEKKDkIQuhBAdhCR0IYToICShCyFEByEJXQghOghJ6EII0UF4ufJikZGRulu3bq68pBBCtHsbNmw4prWOaq6eSxN6t27dWL9+vSsvKYQQ7Z5S6kBL6kmXixBCdBCS0IUQooOQhC6EEB2ES/vQG1NdXU1mZiYVFRXuDqXd8vX1JS4uDovF4u5QhBBu5PaEnpmZSVBQEN26dUMp5e5w2h2tNXl5eWRmZtK9e3d3hyOEcCO3d7lUVFQQEREhyfw0KaWIiIiQbzhCeKrKYigvcMml3J7QAUnmrSQ/PyE82Iv94NluLrmURyR0d8vMzOSCCy4gMTGRnj17Mnv2bKqqqtwdlhCiI6g47rJL/e4Tutaaiy++mAsvvJBdu3aRkZFBSUkJDzzwgLtDE0J0ANWlZiqPu+Z2pdtvirrb8uXL8fX15dprrwXAbDbz4osv0r17d7p3784333xDZWUl+/bt48orr+Thhx8G4MMPP+SVV16hqqqKESNG8Prrr2M2mwkMDGT27Nl8+eWX+Pn58dlnnxEdHe3OtyiEcKO1P51DlSWIyS64lkcl9Ee/2Mb2rCKnnjM5NpiHz+vX5P5t27YxdOhQh7Lg4GC6du1KTU0NP//8M1u3bsXf359hw4YxdepUAgICWLRoEWvXrsVisXDLLbcwf/58Zs6cSWlpKSNHjuTJJ5/kvvvu46233uLBBx906nsSQrQf25KvA+Cs0gJMAWFtei2PSujuoLVu9KbiifJJkyYREREBwMUXX8yaNWvw8vJiw4YNDBs2DIDy8nI6deoEgLe3N+eeey4AQ4cOZdmyZS56J0IIT1awbh0RaW3bTveohH6ylnRb6devH5988olDWVFREYcOHcJsNjdI9koptNZcc801PP300w3OZ7FY7MeYzWZqamraLnghRLtR6teFiDa+xu/+pujEiRMpKyvjX//6FwBWq5W7776bWbNm4e/vz7Jly8jPz6e8vJwlS5YwZswYJk6cyOLFi8nJyQEgPz+fAwdaNBmaEOJ3SgcFt/k1fvcJXSnFp59+yscff0xiYiJJSUn4+vry1FNPATB27FhmzJjBoEGDuOSSS0hNTSU5OZknnniCyZMnM2DAACZNmkR2drab34kQwtNYc3fZt2v9fNv8eh7V5eIu8fHxfPHFF43u69SpE6+++mqD8mnTpjFt2rQG5SUlJfbtSy+9lEsvvdR5gQoh2pXKl84HXgOgxte7za/3u2+hCyFEW1n5yzn27QpT2z/+Ly30k5g1axazZs1ydxhCiHZqX9ep9u3oIP82v5600IUQoo3V6gI6+Xdq8+tIQhdCiDZWylqXXEcSuhBCtDGzCnDJdSShCyFEG7NafFxyHUnoQGBgoMPr9957j9tuuw2AN9980/7QUVPq1xdCiN8q93bNE+MyyqUZN998s7tDEEK0cxXetS65jrTQm/HII4/w17/+FYB169YxYMAARo0axb333kv//v3t9bKyspgyZQqJiYncd9997gpXCOGBssNdk2o9q4X+9f1wZItzzxmTAn945qRVysvLGTRokP11fn4+559/foN61157LfPmzWP06NHcf//9Dvs2bdrEL7/8go+PD7179+b2228nPj7eOe9BCNEumSp/pNZnJJXdylxzPZdcxcP5+fmxadMm+5/HHnusQZ3CwkKKi4sZPXo0AFdeeaXD/okTJxISEoKvry/JyckyWZcQgsBqC+aqY7xx7lyXXM+zWujNtKTdSWt90v0+PnV3sWXaXCF+P/RP86iuicR7zMUN9tWYAjFbjxPu17YLW5zQoha6UupOpdQ2pdRWpdQCpZSvUqq7UuonpdQupdQipVTbzzzjRmFhYQQFBfHjjz8CsHDhQjdHJIRwu2O7yX/lYTbf/golC19qsNtUqzDVuuaGKLQgoSulugB3AKla6/6AGZgOPAu8qLVOBAqA69syUE/wzjvvcOONNzJq1Ci01oSEhLg7JCGEG5WXFLE5cxI/DX+I9Pe/ddhXfWwfJYFJVAQkuiwe1VxXgi2h/wgMBIqAJcDfgflAjNa6Rik1CnhEa332yc6Vmpqq169f71CWnp5O3759T/8duFBJSYl9zPozzzxDdnY2L7/8spujMrSnn6MQHcXxF87hw4x7AAgt3MVVC2+y76t+LJp5WQsAuPXNCa26jlJqg9Y6tbl6zfaha60PK6X+ChwEyoH/AhuAQq31iY7iTKBLK+JtF5YuXcrTTz9NTU0NCQkJvPfee+4OSQjhRtkH99u3C0PrtcS1xlpQ7fJ4mk3oSqkw4AKgO1AIfAz8oZGqjTb1lVI3AjcCdO3a9bQD9QRNLWohhPh9KlsZDiMbltce3sSX++dANOiK/wGta6G3VEtuip4F7NNa52qtq4H/AKOBUKXUiV8IcUBWYwdrredprVO11qlRUVFOCVoIIdxNV1bww4hHGt23LzuHo9HDACjx6+yymFqS0A8CI5VS/spYzn4isB1YAZxYX+0a4LO2CVEIITyPNf8oqLoUailPt2/XbN9TV/Fy13W9NJvQtdY/AYuBjcAW2zHzgDnAXUqp3UAE8E4bximEEB6lIveYw2uzVvbt3L11z6FMTk5zVUgte7BIa/0w8PBvivcCw50ekRBCtAO79x4AQu2vVb32cXlFsX17SPQQl8Ukj/7TcPpcIYRoTk7u0d+UmO1b1TWVrg3GRhK6EEKchorSEofXWtUl9Ep18ud72ook9CYcOHCAiRMnMmDAACZOnMjBgwexWq306NEDrTWFhYWYTCZWrVoFwLhx49i9e7eboxZCuIq1wmiFF1sOoazFWGpMHFplPEhUVV0OwH97vuDSmDxqcq5nf36WHfk7nHrOPuF9mDN8zikfd9tttzFz5kyuueYa/vnPf3LHHXewZMkSkpKS2L59O/v27WPo0KGsXr2aESNGkJmZSa9evZwauxDCc+miQgB2hn/K2D3jKQ4aRNHTc2D8FVht90RLfIpPcgbnkxZ6E3744Qf7FLkzZsxgzZo1gNESX7VqFatWrWLu3LmsWbOGdevWMWzYMHeGK4RwsaiV+wE4K/oMMAUDsCVkBgDmqgoAKrxcNzEXeFgL/XRa0q5iDME3Evqbb75JVlYWjz32GM8//zwrV65k/Pjxbo5QCOFKmwfcAkBkTBJHrTlUACV+xgwo5uoqMMNzE191aUzSQm/C6NGj7VPkzp8/n7FjxwIwYsQIvv/+e0wmE76+vgwaNIh//OMfjBs3zp3hCiHcxBwdT3FQAgCVPsa851ZtzCbeMybOpbFIQgfKysqIi4uz/3nhhRd45ZVXePfddxkwYAAffPCBfVZFHx8f4uPjGTnSmMBh3LhxFBcXk5KS4s63IIRwk+CuPg3K/HONodBBvn4ujcWjulzcpbaJCeiXL1/eaPnq1avt21deeWWD5eiEEB2ctRq/kl1UeUH/mJGs47DD7oNdJwPga2mY7NuStNCFEOIUWYvz8Kr1xqSr8bf4O+7MzbBvmk1mXEkSuhBCnKKK7CyKgxMo8qtosC/nJtdMldsYSehCCHGKjh00Zgsv8T7uUO5TWcB/vR8yXugqV4clCV0IIU7VgYydAFiCjMf/fcu3A8Yol+OhxgOG4Xm/ujwuSehCCHGKTOu+AcDb37byZiNTt+RHuv5hQ0noQghxiioOxQLQtWd3N0fiSBI6cOTIEaZPn07Pnj1JTk7mnHPOISMjg/79+7s7NCGEB8pIvByAcWPSAKhVrh3N0pTffULXWnPRRReRlpbGnj172L59O0899RRHj/52rmMhhDD4VhwAoHNcuK1ENahT5bfLhREZfvcJfcWKFVgsFm6++WZ72aBBg4iPj7e/rqio4NprryUlJYXBgwezYsUKALZt28bw4cMZNGgQAwYMYNcu4x/www8/tJffdNNNWK1W174pIUSbMtXWYKk8YJ/jqdLSMJX6Bx5vUNbWPOpJ0SNPPUVlunOnz/Xp24eYv/ylyf1bt25l6NChJz3Ha6+9BsCWLVvYsWMHkydPJiMjgzfffJPZs2dz1VVXUVVVhdVqJT09nUWLFrF27VosFgu33HIL8+fPZ+bMmU59X0II96g9lom5NgCvmlJ72a9djzEoy3H67Ah/1z72Dx6W0D3VmjVruP322wHo06cPCQkJZGRkMGrUKJ588kkyMzO5+OKLSUxM5Ntvv2XDhg326XTLy8vp1KmTO8MXQjiL1hy86zKKgp8kPG+bvXhr3GYGZY10qGpqpBumrXlUQj9ZS7qt9OvXj8WLF5+0jtaNLyd15ZVXMmLECJYuXcrZZ5/N22+/jdaaa665hqeffrotwhVCuFH6v14gMzMFkiA/ol/djkZyd2WvWNcFZvO770OfMGEClZWVvPXWW/aydevWceDAAfvr8ePHM3/+fAAyMjI4ePAgvXv3Zu/evfTo0YM77riD888/n82bNzNx4kQWL15MTk4OAPn5+Q7nEkK0X/kfLSMjaXojexo2+vqMdf2Qxt99QldK8emnn7Js2TJ69uxJv379eOSRR4iNrfvtesstt2C1WklJSWHatGm89957+Pj4sGjRIvr378+gQYPYsWMHM2fOJDk5mSeeeILJkyczYMAAJk2aRHZ2thvfoRDCWQ6EnGfftoSubbJeUe1XDO482BUhOfCoLhd3iY2N5d///neD8q1btwLg6+vLe++912D/3LlzmTt3boPyadOmMW3aNKfHKYRwr4LwvvbtAJ/6rXLHPpdwN9wQBWmhCyFEi3lXFtq3KzvXrUb02y50s8X1N0RBEroQQrRI6d7tWGrK7a+7jkyyb0cEBDjUVWZJ6EII4bEKN6+jNKAzpea9lFz6K2kD6oYp/nPqsw51TV7uSejShy6EEC2QlWMsZhHoX80tZ93psC8qINLhtTK7p60sLXQhhGiBksJcAPxCcpqtq6QPXQghPJf30h+MvytKmq1r9nLP7IstSuhKqVCl1GKl1A6lVLpSapRSKlwptUwptcv2d1hbB9uWPv30U5RS7NjR9Fwys2bNsj9V+sc//pHt27c3qFNdXc39999PYmIi/fv3Z/jw4Xz99dcAdOvWjWPHjrXNGxBCtKkCf2M6bau1ttm6Zjf1obe0hf4y8H9a6z7AQCAduB/4VmudCHxre91uLViwgLFjx7Jw4cIW1X/77bdJTk5uUP7//t//Izs7m61bt7J161a++OILiouLnR2uEMLFDiScDYDyiWymJnhZvds6nEY1m9CVUsHAeOAdAK11lda6ELgAeN9W7X3gwrYKsq2VlJSwdu1a3nnnHYeErrXmtttuIzk5malTp9of5wdIS0tj/fr1DucpKyvjrbfe4u9//zs+Pj4AREdHc/nllze45gsvvED//v3p378/L730EgClpaVMnTqVgQMH0r9/fxYtWgTAhg0bOOOMMxg6dChnn322PHkqhBt59era+A5d13LX5QGN12ljLRnl0gPIBd5VSg0ENgCzgWitdTaA1jpbKdXqKQVX/zuDY4ea7586FZHxgYy7POmkdZYsWcKUKVNISkoiPDycjRs3MmTIED799FN27tzJli1bOHr0KMnJyVx33XVNnmf37t107dqV4ODgk15vw4YNvPvuu/z0009orRkxYgRnnHEGe/fuJTY2lqVLlwJw/Phxqquruf322/nss8+Iiopi0aJFPPDAA/zzn/889R+GEKLVqjr5NFpe5LOf4KoeAOju8Y3WaWst6XLxAoYAb2itBwOlnEL3ilLqRqXUeqXU+tzc3NMMs20tWLCA6dONCXemT5/OggULAFi1ahVXXHEFZrOZ2NhYJkyY4JTrrVmzhosuuoiAgAACAwO5+OKLWb16NSkpKfzvf/9jzpw5rF69mpCQEHbu3MnWrVuZNGkSgwYN4oknniAzM9MpcQghWqjgAP4l+43tLqWNVsnt+bN9O3GS62dahJa10DOBTK31T7bXizES+lGlVGdb67wz0OhYHq31PGAeQGpqauPz0No015JuC3l5eSxfvpytW7eilMJqtaKU4rnnngOwr0jSEr169eLgwYMUFxcTFBTUZL2mpuNNSkpiw4YNfPXVV8ydO5fJkydz0UUX0a9fP3744YdTe2NCCKfJfPNxar3ORlX8RFr8lY3WGdplEIXpYK4pp2vYABdHaGi2ha61PgIcUkr1thVNBLYDnwPX2MquAT5rkwjb2OLFi5k5cyYHDhxg//79HDp0iO7du7NmzRrGjx/PwoULsVqtZGdn25eea4q/vz/XX389d9xxB1VVVQBkZ2fz4YcfOtQbP348S5YsoaysjNLSUj799FPGjRtHVlYW/v7+XH311dxzzz1s3LiR3r17k5uba0/o1dXVbNu2rcG1hRA26/8Je1c69ZQbNx+nwjeCWipICE5otE6v6D4AaGUi0q/5G6dtoaVPit4OzFdKeQN7gWsxfhn8Wyl1PXAQuKxtQmxbCxYs4P77HXuQLrnkEj766CNef/11li9fTkpKCklJSZxxxhkO9RprvT/xxBM8+OCDJCcn4+vrS0BAAI899phDnSFDhjBr1iyGDx8OGEMgBw8ezDfffMO9996LyWTCYrHwxhtv4O3tzeLFi7njjjs4fvw4NTU1/PnPf6Zfv34Nri2EgF03Pk1g50o6Lz3otHPmhBifVWuXpqfE9Q7wBSrQyn2P96imvv63hdTUVP3bkSHp6en07du3iSM8V0pKCp9//jndu7t+EvvGtNefoxDOUrBuMVXfzSP/7UMA9N2R7rRzL5p1O8d8L2LCHXH0TW68a3hr+ka+e7kQtJVb/zHJadcGUEpt0FqnNldP5nI5DZMmTSIlJcVjkrkQAvL+PIfqvDYY/11wAN8CM3SG6PiYJqtZAn2NDeWep0RBEvppWbZsmbtDEEL8xolk/v3Ix4jI206f0jxUQETrTlp8FF4eQLXJWMgm2N+/yao+vu5Z1KI+mctFCNEh1CovKnzCqPCN4HCXcZTc2Rda2aWcvX8P2xd15mi00YfuZW66Dezj59uqazmDRyR0V/bjd0Ty8xMC1g29j+9HPWF/nbkqguOvnuaMJBnfwJd3cnDzBnb1vLRFh1jMFgp9sljTfcHpXdMJ3N7l4uvrS15eHhEREac05lsYtNbk5eXh6+v+1oEQ7lJ7NJ3SwC4NykvWryPkdE74kTFdR3Hn58mMa9kDhRaThYVDnkU1WJDOddye0OPi4sjMzMRTnyJtD3x9fYmLi2u+ohAdVN7siRDq+LzHvoQ/EFO8moZpvuWan1exjsVkAUDjvm/Mbk/oFotFRosIIVrl2KYQSHMs29f9XI6VVzKkuYO/eQB8giFtToNdvtkbgV4tiuFEQncntyd0IYRore1dG78dWKk6N3usXvsqAKpeQi896k1xpi9eJfvB9oz8oAcan5TrBC+T+9Op+yMQQohWKom5ptFyq7n5TpNd30ZizfOmb70Huvev7ITV7MeRnmON80w5xJj4xq9xglKKCN8IbhxwY8sDdzJJ6EKIdi3nx68p8Tceohx5YQ9+XLLXvi/2SOXJD66tpTrfD22qa+Frrdna748cixxoL7tg7JQWxbJy2sqWB94GPGLYohBCnK4DL79AVO4mAAZN6solc4ba99WaTv7k6P4vXmPd0Dl8N/4ldLmxFkPNoa0OyRyga0Srl3twCUnoQoh27deoWkBTZs7CbDYR0z2EmiHG2sBK15z02E3f/p99uGPt8WPo0nwyzp7eoF57GVItCV0I0a75EUhu1GD8a+tugE4Za4wdz4w7s+kDa61Yj5XbX5bl5bDvlT9SHNjEEnPtgCR0IUS7Vlk92tjQda3ogJDwZo/Txw8TkFs3DXVhdg45u3ayYei9To/RVSShCyHaNbM+BsCZN9Q9XBcSGNjscaXbV1LpU/ccaX7OEazbG445v/D+gQ3KPJUkdCFEu2aqqQAgskvdzIr+Ac1MhVGUTd6Cex0e66/IymDToNkNqnbp1soZG11IEroQot3Ztfgdtr71NACd9htjzQMC6qa2NXvVS22Fhxocn7d3L/mrezgWrqtbt9dLGUskK691zgrZJWQcuhCi3al58K+YgdIEb2pNxoISPj6OQxSjj/zM8ZDuFH61htArH3LYV7x9JwWhxiOgJv9D1JbFU5UZA2HGfu+wAmryOxEYE9rm78WZpIUuhGg3rAXZHL45zf764B1vo20rBJnMjkMLzbVV1JosFOeUNThPQQmk950JQGAX4+GjbX2vBYzJtboPMtYP7j10otPfQ1uSFroQot3Y9/Rcqlce5VhEf6wmb6JzN7Kv+3kAKJNjQs+L6EeVTyhrDxfz25Hlh/OO27ejooIp2gVWL6PfffLsJLp2jaGmFAandWvLt+N0ktCFEO1G3rHlBBPA5pQ/AeC7scC+77cP/1T6GP0nVt1wxIv3yp+guzF6JTQ0yF5uqTxKUl/jRulZ1yY7NXZXkC4XIYRnKz4CR7YA8LN3ODlRg+27Ngy5p9nDu/x8mOKdvziUZcXUJXEVVTeKRemq1kbrVpLQhRAerfihoZQ+arSaa63D2Nrvjw3qDJmS0OTxtcqLg6tXOpRVe9c9QRrYqf6wxJNPFeDpJKELITzO8XdvovTZMVRv+4rMr4M5uCISaqqIrLY2Wn/UhT0blPWaWA0YNz9zrY6zLvqXpwMw62+j6Ncz0V6uqHbWW3ALSehCCM9RVQo56WQ9u4qD7+az/dG77LsOzb2QqEPGIhNnzuhjL4/vG9b4uULrFqQoOF5vWbjaWqi1YK4pJCDAD4AanQVAQJm00IUQolUyb7uQA5elknttH9LHX2wv996sKA6MJz+sDyVf7KPGNhKl98gYTtwDPfvGlEbPGRtbN4bcVGWbhCsnnZp3z8dSGYBXTd3EXD6xxQBUWYJoz2SUixDCraxV5RT/bycAZTiOSKlVZtal3g/AhJW3YjX7oGprMJtNzHhyNBUl1fj4NZ7GIsNjgHzjPBXGWPSjsyeS/0sINQPPwbeybnz6wNSxbPwil5KgeGe/PZeShC6EcKvco1kNyjb3u4FyvyjCCnfZyzJjx3Gw62RMViMRB4X7EhTe9JwtEaGR9u3A7G0A5P8SQlFQVwrC+oCuW55uwPhENn6R2+r34m6S0IUQblWZuZ9y3wiUtlJlCeJodCrHogYB2BefAMhIMh4PMtU2s6ycjZeP2b7ts8PoStmXMMX+IBKqrsfZP9Dob8/3O3L6b8QDSEIXQrjV3uWL2D7yseYr2rR0rLip3pOjFb4RoHVdMgdq+hbWnVMpEm82Ex85rMVxeKIW3xRVSpmVUr8opb60ve6ulPpJKbVLKbVIKXXyxfuEEMIm//AucvcZDwuVV5ubqe1I1bZ8aOHA841VjHYlXkZ1Sb69PKZHMHfccZFD3cmDzqBvXCLt2amMcpkNpNd7/SzwotY6ESgArndmYEKIDqqmku13TyHjzguxZvwP36KC5o+px6eq8bHojek/vO4m5/HCArwrjWtdfO/QdrNO6KloUUJXSsUBU4G3ba8VMAFYbKvyPnBhWwQohOhAaqo4PDWJiE3ehO/wxvqPyyEr76SH+PfY4/Da1MzCz/WFRtaNmikvzEUrMz416zpkMoeW96G/BNwHnBikGQEUam3/yWYCXRo7UAghTjj2v3coOlC3EMWepdH4hAVAKKRMi2ZAcncCw33QtTBv9ncAWAIdR7Icjj69OcorN8yn1vQHTKp9Pzx0Ms220JVS5wI5WusN9YsbqaobKUMpdaNSar1San1ubvsfFiSEaDl9+FeoLLa/LvuPcfPzWER/jnRKBaDCNitiUu/OhEb742UxY6k3QsU73I+u/SLoPTIGgMCKU1sSzteyybjOc6uoNVnwsS1Z1xG1pIU+BjhfKXUO4AsEY7TYQ5VSXrZWehzQcDApoLWeB8wDSE1NbTTpCyE6nvKMDew7/yoUCktcBb3+t4/SVaEUB8bZp7+NOraZKm9joebIqJBGz5PQqxsjpg+kqryGnT8eIXlc7CnF4a2KqQCqLQFokxdhRxsueNFRNJvQtdZzgbkASqk04B6t9VVKqY+BS4GFwDXAZ20YpxCinVn/8hwibV/mqzONbhPjyc+59jrfjX/Rvu1laXy0S0ik0YL39vPi1jcnNFrnZEzKaEd+P+pJAHJie5yservWmrlc5gB3KaV2Y/Spv+OckIQQHcH2yhKH18d3/ki5X2QTtZvm5+fTfKWTUMri8LrCp08TNdu/U3qwSGu9Elhp294LDHd+SEKIjqCLVwC7ek7AvywH74qD1Dx6OTuTHjjl84QEtm7CLO0TAvW6zcdeltSq83kyeVJUCNEmotabWZ96lv116q6bKEw0HtwZeFYc+3fkcjzTeIx/0nUNl3u78N5BbFt/gNCAxvvWW8o7IADqlhCl35iOOyBPps8VQjjV8bWfcPTiOLJixzqUBx+uaz+OvKAnyvaEqJePmaThMQ3O06VnOJOnDW5QfqriBg6ybwfFWTBbOm7a67jvTAjherW1ZF3/IPnbgwgoq5voSmkrRzoNtb82mRRDzu4KwPQH27bnNjohzr593q0D2/Ra7iZdLkII59CarXdP4cRYlaIgY51Pc3QR1qPBFAV3s1c1mU30HR1LzyGd8PZt2zTUfUDdjdiwsOA2vZa7SQtdCOEUvzw5B/PXhwCwmrw4Gm3MXBjmbYwbz4ybgLefFzOeHGU/pq2TOYAydczH/BsjLXQhhFPULl0CmNmafB059bpX+ozqzJpDxkIVVeU1BEf4uTy2P9ycQubOU5sErD2SFroQnq7WCjV1c4D/suRljh7Y4caAGirbswn/AjPsT4raAAAgAElEQVRWk8UhmaekxRHfN9z+uu+Yzu4Ijx6Dohg/reMOVzxBWuhCeLiDt6bhl7uXqMWHqSg8StyHz2H59Gm2H4+jz79XUV5ynIOPXU7kn/9BVLeGw/9cIf2zD9D+0fw0/CGH8tGX9MTLYuaqx0YSEun3u+r+cAdJ6EJ4uNIVxyglmKiqUrJWf03lVuPGnqKUDTePwhzRC7//O8b2mrmc8ap7ZuDI+99XZPedY3897cFheHmb7Y/zh3byb+pQ4USS0IXwZLV1Cxnvndqfol5V+FA3UiPw+ypKIrYAZg5VFzZyAteI26vYlWYMQ5z17BgCQlr3uL44PZLQhfBge//zqn278pAvPocarnIfmGe0ggNVqcvi+q3SAOPBoLGXJUoydyO5KSqEB8vYsaHJfQvHO358C8qLGlY69DNUlzs7rAZyInsB0H3gqU++JZxHEroQHqyiuIgqSyAVPnWr9GTE+vD5yGCuuGcp25+/AQCryZuRP5mNETE2JYd3s/PGaRx6ZnKbxlhTlEdWl3MACIpo+A1CuI4kdCE8SNXeNZQfrGuVR27fzPcjH+f7UU9SZYbP5lxEZtILBPo+zbcv7GVk0rUUhvTku/EvUhCaSHH69/Zjt734OLX7fCmZf4zqb5+C6jZYqUdrtp05mipv243aDrpWZ3shCV0IT6E1e865gf2Trwag+vBeInZ5U2v2BiD4PyuZMeZeh0OWvPgL25LOBqAwpCd5N19j3xf85c/27d23fkD6u4+3KrzSnStJv6wrm/442iiotZI1dwTepSZM1iq84o616vyi9eSmqBAeovjzR+3bldnbOfJrOlWWulXrv3pxW6PHVQb0A8CrppzyXB/2XNyVgOBqwHEx5V+/+y99b3zytGIr+s/zrPvvq8RuCcCHAtCab28eTuyqMjSKWrM3/XoNav5Eok1JQhfCQ2z/31r7gMTtF1+IdbA35cF9m6w/9A8JbPj6gP11pY/xRGbV9gBOPFdaY/bBy2rMOV5hrT7t2A7/5Z/EUjeWvOzoXmJXGWtzrh90EwAlBR138eX2QrpchPAQBQWV9m3fAjMBy61UezX+QE5Mj2BGXtCT6f+vburZg13PcqhT7hvBqnEvsCNpOgARlY0sjlyUxaEZsax7+KoWxVhlCaDUP5o9Z08F4HhwN4pDUwAYdm73Fp1DtB1poQvhCarL6bI5h1IfRUBdXic/zGihB4X7Upxfwbm3DyShX4R9f0SXQIfT5IX1JaIgHYAfRj4GQFbsOLJix9Fz971QegwC6oYW7rp+DDVbQghctxEepYE9b99J2Y//xaRMlPnH8PMwYwm5CStv5XDnMezsfSVgrDgUFd+6peJE60lCF8IDHF70Kl5VilVprxFUtJ9hG583pqCNMVrgM54Y1eQ8KOOmJbFlZSaFR8v4deBtDFv3FN7VxQ3qWb0mkHHucJJW7DUKamup2VI3zFBXl6MsdTMhVuVkUfXX/8MLWJ72msO5NMqezIMjfUkcFt2aty+cRBK6EB4g+70F+JiNJyyLg7uhgZLAePv+k01qNeDMOPIyiyk8anSprBv2l0brVVv8se6ve4rz4IKXHPYf3byWmKF13TabHr2DptrcK9JetcUFM54Y3WRswrWkD10INznyf++S8/KVWPP2UV5RwZHouv7wFWmvURxoLJ02YWafZs/VuVdos3VqTcbwR2tNtbFU3EdvYjVZWJ72GjuSpvPzx2841A/6dhsVPqHkhdXdmA3rHOBQZ/z03s1eV7iOJHTRMVmroaqRm4Aeojwrg4I/P0feG7+QMeYcwvNqORrZyaFOhu1mZnlJ86NTeo+MoTIwt9F9Q6ck0CUplOxOCdQqE9senUXR5jWE7DHzfb1+9vKiAvhlvvEHqLB48f2oJ/l14G0AHA/YwuV/SXU4d0yPkFN746JNSUIXHdK/r+rHW1cPBa3dHUoD+1Z8ztabL3IoM2nF8fAJePuaG9SvrrA2KPstpRR58Q2HDV7x8AhGXtiTwpxyMMeydtRTWD7eyIZP/o4Gqr3rZm7stCeH9CueYNdfHiDn+08oCXb8ZpDca6x9Olww+vUj4xxvygr3kj500eHsWPwCKZuMxFO+ex1+iW27qvypqDiSQcWf5vDbNHgs3Hg4qKqR5N1zSKcGZY2q180+YWYfuvaLsM98WFNlnLfa2+gVj/l4K5mx4x0O9ymMB/ZTs8+PrFseZPNIxxuhwYHGEMqrHhvJ7vU5Mm+LB5IWuuhQ9n7wKPrBt+peX3m1G6OpU1mUS/7OtexLu8BeVl2vMb6718UAXHLfULoNcJyxsMWtYNun+XDsIfqOjnWYxvaS+4Y6VK1VXmQkTQMgdWo3AEoCu9n3+9Rr7F821+hm6TPSmCI3tJM/qed0k3lbPJC00EWHUvnkQofXpmIz1pI8zIERTRzhGj/eMYVOPzr26S8fNRa/MVNI6t2D/A824quNPumptwygpsrK/i15dO7Z8j5q/4Rq2ArVCQ0XQw6LCSBlQhRblueyfvDd9N61yL5v6JQE1i/dT1FwAit7vECXrDV0O/CNcc5wE50SgrnljTMlgbcD0kIX7YquKuPg3HEc+bnxpdb2JhhJ5/Wpdf+1f77hrEbruoouK2iQzBdMSMTidQU1P4Wx/V8F+OruDmO5vbzN9BraiYDQli8WkZgUzJsjZ5MwoPF2mq4xRrkUhfTg15Q/ATDigh6YvUyga8mJGkyt2YdD8RP5fqTxlFFZvrFikiTz9kFa6KJdWf/sHQR+eoxjmfcS80Fd9wXlhdRWV9LjgGZfNPz9b9tIX2oMtwv9xb1zjKS/9df63dtkv/khvVdU28eNn1Brbd0N3IsSL+J45XFmJM9odH9RXt3Poco2v3r3AZFGslYmtKr7JWj1Mh4wConyQ7Qf0kIX7UdtLYHz1wLgt87Mjs+NcdO6upL0waPYOTwNgC3dFIVHy5h3/RB3RVqn1kreD98CsGicif/eNJP0hccbJHMA3coRORaThRsG3ICvV+M3K0ec33CulfDYgEZqGnoOjuLS+1Ob3C88T7MJXSkVr5RaoZRKV0ptU0rNtpWHK6WWKaV22f4Oa/twxe/Z+gfOc3idOe/vAOSsdux+GXzPG8x/+Ed67bmeWlur88Dnr7D/3ZtdE2g9O6f0InLTcXZ1ieT8q5fitXOEfV/qOd246ZUzmPHEKKK6BjH8vLad3KpTQjDn/7luitsL7xzcaFdKZLxxE/as65LxDbC0aUzCuVRzrQKlVGegs9Z6o1IqCNgAXAjMAvK11s8ope4HwrTWc052rtTUVL1+/XrnRC5+d34d0hvvsro2yLFgSOpRSv6mulam1eTFd+Nftr8utazmvGV1N0qDl35Ml579nRdUTRV4eTe6q7aslJ1DjBbub+dCAbj2ubH4Bzd+bFs6nlveoCulpKCC9+d+z5hLe9FnVGcKjpSd0g1Z0baUUhu01s1+XWq2ha61ztZab7RtFwPpQBfgAuB9W7X3MZK8EG2mINCxtRhZhEMyB9jT/VyH1wHV46jfZDmc7rwGxZ7l8/l1WAobxyaya+WSBg8xFW5bB0CZr+MwxDNn9OHWNye4JZlD4/3igWG+XP+3cQw6qyu+ARZJ5u3UKfWhK6W6AYOBn4BorXU2GEkfaOHTD0KcOl1rpTTIyuEIWPKHZHtXSn0f3T2VI+NGApA8NtZevrvnxfbt6uXz2PL+Y06JqejuJ/EuN+F3zIuam+eS3jeZ0o0fo4uyqczdy/cbVwDwY9qV9mPGT08ieUxsU6d0K+leaf9anNCVUoHAJ8CftdZFp3DcjUqp9Uqp9bm5jc81IURzlr1wDz32QEXAQILLb+WzC+pa4k/d34u+O9J55NrnqNkfhI+/F2deXffY+qH4ifbt8K8K8Hp6QatiKTu8nQOL7se3vGF35cErH2L11ePYO24qPV/8NzmRA6GiN17eJm59cwIpaXGturYQJ9OihK6UsmAk8/la6//Yio/a+tdP9LPnNHas1nqe1jpVa50aFRXljJjF701VGRU/folGcaDHjQCEFJ5NhU8YVWZ4Y/pCXrt5OW/ethKAvqM7AzDzqaandd2y8svTDufAxEsoe9i4EZsdBjm/6Z2IyjBaurXKxNb+RrzhnZseTSKEs7RklIsC3gHStdYv1Nv1OXBiifFrgMaf9BCilYpWvUXiVi/2dZvqUP5t2g0svmsYe39wfDJy1MW9AGOVnxODOJanvcbytNfs/eleN9+LrmnZGptH//c2R26IYdfrd1FbXbeckEZRGDmG0Ge/4V+X92xwXKVP3cCvqbcObNG1hGiNlrTQxwAzgAlKqU22P+cAzwCTlFK7gEm210I43S//NNoKh+LGOZT7VSYQs24WqxdlOJSb6i0Gcd7tjivRl9e7QVl4NLNF18+/7W8UrA6j5pWv2XJF3fzf6wffRHaXK1mzcDdxOXexduTjVHqH8NMDN9Dr1618dFkSAFNvGeC2G6Di96Ulo1zWaK2V1nqA1nqQ7c9XWus8rfVErXWi7e98VwQsfn8OmIzl1HJCD2MyK4IjG39wptfQToy9PNGhrEtvx4Uftjw+mzI/o+sv7y8XUlteeEqxeG81Rohsef5uikNSHPZV+oazdvRTVK0fyrzZ39H9kDExmJe3PL8nXEP+pwmPF1JuJTsiioiS3gSG+TDtgeGERvvb94+9PJGrHx/J2Tf0Z+CEeIdjTWbH/+KlS8P5ccQjAFT/VMV3H7zK1scnc2DFh+Tt3sa25+9zGH6Yv+G7BvFsGHwXuUt7GOc3qQYJu7rScQrcxqbEFaItSEIXHu3AptUkbSsnOzYNAN9Ab7z9vLjq0ZH2OgMnxBMS5d/EGRp3Yv7xmBfmY55/iK0vPMbR8y7F9M4X/DwhiV1PTgZg8yN3U2UJZF/8SApDeqKB4yF1/eVxfcK46ZU0zvmTY2s9ot6Utwkp7p3pUfx+yORcwqNlPH8X0V5+FEakATDlxrqnPCf/sR+BYc0vsnDN02PY92suqxbW9bVvHnALad/NxqRrAOixq25y8qBsL2o+OETF7ceI3lVqf8pzH4C1rotm8OSu9sf1YxONrp2ILgFMe3A4Sin73CwyU6FwFUnowu2KMtZQdWQ3hc+/RtQzrxDSbxQA+1YuIW5DCdt7G33RUV2DCAqvS+CJqdGNnu+3AsN8SEmLIzI+iN3rj7J5hXEzND+sN5H525o8bt/wcQ0LzUbinvXsGIcFJHz8LVwyZyihnfztCVwSuXA16XIRbmE9uA1darR2tz00g9wbn6V6VwlZl1xHdfoPaGsNx++8HwBLjTEz4YV3DW7VNTv3DGHctCT7680DbqHUP5qioK5Ue/lRY/ZlT3w/7r6+rrVe4t+5wXm8/bwckvkJMd1D5GlL4VbSQhcuV5S9l8OTLwUg4O1nCd3k2G2y+6LrqLFofKqNFu6h+IkEhPrg7euc/65jL0tkzce7APhp+EMAhOWt43joAGrNPkzdA0ej3iE6dyPfj56JVw30HBLFno3Gk843vDi+yXML4U6S0IXLrXvrIU7MZlL6x8Yn6PSyJfMdf3kSvof4Ps6bnbkwp+Fc5AURwxxeb+t3PXkHKvCq6QrApGv70WNQDiGdTu3mqxCuJF0uwrlqrWT8/TYOfPc5VZWNrxSkM480PEyZ2ZF0BXnBda31Hd3Dyfre6LMefn4Pp4U45OwEAkK8CQw/+fJuRxJuBcDLYsJsMZE0PIbobsFOi0MIZ5MWunCajH88TN6HCwnNNVHGt+xhDgcnVtL1Wx9Ku1sIGtqLmCvvpMuqw5T4d8an6jiWmjIWnX0VUZXGvCtZsWMZ8fNjHIotJSvucfu5/YOc96RlULgvs54dS1V5DW/duYozZ/RhxQc7AJh4TV96j4zh9T+tsNe//IFhTZ1KCI/S7AIXziQLXHRcP8x/mdDH32xR3RqzL6vG/Q0A/8pPKPO5pMm6Fl8zM58YjW9g295s/PdT68g9WMyMJ0YRHOnH0tc3s3/zMW5+LQ2zWb7ICvdq6QIX0kIXp+XAt0vY/90KIrpGo1/6F6E1dUP0jgVBZHHTxxaG9rJvnyyZA1z77FgsPuaT1nGGSdclk7WrkOBI49H+s2/oR2VZjSRz0a5IQhenbNsn/8D0wEv1VjQxkvmWBMXmPiPoXGmsOl8av4LKfV+TG96dAeOnUPGfd5n4fR6bU/7U5Lnrj0CZ/Md+LknmAGExAYTF1E1x62Ux4xXimmsL4SyS0MUp2bv5R46+9CKdcXxopnbpf8h9PpvOdbPLEnDoTAK8ziS8CCq/BOX9GMsu2Yw5z/GcN7w4HpNZoQGLt5FE4/qEEdElECFEy0lCFw50VRnK2xial/6v5wgflkZ03+H2/Zk3XUfnAkWFTyirzkuj1iufnhE3c+j57Bad35w3ADAex/fyNjX6IM7AifENyoQQzZOELux+XfQS3g//A4DiUQEE/VBKPu9SOaSIrOMhdH70b0QVaIoC41mfej9ex4zjDh0xFpgIjvKja3I4Z1zRm8KcMjJ3FJDx8xES+kcQGOrDr8szyT1odK4Hhp18yKAQ4tTJKBdh9825KXTdXXPSOsWBcaxLndug/I8vjsfHr/n2wYGteUR1DZIFH4Q4BTLKRZyST566kT57ajnUJY3OR37AbK3ip2EPEn94BV2y1tjr1U/mVzw0guoq6yk9bJPQX6aSFaKtSEL/HdK1VmpLi9FeXph9/Fl6bgrJe2vJjhnJrsTL2JV4mb3uzqQr2Jl0BQChhbvs5RfdM4TwWFn4WAhPIgndiSqzt+Md1RPl5Vn9w5v/fhM1hw+iVx7AcutVWJ780L7v8KQEeu6tpdI7mPQ+M056nsJQY3m3G14a77SJsoQQziOfylbIyfiVvHv/RNCfbqH4z08CkD7Kl4vf/aVB3drKcn648Q/4d46kx5V/JmTA2DaPz1pyjF/eeZ6AN1ZhH0tiS+YaY/R4l2UH2NP9fA4knN3g+AET4lAoNJqgcF/WLt7N+OlJksyF8FByU/R0aU1632SHIqvJC3NtDQFvPkXXtIsc9qX36dvgFFVjvckJ7ILvhgP0+nAJnRMSG9Q5XWWlRWwdN4KgMigI6UW5XyT7E/5Ayra3HPrBe+xdwt4eFwLG2pyhgzWH8g8zpvtIlEkWaBDCE7T0pqgk9NO0elw/zFU9+GXwnaRs+QdbUm4CIC5zJZ1yFjPq5+32ur/MuRD9TS4/jHwcc005vTMWEJOzweF8R3rWkPZ5Osrcutbv8sduoPNHxk3MCp8wdiZNJy+ifzNHwYwnRxEc4deqawsh2oaMcnGy/a/OJmTy1VhCu7Dl9QcJLvRnzZg7AezJHCAzLo2gop321z/85QpCPstgRdqrAFi9/NiefB3bk68DoNPR9eREpzJs/TNsXPgaQ6+afcqxHd68Fr/gMPatWETnj9ZQ5teJvPC+7Eq83KFeeGwA+VmlhET5MeWm/ix6Yh0AF909RJK5EB2AtNCbozWrz+xP5JFae1G1VwCrxz7XoGrnXiFk7z4OwLj41+k86CyO3Pcmq8a90KJLpX13O/3Sm17j8rcObv6efd98jN+CrwkqU1hNFtalzqXM33GtzVnPjCEg1Mf2drSsdSlEOyMt9FaqKspjz3DjxqUK68PytNvxqSggMm8zh7ucYa+nTApdq7nl9TNRJsUrNy/BTDBeH2wn94PtrLK1zAFu+vsZeFnMlBVVsXnFITZ8fQCAIVMS2Ph/B9jR+ypMn8+j7/k3njQ2a0kRGakj0ECk8iIj8UqyYh1vsgZH+jLlxhSiugY5lEsyF6LjkhZ6I75/dy6hzy5hc/+byYtMabROQv8Izr1tILVWo+Vusk2z+vmHazi0pqpB/ekPDSci1nGyKV2rqSyvweJt5s3bVwIwbs29WGrKKJ0zgyGjR6G6j0R5+1F9vIBjm1YTM/ws0gen8sug2fZhhL91yxtnSuIWogPpcDdFiw/uwi8mHi9v3+YrN6G2qpJvZ56N76E89LljiXp/JTsGhxEwfARVv/yMV3Yh5iorYcUx9sWDT+gzujM7vjcmoBp9SS8GT+ra6DVKCyt57/61DmV/ei3NnvCb8trNyx1eD1v/DL4VeXjVlJHTs4aI/QGU+0YSUHaUlWe83OD4C+4cjFLGyvNmi8zhLURH0qES+roX7qF2wRYCSrOonfc0nWLiiOk5kOr8LMorqqmsLCGqx29GcmhNZWkZ37xyL8czfqbXtlL8K4LY2/1csmLHEnlsM8ciBxB99GcCSo9wtNNQY8S1MlMa0BmAsM7+TLwmmdBof3z8vCjOr0Ap1aKJpaoqaigvriIwzBezV/MJtuBIKR898lOj+/zKcyn3i2pQftWjI/EJ8MIvUOZFEaIj6zB96LVWK9vX+VAy1FgdPuTV3YQXfEuXrDXkhSdTEhhHcNF+OuVuZHf/WBImjGbvihV0OhRJmX8njkcPp9r7DDYNCqHWXJf4jkUa07gejR7ucD2/IAtRYd6cNat/g0fbg8Jb/u3A29frlB7ACYsJwBwA1tKG+xpL5re+OaHF5xZC/D60ixb6R09/ScEB/9O/sNIopYlPjmT0Rb1QJsXOH7MJjfYnMNwXpRQ1lVYSUiLc3vestUZr2L76MEkjYijOqyAvqwSAxKHR5BwsJiTKr9F5xIUQHZNLulyUUlOAlwEz8LbW+pmT1T/dhH5iqF327kJ++HQP4V0C2bflELU1XiSmxuIXaOHnL/Y5HJMwIIzew2KJTQy1D9kTQoj2qM0TulLKDGQAk4BMYB1whdZ6e1PHuGKUS1lRFdWVNYREtaJFL4QQHsQVfejDgd1a6722Cy4ELgCaTOiuYCycIDcJhRC/P60Z39YFOFTvdaatTAghhBu0JqE3dvewQf+NUupGpdR6pdT63NzcVlxOCCHEybQmoWcC9ZdnjwOyfltJaz1Pa52qtU6Nimo4/E4IIYRztCahrwMSlVLdlVLewHTgc+eEJYQQ4lSd9k1RrXWNUuo24BuMYYv/1Fq3fKpAIYQQTtWqJ0W11l8BXzkpFiGEEK3g0idFlVK5wIHTPDwSOObEcNqKxOlcEqdzSZzO5ao4E7TWzd6EdGlCbw2l1PqWDKx3N4nTuSRO55I4ncvT4pR5VoUQooOQhC6EEB1Ee0ro89wdQAtJnM4lcTqXxOlcHhVnu+lDF0IIcXLtqYUuhBDiJCShCyHcQrl7NZkOyKMSulKqp7tjaAmlVLtYLsg2Z73Hf3A8Pb4TlFIhtr896nPzW0qpfkqp019N3XX83B1AS7SXzxF4SEJXSg1RSq0CnlFKBbs7nqYopUba5n1/XinVv9kD3EQpNUYp9T7woFIqXHvojRKl1Ail1FvAHKWUR87cppQyKaWClVJfAq8AaK1r3RxWo5RSA5RSa4AngAh3x9MU2+foE+A1pdTkEwnT07SXz1F9bk/otom9ngAWaa0v01oX2co96rehUuoy4A3gS8AXuMtW7mlx9gBeB1YACcDjSqmp7o3KkVLKrJR6GmOEwFpgCPCwUiravZE1ZEvexYAF6KKUmgYe20p/EFistb5Ia30YPPL/ZxrG/8//ADuBq4Ewd8bUmPbwOWqMJ/ynHALkaa1fA1BKjVJK+Xjgb8NE4Aut9YfAi2B0vXhgnEOBdK31e8DdwCbgXKVU/EmPci0TcBC4zBbnn4GReO5X8D4Yj3e/BFyllArSWtd6SrK0fYvoCZRorV+ylU1SSoViTJznSYk9BVintZ4PfIDxi7LEvSE1ahie/zlqwOUJXSl1uVLqLqXUKFvRAaC3Uuo8pdQy4GHgLaXUFa6Orb5G4twJXKyUug/4AYjF+Mo4zG1BYv/6mlSvaB0Qp5SK11oXYLSAC4GL3BKgzW/irAUWaK0zbL+8szDm1490X4SG+nHWS4K7gSpgn+3PNUqpru78ZV4/Ttu3iBxgnFJqqlJqCXAPRhfRvbY6bom1kf+fq4HLlFIPARuBzsDrtm/AbmPLP7cppUbaitYB8Z72OWqOyxK67Wv2Q8AcW9E8pdQlQC7wBUYXxjNa6ykYX3MmKKX6uCq+k8T5llLqfIyviLOB8cBMW5y5wCVKqRg3xBmqlFoKLAMuV0oF2nZVAGuAy22vd2Ks8xrhjhtljcWptbZqrQsBtNaVSqkgoDuNLJDixjgD6iXBVKDINj30NoxGxxtKKYuru14aixNAa10MvAs8jjGV9dnA28DIeknKnXEG2uLcBEwBugG3aK3TMJLlFKVUXzfE2Vkp9QXGL74w4F2l1Nm2tZJ/wEM+Ry3lsv+MWmsr0Bu4W2v9AvAI8CeMr7O/Av0w+qYBlgNBQKmr4jtJnA8DdwJJWutvMRLmTlv1z4AB7ogTCMCYi/522/Z4W3ku8COQopQabns/h4ExWusKD4hzXCN1RgDbtNZZSqlApVSiKwO0aernCUb3UJBSahFwH7AByNBaV7vhBunJ4vwSI1Ge6JNeDxwFKl0Y3wlN/rtrrX8GooD9tiK3fd4xflmv0VqP11o/DrwM3GDbtwbP+Ry1SJsmdKXUTKXUGba+PDD+c4Uppby01p8AGcB5GF/DngNm21o8k4BwjOTZ5loQ5zZguq0lvge41FZvsKti/E2cwbabXvOAf9tiGK6U6mL7j/cj8Avwoq1l1A84qJTy94A4RyilYm31TszHHwocUkpdi/FVd5AnxYmRIKOAIxj/5n/C6CZ0SYuyBXF2AdBab8Zoad6mlIrEuOHYH8jzkDhP/Lv7AN8Dt9oOnYgxKseVn/c0WxzfAv+qtzsP2GXbPvE5eskdn6PT4fRH/239jjHARxh9pXswfkPfBNyBsajGK1rrQluXyiJgitY6Wyn1DEbfdBxwq9Y63anBtS7OhRi/aAZg/EeMxbiZc5vWeocb4pyttT5mqzMG46vheq31B/WOfQHjZ5mA0U20kzZyinGus91cPnHsB8BVwPvAi7bE5Alx2n+eSqnIevsDAW+tdb6nxWkrvwvogXEj/06t9XZPizkPg7wAAASzSURBVFMp1Q/j228MUI3xOXLH53221vqYMgY4VCul7gCStdY31zvWZZ+jVtNaO+0PYLb9nQR8aNv2whj+8w5GS+wbjK+J/rb9i4C7bNsKCHRmTE6M82OMPj+AQCDFjXH+HfjPb+reiTH8MwQIOnH8iW0PjDP4xL81xnq0l3ponCFAQL2fp8mD4wyqV27x0DhDAT9bmR/QwxPirFfnC+As23anenXb/HPkjD9O6XJRSnkppZ4CnlJKnYHRB20FY+1R4DaMrpUuGL8hp9teA9Rg3BRBG9psCFMr46zC6DtFa12itd7ixjjvAEbZ9p3wFsYvmmXAbqVUrDZuPhZ7aJzfAnuUUp211gu11os9NM5lwN56P8826zN31r+7rX61B8e539Y9WK6Nm49uj1NrbVXGMzG5QIZS6klgmVIqTGtd05afI2dqdUK3/aA2YPQz7sa4y14NnKmUGg72YVWPAs9rrd8H/gvMVEr9gvHbr82SYweNUwOPYdxYPmEqcAvGDeYUbQwF9OQ4N9nizPbwONvLz7O9xHni3/2wB8X5qO0wX2AWRmMjCKOlXtCWcTqdE77OjANm1Hv9OsZNo1nABluZCaP/ajEQbyuLwQVftzp4nP8GutnKLgDGS5wSp8R5WnHGAcMxbpAOclWcTn/fTvjB+QM+1PVBXQU8bdveBNxu207FeJjEPW9U4pQ4JU6Js/E4F7orTmf/aXWXi9a6TGtdqY3hcmCMBMm1bV8L9FXGxEYLMJ4Mc4uOHKftDr7EKXFKnKcX5wZ3xelsXs1XaRllzJimgWjgc1txMfAXjLGw+3Qb95v9//bunzWKKArD+HOQFKKQRjuLIBgUQVP4AVJZiIWFaYRUgmBva2dhFVCiaCdY+wFS2ahYSSJYpxT/gIWKKZI9FmcCIiQkeNlkr88PttjdSXgHwpvlzs49e9Fjzhw+ahwEc7ZlzrYmJWcrLW8sGlEb7XwFLgz//e4Co8x8dRhKcmDOtszZljnbmpScbbRcv6F2zBtRt8zePOj1JHOa05zmnJScLR5N7xSNiFPAIrCUmQexf8SemLMtc7ZlzrYmJWcLzW/9lyQdjMMw4EKS1ICFLkmdsNAlqRMWuiR1wkKXpE5Y6OpWRGxFxGpEfIiItaih37v+zUfETETcGFdGqSULXT37lZlzmXme2svjCjUlZzczgIWuieT30NWtiPiRmcf/eH6amll6ghon9pwaQwY1Au1NRLwFzgHr1Ei8h8B9YJ7ave9RZj4d20lI+2Chq1t/F/rw2jfgLLVB0ygzNyLiDLXV66WImAfuZObV4fhb1Ciye1FDhV8DC5m5PtaTkfag2W6L0oTY3iJ1CliOiDlqLNnsDsdfpjZ1uj48n6aGL1voOnQsdP03hiWXLeAztZb+CbhIXUva2OnHqGEIK2MJKf0DL4rqvxARJ4EnwHLWOuM08DFrjuwicGQ49Ds1T3LbCnA7IqaG3zMbEceQDiE/oatnRyNilVpe2aQugi4N7z0GXkTEAvAS+Dm8/h7YjIg14BnwgPrmy7thos0X4Nq4TkDaDy+KSlInXHKRpE5Y6JLUCQtdkjphoUtSJyx0SeqEhS5JnbDQJakTFrokdeI3Y04Vbn9ymZ8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_stk.drop(['Volume'],axis=1,inplace=True)\n",
    "df_stk.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>Returns</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2006-01-04</th>\n",
       "      <td>2.683214</td>\n",
       "      <td>2.713571</td>\n",
       "      <td>2.660714</td>\n",
       "      <td>2.677500</td>\n",
       "      <td>2.309747</td>\n",
       "      <td>0.002939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-05</th>\n",
       "      <td>2.672500</td>\n",
       "      <td>2.675000</td>\n",
       "      <td>2.633929</td>\n",
       "      <td>2.656429</td>\n",
       "      <td>2.291571</td>\n",
       "      <td>-0.007901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-06</th>\n",
       "      <td>2.687500</td>\n",
       "      <td>2.739286</td>\n",
       "      <td>2.662500</td>\n",
       "      <td>2.725000</td>\n",
       "      <td>2.350724</td>\n",
       "      <td>0.025486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-09</th>\n",
       "      <td>2.740357</td>\n",
       "      <td>2.757143</td>\n",
       "      <td>2.705000</td>\n",
       "      <td>2.716071</td>\n",
       "      <td>2.343021</td>\n",
       "      <td>-0.003282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006-01-10</th>\n",
       "      <td>2.723214</td>\n",
       "      <td>2.924643</td>\n",
       "      <td>2.708214</td>\n",
       "      <td>2.887857</td>\n",
       "      <td>2.491213</td>\n",
       "      <td>0.061328</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Open      High       Low     Close  Adj Close   Returns\n",
       "Date                                                                   \n",
       "2006-01-04  2.683214  2.713571  2.660714  2.677500   2.309747  0.002939\n",
       "2006-01-05  2.672500  2.675000  2.633929  2.656429   2.291571 -0.007901\n",
       "2006-01-06  2.687500  2.739286  2.662500  2.725000   2.350724  0.025486\n",
       "2006-01-09  2.740357  2.757143  2.705000  2.716071   2.343021 -0.003282\n",
       "2006-01-10  2.723214  2.924643  2.708214  2.887857   2.491213  0.061328"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stk['Returns'] = np.log(df_stk[['Adj Close']]).diff()\n",
    "df_stk = df_stk.dropna()\n",
    "df_stk.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a23007278>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEACAYAAAC3adEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXmYFNX1sN8zDPu+yY4DggsoKo64I4oKiopGUGPiFg0xkehPoxGTqBE3YvJpYsQorsTELWoCihviBriwKKCI7IgIsu/7MPf7o6p6qnuququ7ep/zPs880111q+p0Vd177j3n3HPFGIOiKIqiOJTkWgBFURQlv1DFoCiKokShikFRFEWJQhWDoiiKEoUqBkVRFCUKVQyKoihKFKoYFEVRlChUMSiKoihRqGJQFEVRolDFoCiKokRRmmsBUqFVq1amrKws12IoiqIUFDNnzlxnjGmdqFxBKoaysjJmzJiRazEURVEKChH5Nkg5NSUpiqIoUahiUBRFUaJQxaAoiqJEoYpBURRFiUIVg6IoihKFKoYipGJfJau37Mq1GIqiFCiqGIqQ+978hmPuncT6bbtzLYqiKAWIKoYi5L1v1gCweefeHEuiKEohooqhCKk0BoASkRxLoihKIaKKoQhRxaAoShhUMRQhlZXWf9ULiqKkgiqGIkYVg6IoqaCKoQhxTEkiwh3jvuLz5RtzLJGiKIWEKoYixFEMlZWGsZ98y4WPfpJjiRRFKSRUMRQhtl5QFEVJCVUMPlRWGr7ftDPXYqREpa0YHB+D+hoUpXB4cOICjhj5Tk5lUMXgw+j3F3HCqPdYum57rkVJGuPyMQAIqhkUpVD426SFbNqR28mpqhh8mLp4HQCrNhfeqMGxJDkKQvWCoijJoIqhCHEUguoFRVFSIS2KQUQGish8EVkkIiM89vcVkc9FpEJEhsTsu1xEFtp/l6dDnnRQyA7cygKWXVGU3BNaMYhILWA0cCbQA/ixiPSIKbYcuAJ4LubYFsAdwDFAH+AOEWkeVqaaTmXsiEGHDIqiJEE6Rgx9gEXGmCXGmD3AC8BgdwFjzDJjzBygMubYAcBEY8wGY8xGYCIwMA0yhaagG1Pj/LOd0GpMUhQlCdKhGDoA37m+r7C3ZfrYjJJtU9Id477immdnpuVcsSMGRVGUZChNwzm8uqNBm6TAx4rIMGAYQOfOnQOePjzZ6m2P/eTbtJ0r1sdQ0KMfRVGyTjpGDCuATq7vHYGV6T7WGDPGGFNujClv3bp1SoKmggms4/KHyIghx3IoipI6P2zO3fK86VAM04HuItJFROoAFwPjAx77NnCGiDS3nc5n2NuyxpK123jsw8XVthdyL9tRCJFkerkTRVGUFDntgQ9zdu3QisEYUwEMx2rQ5wEvGWPmishIETkXQESOFpEVwFDgMRGZax+7AbgLS7lMB0ba27LGRWM+5b43v2Hb7oqo7YVsn682j6GQtZyi1FBi26Rskg4fA8aYN4A3Yrbd7vo8HctM5HXsU8BT6ZAjFXbYN9/4aIJCjOip8jHoiEFRlOSp8TOfnd50AQ8QquEoOZ3opiiFxYbte3ItAqCKIdKbLmTTUSyqEBSlMHln7g+5FgFQxVDURJSd2pI82V2xjztfm8vmnbnNZKko+YYqhiLGqI8hLq/M/J6npy7jgXfm51oURQGqm7S/yNGyvKoYIrak6M3FYI25Z8I8QKOS/NhnD6kq1Pam5AmxJu1nPl6WEzlqvGKo0gupNQ7Tlm5g8Oip7KmITQOVeyYvtNaUUL1Q2Bhj+HrlllyLodQgVDH4tJpB29JbX53D7O82sXxD4a30pljk+3hh/OyVnPXQZN76alWuRVEyTGwHNVd9uhqvGPxItrHIl6imuSs3V9umA4bCZsHqrQAsWrMtx5IomSa2HcmVGVgVg41fw57oueSb/d6r8ciFjMvX7+CP4+dSWQD2+/x6gkpNJl9qS41XDE6b6fdAgo4E8uWB5gvDn/+cZz5extwCsI3rs1PyFTUl5Qi/Gx/0gRTCBLlcvFyV+XxDbFK5L3NWbPJNn6IoocmTd6vGKwY/Ai8ooXaIgiXZKvjWV6s49+GpvPr59xmRJxF50mbUaN7/Zg2bdmQubUW+PGJVDDa+SfQKrOH38ifk8jcU4noWfixea0WeLVqbXSdwISZyLEY2bt/Dlc9MZ9g/ZzJ92Qa27MrCjPkcPfoarxgSJdEL7mNIvQEsGzEh5WODYf3GL1ds5rXZ1jpIXyzfGLUQyGMfLqZsxAQq9qVnPkYhNGbJSuh0Hkqy/NOKSbkWMnvtujF35WaGPvoJw/45I+3XqBaVlKN6lJa024VM2NuejQe3dddeatcqoV7tWikd74wYznl4ivX/8Pac/8jH1KlVwoJ7zgTgr+8uBGDPvkpKa9X4/oInkfUtCkDp5RPjZn1PwzqlnNajTa5FSQt77Ui7bEw6zNVov0a1AAtXb+WSxz9l19591falGq6a6Hg3e/dVUjZigueKcfE47I/vMPCvHyV1jBsB/mY3/G72pGl0EI98sYtv313BtKXh1oDK1U8pdEV0/QuzuDoDvetsE5uTMhPvQ74ENtQoxXDna1/z8eL1UQ3E+pD5z5PR6DtthfTwe4uSvs6y9TuSPsZhzdbdPPjugpSPT4V8881c/8IsLnzsE9Zt253yOapWxEuTUEo1np+2nNnfbcq1GDkjVi1ouGoWiDdnIYgdd9XmnZSNmMC4WdWjUvJE0SfN+9+sSfs5H5i4gDkrqs/AzgTGGJ6YvIStCRyB81ZZw36v0eKWnXuTWiBF9ULmuPXVLxk8empOrv3t+u28NOO7nFzbIV/akbQoBhEZKCLzRWSRiIzw2F9XRF60938mImX29jIR2Skis+y/R9MhT6ZYsNqKRnl55opQ5zHAna/NDZTiYPOOzEY+vDg9uiKk48V8+L0qs1Wm3/MP5q/l7gnzGPna14HKe/2+1+esovddExMfS+EOGT5dsj7wLPTdFfu4euwMFq3ZmmGp8ovBo6fy25fn5FqMvCC0YhCRWsBo4EygB/BjEekRU+wqYKMxphvwIPAn177Fxpgj7L9rwsoThFzb8bbtruDpqcu4euz0hGV/9I/M9p5KYt6AdExMy2YKDmcEsHVX5hdOr3I+Fxbvfr2ai8d8ytMBUzjP/HYj785bze//+1VmBcszNiXohGXDlFjNlFTAzuc+wCJjzBJjzB7gBWBwTJnBwFj788tAf8lBAp+4l3Q9kfXbdvPNquQiDoKYomLb3CBNsBM7nykcx6Yjf56MZDNGmLeuUO/Nys07AVi6Ltj8iyDO7q9XbuG5z5aHkitfyWXHMfbauQo8SIdi6AC47REr7G2eZYwxFcBmoKW9r4uIfCEiH4rISWmQJyGJHvsZD37EFrsHmuixROZBpPAuZeL9m5ekQoslX2ycyeIoto8XrWPMR8GivhauTtJUYt+cZJWLMYYHJi5g/g/hTDOpPppU07bEK37WQ5P53X+/TFGi/Caf6sDGDM6yjkc6FINXNQniXDfAKqCzMeZI4EbgORFp4nkRkWEiMkNEZqxduzacoF4P3iWhO1Ip0TsStCGu2FeZsWgLYww79liK7B8fJBcK6/zuyMghHaak0GdInUue+Ix73/gmUNmxn3yb0jWS7cVt37OPhyYtZOijHwPw1lc/cMXT04JfL/Rkm/iTONN+vQLHd7JrZKlc7xs0bekGvvo+vUEX73y9Oq3nC0o6FMMKoJPre0dgpV8ZESkFmgIbjDG7jTHrAYwxM4HFwIFeFzHGjDHGlBtjylu3bp2SoFVRSR6PPoX2cH0SoY9/f28Rlz0VrDFYtGYbv/vvl9WchX45WsZ8tIQet7/Nmi27PPfHI3YFu3h6YdZ3mxJG/+SKt+eujprJ7UcYvReJY0+x4dyyq4LJC9dyzb9m8sH84J2bsLo65USPedRzjmX1ll08Py15U9Zzny3n40Xr4pZJtXN04WOfcPbfp6R0bL6RDsUwHeguIl1EpA5wMTA+psx44HL78xDgPWOMEZHWtvMaEekKdAeWpEEmT5wKsnVXBc9+sizwC/DF8o289011zb0vibUGFnvk1/HzS/zi2Rk899lylsTYhIc/94Vn+QlfWit7rQzQMMZSEtPK+f2i3RX7OG/0VH4eYKJSrnqc837I7ExUxzEf5udd+mTwkUK6kLhDZY/yGZMkfVz59HRuffVL1mxN7p3/3X+/5JInPotbJpf6cPPO/Oh4hVYMts9gOPA2MA94yRgzV0RGisi5drEngZYisgjLZOSEtPYF5ojIbCyn9DXGmHDTU+Pg+ANGvvY1t42by9RF6xMfA5z/yMf87JnqDWIy/vMGdZJPZxGrt6YsWsfBt70ZUWhbdu1lwpxVrh5h8q+0xJiS/KKSKu1J0rNizGGDHprMpU9GVzT3UDvTjrxcKCER6HPPu/zkiU+zdr1Qx6fY1K9OstHNJs5ExcoUJ+//7Jnp1d5bh0Sv7E6PuTDpYu3W1CdgppO05EoyxrwBvBGz7XbX513AUI/jXgFeSYcMyeD4EBy7fDzivSPJJFPzyj+0apN3xYuncHbtrcQYq7G48cXZvDtvNU3qhX+MQUxJzv69+yoZ+drXXHtKt1AL8eyrNHy2ZD3Hd2uV8jmi5M1wV899rTVbd7Nm626+XrmFHu093WIZuW42zuO8f9/as+03bN/DwtVbOaZry3iHZZVkzHrvz1/DkrXbuerELpFt78WZ2Bk4aWHAYn3vf5+urRvyzJV9Ep8yT8x3NWvmc8z32N7vum27Oe6+SVHb3ImyYo8POmLw6zVXVBpGv598egyHFRutirurwuo2pfJOxf4Cv0rhnjU+ZdE6nv3022pRKbO+sxexiTnpzj37KBsxgRenR9uEH/toMZc88RkfLkgtmCCbTJizihnLNgLRz/2shyZnTYZUG43Is0vx+Ese/5SLxniPjrbs2sv9b30TyTyaLZKZU3Ll09O56/Wv+W5DsLQy6W6cl2/YkZRPKR+oWYoh5i16JCaCZ9K81ayKsdOPfL1qRu2rn0fPeI4dMRw58p1qyeomzVtNl1vf8I35/vPb84OIHpdkDQVPTK5y42zYsZf/ffE9u/baysWjUpz6lw8ijj5jTEQDuc1OUxau47zRU3lq6rIoeR6YuCAyPP57TI4oJ0rrBzvOPjQBb8SMZRv4T5KpD6597nOmLUvNyuknlrvD8PXKLZSNmMC366vPW3Ea3VTyXX20YC23vmop8KA94dh68o0dZuvVwfnTm9/wyAeLI+ncAb7bsINtuzM/4RBI6uWPN0qA+KOPG1+aRf//96Hnvp170mdaypcU6zVKMcQjyOP436zoYKtY2+3GHXsjlXf2d5soGzEhks46KEvXbY+kypg4zztUbca3G6O+77ZHDLOWBwuHvXvCvMjnjxas5f9enBX57qUYlqzbzp12yom9+wy/fr66E9wZvcyPcQBPXrjO0/EOViQRQBI+/GpEVWaP88xbtYVHY7LZDnn0E24OkfogtgFJdcEW973+z0xLUb07r3rjFcam/cSUpZ7X86NiXyVDH/3Ec5/X8c67V+F6iCfd/z5D/vGx5znmrNiUphXQqguzdN12nv3UPwQ5qOnX+Z1bdu3ln59YQSqvfv49OzwUwJK12zjk9reiOhonjHqPa5/7PNjFfK6da2qYYkivp/LNr1bF2fcDAF8mGdd8/iNVKTDuf8t7NHHhY94V99UvwuVwAmsUsGrzTrreOsE3JtvpDbqHx25zRWzDmagXlEx0VyzxKtIv/zWTM/82mVFvVs1rOOn+91O+lkNsh6DXH98Jfc54ppG0+RgClNkQp9HeZwzLY7L8frZ0vefJv/GZzHfuw1O5wEdppIL7WVzwj4+57X9f+eaEcpSY/7ksnPf1tv99xe3j5vJZnHTtp9qjiImu+Qbfb9rJhDn+bYMfe/dVRinYXFLDFIM/qVS+Ea/6z/xMdUiYKF9LPPYkePGDYIDrn59FpYF/fxZ8AlhVag2PcyZ0aEcXeHDiAspGTIgb0TRj2Ya4GVF73zUxopzTTTIhkpt37PWddBjlN4+zOlwY84L7HgZ5x+NF+Qx+eCp9//w+S1wjwO82JG8GTEeaF6/f4syx2evzI9wj5SDn3mjXRa+MvABbXeayIO7G/36xIm7AS/nd7/LfL6pnbr7qmels3rm3mk80k9QoxRDv4V085pPAJo1ASiQHir9iX/iLGmNSsqXHc3A6DbRTZt6qLRHTE1S/VX+btND3XN+u307ZiAkMefQThj76se8zTSaNdrI8PXVZ3P1lIyZwt+2bun38VzzsE2DgbrSdd6/EQzMket8WrdlG11snMGVh9MStTTv2RGXwjVUwT01ZyqQYc2W8JIpf27P8Y/1wQM4mP7ifvzMnJ9V6ELvMr3PqZwImH0zEDS/O5q7X/bMA+81hmPTNGq56ZjrnjZ4aanSdDDVLMcTZt2z9joiTLiyVlSbjC4WXjZhQLcoi0VA5CO52YeP2vazcFKxHWJUzylQztThpyp1zn/m3yZz4pyqTjt/Q36uR+nhx1dyT2J6nny8jWxhjIr/Fse1vi5P11fl1lZUm0tv1ekcTtQWvfL6CSgM/ffIz5q6sMv+d+/DUqEY89rmMfP1rrhobPT8nSHZdzzIB2qt0zGkZ/tznvPfNas/LlZaEUwwOjpyO0gkSURR0rsjrc1Yx+7tN3DHuqyhFkCgluuNXrEh14kaS1CjFEMZMkwh3BXxh+nc8Py3zC35sj3GGpWOpTrcN/q25P3D8qPcCHVexrypk1q9xWbHRW8ms376HByYuiFQOp0JWGuu8r8xcEdkXa4O95l9VTr6gpgI/Vm7ayfPTljNt6QYemLggKVNa+d3v0uXWN5JaIc65TQ9MXFAV2OA5BPK+n2999QNbdu2NapIGPTSF37w0G7DCJN288vkKLvRxLAPc9+Y8nokZDX3jMZs81U5r7HGfL9/I4Xe+k9AZ/cK05ZSNmMDG7Xt4fc6qqMmm7t9ey1YM8fwk8cjkkp0OW3dVMHj0VMZ+8i0PTqyKMpu6OH6aDof/eZiaMkFaJrgVCqmGG8YyZdE6ykZMiNrmThWRq6yTFVmOJXcTmYlqTNyRywyPZ+CEsR61f3NOPrAqD1alMTw1dRmj3vyG1Vt3cVCbxoEXm4klSG/1yqenMz8m4+o5h7cP5Eh0fv/qLcEVw4szvuOwDk2jwqC9fAxencTl63dwzb9m0v/g/TiobeOofa98voL/d+HhntectmwDgx+e4rlU7GMfVs9GE+tsBm/Fv3PvPrbtrqBRXf8mJdYMMvq9RWzeuZcZyzZyWo82vsc5yQ5XusKavZ6nkxF5+HOfM+G61BM1r9q0i15/TUdAgYk718nd+w/6Wt/yypdcdHTnsKIlpEYphmInHc7nsCRqf5eu83c87qusZNfefZFzGFOVIsCJ0Lr1zINTkytAGS8b7z8+WJxUxtrdFVWjuLkrN8f1a932v+oL4cTmrgIrGigWJ4T1u407qimGRMyOWXZ13KzvGXxEbKZ8i2HPzqy2zatRvmP8XO4YP5dlowb5XjfVRaCceRy1PbIHeDW8S1J0bjunmvFtejqQ974xL66VwlEG7369OqnR6bfrt7N/y4ZhxYuLKoYiIh2mpFQZZ5tCEsXcx2sbnp/2Hb99uWq0tWjNNp50xeEDkYl4meAHj+y0G7YlZ5Z41NXrHvTQFE47ZL+kjv9+4062766gTmkJm3bspXXjup6jpHSuTfzMx8t8FYMXe5Ow4btHCckohv9+sYKm9WsjSMSBXuoaTsU7U9hcRsmM+hy8OgCPT15afaML53ZcHSAxpZtshLSqYigikqmw6WahMykvQf74eBlQY499+P3qkwNTrfSp+j1fTLIBftdnUmJQHn5/EZMXraNjs/pM+HIVC+85M6pB/e3Ls7n3/MMiCtPP6Tk1QWppN18s35SUY3hLnAygsee57oWqyZDJtGc3vDi72jav0YExhpWbdtKuab2o7Qff9iaXHVfGzQMOinudDdv30KReKaW1Sux7aXhoUnKTUoGUQqPXbt3F4Xcmb7LyGlWmG1UMSlZJFOrpxiss0i+mPBFeo4F8ZfZ3myLpQvZVGtz6/qUZK6KUnF8b8ZMEqaVjmRZnElcs8WaNu+/z0nXbo/wzbgU34MGPaN/Masxfm7OSNk3qcVjHpnGv6x59OKd6fc4q7hg/t1rZXXsrGfPREsZ8FD+Lf++7JvKj3h0Y1rdr1kfcXrPcg5CNyGBVDEreMmdF9ZnX6cxLkw3ChhAbUz2U8T8zo2e4p6MDma5U0sfdVxXFNvCvH0Xt27G76hrzV2+NOF/HzVrJuFkreeKy8rjnvn1clU/GmX3vpRSS5dXPv+fVz7MT7ZMOdMSgKDEka9rJNZMXBjfpeGEwkYWYvPjmh62+6SeSu076iVWKsRPFYuehvDYnduHHaNxzWLI10SsfycYaJDVqHoOiFBo9bn87K9d5akp8R2k6iE1mGMu4WfEVg2KhikFRlKwQdmSjZI9kVo5MFVUMiqIoBUQyK0emfI10nEREBorIfBFZJCIjPPbXFZEX7f2fiUiZa9+t9vb5IjIgHfIoiqIUK6mu4Z0MoRWDiNQCRgNnAj2AH4tIj5hiVwEbjTHdgAeBP9nH9gAuBnoCA4FH7PMpiqIoHhTKiKEPsMgYs8QYswd4ARgcU2YwMNb+/DLQXyxD2WDgBWPMbmPMUmCRfT5FURTFiwJRDB0AdwzhCnubZxljTAWwGWgZ8FhFURTFJhvzGNKhGILkCfYrEzjHsIgME5EZIjJj7drE+dEVRVGKkWzMfE6HYlgBdHJ97wjEBiRHyohIKdAU2BDwWACMMWOMMeXGmPLWrVt7FVEURSl6CmXEMB3oLiJdRKQOljN5fEyZ8cDl9uchwHvGyrY1HrjYjlrqAnQHpqVBJkVRCpAOzernWoS8pyAmuNk+g+HA28A84CVjzFwRGSki59rFngRaisgi4EZghH3sXOAl4GvgLeBaY0xhJcNRFCVtfHBzP/4y1HuRIcUiGxPc0pIryRjzBvBGzLbbXZ93AUN9jr0HuCcdciSitESykstcSZ1lowZVWx1PqRnUKS2hdq0ShhzVkS079zLy9a9zLVJeUhAjhkLi09/1z7UIihKK0gBB7InWIMgV3fZr5Ltvxh9OY8YfTot8b68mJV8KxcdQMLRoUCfXIihKKDo0T9xgbtye3Kpz2cJvCdJZt59Oq0Z1aVKvdmRb5xYNIp/vH9IrI/I0qFOLhfecmZFzZ5JCiUoqGEqyMWWwhjOgZxvuGtwzYbm6pTXq1UuZ03u0iXx+8/qTAi2PeWTn5pkUyZNAfgEf0b1SPPRo34QPb+7H0vvO4sLyTh5HhWf0Jb0915FOhn9ffQwA9Wpn9n3u1KKqQ6AjBqXgeOzSclo0rJuw3JCjOmZBmsLBz/xzuL2q2W8HHsQh7Zpgr20TNy3CoF7t0i1eQuK1rxf0tp5143o+Lk2f37J/y4YZdbQ2rOstj7sRTsQBrS3zWP3ayWfycZ7hPecfyskHxg/Bb9e0Sib1MSgFhVM5gry4xx3QkndvPDnwueeNHMhTV0Sv8HXKQZmdzzKsb9dQx8c2FgfHmFLaNrGWtvy/07pz7Sndqh3/+q9P5Od9u3LTGQfysxO6ADD4iPYANPcxizo92GRJ92C6u8uf8JehvVhy71nU82k8k23oPrr5lDCiRfBSZmUtG/DjPp0Dn8NZhS72tzWsk1hR1LJvepeWDSMdgCCoYihw6oQcpuYL955/mOf2P10Qvf2L208HgtlAvVbgGtCzjUdJi/p1atHM1RgOOqwdT18ZPK3Wq786PkqR9O7cLOExYetf7VpVZ3ju58fwwIVHeJa76OjqppILenfk0A5NqVtai+Gndo80PDedcRBf3TmAZ6/yVgAtG6XmR/uXz/lSxf10RYSSEomYwX531sFRZZO9z7EN47Tf92fctSckPC52joRjkll631ncNbgnc+8cwHu/6ccvTz4gsCy79lqKIbYT8P8uTGxau/f8wzjn8PaUl7WIa+b+58+i3/OCyK6q+NOkfm3OObx9rsVIiVaNqsxBB7Ru6FlmR8z6y/UiI4bEL27FPkOs0fmxS+Ov+dvKNlEd0akZo3/SO+E13PTu3Jw+XVpGvjd2OTrd/PUiV+Mdsv7VdTUWxx/QioZ1vXuRXm6Duj4265ISoVHdUnq0bxJOuBjaNK0X6vjY3xC7TjUQUQyxvetUAsg/uKlf5PN+jetxeKcqRe/XWx9+avSozOmxiwiXHldGw7qllJRIYPPVa8NPjPjKeu8f7dcZeGhic15Zq4b8/cdHUqe0JG60WWw9K5TsqooPL19zXMbNHelk0GFVL3MDV+XyqyjNGng3rn716jpXxUxlzd5OLepz+9k9ePiSI5M6zjFruOXyssN3aFaf845MXw7HRCPG2PuUyM4chAC+aU9aBfALxePYri2jvtfxCC5wZBOgS6uG1bYnw/4tG/jue//mfhGzYzuXwmtWP/p9Ddvz7tm+CZ1aNOCVXx7P3ecdWm3/5N9Gm7wcc2DV9auoVeL/rlQaE6U9dQW3AuVnJ3ThycvLKWvVMGrE8PQVR+dQqsQc0q7KBu6257fyME+cfGBrzjvCuxH1i5q4wOVw7tnBv8f74c39+OTWU6ttFxF+dmIXOjb3bxRiad6gNq9fdyIAlx23Pz/u04kvbjvdM9Iltm5mesheWiv6/GN/Fj7jfCqN7JJ7z6Kpj5IPwjs39KV9s/rMsk2JAH8eUt2UYo0SobRWCf/91fGR7SZJoeP16q895QD2a1yPHu0sm727AzLw0LZRZcO0r+/c0Ddi/jlq/+ae/pNOLRpwhiuqzH29K08oi4oei9eHiO1E6YghS/QN0FM7036pji5rzrJRg+L27m4/pwf9D7FeiLDhcNnm1V8dz0c3n0K3/aqURNfWjZhgN64O5x7e3rdy+r23jsJo3qA2Pdv7O9v2b9kwKgojKKMv6V3NdDfqgl7ULbUqbYM6pdz3o140b+hth49VaAe2CR5xcrGHnyARz1zZh1/2OyCqV/ujJEYsLX1+RxB6tKtSzE4D1//g/VI614Ftop3qtWsJh3k4U9s1s35n2yb1ovxFTXzMen7U9mmXTa4VAAAeAUlEQVQZl40axM0DLP9Fi4Z1qFUi3DKwyp8R+76GUQyxv9kP9zslWJ2TJy4r545zekZMWbHlYokNUdYRQ5a4pE/8Sn1d/+7cdra1KN1ph/g7SFPBiTLJJY7JwxjLFt/ZY5jes33TyMzVi8o7xTW5eB0PVRXRMTN0bdWIK44v48A2jeiZBpv5oF7t+L/Tuke+jxzckwE928Y5IprYynn+kR14bfiJTHfNyPXiw5v7MeqC6pOwjDH8++pjfCOFDmjdiFsGHhxV0Y/cP/gchPd+04+pI6JHVibGYu8OER1x5sEsGzWI8cNP4PmfH8sBrRtyw2kHRvb/9Lj9E14zNqzWUZ5QNcLyG2lde0o3Hr+snH4x5tVk5xeVBuhs1SktYfG9Z0WNUgEW33tWROZ4jXGy5kqAyz3un3sUKgIjBx/KaT2qtyHxfAypmgfDkJZcScVOk3qltG9Wn9l3nEETu6I5vcg6tUrYs68y5XP/7eIjuX9ILw76w1tpkTUVTjm4NW/PXZ2wnNNz+XnfrlG9nVj8elOxPZ2SEuGP5yaeDJcKXVo15LLjypI6xq1UwJL3sI5N2b67IiUZDHBCt1aR752SMIEFoWmD2glNQG9efxKXPjmNpeu2R7b16mg5aif9pl/S17z2lG48P205KzbuBODsXh4dG/sxv3ND36h7V7tWSdSEvU9v7c+eiuTrTu1aqfeY3e9tvI732b3aM+KVL9mWxLM/sXtrxn7ybdS2oL37nh38R9Cp+OPCoiOGJGhav3bkQZ9lOy9fddlKE+LzjmQj/MyLQ9o1YdmoQXTfL9iw2Om5pGrjdI7LRQ8oHg3q1GLJvWcx2MdnEpRY82Ls7ywpEe44J3Y5dG9SfSNir9mxebSdO1liQ0uhunnE+4vVQYg3C7tt03q+o8t4uM2zjX0mqcWj6j2Of5eT9X14nS3qXsW53tFlLXz3HRkgtDrd6IiB1Bqqcw9vz+mHtKF+gIksyTD5t6dw0v3vp/WcfgRJyObGGTGkOiU/mwowmSt9cFO/uOaMoD/3ycvL6fb7NyPfY53Lbk50jSQywR3n9ODZT6N7r6m858P6HsAxXVrywfy1EbNnovuR6afsKIa/DD2c8iRMbw7Oe5xuOR2zkTt01v1apXK9OX88I8oH8+KwY1OULjlq/Ijh8E7NOLF7q2p2zyAEVQqJwhDdFa1TC+8e1GOXHhVYrqDEa7i8cIa0qSoGp+L4zYCN5bIANu90sF+TcDH8Dm7b93Wndqs2MQmqGge/uSHp4soTuvCebSpyQkM7JkjA5/dUD+/UjOtP606ZfR6/559Jn6g7Ss4xBw05qmNEpmRw9GO6nbhOx8cdFpuNvEaZoEaPGH5/1iH83E57MPyUbnww33st6VSUhptnrrTCVD9Y4H3+IK9O9zgpix3aNqnHD1t2BZardkx8ZqIOZWQInmJ3onWjutw84KCo+RLxGDn4UP4ZY7NNRCbMVHVLa9GkXilbdgW3N994Rm5SX3v9/ouO7kSXVg3p08XfXJEM7vfVq93LRFt46sHpC/qIzKdIIGeyr5JzPvdxUYohxH2JDSrINDV6xOCOWHE/vzKX3XPZqEFRoZupIBJ8NqUXJ3RrGTWL1v86yZ3XGTEEHTnEMyXFOm69EBGuPaVbSr28XFKrRJjzxwFR28r3b55UsrVYMlXNnZBQNyLCMV1bpq+H7Osryyyp+BO8MCFNon4499ftm4g2JRXO6KFGKwa348t5lr07N+ODNCXpqoZf2uEEL+i/rw5mV7wpyV7qj+ysl2f0sBTkSd3j273PtecINKlfPRImWX9FunBmk/761OpJ6NLN7DvOiORzutRl5kqmwmcyBn3B3WdGpTJJlqCy5ap5e+uGvjx7VfhJgM7vTPfvSOx8TvMFM0iNNiW5cWL0rzoxXEbNTOHuhTRrUJtNO/ZWK3PcAS2rbYuHk/q6R/smLLn3rITx5LcMPJjhp3ajkUfPLXZiWbKml1RpbodrVjOhBKiEJ3Rryead1e+jH03r1/acsJjtYb4fXmkoksHruXpxYJvGLF67vdr2TE+86tCsfrVEeKnw+GXlPPfZ8rhpNVLBUQLuCWmpml0dYt/rbExug5AjBhFpISITRWSh/d8zREBELrfLLBSRy13bPxCR+SIyy/5LbeplEky5xXs00KxBHZaNGpTZXPYhhuDuF2T8tSeyyGPlqTDvjJdSGHpUxyj/SkmJ+Caf279ltHno3RtP5mcndEk5DXRQnN8cZAGbWP599bG8/uuTqm2vU1rCb04/0OOI9OErbgq/46ObT6k2Mz0Vjtq/OX//ceKJXX8Zeji94qSJzkeTyTEu/0q3/Rpx+zk9EjayyT6KyLtY6d7mE9qb54QdMYwAJhljRonICPv7Le4CItICuAMoxzKmzBSR8caYjXaRnxhjZoSUIzDJ5NnJFsk26CLesz/TXSH/HGRVLh/2a1KP2+2Y/SFHdeTlmStSOs+jP+3NpHlrfPdH7Lopnd2bBXf7L/fo1ZgkZ0pKSaS4pDIXwI9zDm/Pr5//AqhKAxNLw7qlnNitFXNWbI7anq8N39L7zsrKdaqcz94+hkQ+jbsG9+S2cXMzIVrShPUxDAbG2p/HAud5lBkATDTGbLCVwURgYMjr1mj8wmSTaXT8MqNmgvsv6MX8u1N75AMPbRdXQYUZMaRCuho/X/NTHhmir+ufOKDATVUYaPplCUOqwR///vkxERMzwKkJckk5HQT3RGW3MrimX/x1Hi71mqkfyUhb3bGdScKOGNoYY1YBGGNW+ZiCOgDfub6vsLc5PC0i+4BXgLuNzy8XkWHAMIDOnYOvsJQq/Q5qHShENB0EeWk7Nq/PVSd24cjOzao5GJ//+bG0alQncKO1bNSgFKRMnZISoW5JeicCRs5dLUYwOxUn1fqZ8Bnl27TwGkzvzs2557xDuWjMp4BlQut910Tf8pFq7KEYbj+7RyAfTv3atdi5d1+17Q9cdDiPfbiE8jgzpNNJQklF5F3Aa0z5+4DX8KoLzq37iTHmexFpjKUYLgX+6XUSY8wYYAxAeXl5xmvPM0msDpYNRCSSyC8Wx+m8Zmv8OQxnHtq22qIfYXn1V8ez1MMRmS2cl6taBspMXS/PesO5JlvO0Fzh7v23SJDNtiRi1jTVtgUd0bZrVo8lHvWpY/MG3OWx5kOmSKgYjDG+qSVFZLWItLNHC+0AL2PwCqCf63tH4AP73N/b/7eKyHNAH3wUQyFw1+CefLRwnX8B17vRq2NTftE3+BKCXoy59KioqJpEtu5//DT9s6d7d25O7zj5cDJNpDJmuaMdNhKpmAcGxaQqnOd8bNfEPfWSiFmzatvRZc15aurSqDTnCS4YRbLZCdJFWB/DeMCJMrocGOdR5m3gDBFpbkctnQG8LSKlItIKQERqA2cDX4WUJ6dcelwZj18Wf3lKhz5lLUJHQJ3Rsy1DXQvOFHnnzRMnu2hTj7kVmSDsLXbSbyQKgsjlszwo4FoDsWTL/p1NgibcA5fz2XUfzjysHZ/9rj/Hp5gbq2GaJvUlS9irjgJeEpGrgOXAUAARKQeuMcZcbYzZICJ3AdPtY0ba2xpiKYjaQC3gXeDxkPLkNxmu7IWalyUMQ3p3pGKficzJaFrfGu4nO6cjWfzawCcuK2fV5p2+x53Row1PXVHOyQdmPDI7ZxSTeSmScC/QT/KOkGuTTC4u13XCLMQUllCKwRizHujvsX0GcLXr+1PAUzFltgPpt20UCFefFGwi3R8GHRL4nEFWGis2SkqES46pCkZo3bguH9zUjw4JEsalirhMV17KwWsRltjj05n3J58ovvFCciMGZ530lmHWz87y2s5+6MznHNG2abBeRFAFAsGzvRY7mczFVDx94cTEa5eKUQl4UTViSPzkD2nXhPuH9GJAj+CrBsYjlwOvGp0rKdu0CtOTUPKOIrKYRHGM7WhtVj93pox8IZKJNWD5C8s7JVxVLyg5Sj8G6IghLdx65sEcGmdpPgevBdKVwqSYe8y3nd2Dy44rCzyqdahXao1Yf3psdtbRyAZOVFIuGukGdXLXPNdIxfDc1cfQqF76fvovTg4XdqoUDqcesh+vfvE9Pds3KdqQ09q1SqJm/AalTmkJi+89K6c93XTj5D3KVmCH+zJ3Zmg99CDUSFPS8d1aRRZErymcEnKxIcXi7F7tmTdyIIe44tLT2WYUiq65oHdHSgTOjgm5rlUSbu2RfMNZ3yJb6y67w92zmbYmlho5YqhpTPt9f/ZrnJ7lKxV18oOVoXTJfdlNrZILerZvysQb+nJA6+ykx+nquk4uR6SqGLLMhOtOzOoD/9vFR6hSyDDpfJ7F09cuHrqnOOGvkFHFkGV6tvd2QP/nmuPS3ij866pjODHBqmxKflEopiSluFHFkCccnYGsiaoUskMmTOr5uNiNUnOokc5nRVEUxR9VDHlI22RyqyiKoqQZVQx5yGSfdakVRak55NLfpIohD6ntsZ6zUjNwJpbpLHkll6jzWVHyiOMPaMX7N/WjrGX89RoUJZOoYlCUPKNLBrPDKkoQ1GahKClSrLmSFEUVg6IoihKFKoYi4+oTu+RaBEVR0kAu19AOpRhEpIWITBSRhfb/5j7l3hKRTSLyesz2LiLymX38iyKiK4MoBUMRJRFVlCjCjhhGAJOMMd2BSfZ3L/4MXOqx/U/Ag/bxG4GrQspT49HGKnuoj0EpVsIqhsHAWPvzWOA8r0LGmEnAVvc2sZK2nwq8nOh4JXluHnBQrkVQFCUEhTzBrY0xZhWA/X+/JI5tCWwyxlTY31cAHULKU+NxFkkpLaZltBRFySoJ5zGIyLtAW49dvw95ba+Wy1dJisgwYBhA586dQ166+FErh6IUNrns2iVUDMaY0/z2ichqEWlnjFklIu2ANUlcex3QTERK7VFDR2BlHDnGAGMAysvLtd3zQccJilIcFLIpaTxwuf35cmBc0AONFYv1PjAkleMVRVGyzRXHl1G/dmaXds3lWs8OYRXDKOB0EVkInG5/R0TKReQJp5CITAb+A/QXkRUiMsDedQtwo4gswvI5PBlSHkVRlIzxx3N7Mu+ugRm9RlnL3KdECZUryRizHujvsX0GcLXr+0k+xy8B+oSRQfFGQykVpbDJZR3Wmc/FhjoZFKWgyYe5SKoY8pQjOzcLdbzRuCRFKUjyYbSvabfzlJd+cRwV+5J/Q3QReUVRwqKKIU+pXauEMMEP+dDrUBQledSUpKSdfHipFEUpbFQxKEpIVBkrmaFA024r+Ye2UdlHzXZKOsmHOqyKocjQNkpRlLCoYlAURckj8qFzp4qhyMiHYWhNQ30MSrGhiqFIyeV6sYqipE4+9DNUMRQZ2nvNHvVqW9VH9KYrRYZOcCtSdMCQeZ65sg/jZn1P+6b1ci2KUoTksg6rYigyNCVG9ujUogHDT+2eazGUIiMfRqBqSipSdMCgKEqqqGIoMvKgs6EoSghGDu7JMV1acGiHpjmTQU1JRYr6GBSlMOnZvikv/uK4nMqgIwZFURQlilCKQURaiMhEEVlo/2/uU+4tEdkkIq/HbH9GRJaKyCz774gw8iiKoijhCTtiGAFMMsZ0BybZ3734M3Cpz76bjTFH2H+zQspT41EXg6IoYQmrGAYDY+3PY4HzvAoZYyYBW0NeS0kCXdpTUZRUCasY2hhjVgHY//dL4Rz3iMgcEXlQROqGlEfRsCRFUUKSMCpJRN4F2nrs+n0arn8r8ANQBxgD3AKM9JFjGDAMoHPnzmm4dHGjUUmKoqRKQsVgjDnNb5+IrBaRdsaYVSLSDliTzMWd0QawW0SeBm6KU3YMlvKgvLxcmz0fdLygKEpYwpqSxgOX258vB8Ylc7CtTBBrDvh5wFch5VFsVHMqipIqYRXDKOB0EVkInG5/R0TKReQJp5CITAb+A/QXkRUiMsDe9W8R+RL4EmgF3B1SnhqPuhgURQlLqJnPxpj1QH+P7TOAq13fT/I5/tQw11fioE4GRVFSRGc+K4qiKFGoYlAURVGiUMVQZOh6DIqihEUVQ5GiHgZFUVJFFUORoVFJiqKERRVDkaJBSYqipIoqhiJDBwyKooRFFYOiKIoShSqGIkXTbiuKkiqqGIoMdT4rihIWVQyKoihKFKoYFEVRlChUMRQpGq6qKEqqqGIoMkSdDIqihEQVQ5GiAwZFUVJFFYOiKIoShSqGIkV9DIqipIoqhiJDXQyKooQllGIQkRYiMlFEFtr/m3uUOUJEPhGRuSIyR0Qucu3rIiKf2ce/KCJ1wsijVKEznxVFSZWwI4YRwCRjTHdgkv09lh3AZcaYnsBA4K8i0sze9yfgQfv4jcBVIeWp8ehCPYqihCWsYhgMjLU/jwXOiy1gjFlgjFlof14JrAFaixVXeSrwcrzjleTovl8jAA5u2zjHkiiKUqiUhjy+jTFmFYAxZpWI7BevsIj0AeoAi4GWwCZjTIW9ewXQIaQ8NZ7TerThrf87iYPaqGJQFCU1EioGEXkXaOux6/fJXEhE2gHPApcbYyrFeyaWr2FcRIYBwwA6d+6czKVrHAe3bZJrERRFKWASKgZjzGl++0RktYi0s0cL7bDMRF7lmgATgD8YYz61N68DmolIqT1q6AisjCPHGGAMQHl5uXpWFUVRMkRYH8N44HL78+XAuNgCdqTRf4F/GmP+42w3xhjgfWBIvOMVRVGU7BJWMYwCTheRhcDp9ndEpFxEnrDLXAj0Ba4QkVn23xH2vluAG0VkEZbP4cmQ8iiKoighEVOAU2TLy8vNjBkzci2GoihKQSEiM40x5YnK6cxnRVEUJQpVDIqiKEoUBWlKEpG1wLcpHt4KKyIq31E504vKmV5UzvSSLTn3N8a0TlSoIBVDGERkRhAbW65ROdOLypleVM70km9yqilJURRFiUIVg6IoihJFTVQMY3ItQEBUzvSicqYXlTO95JWcNc7HoCiKosSnJo4YFEVRlDioYlAUpaDxydSshKAoFYOIHJBrGYIgIrVzLUMQRKSW/T+vK2C+y+cgIk3t/3ld/0Skp4jUy7UcAaifawGCUCj1CIpMMYhIbxH5CBhlp/rOS0TkWBF5AfiziByaa3n8EJETRGQs8AcRaWHy1CElIseIyOPALSKScPJOLhCREhFpIiKvAw8BGGMqcyyWJyLSS0SmAHdjJbfMS+x69AowWkTOcBrefKNQ6pGbolEMdnrvu4EXjTFDjTFb7O15pZ1FZCjwD+B1oB5wo7093+TsCjyClRp9f+AuERmUW6miEZFaInIfVkTHVKA3cIeItMmtZNWxlcBWoDbQQUQugrwdNfwBeNkYc74x5nvIy/ezH9b7+SowH/gp0DyXMnlRCPXIi3x8KVOlN7DeGDMaQESOE5G6eaiduwOvGWP+BTwIlkkpD+U8CphnjHkG+A0wCzhbRDrlVKpoSoDlwFBbzv8DjiV/TQsHY6U9+CvwExFpHGc1w6xjj2oOALYZY/5qbztdRJoB+WYGOQyYboz5N9bKkLWBbbkVyZOjyf96VI2CVQwicqGI3Cgix9mbvgUOEpFzRGQicAfwuIj8OHdSeso5H/iRiPwW+ARojzUUPjpnQhIZlh/o2jQd6CginYwxG7F65JuA83MioE2MnJXA88aYBXYnYCXW2uGtciehhVtOV2O6CNgDLLX/LheRzrnsFLjltEc1a4CTRGSQiPwPuAnL9HWzXSYnsnq8n5OBoSJyO/A50A54xB6R5wy7/RkuIsfam6YDnfKtHiWi4BSDbT64HWuRH4AxInIBsBZ4Dcs0M8oYMxBr+HaqiBycB3I+LiLnYg19r8davOgyW861wAUi4rW2dqblbCYiE4CJwIUi0sjetQuYgrXQElgK7WugZS4ckl5yGmP2GWM2ARhjdotIY6ALcZaIzYGcDV2NaTmwxRgzF5iL1Xn5h4jUzrZJyUtOAGPMVuBp4C7gKWPMAOAJ4FhXY5dLORvZcs4CBgJlwK+MMf2wGt2BInJIDuRsJyKvYSnQ5sDTIjLAGLMEqwOYF/UoKAWnGIwx+4CDgN8YYx4A/gj8EmuYPhvoiWW7B3gPaAxszwM57wBuAA40xkzCanjn28XHAb1yISfQEHgb+LX9ua+9fS3wKXCYiPSxf8/3wAnGmF15IOdJHmWOAeYaY1aKSCMR6Z5NAW387idYZq/GIvIi8FtgJrDAGLM3B47oeHK+jtXgOjb7GcBqYHcW5XPwfe7GmGlAa2CZvSln9R1L6U8xxvQ1xtwF/A34ub1vCvlTjwJREIpBRC4TkZNtWydYL2lzESk1xrwCLADOwRpe3g9cb/fATgdaYDXC+SDnXOBie2SwmKr1ro/MlowxcjaxnYtjgJdsGfqISAf7Bf4U+AJ40O6p9QSWi0iDPJDzGBFpb5crtQ9pBnwnIldiDeGP8DpvruTEamhbAz9gPfNfYpk/s9LDDSBnBwBjzBysnu9wEWmF5dg9FFifJ3I6z70u8DFwrX1of6woqmzW9362HJOAf7p2rwcW2p+devTXXNSjVMjblBi2XbYt8ByWLXkxVo/hF8B1QCnwkDFmk20qehEYaIxZJSKjsGz3HYFrjTHz8kjOF7AUVi+sF7o9ltNsuDHmmxzIeb0xZp1d5gSsIe8MY8yzrmMfwLqX+2OZv+aTIZKUc7rtxHeOfRb4CTAWeNBu4PJBzsj9FJFWrv2NgDrGmA35Jqe9/UagK1bAxA3GmK/zTU4R6Yk1Gm8L7MWqR7mo79cbY9aJFUiyV0SuA3oYY65xHZu1ehQaY0ze/QG17P8HAv+yP5dihX09idUzfBtr+NvA3v8icKP9WYBGeSrnf7BsogCNgMNyKOffgVdjyt6AFfbbFGjsHO98zkM5mzjPGrgYGJKncjYFGrruZ0key9nYtb12nsrZDKhvb6sPdM0HOV1lXgNOsz/v5yqb8XqUjr+8MiWJSKmI3AvcKyInY9no9wEYYyqA4Vgmow5YGvti+ztABZbzCWORsdC1kHLuwbItY4zZZoz5ModyXgccZ+9zeBxLYU0EFolIe2M5ebfmqZyTgMUi0s4Y84Ix5uU8lXMisMR1PzPmU0jXc7fL781jOZfZZs+dxnLy5lxOY8w+seZUrQUWiMg9wEQRaW6MqchkPUoneaMY7Bs+E8sOuwgrKmIvcIqI9IFION2dwJ+NMWOBd4DLROQLLG2csUa2SOU0wEgsB77DIOBXWI78w4wVAprPcs6y5VyV53IWyv0sFDmd5/59Hsl5p31YPeAKrE5LY6yRw8ZMypl2cj1kcf6wog0udX1/BMs5dwUw095WgmXfexnoZG9rSxaGkUUu50tAmb1tMNBX5VQ5Vc6U5OwI9MFyRB+RLTnT/rtzLYDrhjcA6lJlo/sJcJ/9eRbwa/tzOdakJpVT5VQ5Vc58kvOFXMmZ7r+8MSUZY3YYY3YbK0wSrMidtfbnK4FDxEpA9jzWTMecUMxy2hEXKqfKqXKmJufMXMmZbkoTF8kuYmVINEAbYLy9eSvwO6xY6qUmw3bFIBSjnMbu+uQClTO9qJzppVDkTBd5M2JwUYmVEGsd0MvWxrcBlcaYKfnQ2NqonOlF5UwvKmd6KRQ500OubVlef1gZMiuxppJflWt5VE6VU+VUOQtFznT85eXMZxHpCFwKPGCMyUV+lkConOlF5UwvKmd6KRQ500FeKgZFURQld+Sjj0FRFEXJIaoYFEVRlChUMSiKoihRqGJQFEVRolDFoCiKokShikFREiAi+0RklojMFZHZInKjJFijWUTKROSSbMmoKOlEFYOiJGanMeYIY0xPrFw5Z2GtGhaPMkAVg1KQ6DwGRUmAiGwzxjRyfe+KtaZ0K6xlGp/FWt4RrKUlPxaRT4FDgKVYS40+BIwC+mFl6xxtjHksaz9CUZJAFYOiJCBWMdjbNgIHYyVSqzTG7BKR7lgpostFpB9wkzHmbLv8MKwlHu8Wa/H4qcBQY8zSrP4YRQlA3mVXVZQCwUmtXBt4WESOwFru8UCf8mdgJV8bYn9vCnTHGlEoSl6hikFRksQ2Je0D1mD5GlYDh2P57Hb5HYa1qMvbWRFSUUKgzmdFSQIRaQ08CjxsLDtsU2CVsdb5vhSoZRfdirXer8PbwC9FpLZ9ngNFpCGKkofoiEFRElNfRGZhmY0qsJzND9j7HgFeEZGhwPvAdnv7HKBCRGYDzwB/w4pU+txe4WstcF62foCiJIM6nxVFUZQo1JSkKIqiRKGKQVEURYlCFYOiKIoShSoGRVEUJQpVDIqiKEoUqhgURVGUKFQxKIqiKFGoYlAURVGi+P/crCl5YPQYWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_stk['Returns'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a23000390>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFjNJREFUeJzt3X+QXXV9xvH300QikkoCgW2apN1QU1s0lsI20DrtbIyFBJQwU5iGSWXBdDKtaJkSR5bSDjM4TLEdSmXG4mwlElpLQNSSIVFMA1frjEEIQkJAzBJS2CQSMSF1AdGtn/5xv7HXZX/de+7eu6ff5zVzZ8/9nu+559mTu3n2nvtjFRGYmVl+fqHdAczMrD1cAGZmmXIBmJllygVgZpYpF4CZWaZcAGZmmXIBmJllygVgZpYpF4CZWaamtzvAWObMmROdnZ11bfPKK69wwgknTE6gSVbW7GXNDeXNXtbcUN7sZcq9Y8eOlyLilPHmTekC6Ozs5NFHH61rm0qlQnd39+QEmmRlzV7W3FDe7GXNDeXNXqbckv5rIvN8CsjMLFPjFoCk9ZIOSXpyhHUflRSS5qTrknSrpH5JOyWdWTO3R9KedOlp7rdhZmb1msgjgDuA5cMHJS0A/hB4vmZ4BbAoXdYCt6W5JwHXA2cDS4DrJc0uEtzMzIoZtwAi4uvA4RFW3QJ8DKj9POmVwJ1RtR2YJWkucB6wNSIOR8QRYCsjlIqZmbVOQ88BSLoQ2B8RTwxbNQ94oeb6QBobbdzMzNqk7lcBSXoLcB1w7kirRxiLMcZHuv21VE8f0dHRQaVSqSvf4OBg3dtMFWXNXtbcUN7sZc0N5c1e1txjaeRloL8GLASekAQwH3hM0hKqv9kvqJk7HziQxruHjVdGuvGI6AP6ALq6uqLel12V6aVaw5U1e1lzQ3mzlzU3lDd7WXOPpe5TQBGxKyJOjYjOiOik+p/7mRHxPWATcFl6NdA5wNGIOAg8AJwraXZ68vfcNGZmZm0ykZeB3gV8E3i7pAFJa8aYvgXYC/QD/wx8CCAiDgMfBx5JlxvSmJmZtcm4p4Ai4tJx1nfWLAdw5Sjz1gPr68xnZsN09m5m3eIhLu/d3ND2+266oMmJrKz8TmAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0yNWwCS1ks6JOnJmrG/l/QdSTslfUnSrJp110rql/SMpPNqxpensX5Jvc3/VszMrB4TeQRwB7B82NhW4J0R8S7gu8C1AJJOB1YB70jb/JOkaZKmAZ8CVgCnA5emuWZm1ibjFkBEfB04PGzsqxExlK5uB+an5ZXAxoh4PSKeA/qBJenSHxF7I+LHwMY018zM2qQZzwF8EPhyWp4HvFCzbiCNjTZuZmZtMr3IxpKuA4aAzx0bGmFaMHLRxCi3uRZYC9DR0UGlUqkr0+DgYN3bTBVlzV7W3FDO7OsWD9FxfPVrI9r9/ZbxmEN5c4+l4QKQ1AO8D1gWEcf+Mx8AFtRMmw8cSMujjf+ciOgD+gC6urqiu7u7rlyVSoV6t5kqypq9rLmhnNkv793MusVD3LyrsR/ffau7mxuoTmU85lDe3GNp6BSQpOXANcCFEfFqzapNwCpJMyQtBBYB3wIeARZJWijpOKpPFG8qFt3MzIoY91cISXcB3cAcSQPA9VRf9TMD2CoJYHtE/FlE7JZ0D/AU1VNDV0bE/6Tb+TDwADANWB8Ruyfh+zEzswkatwAi4tIRhm8fY/6NwI0jjG8BttSVzszMJo3fCWxmlikXgJlZplwAZmaZcgGYmWWq0BvBzKwxnb2b2x3BzI8AzMxy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTI1bAJLWSzok6cmasZMkbZW0J32dncYl6VZJ/ZJ2SjqzZpueNH+PpJ7J+XbMzGyiJvII4A5g+bCxXmBbRCwCtqXrACuARemyFrgNqoUBXA+cDSwBrj9WGmZm1h7jFkBEfB04PGx4JbAhLW8ALqoZvzOqtgOzJM0FzgO2RsThiDgCbOWNpWJmZi3U6HMAHRFxECB9PTWNzwNeqJk3kMZGGzczszZp9h+F1whjMcb4G29AWkv19BEdHR1UKpW6AgwODta9zVRR1uxlzQ3ty75u8VCh7TuOb/w22v1vVdb7S1lzj6XRAnhR0tyIOJhO8RxK4wPAgpp584EDabx72HhlpBuOiD6gD6Crqyu6u7tHmjaqSqVCvdtMFWXNXtbc0L7sl/duLrT9usVD3LyrsR/ffau7C+27qLLeX8qaeyyNngLaBBx7JU8PcF/N+GXp1UDnAEfTKaIHgHMlzU5P/p6bxszMrE3G/RVC0l1Uf3ufI2mA6qt5bgLukbQGeB64JE3fApwP9AOvAlcARMRhSR8HHknzboiI4U8sm5lZC41bABFx6Sirlo0wN4ArR7md9cD6utKZmdmk8TuBzcwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8uUC8DMLFMuADOzTLkAzMwy5QIwM8tUoQKQ9JeSdkt6UtJdkt4saaGkhyXtkXS3pOPS3Bnpen9a39mMb8DMzBrTcAFImgf8BdAVEe8EpgGrgE8At0TEIuAIsCZtsgY4EhFvA25J88zMrE2KngKaDhwvaTrwFuAg8B7g3rR+A3BRWl6ZrpPWL5Okgvs3M7MGKSIa31i6CrgReA34KnAVsD39lo+kBcCXI+Kdkp4ElkfEQFr3LHB2RLw07DbXAmsBOjo6ztq4cWNdmQYHB5k5c2bD31M7lTV7WXND+7Lv2n+00PYdx8OLrzW27eJ5Jxbad1Flvb+UKffSpUt3RETXePOmN7oDSbOp/la/EHgZ+DywYoSpxxpmpN/239A+EdEH9AF0dXVFd3d3XbkqlQr1bjNVlDV7WXND+7Jf3ru50PbrFg9x867Gfnz3re4utO+iynp/KWvusRQ5BfRe4LmI+H5E/AT4IvB7wKx0SghgPnAgLQ8ACwDS+hOBwwX2b2ZmBRQpgOeBcyS9JZ3LXwY8BTwEXJzm9AD3peVN6Tpp/YNR5PyTmZkV0nABRMTDVJ/MfQzYlW6rD7gGuFpSP3AycHva5Hbg5DR+NdBbILeZmRXU8HMAABFxPXD9sOG9wJIR5v4IuKTI/szMrHn8TmAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0wVKgBJsyTdK+k7kp6W9LuSTpK0VdKe9HV2mitJt0rql7RT0pnN+RbMzKwRRR8BfBL4SkT8BvBbwNNAL7AtIhYB29J1gBXAonRZC9xWcN9mZlZAwwUg6a3AHwC3A0TEjyPiZWAlsCFN2wBclJZXAndG1XZglqS5DSc3M7NCFBGNbSidAfQBT1H97X8HcBWwPyJm1cw7EhGzJd0P3BQR30jj24BrIuLRYbe7luojBDo6Os7auHFjXbkGBweZOXNmQ99Tu5U1e1lzQ/uy79p/tND2HcfDi681tu3ieScW2ndRZb2/lCn30qVLd0RE13jzphfYx3TgTOAjEfGwpE/yf6d7RqIRxt7QPhHRR7VY6Orqiu7u7rpCVSoV6t1mqihr9rLmhvZlv7x3c6Ht1y0e4uZdjf347lvdXWjfRZX1/lLW3GMp8hzAADAQEQ+n6/dSLYQXj53aSV8P1cxfULP9fOBAgf2bmVkBDRdARHwPeEHS29PQMqqngzYBPWmsB7gvLW8CLkuvBjoHOBoRBxvdv5mZFVPkFBDAR4DPSToO2AtcQbVU7pG0BngeuCTN3QKcD/QDr6a5ZmbWJoUKICIeB0Z6omHZCHMDuLLI/szMrHn8TmAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPLlAvAzCxTRf8imJmVTGeBP0i/76YLmpjE2s2PAMzMMuUCMDPLlAvAzCxTLgAzs0wVLgBJ0yR9W9L96fpCSQ9L2iPpbknHpfEZ6Xp/Wt9ZdN9mZta4ZjwCuAp4uub6J4BbImIRcARYk8bXAEci4m3ALWmemZm1SaECkDQfuAD4TLou4D3AvWnKBuCitLwyXSetX5bmm5lZGxR9BPCPwMeAn6brJwMvR8RQuj4AzEvL84AXANL6o2m+mZm1QcNvBJP0PuBQROyQ1H1seISpMYF1tbe7FlgL0NHRQaVSqSvX4OBg3dtMFWXNXtbc0L7s6xYPjT9pDB3HF7+NRjTjWJX1/lLW3GMp8k7gdwMXSjofeDPwVqqPCGZJmp5+y58PHEjzB4AFwICk6cCJwOHhNxoRfUAfQFdXV3R3d9cVqlKpUO82U0VZs5c1N7Qv++UF3o0L1f/8b97V+jfy71vdXfg2ynp/KWvusTR8Cigiro2I+RHRCawCHoyI1cBDwMVpWg9wX1relK6T1j8YEW94BGBmZq0xGe8DuAa4WlI/1XP8t6fx24GT0/jVQO8k7NvMzCaoKY8hI6ICVNLyXmDJCHN+BFzSjP2ZmVlxfiewmVmmXABmZplyAZiZZcoFYGaWKf9FMLMGFfnLWmZTgR8BmJllygVgZpYpF4CZWaZcAGZmmXIBmJllygVgZpYpF4CZWaZcAGZmmXIBmJllygVgZpYpF4CZWaZcAGZmmXIBmJllygVgZpYpF4CZWaZcAGZmmWq4ACQtkPSQpKcl7ZZ0VRo/SdJWSXvS19lpXJJuldQvaaekM5v1TZiZWf2KPAIYAtZFxG8C5wBXSjod6AW2RcQiYFu6DrACWJQua4HbCuzbzMwKargAIuJgRDyWln8IPA3MA1YCG9K0DcBFaXklcGdUbQdmSZrbcHIzMyukKc8BSOoEfht4GOiIiINQLQng1DRtHvBCzWYDaczMzNpAEVHsBqSZwNeAGyPii5JejohZNeuPRMRsSZuBv42Ib6TxbcDHImLHsNtbS/UUER0dHWdt3LixrjyDg4PMnDmz0PfULmXNXtbcUCz7rv1Hm5xm4jqOhxdfa/1+F887sfBtlPX+UqbcS5cu3RERXePNm15kJ5LeBHwB+FxEfDENvyhpbkQcTKd4DqXxAWBBzebzgQPDbzMi+oA+gK6uruju7q4rU6VSod5tpoqyZi9rbiiW/fLezc0NU4d1i4e4eVehH9+G7FvdXfg2ynp/KWvusRR5FZCA24GnI+IfalZtAnrScg9wX834ZenVQOcAR4+dKjIzs9Yr8ivEu4EPALskPZ7G/gq4CbhH0hrgeeCStG4LcD7QD7wKXFFg32ZmVlDDBZDO5WuU1ctGmB/AlY3uz8zMmsvvBDYzy5QLwMwsUy4AM7NMuQDMzDLlAjAzy5QLwMwsU61/K6GZlVZngXc/77vpgiYmsWbwIwAzs0y5AMzMMuUCMDPLlJ8DsKzt2n+0rZ/qadZOfgRgZpYpF4CZWaZcAGZmmXIBmJllygVgZpYpF4CZWaZcAGZmmXIBmJllygVgZpYpvxPYSq3Ip1MCrFvcpCA2rmP/VusWD9X97mt/kujkaPkjAEnLJT0jqV9Sb6v3b2ZmVS0tAEnTgE8BK4DTgUslnd7KDGZmVtXqU0BLgP6I2AsgaSOwEniqxTlsCil6Gsf+//MfopkcrS6AecALNdcHgLNbnMFG0egPWSPndM1apVm/YLT6ft6K4lJETPpOfrYz6RLgvIj403T9A8CSiPhIzZy1wNp09e3AM3XuZg7wUhPitkNZs5c1N5Q3e1lzQ3mzlyn3r0bEKeNNavUjgAFgQc31+cCB2gkR0Qf0NboDSY9GRFej27dTWbOXNTeUN3tZc0N5s5c191ha/SqgR4BFkhZKOg5YBWxqcQYzM6PFjwAiYkjSh4EHgGnA+ojY3coMZmZW1fI3gkXEFmDLJO6i4dNHU0BZs5c1N5Q3e1lzQ3mzlzX3qFr6JLCZmU0d/iwgM7NMlbIAJJ0kaaukPenr7BHmnCHpm5J2S9op6Y9r1i2U9HDa/u70hPSUyZ7mfUXSy5LuHzZ+h6TnJD2eLmeUJHcZjnlPmrNHUk/NeCV9fMmxY37qJOcd8+NSJM1Ix7A/HdPOmnXXpvFnJJ03mTmblVtSp6TXao7vp1uZe4LZ/0DSY5KGJF08bN2I95tSiIjSXYC/A3rTci/wiRHm/DqwKC3/MnAQmJWu3wOsSsufBv58KmVP65YB7wfuHzZ+B3DxVDzm4+Se0sccOAnYm77OTsuz07oK0NWirNOAZ4HTgOOAJ4DTh835EPDptLwKuDstn57mzwAWptuZVoLcncCTrbo/NJi9E3gXcGftz99Y95syXEr5CIDqx0dsSMsbgIuGT4iI70bEnrR8ADgEnCJJwHuAe8fafhKNmx0gIrYBP2xVqAloOHdJjvl5wNaIOBwRR4CtwPIW5av1s49LiYgfA8c+LqVW7fdzL7AsHeOVwMaIeD0ingP60+1N9dztNm72iNgXETuBnw7bdqrcbxpS1gLoiIiDAOnrmA/JJS2h2uzPAicDL0fEUFo9QPUjKlqlruyjuDGd1rpF0ozmxhtVkdxlOOYjfUxJbcbPptMTfzPJ/2mNl+Pn5qRjepTqMZ7ItpOlSG6AhZK+Lelrkn5/ssOOliup57i185gXNmX/HoCk/wB+aYRV19V5O3OBfwF6IuKno/zwNvWlUM3KPoprge9RLbQ+4Brghibc7mTmLsMxHyvj6ojYL+kXgS8AH6B6KmAyTORYjTZn0o/zGIrkPgj8SkT8QNJZwL9LekdE/HezQ46iyHFr5zEvbMoWQES8d7R1kl6UNDciDqb/4A+NMu+twGbgryNiexp+CZglaXr6LeQNH0cxFbKPcdsH0+Lrkj4LfLRA1OG3PVm5y3DMB4DumuvzqZ77JyL2p68/lPRvVE8ZTFYBjPtxKTVzBiRNB04EDk9w28nScO6onkx/HSAidkh6lupzeI9Oeuqfz3VMPcdt1PtNGZT1FNAm4Niz7T3AfcMnpFeZfAm4MyI+f2w83dkeAi4ea/tJNG72saT/wI6dV78IeLKp6UbXcO6SHPMHgHMlzU6vEjoXeEDSdElzACS9CXgfk3vMJ/JxKbXfz8XAg+kYbwJWpVfbLAQWAd+axKxNyS3pFFX/VgiSTku597YoNxT7iJoR7zeTlLP52v0sdCMXqucNtwF70teT0ngX8Jm0/CfAT4DHay5npHWnUf3B6Ac+D8yYStnT9f8Evg+8RvW3jPPS+IPALqr/Cf0rMLMkuctwzD+Y8vUDV6SxE4AdwE5gN/BJJvmVNcD5wHepPmd1XRq7AbgwLb85HcP+dExPq9n2urTdM8CKVh3jIrmBP0rH9gngMeD9rcw9wey/k+7PrwA/AHaPdb8py8XvBDYzy1RZTwGZmVlBLgAzs0y5AMzMMuUCMDPLlAvAzCxTLgAzs0y5AMzMMuUCMDPL1P8Czn09qN1iwBYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_stk['Returns'].hist(bins=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    3646.000000\n",
       "mean        0.001006\n",
       "std         0.020630\n",
       "min        -0.197470\n",
       "25%        -0.008248\n",
       "50%         0.000951\n",
       "75%         0.011390\n",
       "max         0.130194\n",
       "Name: Returns, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stk['Returns'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/statsmodels/compat/pandas.py:49: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
      "  data_klasses = (pandas.Series, pandas.DataFrame, pandas.Panel)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "============================================\n",
      "               CAPM       FF3        FF5    \n",
      "--------------------------------------------\n",
      "Intercept   -0.0041*** -0.0043*** -0.0043***\n",
      "            (0.0004)   (0.0004)   (0.0004)  \n",
      "MKT         1.1307***  1.2145***  1.1735*** \n",
      "            (0.0366)   (0.0347)   (0.0357)  \n",
      "SMB                    -0.2240*** -0.2386***\n",
      "                       (0.0632)   (0.0639)  \n",
      "HML                    -0.4371*** -0.3320***\n",
      "                       (0.0547)   (0.0571)  \n",
      "RMW                               0.4680*** \n",
      "                                  (0.1018)  \n",
      "CMA                               -0.5962***\n",
      "                                  (0.1499)  \n",
      "N           1130       1130       1130      \n",
      "Adjusted R2 0.5723     0.6177     0.6349    \n",
      "============================================\n",
      "Standard errors in parentheses.\n",
      "* p<.1, ** p<.05, ***p<.01\n"
     ]
    }
   ],
   "source": [
    "df_regOutput = assetPriceReg(df_stk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_45780/3261599062.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[0mdf_factors\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'CMA'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf_factors\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'CMA'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m100\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 12\u001b[1;33m \u001b[0mdf_stock_factor\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmerge\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf_stk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdf_factors\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mleft_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mright_index\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# Merging the stock and factor returns dataframes together\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     13\u001b[0m \u001b[0mdf_stock_factor\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'XsRet'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf_stock_factor\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'Returns'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m-\u001b[0m \u001b[0mdf_stock_factor\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'RF'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;31m# Calculating excess returns\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "    import pandas_datareader.data as web  # module for reading datasets directly from the web\n",
    "    \n",
    "    # Reading in factor data\n",
    "    df_factors = web.DataReader('F-F_Research_Data_5_Factors_2x3_daily', 'famafrench')[0]\n",
    "    df_factors.rename(columns={'Mkt-RF': 'MKT'}, inplace=True)\n",
    "    df_factors['MKT'] = df_factors['MKT']/100\n",
    "    df_factors['SMB'] = df_factors['SMB']/100\n",
    "    df_factors['HML'] = df_factors['HML']/100\n",
    "    df_factors['RMW'] = df_factors['RMW']/100\n",
    "    df_factors['CMA'] = df_factors['CMA']/100\n",
    "    \n",
    "    df_stock_factor = pd.merge(df_stk,df_factors,left_index=True,right_index=True) # Merging the stock and factor returns dataframes together\n",
    "    df_stock_factor['XsRet'] = df_stock_factor['Returns'] - df_stock_factor['RF'] # Calculating excess returns\n",
    "\n",
    "    # Running CAPM, FF3, and FF5 models.\n",
    "    FF5 = sm.ols( formula = 'XsRet ~ MKT + SMB + HML + RMW + CMA', data=df_stock_factor).fit(cov_type='HAC',cov_kwds={'maxlags':1})\n",
    "    FF5_2 = sm.ols( formula = 'XsRet ~ MKT + SMB + HML + RMW + CMA', data=df_stock_factor).fit()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>XsRet</td>      <th>  R-squared:         </th> <td>   0.637</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.635</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   290.6</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Sat, 02 Jan 2021</td> <th>  Prob (F-statistic):</th> <td>1.32e-199</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>23:22:35</td>     <th>  Log-Likelihood:    </th> <td>  3471.5</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>  1130</td>      <th>  AIC:               </th> <td>  -6931.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>  1124</td>      <th>  BIC:               </th> <td>  -6901.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     5</td>      <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>         <td>HAC</td>       <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>   -0.0043</td> <td>    0.000</td> <td>  -11.876</td> <td> 0.000</td> <td>   -0.005</td> <td>   -0.004</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>MKT</th>       <td>    1.1735</td> <td>    0.036</td> <td>   32.836</td> <td> 0.000</td> <td>    1.103</td> <td>    1.244</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>SMB</th>       <td>   -0.2386</td> <td>    0.064</td> <td>   -3.734</td> <td> 0.000</td> <td>   -0.364</td> <td>   -0.113</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>HML</th>       <td>   -0.3320</td> <td>    0.057</td> <td>   -5.814</td> <td> 0.000</td> <td>   -0.444</td> <td>   -0.220</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>RMW</th>       <td>    0.4680</td> <td>    0.102</td> <td>    4.596</td> <td> 0.000</td> <td>    0.268</td> <td>    0.668</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CMA</th>       <td>   -0.5962</td> <td>    0.150</td> <td>   -3.976</td> <td> 0.000</td> <td>   -0.890</td> <td>   -0.302</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>136.316</td> <th>  Durbin-Watson:     </th> <td>   1.674</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td>1163.046</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 0.171</td>  <th>  Prob(JB):          </th> <td>2.80e-253</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 7.958</td>  <th>  Cond. No.          </th> <td>    353.</td> \n",
       "</tr>\n",
       "</table><br/><br/>Warnings:<br/>[1] Standard Errors are heteroscedasticity and autocorrelation robust (HAC) using 1 lags and without small sample correction"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                  XsRet   R-squared:                       0.637\n",
       "Model:                            OLS   Adj. R-squared:                  0.635\n",
       "Method:                 Least Squares   F-statistic:                     290.6\n",
       "Date:                Sat, 02 Jan 2021   Prob (F-statistic):          1.32e-199\n",
       "Time:                        23:22:35   Log-Likelihood:                 3471.5\n",
       "No. Observations:                1130   AIC:                            -6931.\n",
       "Df Residuals:                    1124   BIC:                            -6901.\n",
       "Df Model:                           5                                         \n",
       "Covariance Type:                  HAC                                         \n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept     -0.0043      0.000    -11.876      0.000      -0.005      -0.004\n",
       "MKT            1.1735      0.036     32.836      0.000       1.103       1.244\n",
       "SMB           -0.2386      0.064     -3.734      0.000      -0.364      -0.113\n",
       "HML           -0.3320      0.057     -5.814      0.000      -0.444      -0.220\n",
       "RMW            0.4680      0.102      4.596      0.000       0.268       0.668\n",
       "CMA           -0.5962      0.150     -3.976      0.000      -0.890      -0.302\n",
       "==============================================================================\n",
       "Omnibus:                      136.316   Durbin-Watson:                   1.674\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):             1163.046\n",
       "Skew:                           0.171   Prob(JB):                    2.80e-253\n",
       "Kurtosis:                       7.958   Cond. No.                         353.\n",
       "==============================================================================\n",
       "\n",
       "Warnings:\n",
       "[1] Standard Errors are heteroscedasticity and autocorrelation robust (HAC) using 1 lags and without small sample correction\n",
       "\"\"\""
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter Terminology\n",
    "\n",
    "In general, we do not know the underlying true values of the coefficients β. We will denote $\\widehat{\\beta}$ the estimation of the unknown parameter β.\n",
    "\n",
    "We note that we can talk about $\\widehat{\\beta}$ in two ways:\n",
    "\n",
    "- $\\widehat{\\beta}$ as a concrete estimated value, or more generally, as a result. Then we say that $\\widehat{\\beta}$ is an estimate of β based on an observed data sample;\n",
    "\n",
    "- $\\widehat{\\beta}$ as a random variable, or more generally, as a rule for calculating an estimate based on observed data. Then we say that $\\widehat{\\beta}$ is an estimator of β.\n",
    "\n",
    "When we are talking about $\\widehat{\\beta}$ as an estimator, we can also talk about its mean, $\\mathbb{E}(\\widehat{\\beta}_1)$ , variance $\\mathbb{V}{\\rm ar}(\\widehat{\\beta}_1)$ and distribution, which are very important when determining if a particular estimation method is better than an alternative one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key assumptions in Regression Analysis\n",
    "\n",
    "<p>Let our <em>(random) samples</em> of <span class=\"math inline\">\\(\\epsilon\\)</span>, <span class=\"math inline\">\\(X\\)</span> and <span class=\"math inline\">\\(Y\\)</span> be: <span class=\"math inline\">\\(\\boldsymbol{\\varepsilon} = (\\epsilon_1, ..., \\epsilon_N)^\\top\\)</span>, <span class=\"math inline\">\\(\\mathbf{X} = (X_1,....,X_N)^\\top\\)</span>, and <span class=\"math inline\">\\(\\mathbf{Y} = (Y_1, ..., Y_N)^\\top\\)</span>.</p>\n",
    "<p>The required conditions are:</p>\n",
    "\n",
    "\n",
    "<p>\n",
    "<strong>(UR.1)</strong> The Data Generating Process (<strong>DGP</strong>), or in other words, the population, is described by a linear (<em>in terms of the coefficients</em>) model: <span class=\"math display\"><span class=\"math display\">\\[Y = \\beta_0 + \\beta_1 X + \\epsilon\\]</span></span>\n",
    "</p>\n",
    "</div>\n",
    "\n",
    "<p>Another example of a linear model is: <span class=\"math display\">\\[\\log (Y) = \\beta_0 + \\beta_1 \\dfrac{1}{X} + \\epsilon \\iff U = \\beta_0 + \\beta_1 V + \\epsilon,\\ \\text{where}\\ U = \\log(Y),\\ V = \\dfrac{1}{X}\\]</span> In a linear model <span class=\"math inline\">\\(\\epsilon\\)</span> and <span class=\"math inline\">\\(Y\\)</span> are <strong>always dependent</strong>, thus <span class=\"math inline\">\\(\\mathbb{C}{\\rm ov} (Y, \\epsilon) \\neq 0\\)</span>. However, <span class=\"math inline\">\\(\\epsilon\\)</span> may (or may not) depend on <span class=\"math inline\">\\(X\\)</span>. This leads us to further requirements for <span class=\"math inline\">\\(\\epsilon\\)</span>.</p>\n",
    "\n",
    "\n",
    "<p>\n",
    "<strong>(UR.2)</strong> The error term <span class=\"math inline\"><span class=\"math inline\">\\(\\epsilon\\)</span></span> has an expected value of zero, given any value of the explanatory variable: <span class=\"math display\"><span class=\"math display\">\\[\\mathbb{E}(\\epsilon_i| X_j) = 0,\\ \\forall i,j = 1,...,N\\]</span></span>\n",
    "</p>\n",
    "</div>\n",
    "\n",
    "\n",
    "<p>This means that whatever are the observations, <span class=\"math inline\">\\(X_j\\)</span>, the errors, <span class=\"math inline\">\\(\\epsilon_j\\)</span>, are on average 0. This also implies that <span class=\"math inline\">\\(\\mathbb{E}(\\epsilon_i X_i) = 0\\)</span> and <span class=\"math inline\">\\(\\mathbb{E}(\\epsilon_i) = \\mathbb{E}\\left( \\mathbb{E}\\left( \\epsilon_i | X_i\\right) \\right)=0\\)</span>. An example would be the case, where <span class=\"math inline\">\\(X_j\\)</span> and <span class=\"math inline\">\\(\\epsilon_i\\)</span> are independent r.v.’s and <span class=\"math inline\">\\(\\mathbb{E}(\\epsilon_i) = 0\\)</span> <span class=\"math inline\">\\(\\forall i,j = 1,...,N\\)</span>. Furthermore, This property implies that:\n",
    "<span class=\"math display\">\\[\\mathbb{E}(Y_i|X_i) = \\beta_0 + \\beta_1 X_i\\]</span>\n",
    "On the other hand, if <span class=\"math inline\">\\(\\mathbb{C}{\\rm ov}(X_i, \\epsilon_i) \\neq 0\\)</span>, then expressing the covariance as <span class=\"math inline\">\\(\\mathbb{C}{\\rm ov}(X_i, \\epsilon_i) = \\mathbb{E}\\left[ (X_i - \\mathbb{E}(X_i))(\\epsilon_i - \\mathbb{E}(\\epsilon_i))\\right] = \\mathbb{E}\\left[ (X_i - \\mathbb{E}(X_i))\\epsilon_i\\right] = \\mathbb{E} \\left[ (X_i - \\mathbb{E}(X_i))\\mathbb{E}(\\epsilon_i | X_i) \\right] \\neq 0\\)</span>, which implies that <span class=\"math inline\">\\(\\mathbb{E}(\\epsilon_i| X_j) \\neq 0\\)</span>, i.e. assumption <strong>(UR.2)</strong> does not hold.</p>\n",
    "\n",
    "\n",
    "<p>\n",
    "<strong>(UR.3)</strong> The error term <span class=\"math inline\"><span class=\"math inline\">\\(\\epsilon\\)</span></span> has the same variance given any value of the explanatory variable (i.e. homoskedasticity): <span class=\"math display\"><span class=\"math display\">\\[\\mathbb{V}{\\rm ar} (\\epsilon_i | \\mathbf{X} ) = \\sigma^2_\\epsilon,\\ \\forall i = 1,..,N\\]</span></span> and the error terms are not correlated across observations (i.e. no autocorrelation): <span class=\"math display\"><span class=\"math display\">\\[\\mathbb{C}{\\rm ov} (\\epsilon_i, \\epsilon_j) = 0,\\ i \\neq j\\]</span></span>\n",
    "</p>\n",
    "</div>\n",
    "<p>This implies that the conditional variance-covariance matrix of a <em>vector</em> of disturbances <span class=\"math inline\">\\(\\boldsymbol{\\varepsilon}\\)</span> is a unit (or identity) matrix, times a constant, <span class=\"math inline\">\\(\\sigma^2_\\epsilon\\)</span>:\n",
    "\n",
    "    \n",
    "$\\mathbb{V}{\\rm ar}\\left( \\boldsymbol{\\varepsilon} | \\mathbf{X} \\right) =\n",
    "\\begin{bmatrix}\n",
    "\\mathbb{V}{\\rm ar} (\\epsilon_1) & \\mathbb{C}{\\rm ov} (\\epsilon_1, \\epsilon_2) & ... & \\mathbb{C}{\\rm ov} (\\epsilon_1, \\epsilon_N) \\\\\n",
    "\\mathbb{C}{\\rm ov} (\\epsilon_2, \\epsilon_1) & \\mathbb{V}{\\rm ar} (\\epsilon_2) & ... & \\mathbb{C}{\\rm ov} (\\epsilon_2, \\epsilon_N) \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "\\mathbb{C}{\\rm ov} (\\epsilon_N, \\epsilon_1) & \\mathbb{C}{\\rm ov} (\\epsilon_N, \\epsilon_2) & ... & \\mathbb{V}{\\rm ar} (\\epsilon_N)\n",
    "\\end{bmatrix} =\n",
    "\\sigma^2_\\epsilon  \\mathbf{I}$\n",
    "    \n",
    "    \n",
    "This means that the disturbances $\\epsilon_i$ and $\\epsilon_j$ are independent $\\forall i \\neq j$ and independent of $\\mathbf{X}$, thus:\n",
    "\n",
    "$\\mathbb{C}{\\rm ov} (\\epsilon_i, \\epsilon_j | \\mathbf{X} ) = \\mathbb{E} (\\epsilon_i \\epsilon_j | \\mathbf{X} ) = 0,\\ \\forall i \\neq j.$\n",
    "</p>\n",
    "\n",
    "\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "<strong>(UR.4) (optional)</strong> The residuals are normal: <span class=\"math display\"><span class=\"math display\">\\[\\boldsymbol{\\varepsilon} | \\mathbf{X} \\sim \\mathcal{N} \\left( \\mathbf{0}, \\sigma^2_\\epsilon \\mathbf{I} \\right)\\]</span></span>\n",
    "</p>\n",
    "</div>\n",
    "<p>This <strong>optional condition</strong> simplifies some statistical properties of the parameter estimators.</p>\n",
    "<p>We can combine the requirements and restate them as the following:</p>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "The Data Generating Process <span class=\"math inline\"><span class=\"math inline\">\\(Y = \\beta_0 + \\beta_1 X + \\epsilon\\)</span></span> satisfies <strong>(UR.2)</strong> and <strong>(UR.3)</strong>, if: (conditionally on all <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{X}\\)</span></span>’s) <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbb{E} (\\epsilon_i) = 0\\)</span></span>, <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbb{V}{\\rm ar}(\\epsilon_i) = \\sigma^2_\\epsilon\\)</span></span> and <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbb{C}{\\rm ov} (\\epsilon_i, \\epsilon_j) = 0\\)</span></span>, <span class=\"math inline\"><span class=\"math inline\">\\(\\forall i \\neq j\\)</span></span> and <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbb{C}{\\rm ov} (\\epsilon_i, X_j) = \\mathbb{E} (\\epsilon_i X_j) = 0\\)</span></span>, <span class=\"math inline\"><span class=\"math inline\">\\(\\forall i,j\\)</span></span>.\n",
    "</p>\n",
    "</div>\n",
    "<p>The linear relationship <span class=\"math inline\">\\(Y = \\beta_0 + \\beta_1 X + \\epsilon\\)</span> is also referred as the <em>regression line</em> with an <em>intercept</em> <span class=\"math inline\">\\(\\beta_0\\)</span> and a <em>slope</em> <span class=\"math inline\">\\(\\beta_1\\)</span>. From <strong>(UR.2)</strong> we have that the regression line coincides with the expected value of <span class=\"math inline\">\\(Y_i\\)</span>, given <span class=\"math inline\">\\(X_i\\)</span>.</p>\n",
    "<p>In general, we do not know the true coefficient <span class=\"math inline\">\\(\\beta_0\\)</span> and <span class=\"math inline\">\\(\\beta_1\\)</span> values but we would like to estimate them from our sample data, which consists of points <span class=\"math inline\">\\((X_i, Y_i)\\)</span>, <span class=\"math inline\">\\(i = 1,...,N\\)</span>. We would also like to use the data in the <em>best</em> way possible to obtain an <em>estimate of the regression</em>: <span class=\"math display\">\\[\\widehat{Y} = \\widehat{\\beta}_0 + \\widehat{\\beta}_1 X\\]</span></p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Properties of the OLS estimator\n",
    "\n",
    "From the construction of the OLS estimators the following properties apply to the sample:\n",
    "\n",
    "1. The sum (and by extension, the sample average) of the OLS residuals is zero:\n",
    "\n",
    "$\\begin{equation}\n",
    "\\sum_{i = 1}^N \\widehat{\\epsilon}_i = 0\n",
    "\\end{equation}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a236de828>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEICAYAAABbOlNNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXeY3NS1wH9ndte9V1wwBmwwNh1jqsE0Y6oJJZSQQAIhISEJJQkkIfQkQBJIQnmEAO8RCC0QejFgegnYBmNjwAVjwMYYd+Oy3jL3/SFpVtJIM5qRpqz3/L5vv53RaKQz0tU995R7rhhjUBRFURSHVKUFUBRFUaoLVQyKoiiKB1UMiqIoigdVDIqiKIoHVQyKoiiKB1UMiqIoigdVDIqiKIoHVQyKoiiKB1UMiqIoiofaSgtQDH369DFDhw6ttBiKoiitimnTpi0zxvTNt1+rVAxDhw5l6tSplRZDURSlVSEin0bZT11JiqIoigdVDIqiKIoHVQyKoiiKB1UMiqIoigdVDIqiKIoHVQyKoiiKB1UMVcCiVRtYuHJ9pcVQFEUBWuk8hk2Nfa5+AYAFVx9RYUkURVHUYlAURVF8qGJQFEVRPCSiGERkgojMFpF5InJRwOftReR++/O3RGSo67MdReRNEZklIjNFpEMSMimKoijFEVsxiEgNcBNwGDASOFlERvp2OwNYaYwZBlwPXGN/txa4G/ihMWYUMA5ojCuToiiKUjxJWAxjgHnGmPnGmAbgPmCib5+JwJ326weBg0REgPHADGPMewDGmOXGmOYEZFIURVGKJAnFMAj43PV+ob0tcB9jTBOwGugNbAMYEZkkIu+IyC8TkEdRFEWJQRLpqhKwzUTcpxbYF9gdWA9MFpFpxpjJWScROQs4C2DIkCGxBFYURVHCScJiWAhs7no/GPgibB87rtAdWGFvf9kYs8wYsx54Ctg16CTGmFuNMaONMaP79s27zoRSJM/O+pK/TZ5baTEURakgSSiGKcBwEdlSRNoBJwGP+fZ5DDjNfn088IIxxgCTgB1FpJOtMPYHPkhAJqVIzrprGtc9N6fSYiiKUkFiu5KMMU0icg5WJ18D3GGMmSUiVwBTjTGPAbcDd4nIPCxL4ST7uytF5Dos5WKAp4wxT8aVSVEURSmeREpiGGOewnIDubdd4npdD5wQ8t27sVJWFUVRlCpAZz4riqIoHlQxKIqiKB5UMSiKoigeVDEoiqIoHlQxKIqiKB5UMShKlfPV1/UMvehJpn26otKiKG0EVQyKUmLqG5ux5nMWx1vzLYVwx2sLEpKocqxY18Dj7/kLIyjVhioGRSkhq9c3MuK3z3DDC/OKPoYEVRprpfzgrqn85N53WbKmvtKiKDlQxaAoJWTp2o0APPLuoqKPIXYNSpNVm7L18cUqSyE0NKUrLImSC1UMilIOYoz6HYshhjdKUQpCFYOiKIriQRWDorQS1GJQyoUqBkUpKfF7c8cLtSnEGJTWgSoGRalyNMaglBtVDIpSUpLINd2E8lWVVoEqBkUpKQm4khyLIfaRFCUaqhgUpQzEGfNnYgyqGZQyoYpBqUouffR9bnxhbqXFUJQ2SSKKQUQmiMhsEZknIhcFfN5eRO63P39LRIb6Ph8iImtF5OdJyKO0fu5881P+9OycSotRFYi05CUpSjmIrRhEpAa4CTgMGAmcLCIjfbudAaw0xgwDrgeu8X1+PfB0XFkUpdpIwv2jriSl3CRhMYwB5hlj5htjGoD7gIm+fSYCd9qvHwQOEnsYJCLHAPOBWQnIoihViSRQCU/1glIuklAMg4DPXe8X2tsC9zHGNAGrgd4i0hm4ELg8ATkUZZNkU6quqrQOklAMQc3WP7gJ2+dy4HpjzNq8JxE5S0SmisjUpUuXFiGmopSfJEb5LRPc1GZQykNtAsdYCGzuej8Y8K/E4eyzUERqge7ACmAP4HgRuRboAaRFpN4Yc6P/JMaYW4FbAUaPHq1PiNKqiJeu6pTdVpTykIRimAIMF5EtgUXAScApvn0eA04D3gSOB14w1vBnrLODiFwGrA1SCoqiKEr5iO1KsmMG5wCTgA+BB4wxs0TkChE52t7tdqyYwjzgfCArpVVRlBA2oVpJbSFe8uysL7n88dadS5OExYAx5ingKd+2S1yv64ET8hzjsiRkUZRqItF01fiHUsrAWXdNA+DSo0ZVWJLi0ZnPrYSv6xvZ0NBcaTGUIklipKzBZ6VcqGJoJexw2bOMvfbFSouhFEmcPj2JORBJsGztRv7w1Ic0p1VBlYN02nDbq/NZ39BU9nOrYmhFLLMXlleUSvDbR97n76/M5+U5X1ValDbBM7O+5KonP+TaZ2aX/dyqGIAla+o5/4Hp1Deqq0YpDXEG/dVSEqOhKQ1AOh3/WHF+y+QPl3Dufe/GF6LKcfqj1Rsay35uVQzA5Y/P4j/vLOL5D5dUWhRFyaJKPElVwxl3TuWR6f6pUpseKfvGpyswIlDFQMsIqEafQKUKaZngpr79toTTHVUipKOKAWi2NXK1BPmUTYckOvNqWfNZ1VJ5cfqjSmSjqWLAiv4D1KRUMSilQRJYt7nSikEpL5WMLbVZxfDkjMVM+MsrGGMyFkNNm70aybNiXQOr15c/aFZtJDnBrdIkIYca5dFxYgyVcCEmMvO5NXLu/e/S2GxoaE5nfHjqSkqOXa98DoAFVx9RYUkqS5KjPY0xtC0cB0YSWWAFn7v8p6wOMtrYuFxJqhiUhHE681hNS2MMbZKW4LPGGMqGoxia0yYzk7PYGMPajU3MX5p3SQmlzHy5ur7SIiTkSqquAYuOn8qDZNJVy3/uNqsYHCXQlDYZjZwqssWf8o//cuCfX2bZ2o0MvehJHntv08+xrnaeeX8xe/5hMq/NXVZROW55+WMANiQwebJaRuxJKDt1i+WnJfisFkNJ+XJ1faasRMZM8yiG4o47Y+FqAD5a/DUA9739WTxBE+DIG17ltlfnV1qMivH2JysB+OjLNRWV44kZi4GEZq9WuC9N0lCotFuskpx737tc+uj7ADQ1p0MrLrQEn8tPm1IMp97+Fpc+atVJd1sMjispKPj8xaoNfLg4WufS0Gzd4Ha1pbus6Yh25fuL1nDVkx+WTI5qp7HZitiV8l4UQhKdqo6yNw0emf4Fd775KQDfuu0tRvz2mcD9nD5KYwxlwHm43NPNnb426AbsffULHPbXVyMd26kl066Eea/NBTaSxuY0z39Q3lIfTc0VSKPw4SiGuk0gB7laFEKSUlT6F72/aDXn3PNOxSvFvvXJivAPdeZzeRBaTFhHMbhjDHE180ZbMbSvq4l1nFwU2pCvf24OZ/5zKq/PK5+vvb6p8oqhocoUw8o4czrsW14t7pdqCT7H8b3/+J53eGLGYj5fsT5BiZIl1dpnPovIBBGZLSLzRCRr2U4RaS8i99ufvyUiQ+3th4jINBGZaf8/MAl5wuV0Kwbrf9rlSoo7jNlYDouhQMXwmd3wl69rKIU4gVTDgkKO9VZXUyW9WAJUiV5IJvhcws7u1w/P5PwHpufcx3lGN1bBICaMVj3zWURqgJuAw4CRwMkiMtK32xnASmPMMOB64Bp7+zLgKGPMDsBpwF1x5ckpK5Ixy71ZSdbncU22jCuplDGGIltJOUcd1VC+PBNjqBKLIQ7VohASmflMcgHVsCZ9z1uf8Z93FuX8bvs6q100lFkxhD2HQe5Xk/nfOi2GMcA8Y8x8Y0wDcB8w0bfPROBO+/WDwEEiIsaYd40xTm7nLKCDiLRPQKZARGDSrCU88/6XrnkM6UxANzFXUgkVQ6ESFjubO44iqQ7FkMDEsiqj0kt7VouCcogjT4vFULq2GnS/6huDFVGQ+9X5fmud+TwI+Nz1fqG9LXAfY0wTsBro7dvnOOBdY0zJlilbud5yp/zw7mmk7F/enG4J6MZt+E4jK6liSMM7n63k6/pGPlm2jjc/Xp5z/1KZo8vXbuSuNxcEfpZEzn5cHIshqTj43ybP5cYX5iZzsAIxyXg6EyOZ9avjHyMOjlVfSosh6DeuC1mmM2gw5Xy9EllJSdRKCmom/l+Scx8RGYXlXhofehKRs4CzAIYMGVK4lHj98zWZ4HOaeV9Zs5bj3gBnNPD3V+Zz8Mj+7D60V0Hfn7lwNTsM7p5zn7UNTRx78xuMHd6HV+3JW7nqEWVKNhfYrRiTuwM49/7pvDp3GWO27M22m3X1fBY2KionGcWQ0EN13XNzADjnwOGJHK8QqiUrqdqwRtTFaan2tVaCSCljDEF3LSz+FiSHSWjAWgxJDG0XApu73g8G/FN/M/uISC3QHVhhvx8MPAx8xxjzcdhJjDG3GmNGG2NG9+3btyhB3X2E40p62OWLjGuqb3Rp/RNuebPg7x9142ssz7Ous3MOZ1JdPkpnMVjWV2PAkLw6LAbrB1/x+AexjmOM4YbJlbEU/FR6lJ0s8X9MnCM42WoNJUytDupP3BbD4tUbMq8DLYZMNlrrjDFMAYaLyJYi0g44CXjMt89jWMFlgOOBF4wxRkR6AE8CvzLGvJ6ALJFJ2cHn2Uu+zmyLe/0LGX0YY7jt1fl8Xe9NY1zf0MyHi9eENoZMQCqHsO7PxFUssBDiXApnVFTJjCBHYS1buzFWzGPFugb+bFsL+fhydT1zXe0pKarNlZRMVlL8Y4C1XnvUSZ9uHHdv0MAmKaYsWMmiVRs829ZtbGmLe/3hhczroDaaSZY0cNur8/nf1z8pjaABxFYMdszgHGAS8CHwgDFmlohcISJH27vdDvQWkXnA+YCT0noOMAz4rYhMt//6xZUpVFbX66CRdNyspEI6oFfmLuOqJz/kct+I9s/Pzuawv77K/VM+D/xelE7C/ZsyvzOyZM4xcn8jbFWx9Q1N/PDuaQDUpsKb1z5Xv8Dd//20QKmi4/YdL/26+LBVIYUV9/zDZA65/pWiz5WXTctkiM3nK9azx+8nc9OL8wr+rjNoKaViOPkf/2W/a1/0bFsfEmN497NVWanomeCzMYF9RSlJJEpqjHnKGLONMWZrY8zv7G2XGGMes1/XG2NOMMYMM8aMMcbMt7dfZYzpbIzZ2fX3VRIyRcUdV8gXYzjv/umc/8B0Pl2+zmMGOhSiGJx9V/kmPjmLnM/6IqwMR/7OIR2kGcrEp8tbJgzlilEsWrWBix95P/Hz3/jCXCbN+tLzwC9Zk11ltaEpnTWaC6LCE2OB4iwFYwyr1pdm7koiwef4h+Dzldb9e7WIyZuOK6mxqXhJPvpyDe99vsqz7dPl6zzv/Z39etua9g84Ln7kfa595iPPtpbgc9EiFk3rT/KOibsTdfenq9c3Zo00H353Ef95ZxH7//EljxnoUIhvvaWSa2F33XkYcumwoI8K9VO69/5k2TqufOKDSCa7+0EIO2Uxpn9U/vTsHH5w1zSP7/irAIvhoodmsM/VL+SdjFfpkgluCpHkX299xs5XPMf9Uz7jF/9+r+KprqWgMcaE0rra+DGGCX95lYk3eT3gR9+Y2yPuWAydAqojvOHLMGztwedWjfu5dz88u//ueXb/3fMFHWtDAdk4jloI63fCMlF+cJflplm7MdgktY7pijHYZ/q6vqmgGcnufuSsf07l9tc+Yf4yazT0xsfL+My2DPxyRsnsagr40c9/sIQf/WtaZPny4R4JBlkMz9n1o/J1DEl2qMUeK9NBFPD1l2ZbhveFD83k39MWxs6+aWpO88JHyRnzUX7Luo1NOQcRjlVY64plRb3GjjLJl67a1Jzm0+Xr+M4db0eqkJtvHyfGEFQ2x/9MBwWfP1+xvqQDK4c2pRiCGs2MhS2moHO9121sKmokUcgNc9zvoUHmkENFybv2Zl9Z/6944gP2++OLwV/Ig9ORO0bOKf94i69DFJO709/Q2BwYjA0ahZ/5z6k8NfPLgmV7dPoifvHv93LK4ZRan7FwVebBddJY84UQknwGgxRiFNzfWrexibPvnuZRdnOWfM2/p3pjUv72U+y5Hf739QWhxy4F9Y3NjLp0Us4KwUH1sJxstHy0ixh8Pv1/p7D/H1/ilTlLeeTd3LOpo+BYDB3bZXe9X9d7nyl38Nlh7LUvljSTyqFtKQbX66V2Z+HOuU8bw8dL1zLq0klFHb8Qt0O+1ZkKffbuenNBxr/psRhcHV8hQVi3JeAoryiuZf81+Mm972bt05TgVM6f3Tedf09bCHgVs1vhNjYbjDEcfePrfOeOtz1y5rvOSc2DcJ+zWAyGx977gqff/5LrXZlS469/hV88OCPnd6NUvH3ugyX8+F/vBH62LE8adaHkm5vh+OL/8+7CzLaNTc1c/viszHtHCbiz3/J1mje9OI9Hpy+KHHx+zRW/SBW7YItNY3M6YzF0qM22GPzX2LlGfiu8lBNoHdqUYnDjD/qC1UnEyXv338BcZm2LKyl4n0LWn97Q0MxvH52VmTuRxCh3bb01Ml369cbMIxxlhTt/5+c2mV+du5SdLn+WNfbIKOg581tdPw+wBsJwu0vcHbq7UKITLExHdM9EsQJ/eNc0jrwhf2n2ojNgAkaOhRJllPn9f07lyZmLAz9zB0uLCT5PWbCCr+sbQ7PZ/AQNRh6atshjuWRcSa7st3wW9R8nzeZn902nJhXuSrrooRk8FXAdwp5JYww3vTgvdOA1ZcEKPvpyDcN/83Tm+s79Kngp4C9cCRHONfI3wWLL3BRCm1UMQRhjYmVc+Bv74tX1GGMCR4r5bm5tAXMAnMbtmKKeeQwRxvmfLl/HFY97g8t/mzyXp9//khtemJtVqtyN/zf7f6u78//jpNms3tDIR/bCR0HHq29qZuHKlsymB6ctzNonDHfw3/1bmo3JcqU47/P5pP2KO2ip0Gdmfcn7i/Iv5tSUx82xan1DznWqjWm53kHN56mZi7nl5Y9Zua4hazwe1cUSRq3LXfNFnmyujU3Nnuv6dX0jJ9zyJmffHWyNBOF82/2cNPjqGgWtuVFoiYuNAQrzvimf86MAyynMYJixcDV/nDSb8+4Pruh6wi1vZgYk80IUgsMKuwry5yvWZ6zt1jrBbZOh0JIYK9Y1MPvLFh+63+3Q1Gz45t/fZOtfP8XqDY08On1RJk11gR3IDTul09jXNzTlLfTVaLtmnGfo6qdb0t7unxo8H8LND+6axh2vf8L8ZS2N1llhqiYlmesikj2C9l8zv2Jwv3eeK6dTDlIMv3hwBvteU1wsxJ0u7BbLmGy5/KOxDQ3NTJqVHePw6/Sn3g8eUYex99YtJcEa87jQdr7iOfb8w+Ss7WFulzPvnMLP7mtx1V3x+Adc/fRH/PrhmVmdSWOODnPV+gbP/kEdUZ2rV/zto7NCXUtrNzax7cXP8FfXbHHHkvvAtRJifovB+u9uIf57EVRaPUgxfLJsHRsamvl4qatTtk9QiCJJpYQNDc1ZQWKnPYfVQYJoAzRoiX2Mdc1/KHcFWEimVtImQ67G6s9XBjjyb6/yhWuE5+8kG9Nppiyw1h7+2X3v8tLspXTrUMuMyw7l0sdmBX7HodZ+EEdeMolRA7vllHvuEqvBO03vX28Vtua04889+LrsyVk1Ip7r4ndJ/PS+d/nf03dnWD+rXpJfOXo6ZKc+VY7Kpy8FZL58sWoD732+ii37dmbEZuHXwmMx+OanhI3Wnf0ufuR9HnpnIU/9dCwjXdfbf3/co8YHpnzOo+/lDkj26dJSLDifxZAPg1tJCM9/6L1Wzr15+v1sBed2Y63d2MQNL8zl/EO24as1Gxl77YtcfMR2mc/TBvwGa60vJXTdxibPb3NYaY94/z11IecevE32byjwErjbSNbzlYkxuCwGX/tsbE5zwJ9eYkivTpm1SaDFIilIMYiw7zUvsHxdg68+WYQfFdEBEGSVONmA5aRNKYa8/mQTvo8/XxnwKAXn+27cnaIz8WuNL/Pgs5AVpNyHCp/sZnHyP/4LwLqGZmbmqaEUVKgvl0VSUyOedEn/g/f5ig1c8MB7PHrOvkB25xdsMVjHCLIYOtTVsM6XVrv31S1zRnIVDKz3KAavDGEB79FXPc++w/pkSpP4f5/fQqoR4cf/eoejdhrILx/KHfAFr6IsVjGsWGfJFselcMj1r2Su3V+fn8M/Xv2EIb06MbR3ZwAmu5RM2hhqfD2Zv7yJX1EA3Pv2Z3Rub3Upi1ZtYNqnK9hti16Bo/98weegz7MVQ3a6qr+jX28He/3PmRMELiSNd/KHS3IueJWE53/txuaSzsaOSptyJeV7sFatbwidsh4FfyfivsHuc7s7y4Urg/21xeYqH3XjawV/nuvhqBHxlP8NHGG5Onh/o/5sxfos09sZ6QWNjuKsH+EohpT4LYbc6ZqvzVsWuvpeUODvyZmLM2U/osjlkM+V5GfWF6t5+N2FBQXgo+BYiOm0CYxZBMXEopQG+dV/ZvJTVxbacf/jLSQpEl5KxeG8+6dz3bOzXZ+3nNcvVmCMwdX+Tr71v6HunTvsukO5LIanfQHoIEssKlFb9TE3vc6pt71V9HmSok1ZDPnIlTMdhZmLvKP1letaMp/cDTDMSnBTjhm3zWnDxY/MDMzQcqhJSeYBfHLm4sBZpu4+w68Y1jc0c+zNr/PseftnOgUndTIo/a85T+e5fO1GenZqx5Kv6xnQvWNme9q0pB63q035ZmBnB5/9hK2+F7cUu1uOQi2GI/6WrcRzzniPKGs6owxaVjR0K4agw9T67lWpJlk9bM8VcAZMuRRWQyYrKdhieHP+8rwDveXrNvLwuwv5xi6Dsz47OyR110+Uy17IgOetT1ZE3rdUqGIoIafe3qL53aPytfX5rZJmY0qWjfDEjC+47LEPmLB9f+59O3dwuiYlmRH/HyfNDtwnJeGmPMAcXwzE6aSb04YnZnzBETsMyOybrwPf7arn+cWh2/LHSbO59du7ZbafettbGZdCu5oU65pb3ErNaUNznk45zGryd0b/98aCnMfx4/56XBeBMe5snYDP83x/0aoNDOrRMdOualLicvO0HDBo7obfdeS/LmHzJH798Ew+WVq4j/w/toLwuJ+yAuomSzZ/+1u9Ifez9vq85bw+bzmjBnZnSK9OmXhdIRxvp4nn6vxb20KCqhjKhLvjiTLBK502JVtE5Jx7LHP/7v/mD1LXiOSdF+G1GIJ3bmhK885nVgDf6VTWNzRzzj3vUntqChGr44syqn55zlIAzrqrxZ3z5vyWOjPtams87qu0ye/Gca519lyUvOLkxD2yvvChGTz507FFH8tguMdOLCimo7n2mY/460m7ZK5/SoIVjf8azFy4Omsw41ceQUtTAhl5i8Url/ezoNhYQ7N3W9SJec1pw6hLJ5XMUm9tS8y2KcVQeudMOO6RzJI1+RtrUwkVQyHU1abyzpp1j5TClN5bn7R03H/zLXxz44tzEaz7E0Vp5rOk2temPJ1IOmQuiRsnR97fKRbjSlr69UZWrGtg2826er7vTiJ4be4ydty8O9061EU+7tf1TSzOM88hF06tLI8rKeBLfjdRUFzKvc9HX65hwl/yT/Bzq7NiFO51vnUxMsrcJYvfYnAWlMpHlDbSlmhTwedK4g6KRQlc/uutz9jp8mdLKVIkOrWrKchiCAvmuWen+jM73l+0JnOOfK6krfp0ziuPP1D68LuL8qYl1mc6Gaujcdw+NxZR6//AP73EoX95hTX1jXwc4EZZsqaeU29/i/PvL6zqqVsp9OrcLuvzfK4q5zc6npd02gROJGup0WN457OVgcdyWwxRi+t54hgRh2m58v8dxeC+R/7BVNQyMEGlW5KkWIthq76dkxUkIm1LMWxiA4Iu7Utv8EUpaZwSYX1DEwf9+aXQAH7Umdz5+sk+XdvnHcUHBfc/+jL3ymoNmU5mLsN+8xTDf/M00FKFtRCcAoPH3fxGoCyOe+P5D5cUPZkvaBGkvIrBthgcn3xjuiVokQoI8j46/QuOvfmNwGO5R9dJrPH94kdfMfSiJ7O2N6XTocrTvZTuOvua+wcA00IUm5/5BcZBgq71mhyVVaNOcPNz5cTti/peXNqUK8lhUI+OkRZpqXbKYfpe9J+ZefdJifDe56sDR8cOhayElgtjTFG1oNyZK7kyVYqp8BpGWD0ct79+0aoNLFq1geP/J7gDDmPB8uxrna/sRb3tLnNmMTc1pzNKdoXLinM64rlfhStTt8dvYxFLpxoD/3xzAW/MW07Pzu1Cj7FsbQN3vrGA0/fZMuszd1zjyRmL+ebum2fNQ3nFjkclzQ6XZRfaDLvfULzFkNRzUyhty2Kw6dk5v1+3mMU/yk0hCwOVkqZ0OivN00+UAnxRsCYhFq4Z3FbGyEuKq56bFP7yyr95eGbO2EEQQSsI5mODz2Joam6ZxzDDNTHSUby5DBC3K6mYdrihsZlLHp3FM7O+5N63P8tpBT4y/YvAe+5WJs5kw3KVjyjUSip2eVl/mnC5gtiJ9H4iMkFEZovIPBG5KODz9iJyv/35WyIy1PXZr+zts0Xk0CTkCcNpWj07Zftn/ZSjtO2mwn/nr+DFPH7m5QmVbV69obGogPD5DxQ+SSyqAtq8V8f8O9lc9+zsrAl/73wazd3hpl1A2eZ8OBaD09k0pYM9/U6nn+s6e11J0RRDrj4tl7EjErwwVVC+fyXqCkWh2DlSfqVbl2Md9SSJfRYRqQFuAg4DRgIni8hI325nACuNMcOA64Fr7O+OBE4CRgETgJvt45WU7h3zWwxxa6+Xg1P2GFJpETLkC9KecefURM4z76u1kSqZJkFY4NXP5yuij97/9kJ2eeZiHIL+SqNR2NDgBJ9bXEm5spJyuSrTHoshWmcs0qIc/Kd9/L0vQr+XEuE0ex2NXJx559SqVQzFMmqgt3xNuVxLSaifMcA8Y8x8Y0wDcB8w0bfPROBO+/WDwEFipUFMBO4zxmw0xnwCzLOPV1J6dIqeIlitnLnvlvQNKGKWj472+ggn7b556D6Hjuof6Vi/PdKv/zctknJ/+fndU97Ro7+TPPrG17j9tU9yHqOYDnBDQxOfr1if6VwaQyyGtDE8/8GSnDK4lUYhS8Y6FOIOTAmZOTC5eP7DJWVZ3axcnDxmiCf7bLctenLDybuU5dxJKIZBgHv67EJ7W+A+xpgmYDXQO+J3ARCRs0RkqohMXbq0uICSY47EOpMOAAAgAElEQVT26JjfldQaFk8vpt/abkBXFlx9BEfuODDrs75d2/PCBftz87d2C/iml9cvOpAz9s0OCBbKCbtllyKoFuKWw3AzsHuHyOeZsXA1Vz6Re8GoKB2ln3UNzYy99kX+/vJ8wMqsCWrnzWnDmf/MbeGljeGNj5dx1A2vZdaXLoRCrmwh5SRuefnjgmWpJCKww6DuIZ96r9JDZ+/NwSOjDdrikoRiCLpr/vsetk+U71objbnVGDPaGDO6b9++BYroJcoDP7x/16KP/8ZFBxb93ULYa6ve+Xfy4Yz0BvTwdlS/nLAtU35zMFv17RLJXE3Kov1uQLZJtZDUBMPh/brkbE9JKqAoOKPqxqZ0YHrwRQ/lz0Rbv7GZH/xzGjMXrS5qPelCfvPbBdQOirsgUbnp3K6WoX2C5ypUcmyahGJYCLj9EoMBv8Mws4+I1ALdgRURv5s4vSO4YPp3a59xt/zkwGHc+u3duOP00ZGO788kiMLA7h3YyS6H7S9xHIQB9tiqN0fvlD3yz8Xoob0AGOAbwbYvIJh55I4D6Nc1fARcCJ3blyakFOUa5iNJf3X/buFtLs48gM7tir9+jc3pwFHY2wvyd8SfrVifma8RFXcuv2O1tHV6dKoLHWQ5iuGWU3fj2uN3LJ9QJKMYpgDDRWRLEWmHFUx+zLfPY8Bp9uvjgReMZcM+BpxkZy1tCQwH8keZYtIjQvBZkMy6AMP6dWH8qM04cER/+nXNr1TcgesOdSm+sUugd8zDibsP4XJ7MkvnAiauFbIEKMBFh40AoFM77zn86aZhjXW7Ad248ZRdEwuC5UtzLZaz99869jGSVAwd6kqjAAtpK34a08WXgci3vGcQIrDAXpfktXnZS6S2RY7bdTBbhlgMDhO234xvjg6PCZaC2E+lHTM4B5gEfAg8YIyZJSJXiMjR9m63A71FZB5wPnCR/d1ZwAPAB8AzwI+NMSVPzo/6KGywJ0K5O9Ews89Nj451Gf//tpt145gIiiElLRUqO7vO91S+omsFPtd1IfMz/Om58353OGOH9wmUM0lKNV+ke4SU5HzcPyX/sqhBnBewclmpAtlh9zMKjU1p7n27uCJ3t+UJjivROPfg4ZxzwDBu/taumW3OanplykwNJJFTG2OeMsZsY4zZ2hjzO3vbJcaYx+zX9caYE4wxw4wxY4wx813f/Z39vW2NMU8nIU8EefPus3mvTpkFTdzm+k2n7MpPDxyW87u1NSlu/fbozLlqQjqF7Qa0LCGZSknGP9q1Q4tiGJlnWc+k3JB+xZBKCXedsUeWcki6f0vaYjhwRD8AugaMpP3us3xM9s3NGNSjI9cevyPnHJD7/g/qmT2voVTZMnHm2yxeXc8bHy/Pv+MmzJUTR0Xa78IJI0pyfhGhtibFAdv2y2zrmOlvKpcy3yZncUXpTC8Yvw2//8YOHL/b4IxfHqzMnfPHb5v3+46LJ21M6Cj77jNaMnNTIhmzvhD3QP9uuTu7XCP8u8/Yg58dNByAXYf0DNzn9L2H+o7nPeCPxoW7bPbbJneSwOPn7BtrxOtn1yE9MvIFuboKVQx+zhy7Jd8cvTlH5YnrdAxwG210xRHiyuEmzvUrd85/obO7iyVpq/acA4Zxdo52ngTu9hq0ql65aZuKIYLFUFeTYmifzvzphJ2KGtU6AejmdPhkua6uksspgREDrMyVsM72wR/ulXnt/ITzDhnOX07cOVSOp37W4opyL2wDsO/wPpx3yDYsuPoINu/VKfD7B23XnwVXH8F/frQ3kJ06+MscI6mzxm4V+hnAFn06JTrDPG1ags5BZvjgnsG/MSqO0gmqbOqmky8gbGiZdQxw2dHRRqlRqKstvveoL2KSXKk5fIfNYh+jkPjXwB75Z60HLVyUNO6ElUzF25KfNZw2qRjSBl795QElPYdT/TKdNqH+5Xa1KUba7qSUCH26tGfB1Udw0HbBuco7DM7Od25fW5MzhjFis24MsTv9bWKk4DrPRtAz1yckyyvfiKdGBBHJLFIP4Vk2x+6aP05jaOkUgqpZ+lN0rz2usEwP57f3zDNBMijQ7FgMt5y6G4eOCu78eudROJCd8RYnRlPMxLRSMqhHx6LX/HYTlrK6bUD7D3vW3EQt+REHz+DRftjUYigzzWkTOkIGOGLHAaGfRaXO5Upynl3/SNJNlEbgVjA/HJd7NO7GcWtFWQQnDMfKClJyUy8+mIk7W+6VrV314/P9pKCR3czLgstlBZ23Q52v+ZpwJQwwsLt3dOivdf/dfYZyy6m7cv4h2cFjaLGW/Mtc+gmaWe+sNtbeL7OLXYb0ZI8te4V+DrCZzw01JEc7zsf6MigGdxwtF3ts2YsHfrhXpjR4NeCUznEUw/Pn71+WqgnpzCBMYwxlJd/kmuu/Ge6aiYrT6aWNyXQo7qCyg3Pv/Y3ge/tsyS2nel0/7n2izCNwHkp30bRicWIZ40LiBo5sP3YHZvO066D5HmHPQkrgpZ+P468ntdwb/8gw38/zuw1EWoLVAJceNYoJ2w8IjbcEPaj+NTHuOmNMYC0ux2LI7Tor/P50ipGumnTp+T+fsFOWmy1LeYdwwfhtGdSjY8FzI/xsN6Abo7douX9BmXVRcQZ3zkTHYf26ZA0uCuXO743hum/ulHMfZxCmrqQy41+60M2HV0xIJFPGCQqmDZmspFwL6/j7yEuOGsmE7TfLuU8Q7sCoE1juYaduxnGVbt6rE2//+iBvx+/Cmdbv9uP73TnH7OwN2gZZDGGuhJQIQ/t0Zr/hLYrJn4M/dnifnJZX7y5+V40EDgLCJscFXf9XfC7Jvl3bB7Yfx2LIN58h3y3y38OwNvHceftxSZlrWR232+Asq9itCMcMDbeGetml8OMuPnXf9/ekr2uuUfvaFC/9fBwPnb130cd09xdxow39urbPG+sKWlWv3LRJxZBr1nzHImeSukeybmpSkgledcmxvm+UgFmUhnLhhJaMKccyuvGUXbhwwghGbFZ8jAGgX7cOoYH07+4zlKd/NpYxLleIX9y/nOQtAFZIw3f2rQnotMcO78PkC/bngjzZYkGj9aDgbV3IwCDIYvBfjtpUKjBT6KcHDaddTYrh/brkkFBYsiZ65s6kc/cLTYUe3r9rTtdluXArwly327lmfldZoaRSXo9Au1oriWS3LYKtwCi4Le24NdRSInkHeNVQpq1NKoZS6OFDfMWtnIJ9PTvVZZYd7Nq+lmuO24HfHjmSJ36yryWLLUxSowPv2r1WC+vXtQNnj9u6pCMQEcm4rvwprhB/Ipvjew9yP/3y0BFs7avxFFQ31F/2ozYlgXIFzYEAGDUo21/uv6ZpY+jesS5LARy0XX/m/O4wTyZaELuFuLGCKt5u078L85aGrxqWZCpwVPxNzK2Mc82ydpRuWHwnKjUp8SwwlMQ1cCuaqJ32hRNG8N9fHZS1XST/s25c+1aKNqkYgtYxuOuMMfyPa/ZhofhHkyMHdqNf1/ZcOGFEJvuje6c6Ttx9CGfsuyXb+yoqJhVoch+lDCt/BnLZ0aNYcPURGVl226InH105Ied3HvzhXvzze8EV1y86bEQm8yrIsnL7sXNdRbcSOP+QbdhxcPfAQHLYeh3+2vhB8jhzA/LN4QhCBH5/7A7868w9sj47N2A2tYjw+rzwCWqlKjdSCG6LIVfap2OJ9unSnuN2Da64O6B7B/5y4s456walRDyj+jilW5zDuBWaoyRuztNX9O7SLtD6EfK7hHMlepSLNrnmc5Cfd+zweBVb/XTrUMfbvzkYsDqL0/bagp/YPn83jh8+qUk57raUK5ZSDpyRkZB/4aPROfzP7vpUtQETFKJOCHR/9acB98KhW5SFnMRSvCmBOVcdxherNvB/byzIWE3Fdkgd6moy2VLjtu3LlRO356U5SzOJC4W4MiphMfhxpzLnshjcLrEtegf74I/bdTDH7DKIf08NL1WSEvHYisUUtPTjltux/kcN7MY2/bswZ0mwxRZ2VpH8bSMzwa1gSZOj8i1nEyGXcm9Xm+LyidsH5vvPt10BYXMBCpbD1ZzKXc7Zj3NNkpQi6JmKrBgijsCiFLxzjpUSyfixLzt6VMs8iiKeaucrA7p35K4zxnDDybuwea9OfHvPLTJKNt+1PHyHzTK1dtrZ8ZN8KbCFMOOy8Z73lx4VHuA+dFR/z5oduV1JLa9/NG5rHvjBXln7RLmm/k43yCJ87Jx98h8I6GlnWLmXAl5T37KmS9CaJvmRvO3QcYNW0pXUJi0GP8+fv1/sYwRNqIqCU8F13xxpdSePaams+P2xW3rqqmTJ4RKj2MqZSeGIUmzA7oBt+/LibO+iTEH+2aBJcUGnTHJZxF6d2/FVjgXew4LCuXCL7LdgW65l7mOcc8DwTH2tGttEalebomv72tipoG45HBx3VVA20a5DenpdSbkUg+ve1NakPEkMWTLkuLbZyQDZ+0ZxsfXp0p7//GhvHn/vCybu3DK58thdB/HPNz+la4fanFZ+eHZd/g7fsfQKKYWfNKoYiDYtPh/Favf7z9qTJV9vzDlK/cOxLT7V3xwRPEK77Tuj6dWlnefBrXR2Qy6LIZ+PFlrSbPPhHhXm6jSSHIE98IO9eHnO0tD7lk8JTb5gfz5cvIZz7nk3s23fYeGDg6iyu1NtHRGiWo7f2mMI/3ord7VVEWFwz44sXLnBPl+KW7+9W8aF5h4gNTSlPZ1wlOBzFHKtxe2//1EGA7efNppendvxjZvfyGz7xaHb0K1DHd/aYwvPvpcdNYpfHbYdqZR4zvXLCdty7TOzW+QIOZch/289ZY8hLFmzkR8dUNr6TLlQxZAQxfY5exSxClsQzpJ/X7nSHePU6k+CIb0sX/mxAcHEKOUf8nHVMdtnOqhciFhKMslg3tA+nXOWYHd3GkEW09Z9u2TJ/p29tsjaz4/BcNcZYzLrGvhxxxV6d7bckzWpVKQGesSOA/IqBoDXLjyQoRc9CVhZR+NDSnw0NKc9iiqXeirEwmoOyTffPiBrLEqMIUpZDIdUSjIp7Y7Ie23VO2vCadjPac5RIueVXxxAx3Y1tK+tyaybUik0xkDxbqCqxPVTkihIFoe+Xdsz//eHc2pAFliU1FlnPejdQwLTp+65Rc4H6L6z9uTlX4zLXJJi3DvF4j5TWMltf3XTXNekU52l5HfZvCdjh/fl23sGKxH3wk3bDejKbw7fjqsmbh9JKW6MsJKcY33sM8wa0ORKQ97YlPZ8nmvWd1DRwx+N25qDt2txmzq/IKysyJ5bWjJ5s5KK6+L65alabMljSVRbI1kzvMOud2NzOtQFNaR3J8/kvEqiFgPJuBgqOUvRTZ/O7fnm6MGc6gpYVpKwbKR8A7mxw/uw97A+ngJ7Dtcct0Ng6qgbY2BP2xoT22RIifDwj/aOVCPoymO257ePvJ93v/Dzt3ROYcXq3Ioh38zc7p3qeOIn+7J1X+/8iNqUeCZgtfO51b6/n1VTK8pSp7sM6UG3DrWZAGsQzvGd//7MJ3fJ8YamdKYNDu3dKae7NKgjdSr3OtaJg/v3vvjzcfzPS/N4YOrCjNvIbU8cv1v+4ot+7j5jj5wxvxaZrf91NamsUuu5LAaJURG3XKjFkBDVcqtTKeHa43dix8E9Ki1KTnIprQ+uOJQ7Tt899PMTdx+SNQ8kc9yAbc4DnEpZher2yeHLd9jet0BSoaa9252+bmOIYmi2th+908BIM3O3H9Q9a2a+/zqEXdf/+27wHBE3PTq1Y0ZIEUOw7ovTuTt1qvyB3NtOG51JL3YspYfO3ouHzt6bXx++nUtO77ELSQxocllgW/bpzBa9LZeefxByx+mjGdYv+mz/43YdTF2NRFIK0KLMalOSpRhGhhQPbEqHr89STcRSDCLSS0SeE5G59v/A1i0ip9n7zBWR0+xtnUTkSRH5SERmicjVcWSJQzIWQ/xjtCVyjWA7tatNNAe/Za5I9Jvkny+Rb20JP+6A7w/2D/6uYzHEWZPif10K9MTRm4fGboblLMWRm9tPG83pew/1LHF7mO2m9B93816duGC8NRnP+X27bdGL3l3ae5Sf360X6d7Y+/iLJzpB7biuwj9/cyfm/u7wyPs7p6urTdHBVtgjNuvK7KsmMDykxH372lRFJ65FJe7TdxEw2RgzHJhsv/cgIr2AS4E9gDHApS4F8idjzAhgF2AfETkspjwVoxrcNq2JuIvmFEJYBdtc+Eew+Sbo+XG6rl8cum3grGWAQ0dtxqiB3UILE0ahp0sRXHP8jgXLGYWDtuuftbjQKWOG8NGVEwIz+hwrItcKcX45o4j9XbvUSqMvZuO4lvz3rNSxQ+eZr3NZDBub0oFppntt1ZsbTt6FUQO7twnFMBG40359J3BMwD6HAs8ZY1YYY1YCzwETjDHrjTEvAhhjGoB3gOC58CWmmAZUDeUGWjP5FruJi3tM6dyroPpJYcSd8+BYDLn6gB6d2vHkT8fmzG5KimL7or1CsuZEJDRm0K7G2p5LMfhH91Gut6MEm2yLwSlh0862PostgFksjsS1NanMtdgYsqhP9451mcrHrUAvxA4+9zfGLAYwxiwWkaCZV4MA9xz2hfa2DCLSAzgK+GtMeYqimBvVqV1N2dfM3ZQomYUVcNh//3AvnpqxOHAt5tDDxBSvZcW7VtAL5OCs/QpzoYHLYgjJxoJsC6GQ9nDsroN4ZtaXmRUNzxy7FRsamzPFG8s1f6cl+NxiMdSH9AnubDHHWqrmppFXMYjI80BQ3uNvIp4j6Odnbp2I1AL3An8zxszPIcdZwFkAQ4Zkpz/GoZj706muhlU0JipHW6Bnp7pY/u6ouLOCRmzWjRGbha8kNvXig7OWb4xbTqQaFltxU86UbCd1M6kJbX7Gj9rMk63Woa6GXxyaPznge/sO5cKHZjIg5mI7ftxZSWEWgztm5iiUamkbQeRVDMaYg8M+E5ElIjLAthYGAF8F7LYQGOd6Pxh4yfX+VmCuMeYveeS41d6X0aNHF/XUXnLkSD5cvCZrezGj1yCzdWD3DuwdIeOlLfPuJeNj17RPmqA6VTFWQbW+X2UWQz4xRsdYr8DPXlv15rS9tuAH+4fP3G1Xm4LwiiIeBvXoyNA+0WNSW/bpzMtzlmYtw3ni7kM4cXdrUHnsLoPoH3PtBye2UZtK0aGd1fFvDLEYvDPSC0+GKDdxXUmPAacBV9v/Hw3YZxLwe1fAeTzwKwARuQroDpwZU45IfM9V0Csu7gwNhzcC6q8r2ZR0XQiyc9mLIb7FYMtTvc++hwdjrHDmp7bGKhoZxFM/HcsXqzZw2eOzYF20471+0YEFnf9Xh49gv236sEvI2hYA150Yf/leJzuqrqZlXY+wtGZv2Rbv/2okbgT1auAQEZkLHGK/R0RGi8htAMaYFcCVwBT77wpjzAoRGYzljhoJvCMi00WkLArCTzH3pxB/tVI+knrYtuzTObA4X1TSZaypf8QOAxg/MndZh0KkeOnn4xIpWRLEyIHdOHhkfzoHDKySon1tDQeOiF7molic+RS1NVbdpAVXH8GZIWnNda6gihN4r+aKC7HujjFmOZA1TDbGTMVlBRhj7gDu8O2zkCpxsxXz7JY7A0IpL53b1zLriglZs26jYiJkJSXFTREKEuay0n7sK9Y2tE9nth/UnZfnLC3ZE7opPD+NLldSPmo9MYbqDzJoziXFuTaqYT1dJZtx21rlqkcNDA82lwPHEVXNfmSHKIHbpHE/Pz/LsWhSNeNYDFFS193zPVpiDKWRKwm0VlKRXDB+Wxau3MDMRasrLYri4sgdB3LAtv0qXlm2xZVUUTEyhIlxT8AyouXAidHdcuquTNh+QEVkiIsz0S5KBVf3Ouhi65FqHjSoxVAkw/p14fGf7FtpMZQAKq0UwFUrqUoe/jAx9gyZwFbqvDHHYqiPUNG1WnGCz0GrxPlxT+AT3/9qRBWDopSAapvH4ObWb++WeV0pveUohiiVbquVJjunOUrlWjetwc2oikFRclBs6Q5nHkS1PPzuOJq7BEelanx1zCiG+MuNVoqmTLpqYd2o0yaG9C5fvbBCqbzNrShVzAsXjGNNfeEz3Lt2sB6tzu2rL0khybWvi8WxGMLWqmgNNESMMQzyFRrs0r6WW07djdFDk5tUmDSqGBQlBz07t/NUMI3K+eO3oU/X9hy548ASSBWPKMHSUuMEn9eHlJBoDUSxGOZcdVigu27C9pVdXTEfqhgUpQR0alfLD3OUhKgkhbi3SqVCnJXoBsQsS1FJnBhDbY4YQ2utwtw6pVYUpWhydWTlYsL2m3HP9/fg1D2C165uDRyxg2UN7hCymmBrRi0GRWljxF3pLCn23rp1F5w8YscBHL7D4ZvkIl1qMShKG6Mags+bCpuiUgBVDIrS5qhJSWbNBEUJQl1JitLGSKWEZ8/dnw8C1iZxqLY1M5TyoopBUdoYtSlhSO9OVT3BSqksak8qShujWmZjK9WLKgZFaWNUwwQ3pbpRxaAobQzNSlLyEUsxiEgvEXlOROba/wOLf4jIafY+c0XktIDPHxOR9+PIoihKNApJsdxU0zGV3MS1GC4CJhtjhgOT7fceRKQXcCmwBzAGuNStQETkWGBtTDkURVGUhIirGCYCd9qv7wSOCdjnUOA5Y8wKY8xK4DlgAoCIdAHOB66KKUdROMtAKoqiKC3ETVftb4xZDGCMWSwi/QL2GQR87nq/0N4GcCXwZ2B9TDmK4h/fGc3Gpta7gpSiKEopyKsYROR5IKhG7G8iniPISWlEZGdgmDHmPBEZGkGOs4CzAIYMGRLx1Lmpq0kVvMiGoijKpk5exWCMOTjsMxFZIiIDbGthAPBVwG4LgXGu94OBl4C9gN1EZIEtRz8ReckYM44AjDG3ArcCjB49WqdlKoqilIi4w+XHACfL6DTg0YB9JgHjRaSnHXQeD0wyxvyPMWagMWYosC8wJ0wpKIqiKOUjrmK4GjhEROYCh9jvEZHRInIbgDFmBVYsYYr9d4W9TVEURalCYgWfjTHLgYMCtk8FznS9vwO4I8dxFgDbx5FFUZTk0VkMbRONvCqKEooG89omqhgURVEUD6oYFEVRFA+qGBRFURQPqhgURQlFg89tE1UMiqIoigdVDIqiKIoHVQyKoiiKB1UMiqIoigdVDIqiZGF0ZlubRhWDoiiK4kEVg6IoiuJBFYOiKIriQRWDoiiK4iHums9tnsuOGsmcr9ZWWgxFKQmiU5/bJKoYYnL6PltWWgRFUZREUVeSoiiK4kEVg6IoiuIhlmIQkV4i8pyIzLX/9wzZ7zR7n7kicpprezsRuVVE5ojIRyJyXBx5FEVRlPjEtRguAiYbY4YDk+33HkSkF3ApsAcwBrjUpUB+A3xljNkGGAm8HFMeRVFCEIGjdhoYaV+ji3q2aeIGnycC4+zXdwIvARf69jkUeM4YswJARJ4DJgD3At8DRgAYY9LAspjyKIoSwid/OKLSIiithLgWQ39jzGIA+3+/gH0GAZ+73i8EBolID/v9lSLyjoj8W0T6h51IRM4SkakiMnXp0qUxxVYURVHCyKsYROR5EXk/4G9ixHMEZUIbLGtlMPC6MWZX4E3gT2EHMcbcaowZbYwZ3bdv34inVhQlDqJruLVJ8rqSjDEHh30mIktEZIAxZrGIDAC+CthtIS3uJrCUwUvAcmA98LC9/d/AGdHEVhRFUUpFXFfSY4CTZXQa8GjAPpOA8SLS0w46jwcmGWMM8DgtSuMg4IOY8iiKoigxiasYrgYOEZG5wCH2e0RktIjcBmAHna8Epth/VziBaKxA9WUiMgP4NnBBTHkURVGUmMTKSjLGLMca6fu3TwXOdL2/A7gjYL9Pgf3iyKAoiqIki858VhQlC13BrW2jikFRFEXxoIpBURRF8aCKQVEURfGgikFRFEXxoIpBUZRQdAW3tokqBkVRFMWDKgZFUbI4ZKRVz3JIr04VlkSpBLrms6IoWZy+91CO220w3TrUVVoUpQKoxaAoShYiokqhDaOKQVEURfGgikFRFEXxoIpBURRF8aCKQVEURfGgikFRFEXxoIpBURRF8SCmFRZeF5GlwKdFfLUPsCxhcYpB5fCicnhRObyoHF7iyLGFMaZvvp1apWIoFhGZaowZrXKoHCqHyqFyhKOuJEVRFMWDKgZFURTFQ1tTDLdWWgAblcOLyuFF5fCicngpuRxtKsagKIqi5KetWQyKoihKHlQxKIqiKB5UMZQIETlFRHayX7f5BRKr6XqISMXbvYgcLSJbV1qOaqCa2oYtQ0XbRzW0jYo/IEkhIseIyJVVIMfBIvIq8BdgFwBTgUCOXo8sOY4WkfPLfd4AOQ4WkTeB24EBFZSj4u2jWtqGLUvF20e1tA1o5Su42aOLFPBd4CJgCxF51hjzagXk6ADcCfQDrgImAp3sz2uMMc1lkkOvh1eWWuAC4GxgiIi8YIyZXmYZBOgM3At0BS4GzgW2AF4TkZQxJl0mOSraPqqpbdjnq2j7qJa2kYUxptX/AePsi/p94KUKyjHR9fpU4E29HlVxPY7B6ozOBd6q4PU40fX6x8ADbbV9VEvbqJb2US1tw/lrla4kEfmpiPxDRM60N71sjPnaGPMPoLOInGHvV9Lf55Lj+wDGmEft7TXAJ8AsEdm8lDL45NDr0SLH1SLyTXvTk8aYemPMX4B+InKKvV9J1650yXECgDHmfnt7DbAK+FxE2pdSBpccFW0f1dI2XLJUtH1US9sIpZJaqUjNejrwX2AC8DLwK2Br1+eHAbOAnmWW49fAVq7PdwCmAF31epTnegACnAe8DhwPfGjL1c+1zzeARSW+FmFy9HXtszfwUSnlqJb2UQ1to1raRzW1jVx/rdFiOAi4xhjzDJZvsAPwLedDY8zTWBf7LBHp6mjkMsjRDsskduSYCWwATirR+cPkaLPXw1hP1QHAxcaYB7EewJ2AQ137PAzMEZGfgxXwK6McE1z7vAEsFJGjkz6/j1+Hps4AAAmFSURBVGpoHxVvG/Z5Kt4+qqxthNJqFIPLzH0XOBLAGDMVeBMYKCL7uHa/EPgDMBfYrExy/Ncthx1UehboUIoUvLZ+PfzHcMkxFRhry/EMMAcYJSLbunY/G7hWRL4EBpVZjhH2ft2Aj4DGOOfPIVfF20e1PCs+WcraPgqUoSxtIwpVqxhEZJSIdHDem5bI/OtASkT2s9+/DywGBtrfGwbcDDwC7GqMuaESctgjg37AOvt1LERkH3HlNlfwehQlR9LXA+jofuOSYx7QVUR2sN+/DHTHCrYiIjsD/wAewroed5ZZji72fmuAwUD/mOcHMr7pjKKqRPsoVoYStI1cspStfRQhQ0naRjFUnWIQkR1F5DWsNLberu2OrHOxfKInipVSthBrlDPU/nw1cI4x5lhjzBcVlAPg58aYO4qVwT7friLyLPACVuMpVI6krkdcOSCZ67GniDwE3CQi410Pn5N6/TbQDBwiIrXGmA+wRn1O/frlwI+MMSfEvB5x5QA4yRjzf8XKYJ9vLxH5B3CeiHRzOlaXHCVvHwnIAAm0Dfuc+4jIncDFItLLJYsTSC55+0hABkigbcSh6hQDVh7vg8aYbxhjFkEmt9nRtl8Dr2L5Kf9kX+yeWDcUY8xSY8zcSsthy9JQ7MlFpE5E/o5VSfFvwCSsNMOyXo+k5LBlKfp62OcbhzW6/Q8wG8tP3VOsXO8m+xzzsAKZw7By9QE2Yq/4Z4z53PZpV0qOBc5xjDH1MeXYD7gRS1kPBH4lIuPtYzfZu5W6fcSWwd43VtuwZdkK6768iDUP4EoROdw+fqP9v6TtI6YMC5zjxG0bcakaxSAiKdtFsdZYaWOIyCEi0gMrko+IXAXcgzXKuQSrcb1qv4/rEqgqOYD2wCvAWGPME1id0Hb2CKPZluPyNiQHwI7AFGPMv4C7gTqs+5S25bhKRG4HpmEpsTEiMg1YgaXQqkGOZxOUYzTwujHmXizLtj9wsoj0d+Sg9PelGmRwGAN8aI+0fw5MB44SkQGOLGVoH3FkSLJtxMNUMCUK2BPYxvW+K5bZeSSWz3MS8E+sFLuhWA1smGv/FAmkuFWjHGCVRHd9dgZwi/MZVud0D97Uw01WDvv9zlgP0KXAEuAl4A7gRKwUP/996QL02ITlOAKrcx1ov/+bfe6zgG1KcV+qQQbXsY4CzgH2tN9vhRXPGGK/H4kVVD8X2LcU96UaZCjFX2VOCj2AJ7FMzIuBzq7Pfg28Axxtv98PeBTYy9242oIcWB1vyn49zO6EejqftSE5urg+G4PVCR9nvz8DK1i4U1uTA6vjvQFrpPkQ8DDwCyx/PUnKUQ0yuI41AHgcy5L9LVa67aH2Z38CLrBf1wDfxlLg3RO+HhWXoZR/lXIldcYahf/Efr2f67MnsEblvez3U4EvgXqwXD0mudohVS2HsUjbAd4F9j77O5+1ITnGOh8YY94G+mL7hLF82z2AlW1IDue+zMGaF/AH4N/GmG9gZfwc4HwxQTmqQQaH0cBrxpj9jDFXAn/FslAAXgN2EJE9jOXqXATsZ4xZnbAs1SBDySibYhCR74jI/nbWwiKsYOYDWB3tHiIyCMAYMwNrxPFjEemDFdjbgZZgWawL2orkcNIJxT6XkzLrKCZ/ClxbkaM98AbwI/urB2Ep7/o2JMcYRw5jTIMx5kVjzH32V3cDnnaOE0eOapDBJ8s4+7pPxnLtOizHmgsA1hyJd4HrRaQLMAr4VEQ6xZWlGmQoFyVd2tPuNDbD8qulgY+xRhs/M8Yss/fZB/gmMNUYc5fru+dj+euGA+cZK6WrLckxxRhzt72txhjTLCJ3AR8bYy4rVoZWLEfmvojIKCzTfDOsSUDnGGM+bGNyZO6LvX1frFHrMuAHxpgFrVWGqLKISJ0xplFEfgqMNMb80PXd67DmAmwBfMcYM7u1ylARSuWjAmpMiw/ybvt1LZY/8j++fc/DymjojiswBdSpHHRSOTJy9AA62ts64qq300bl6E5LHGggcHhrl6EQWVz7PA4cbL/u59o3bqC94jJU6i/5A1oX4/fANVh+6KOAO12fC9asx/1d27pgLdbxNlZgc6DKoXIEyDHFlmOQyuG5L4NbuwzFyoI1N+IOYAjwO+A9YhYErAYZKv2XaIxBRPbHys/tiTXt+0osE/sAERkDmWDlFcBlrq8egeWrfQ/YwcSYkapybNJyTLflWKRyeO7LwtYsQ5GyXG5/rQNWddLJWGnmBxtjVrZmGaqCJLUMVtbGt13vb8YqSHU6MM3elsLy2T0ADLW3TcSK2qscKofKUUY5qkGGGLIMxkob/iew86YiQzX8JXswa3m+9rT43b4F/MF+PR34if16NHBvyX6UyqFyqBytRoYiZblvU5WhGv4SdSUZY9YbYzaalrVSDwGW2q+/i1VK4Qms9U3fgZZ0R5VD5VA5yi9HNchQpCzTSiFLNchQFZRI69ZgmVtPY0//xpox2wNrWnjsoJ3KoXKoHJuWDNUkSzXIUMm/Uk1wS2MVFlsG7Ghr2N8CaWPMayZm0E7lUDlUjk1ShmqSpRpkqBwl1Lh7Yl3c14AzKqX5VA6VQ+VoPTJUkyzVIEOl/ko281lEBmMVj7rOGLOxJCdROVQOlWOTkqGaZKkGGSpFSUtiKIqiKK2PqlmoR1EURakOVDEoiqIoHlQxKIqiKB5UMSiKoigeVDEoiqIoHlQxKEoeRKRZRKaLyCwReU9EzhdrmdNc3xkqIqeUS0ZFSRJVDIqSnw3GmJ2NMaOwauccjrVyWy6GAqoYlFaJzmNQlDyIyFpjTBfX+62wFuvpg7Vs411Yyz2CtbznGyLyX2A74BPgTuBvwNXAOKzqnTcZY/5eth+hKAWgikFR8uBXDPa2lcAI4Gus+jn1IjIcqzT1aBEZB/zcGHOkvf9ZWEs+XiXWYvKvAycYYz4p649RlAjUVloARWmlOKWW64AbRWRnoBlrfeAgxmMVYzveft8dGI5lUShKVaGKQVEKxHYlNQNfYcUalgA7YcXs6sO+hrXIy6SyCKkoMdDgs6IUgIj0BW4BbjSWH7Y7sNgYk8YquFZj7/o11vq/DpOAs0Wkzj7ONiLSGUWpQtRiUJT8dBSR6VhuoyasYPN19mc3Aw+JyAnAi8A6e/sMoElE3gP+D/grVqbSO/aKX0uBY8r1AxSlEDT4rCiKonhQV5KiKIriQRWDoiiK4kEVg6IoiuJBFYOiKIriQRWDoiiK4kEVg6IoiuJBFYOiKIriQRWDoiiK4uH/AasM7Ced6k0AAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "FF5.resid.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.436689408707376e-15"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.resid.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. The sample covariance between the regressors and the OLS residuals is zero:\n",
    "\n",
    "$\\begin{equation}\n",
    "\\sum_{i = 1}^N X_i \\widehat{\\epsilon}_i = 0\n",
    "\\end{equation}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-7.109926796616817e-16"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(FF5.resid.values * df_stock_factor['MKT'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.25456703e-04, 5.17198916e-20],\n",
       "       [5.17198916e-20, 1.54769773e-04]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.cov(FF5.resid.values, df_stock_factor['MKT'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that both the sum and the sample covariance are very close to zero."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ol start=\"3\" style=\"list-style-type: decimal\">\n",
    "<li>The point <span class=\"math inline\">\\((\\overline{X}, \\overline{Y})\\)</span> is <strong>always on the OLS regression line</strong> - if we calculate <span class=\"math inline\">\\(\\widehat{\\beta}_0 + \\widehat{\\beta}_1 \\overline{X}\\)</span>, the resulting value would be equal to <span class=\"math inline\">\\(\\overline{Y}\\)</span>.</li>\n",
    "</ol>\n",
    "\n",
    "However, these properties are not the only ones, which justify the use of the OLS method, instead of some other competing estimator. The main advantage of the OLS estimators can be summarized by the following Gauss-Markov theorem:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gauss-Markov theorem\n",
    "\n",
    "<p>\n",
    "Under the assumption that the conditions <strong>(UR.1) - (UR.3)</strong> hold true, the OLS estimators <span class=\"math inline\"></span> and <span class=\"math inline\"><span class=\"math inline\">\\(\\widehat{\\beta}\\)</span></span> are <strong>BLUE</strong> (<strong>B</strong>est <strong>L</strong>inear <strong>U</strong>nbiased <strong>E</strong>stimator) and <strong>Consistent</strong>.\n",
    "</p>\n",
    "\n",
    "## What is an Estimator?\n",
    "\n",
    "\n",
    "<p> An <strong>estimator</strong> is a rule that can be applied to any sample of data to produce an <strong>estimate</strong>. In other words the <strong>estimator</strong> is the rule and the <strong>estimate</strong> is the result.</p>\n",
    "\n",
    "<p>The remaining components of the acronym <strong>BLUE</strong> are provided below.</p>\n",
    "\n",
    "### OLS estimators are Linear\n",
    "\n",
    "\n",
    "<p>From the specification of the relationship between <span class=\"math inline\">\\(\\mathbf{Y}\\)</span> and <span class=\"math inline\">\\(\\mathbf{X}\\)</span> (using the matrix notation for generality):\n",
    "<span class=\"math display\">\\[\n",
    "\\mathbf{Y} = \\mathbf{X} \\boldsymbol{\\beta} + \\boldsymbol{\\varepsilon}\n",
    "\\]</span>\n",
    "We see that the relationship is <strong>linear</strong> with respect to <span class=\"math inline\">\\(\\mathbf{Y}\\)</span>.</p>\n",
    "\n",
    "\n",
    "### OLS estimators are Unbiased\n",
    "\n",
    "<p>Using the matrix notation for the sample linear equations (<span class=\"math inline\">\\(\\mathbf{Y} = \\mathbf{X} \\boldsymbol{\\beta} + \\boldsymbol{\\varepsilon}\\)</span>) gives us the following:</p>\n",
    "\n",
    "$\\begin{aligned}\n",
    "\\widehat{\\boldsymbol{\\beta}} &= \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbf{Y} \\\\\n",
    "&= \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\left( \\mathbf{X} \\boldsymbol{\\beta} + \\boldsymbol{\\varepsilon} \\right) \\\\\n",
    "&= \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbf{X} \\boldsymbol{\\beta} + \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \\\\\n",
    "&= \\boldsymbol{\\beta} + \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \n",
    "\\end{aligned}$\n",
    "\n",
    "\n",
    "<p>If we take the expectation of both sides and use the law of total expectation:</p>\n",
    "<p>\n",
    "    $\\begin{aligned}\n",
    "\\mathbb{E} \\left[ \\widehat{\\boldsymbol{\\beta}} \\right] &= \\boldsymbol{\\beta} + \\mathbb{E} \\left[ \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \\right] \\\\\n",
    "&= \\boldsymbol{\\beta} + \\mathbb{E} \\left[ \\mathbb{E} \\left( \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \\biggr\\rvert \\mathbf{X}\\right)\\right] \\\\\n",
    "&= \\boldsymbol{\\beta} + \\mathbb{E} \\left[ \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbb{E} \\left(  \\boldsymbol{\\varepsilon} | \\mathbf{X}\\right)\\right] \\\\\n",
    "&= \\boldsymbol{\\beta}\n",
    "\\end{aligned}$\n",
    "    \n",
    "    \n",
    "since $\\mathbb{E} \\left( \\boldsymbol{\\varepsilon} | \\mathbf{X}\\right) = \\mathbf{0}$ from <strong>(UR.2)</strong>.</p>\n",
    "<p>We have shown that <span class=\"math inline\">\\(\\mathbb{E} \\left[ \\widehat{\\boldsymbol{\\beta}} \\right] = \\boldsymbol{\\beta}\\)</span> - i.e., the OLS estimator <span class=\"math inline\">\\(\\widehat{\\boldsymbol{\\beta}}\\)</span> is an <strong>unbiased</strong> estimator of <span class=\"math inline\">\\(\\boldsymbol{\\beta}\\)</span>.</p>\n",
    "<p>Unbiasedness does not guarantee that the estimate we get with any particular sample is equal (or even very close) to <span class=\"math inline\">\\(\\boldsymbol{\\beta}\\)</span>. It means that if we could <em>repeatedly</em> draw random samples from the population and compute the estimate each time, then the average of these estimates would be (very close to) <span class=\"math inline\">\\(\\boldsymbol{\\beta}\\)</span>.</p>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "### OLS estimators are Best (Efficient)\n",
    "\n",
    "<p>When there is more than one unbiased method of estimation to choose from, that estimator which has the lowest variance is the <strong>best</strong>. In other words, we want to show that OLS estimators are <em>best</em> in the sense that <span class=\"math inline\">\\(\\widehat{\\boldsymbol{\\beta}}\\)</span> are <strong>efficient</strong> estimators of <span class=\"math inline\">\\(\\boldsymbol{\\beta}\\)</span> (i.e. they have the <strong>smallest variance</strong>).</p>\n",
    "<p>To do so we will calculate the variance - the average distance of an element from the average - as follows (remember that for OLS estimators, condition <strong>(UR.3)</strong> holds true):</p>\n",
    "<p>From the proof of <strong>unbiasedness</strong> of the OLS we have that:\n",
    "<span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "\\widehat{\\boldsymbol{\\beta}} =  \\boldsymbol{\\beta} + \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \\Longrightarrow \\widehat{\\boldsymbol{\\beta}} - \\boldsymbol{\\beta} = \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \n",
    "\\end{aligned}\n",
    "\\]</span>\n",
    "Which we can then use this expression for calculating the <strong>variance-covariance matrix</strong> of the OLS estimator:\n",
    "    \n",
    "$\\begin{aligned}\n",
    "\\mathbb{V}{\\rm ar} (\\widehat{\\boldsymbol{\\beta}}) &= \\mathbb{E} \\left[(\\widehat{\\boldsymbol{\\beta}} - \\mathbb{E}(\\widehat{\\boldsymbol{\\beta}}))(\\widehat{\\boldsymbol{\\beta}} - \\mathbb{E}(\\widehat{\\boldsymbol{\\beta}}))^\\top \\right] \\\\\n",
    "&= \\mathbb{E} \\left[(\\widehat{\\boldsymbol{\\beta}} - \\boldsymbol{\\beta})(\\widehat{\\boldsymbol{\\beta}} - \\boldsymbol{\\beta})^\\top \\right] \\\\\n",
    "&= \\mathbb{E} \\left[  \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon}  \\left( \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \\right)^\\top \\right] \\\\\n",
    "&= \\mathbb{E} \\left[  \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\boldsymbol{\\varepsilon} \\boldsymbol{\\varepsilon}^\\top \\mathbf{X} \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\right] \\\\\n",
    "&= \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top  \\mathbb{E} \\left[  \\boldsymbol{\\varepsilon} \\boldsymbol{\\varepsilon}^\\top\\right]  \\mathbf{X} \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\\\\n",
    "&= \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top  \\left(\\sigma^2 \\mathbf{I} \\right)  \\mathbf{X} \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\\\\n",
    "&= \\sigma^2 \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbf{X} \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1} \\\\\n",
    "&= \\sigma^2 \\left( \\mathbf{X}^\\top  \\mathbf{X}\\right)^{-1}\n",
    "\\end{aligned}$\n",
    "\n",
    "\n",
    "### Estimating the variance parameter of the error term\n",
    "\n",
    "\n",
    "<p>We see an immediate problem from the OLS estimator variance formulas - we do not know the true error variance <span class=\"math inline\">\\(\\sigma^2\\)</span>. However, we can estimate it by calculating the <strong>sample</strong> residual variance:\n",
    "<span class=\"math display\">\\[\n",
    "\\widehat{\\sigma}^2 = s^2 = \\dfrac{\\widehat{\\boldsymbol{\\varepsilon}}^\\top \\widehat{\\boldsymbol{\\varepsilon}}}{N - k} = \\dfrac{1}{N-k} \\sum_{i = 1}^N \\widehat{\\epsilon}_i^2\n",
    "\\]</span></p>\n",
    "\n",
    "<p>Note that this is an <strong>estimated variance</strong>. Nevertheless, it is a key component in assessing the accuracy of the parameter estimates (when calculating test statistics and confidence intervals).</p>\n",
    "<p>Since we estimate <span class=\"math inline\">\\(\\widehat{\\boldsymbol{\\beta}}\\)</span> from the a random sample, the estimator <span class=\"math inline\">\\(\\widehat{\\boldsymbol{\\beta}}\\)</span> is a random variable as well. We can measure the uncertainty of <span class=\"math inline\">\\(\\widehat{\\boldsymbol{\\beta}}\\)</span> via its standard deviation. This is the <em>standard error</em> of our estimate of <span class=\"math inline\">\\(\\boldsymbol{\\beta}\\)</span>:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "The square roots of the diagonal elements of the variance-covariance matrix <span class=\"math inline\"><span class=\"math inline\">\\(\\widehat{\\mathbb{V}{\\rm ar}} (\\widehat{\\boldsymbol{\\beta}})\\)</span></span> are called <strong>the standard errors (se)</strong> of the corresponding OLS estimators <span class=\"math inline\"><span class=\"math inline\">\\(\\widehat{\\boldsymbol{\\beta}}\\)</span></span>, which we use to <strong>estimate</strong> the standard <strong>deviation</strong> of <span class=\"math inline\"><span class=\"math inline\">\\(\\widehat{\\beta}_i\\)</span></span> from <span class=\"math inline\"><span class=\"math inline\">\\(\\beta_i\\)</span></span> <span class=\"math display\"><span class=\"math display\">\\[\n",
    "\\text{se}(\\widehat{\\beta}_i) = \\sqrt{\\widehat{\\mathbb{V}{\\rm ar}} (\\mathbf{\\widehat{\\beta}_i})}\n",
    "\\]</span></span>\n",
    "</p>\n",
    "<p>\n",
    "The standard errors describe the accuracy of an estimator (the smaller the better). The standard errors are measures of the <strong>sampling variability</strong> of the least squares estimates <span class=\"math inline\"><span class=\"math inline\">\\(\\widehat{\\beta}_1\\)</span></span> and <span class=\"math inline\"><span class=\"math inline\">\\(\\widehat{\\beta}_2\\)</span></span> in <strong>repeated samples</strong> - if we collect a number of different data samples, the OLS estimates will be different for each sample. As such, the OLS estimators are <strong>random variables</strong> and have their own distribution.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept    0.000365\n",
       "MKT          0.035705\n",
       "SMB          0.063897\n",
       "HML          0.057064\n",
       "RMW          0.101819\n",
       "CMA          0.149880\n",
       "dtype: float64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.bse\n",
    "\n",
    "# Note: the b in bse stands for the parameter vector β, and se - standard errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confidence Intervals\n",
    "\n",
    "In this section we will introduce the notion of interval estimation - a procedure for creating ranges of values, called confidence intervals, in which the unknown parameters are likely to be located. Confidence interval creation procedures rely heavily on (UR.4) assumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Intercept</th>\n",
       "      <td>-0.005060</td>\n",
       "      <td>-0.003629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MKT</th>\n",
       "      <td>1.104115</td>\n",
       "      <td>1.244078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SMB</th>\n",
       "      <td>-0.363729</td>\n",
       "      <td>-0.113258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HML</th>\n",
       "      <td>-0.444311</td>\n",
       "      <td>-0.220624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RMW</th>\n",
       "      <td>0.269044</td>\n",
       "      <td>0.668167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CMA</th>\n",
       "      <td>-0.890928</td>\n",
       "      <td>-0.303408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  0         1\n",
       "Intercept -0.005060 -0.003629\n",
       "MKT        1.104115  1.244078\n",
       "SMB       -0.363729 -0.113258\n",
       "HML       -0.444311 -0.220624\n",
       "RMW        0.269044  0.668167\n",
       "CMA       -0.890928 -0.303408"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.conf_int()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hypothesis Testing\n",
    "\n",
    "in practice, we usually want to answer very specific questions about the effects of specific variables:\n",
    "\n",
    "Does income affect expenditure?\n",
    "Do more years in education lead to an increase in wage?\n",
    "Hypothesis tests use the information about a parameter from the sample data to answer such yes/no questions (though not necessarily in such strong certainty)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Null Hypothesis\n",
    "\n",
    "<p>The <strong>null hypothesis</strong> is denoted by <span class=\"math inline\">\\(H_0\\)</span>, and for the univariate regression can be stated as:\n",
    "<span class=\"math display\">\\[\n",
    "H_0: \\beta_i = c\n",
    "\\]</span>\n",
    "where <span class=\"math inline\">\\(c\\)</span> is a constant value, which we are interested in. When testing the null hypothesis, we may either <strong>reject</strong> or <strong>fail to reject</strong> the null hypothesis.</p>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "<strong>The null hypothesis, is presumed to be true, until the data provides sufficient evidence that it is not.</strong>\n",
    "</p>\n",
    "</div>\n",
    "<p>If we fail to reject the null hypothesis, it does not mean the null hypothesis is true. A hypothesis test does not determine which hypothesis is <strong>true</strong>, or which is most likely: it only assesses whether available evidence exists to <strong>reject</strong> the null hypothesis.</p>\n",
    "</div>\n",
    "\n",
    "\n",
    "###  The Alternative Hypothesis\n",
    "\n",
    "<p>Once, we state our null hypothesis, we must test it <em>against</em> and <strong>alternative hypothesis</strong>, denoted <span class=\"math inline\">\\(H_1\\)</span>.</p>\n",
    "<p>For the null hypothesis <span class=\"math inline\">\\(H_0: \\beta_i = c\\)</span> we may specify the alternative hypothesis in thee possible ways:</p>\n",
    "<ul>\n",
    "<li><span class=\"math inline\">\\(H_1: \\beta_i &gt; c\\)</span> - rejecting <span class=\"math inline\">\\(H_0\\)</span>, leads us to “accept” the conclusion that <span class=\"math inline\">\\(\\beta_i &gt; c\\)</span>. Economic theory frequently provides information about the <strong>signs</strong> of the variable parameters. For example: economic theory strongly suggests that food expenditure will rise if income increases, so we would test <span class=\"math inline\">\\(H_0: \\beta_{INCOME} = 0\\)</span> against <span class=\"math inline\">\\(H_1: \\beta_{INCOME} &gt; 0\\)</span>.</li>\n",
    "<li><span class=\"math inline\">\\(H_1: \\beta_i &lt; c\\)</span> - rejecting <span class=\"math inline\">\\(H_0\\)</span>, leads us to “accept” the conclusion that <span class=\"math inline\">\\(\\beta_i &lt; c\\)</span>.</li>\n",
    "<li><span class=\"math inline\">\\(H_1: \\beta_i \\neq c\\)</span> - rejecting <span class=\"math inline\">\\(H_0\\)</span>, leads us to “accept” the conclusion that <span class=\"math inline\">\\(\\beta_i\\)</span> is either greater <em>or</em> smaller than <span class=\"math inline\">\\(c\\)</span>.</li>\n",
    "</ul>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "<strong>We usually talk about hypothesis testing in terms of the null</strong>, i.e. we either <strong>reject</strong> or <strong>fail to reject</strong> the null - we <strong>never</strong> <em>accept</em> the null. As such, if we reject the null, then we “accept” (i.e. we are left with) the alternative.\n",
    "</p>\n",
    "</div>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "### The Test Statistic\n",
    "\n",
    "\n",
    "<p>The <strong>test statistic</strong> is calculated under the null hypothesis (i.e. assuming the null hypothesis is <em>true</em>). Under the null hypothesis the distribution of the statistic is <em>known</em>. Based on the value of the test statistic, we decide whether to reject, or fail to reject the null.</p>\n",
    "<p>Under the null hypothesis <span class=\"math inline\">\\(H_0: \\beta_i = c\\)</span> of our univariate regression model, we can calculate the following <strong><span class=\"math inline\">\\(t\\)</span>-statistic</strong>:\n",
    "<span class=\"math display\">\\[\n",
    "t_i = \\dfrac{\\widehat{\\beta}_i - c}{\\text{se}(\\widehat{\\beta}_i)} \\sim t_{(N-2)}\n",
    "\\]</span>\n",
    "If the null hypothesis is <em>not true</em>, then the <span class=\"math inline\">\\(t\\)</span>-statistic does not have a <span class=\"math inline\">\\(t\\)</span>-distribution with <span class=\"math inline\">\\(N-2\\)</span> degrees of freedom, but some other distribution.</p>\n",
    "</div>\n",
    "\n",
    "\n",
    "### The Rejection Regions\n",
    "<p>The <strong>rejection region</strong> consists of values that have low probability of occurring when the null hypothesis is true. The rejection region depends on the specification of the alternative hypothesis. If the calculated <em>test statistic</em> value falls in the rejection region (i.e. an unlikely event to occur under the null), then it is unlikely that the null hypothesis is holds.</p>\n",
    "<p>The size of the rejection regions are determined by choosing a <strong>level of significance <span class=\"math inline\">\\(\\alpha\\)</span></strong> - a probability of the unlikely event, usually <span class=\"math inline\">\\(0.01\\)</span>, <span class=\"math inline\">\\(0.05\\)</span>, <span class=\"math inline\">\\(0.1\\)</span>.</p>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "To determine, whether to reject the null hypothesis or not, we will compare the calculated <span class=\"math inline\"><span class=\"math inline\">\\(t\\)</span></span>-statistic <span class=\"math inline\"><span class=\"math inline\">\\(t_i\\)</span></span> to the critical value <span class=\"math inline\"><span class=\"math inline\">\\(t_c\\)</span></span>.\n",
    "</p>\n",
    "</div>\n",
    "\n",
    "\n",
    "### Type I and Type II Errors\n",
    "<p>When deciding whether to reject the null hypothesis or not, we may commit one of two types of errors:</p>\n",
    "<ul>\n",
    "<li><strong>Type I error</strong> - <strong>to reject the null hypothesis when it is true</strong>. The probability of committing Type I error is <span class=\"math inline\">\\(\\mathbb{P}(H_0 \\text{ rejected} | H_0 \\text{ is true}) = \\alpha\\)</span>. Any time we reject the null hypothesis, it is possible that we have made such an error. We can specify the amount of Type I error, that we can tolerate, by setting the level of significance <span class=\"math inline\">\\(\\alpha\\)</span>. If we want to avoid making a <strong>Type I</strong> error, then we set <span class=\"math inline\">\\(\\alpha\\)</span> to a very small value.</li>\n",
    "<li><strong>Type II error</strong> - <strong>to not reject the null hypothesis when it is false</strong>. We cannot directly calculate the probability of this type of error, since it depends on the unknown parameter <span class=\"math inline\">\\(\\beta_i\\)</span>. However, we do know that by making <span class=\"math inline\">\\(\\alpha\\)</span> smaller we increase the probability of <strong>Type II</strong> error.</li>\n",
    "</ul>\n",
    "<p>It is believed that a <strong>Type I</strong> error is more severe, hence, it is recommended to make the probability <span class=\"math inline\">\\(\\alpha\\)</span> small.</p>\n",
    "</div>\n",
    "</div>\n",
    "\n",
    "\n",
    "### The <span class=\"math inline\">\\(p\\)</span>-value\n",
    "\n",
    "<p>When reporting the outcome of statistical hypothesis tests, we usually report the <span class=\"math inline\">\\(p\\)</span>-value of the test. The <span class=\"math inline\">\\(p\\)</span>-value is defined as the probability, under the null hypothesis, of obtaining a result, which is equal to, or more extreme, than what was actually observed.</p>\n",
    "<p>Having the <span class=\"math inline\">\\(p\\)</span>-value allows us to easier determine the outcome of the test, as we do not need to directly compare the critical values.</p>\n",
    "<div class=\"THEOREM\">\n",
    "<ul>\n",
    "<li>\n",
    "If <span class=\"math inline\"><span class=\"math inline\">\\(p \\leq \\alpha\\)</span></span>, we <strong>reject</strong> <span class=\"math inline\"><span class=\"math inline\">\\(H_0\\)</span></span>.\n",
    "</li>\n",
    "<li>\n",
    "If <span class=\"math inline\"><span class=\"math inline\">\\(p \\geq \\alpha\\)</span></span>, we <strong>do not reject</strong> <span class=\"math inline\"><span class=\"math inline\">\\(H_0\\)</span></span>.\n",
    "</li>\n",
    "</ul>\n",
    "</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept     1.147488e-32\n",
       "MKT          3.858437e-237\n",
       "SMB           1.896132e-04\n",
       "HML           5.669656e-09\n",
       "RMW           4.177710e-06\n",
       "CMA           6.767988e-05\n",
       "dtype: float64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.pvalues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goodness-Of-Fit\n",
    "\n",
    "<p>In order to have an accurate prediction of <span class=\"math inline\">\\(Y\\)</span>, we hope that the independent variable <span class=\"math inline\">\\(X\\)</span> helps us <em>explain</em> as much variation in <span class=\"math inline\">\\(Y\\)</span> as possible (hence why <span class=\"math inline\">\\(X\\)</span> is usually referred to as an <em>explanatory</em> variable). Ideally, the variance of <span class=\"math inline\">\\(X\\)</span> will help <em>explain</em> the variance in <span class=\"math inline\">\\(Y\\)</span>. Having said that, we would like to have a way to <strong>measure</strong> just how <em>good</em> our model is - how much of the variation in <span class=\"math inline\">\\(Y\\)</span> can be explained by the variation in <span class=\"math inline\">\\(X\\)</span> using our model - we need a <strong>goodness-of-fit</strong> measure.</p>\n",
    "<p>Another way to look at it is - a <strong>goodness-of-fit</strong> measure aims to quantify how well the estimated model <em>fits</em> the data. Fortunately, there are many ways to measure the goodness-of-fit of the estimated model.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### R-squared, <span class=\"math inline\">\\(R^2\\)</span>\n",
    "\n",
    "\n",
    "<p>It is often useful to compute a number that summarizes how well the OLS regression fits the data. This measure is called the <strong>coefficient of determination</strong>, <span class=\"math inline\">\\(R^2\\)</span>, which is the ratio of explained variation, compared to the total variation, i.e. the proportion of variation in <span class=\"math inline\">\\(Y\\)</span> that is explained by <span class=\"math inline\">\\(X\\)</span> in our regression model:\n",
    "<span class=\"math display\">\\[\n",
    "R^2 = \\dfrac{\\text{ESS}}{\\text{TSS}} = 1 - \\dfrac{\\text{RSS}}{\\text{TSS}}\n",
    "\\]</span></p>\n",
    "<ul>\n",
    "<li>The closer <span class=\"math inline\">\\(R^2\\)</span> is to <span class=\"math inline\">\\(1\\)</span>, the closer the sample values of <span class=\"math inline\">\\(Y_i\\)</span> are to the fitted values <span class=\"math inline\">\\(\\widehat{Y}\\)</span> of our regression. Ir <span class=\"math inline\">\\(R^2 = 1\\)</span>, then all the sample data fall exactly on the fitted regression. In such a case our model would be a <em>perfect fit</em> for our data.</li>\n",
    "<li>If the sample data of <span class=\"math inline\">\\(Y\\)</span> and <span class=\"math inline\">\\(X\\)</span> do not have a linear relationship, then <span class=\"math inline\">\\(R^2 = 0\\)</span> of a univariate regression.</li>\n",
    "<li>Values <span class=\"math inline\">\\(0 &lt; R^2 &lt; 1\\)</span>, the interpretation of <span class=\"math inline\">\\(R^2\\)</span> is as <em>the proportion of the variation in <span class=\"math inline\">\\(Y\\)</span> around its mean, that is explained by the regression model</em>. For example <span class=\"math inline\">\\(R^2 = 0.17\\)</span> means that <span class=\"math inline\">\\(17\\%\\)</span> of the variation in <span class=\"math inline\">\\(Y\\)</span> is explained by <span class=\"math inline\">\\(X\\)</span>.</li>\n",
    "</ul>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "When comparing <span class=\"math inline\"><span class=\"math inline\">\\(\\text{RSS}\\)</span></span> of different models, we want to choose the model, which better fits our data. If we want to choose a model based on its <span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> value we should note a couple of things:\n",
    "</p>\n",
    "<ul>\n",
    "<li>\n",
    "<p>\n",
    "<span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> comparison <strong>is not valid</strong> for comparing models, that <strong>do not have have the same transformation of the dependent variable</strong>, for example two models - one with <span class=\"math inline\"><span class=\"math inline\">\\(Y\\)</span></span> and the other with <span class=\"math inline\"><span class=\"math inline\">\\(\\log(Y)\\)</span></span> dependent variables cannot be compared via <span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span>.\n",
    "</p>\n",
    "</li>\n",
    "<li>\n",
    "<p>\n",
    "<span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> does not measure the predictability power of the model. For example, a linear model may be a good fit for the data, but its forecasts may not make economic sense (e.g. forecasting negative wage for low values of years in education via a simple linear model).\n",
    "</p>\n",
    "</li>\n",
    "<li>\n",
    "<p>\n",
    "<span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> is based on the sample data, so it says nothing whether our model is close to the true population DGP.\n",
    "</p>\n",
    "</li>\n",
    "<li>\n",
    "<p>\n",
    "<span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> may be low if: the error variance, <span class=\"math inline\"><span class=\"math inline\">\\(\\sigma^2\\)</span></span>, is large; or if the variance of <span class=\"math inline\"><span class=\"math inline\">\\(X\\)</span></span> is small.\n",
    "</p>\n",
    "</li>\n",
    "<li>\n",
    "<p>\n",
    "<span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> may be large even if the model is wrong. For example, even if the true relationship is non-linear, a linear model may have a larger <span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span>, compared to the quadratic, or even the log-linear model.\n",
    "</p>\n",
    "</li>\n",
    "<li>\n",
    "<p>\n",
    "On the other hand, <strong>the goodness-of-fit</strong> of the model does not depend on the unit of measurement of our variables (e.g. dollars vs thousands of dollars). Furthermore, comparisons of <span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> are valid, if we compare a simple linear model to a linear-log model, as they both have the same dependent variable, <span class=\"math inline\"><span class=\"math inline\">\\(Y\\)</span></span>.\n",
    "</p>\n",
    "</li>\n",
    "</ul>\n",
    "<p>\n",
    "In any case, a model should not be chosen <strong>only</strong> on the basis of model fit with <span class=\"math inline\"><span class=\"math inline\">\\(R^2\\)</span></span> as the criterion.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6371536242060707"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.rsquared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6355438354935865"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FF5.rsquared_adj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression Diagnostics and Graphics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>In many cases while carrying out statistical/econometric analysis, we are not sure, whether we have correctly specified our model. As we have seen, the <span class=\"math inline\">\\(R^2\\)</span> can be artificially small (or large), regardless of the specified model. As such, there are a number of regression diagnostics and specification tests.</p>\n",
    "<p>For the univariate regression, the most crucial assumptions come from <strong>(UR.3)</strong> and <strong>(UR.4)</strong>, namely:</p>\n",
    "<ul>\n",
    "<li><span class=\"math inline\">\\(\\mathbb{V}{\\rm ar} (\\epsilon_i | \\mathbf{X} ) = \\sigma^2_\\epsilon,\\ \\forall i = 1,..,N\\)</span></li>\n",
    "<li><span class=\"math inline\">\\(\\mathbb{C}{\\rm ov} (\\epsilon_i, \\epsilon_j) = 0,\\ i \\neq j\\)</span></li>\n",
    "<li><span class=\"math inline\">\\(\\boldsymbol{\\varepsilon} | \\mathbf{X} \\sim \\mathcal{N} \\left( \\mathbf{0}, \\sigma^2_\\epsilon \\mathbf{I} \\right)\\)</span></li>\n",
    "</ul>\n",
    "<p>We note that the residuals are defined as:\n",
    "    \n",
    "    \n",
    "$\\begin{aligned}\n",
    "\\widehat{\\boldsymbol{\\varepsilon}} &= \\mathbf{Y} - \\widehat{\\mathbf{Y} } \\\\\n",
    "&= \\mathbf{Y} - \\mathbf{X} \\widehat{\\boldsymbol{\\beta}} \\\\\n",
    "&= \\mathbf{Y} - \\mathbf{X} \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbf{Y} \\\\\n",
    "&= \\left[ \\mathbf{I} - \\mathbf{X} \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\right]\\mathbf{Y}\n",
    "\\end{aligned}$\n",
    "    \n",
    "   \n",
    "Hence, for the OLS <strong>residuals</strong> (i.e. not the true unobserved errors) the expected value of the residuals is still zero:\n",
    "\n",
    "\n",
    "$\\begin{aligned}\n",
    "\\mathbb{E} \\left( \\widehat{\\boldsymbol{\\varepsilon}}| \\mathbf{X} \\right) &=  \\mathbb{E} \\left( \\left[ \\mathbf{I} - \\mathbf{X} \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\right]\\mathbf{Y} | \\mathbf{X} \\right)\\\\\n",
    "&= \\mathbb{E} \\left( \\left[ \\mathbf{I} - \\mathbf{X} \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\right] \\left( \\mathbf{X} \\boldsymbol{\\beta} + \\boldsymbol{\\varepsilon} \\right) | \\mathbf{X} \\right) \\\\\n",
    "&= \\mathbf{X} \\boldsymbol{\\beta} + \\mathbb{E} (\\boldsymbol{\\varepsilon}) - \\mathbf{X} \\boldsymbol{\\beta} - \\mathbf{X} \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top \\mathbb{E} (\\boldsymbol{\\varepsilon}) \\\\\n",
    "&= 0\n",
    "\\end{aligned}$\n",
    "\n",
    "\n",
    "For simplicity, let $\\widehat{\\boldsymbol{\\varepsilon}} = \\left[ \\mathbf{I} - \\mathbf{H}\\right]\\mathbf{Y}$, where $\\mathbf{H}\\ = \\mathbf{X} \\left( \\mathbf{X}^\\top \\mathbf{X}\\right)^{-1} \\mathbf{X}^\\top$.</p>\n",
    "\n",
    "\n",
    "<p>Consequently, the variance-covariance matrix of the residuals is:</p>\n",
    "\n",
    "$\\begin{equation}\n",
    "\\begin{aligned}\n",
    "\\mathbb{V}{\\rm ar} \\left( \\widehat{\\boldsymbol{\\varepsilon}}| \\mathbf{X}\\right) &= \\mathbb{V}{\\rm ar} \\left( \\left[ \\mathbf{I} - \\mathbf{H}\\right]\\mathbf{Y}|\\mathbf{X}\\right) \\\\\n",
    "&= \\left[ \\mathbf{I} - \\mathbf{H}\\right]\\mathbb{V}{\\rm ar} \\left( \\mathbf{Y} | \\mathbf{X}\\right) \\left[ \\mathbf{I} - \\mathbf{H}\\right]^\\top \\\\\n",
    "&= \\left[ \\mathbf{I} - \\mathbf{H}\\right] \\sigma^2 \\left[ \\mathbf{I} - \\mathbf{H}\\right]^\\top \\\\\n",
    "&= \\sigma^2 \\left[ \\mathbf{I} - \\mathbf{H}^\\top - \\mathbf{H} + \\mathbf{H} \\mathbf{H}^\\top\\right] \\\\\n",
    "&= \\sigma^2 \\left[ \\mathbf{I} - \\mathbf{H}^\\top - \\mathbf{H} + \\mathbf{H}^\\top\\right] \\\\\n",
    "&= \\sigma^2 \\left[ \\mathbf{I} - \\mathbf{H}\\right]\n",
    "\\end{aligned}\n",
    "\\end{equation}$\n",
    "\n",
    "\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "This result shows an important distinction of the residuals from the errors - the residuals may have different variances (which are the diagonal elements of <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbb{V}{\\rm ar} \\left( \\widehat{\\boldsymbol{\\varepsilon}}| \\mathbf{X}\\right)\\)</span></span>), even if the true errors (which affect the process <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{Y}\\)</span></span>) all have the same variance <span class=\"math inline\"><span class=\"math inline\">\\(\\sigma^2\\)</span></span>.\n",
    "</p>\n",
    "</div>\n",
    "<p>The variance for the fitted values is smallest for observations near the mean and the largest for values, which deviate the most from the process mean.</p>\n",
    "\n",
    "\n",
    "### Residual Diagnostic Plots\n",
    "\n",
    "<p>One way to examine the adequacy of the model is to visualize the residuals. There are a number of ways to do this:</p>\n",
    "<ul>\n",
    "<li>Plotting the residuals <span class=\"math inline\">\\(\\widehat{\\epsilon}_i\\)</span> against the fitted values <span class=\"math inline\">\\(\\widehat{Y}_i\\)</span>;</li>\n",
    "<li>Plotting the residuals <span class=\"math inline\">\\(\\widehat{\\epsilon}_i\\)</span> against <span class=\"math inline\">\\(X_i\\)</span></li>\n",
    "<li>Plotting the residual Q-Q plot, histogram or boxplot.</li>\n",
    "</ul>\n",
    "<p>In all cases, if there are no violations of our <strong>(UR.2)</strong> or <strong>(UR.3)</strong> assumptions - the plots should reveal <strong>no patterns</strong>. The residual histogram, Q-Q plot should be approximately normal so that our assumption <strong>(UR.4)</strong> holds.</p>\n",
    "<p>As we are not guaranteed to specify a correct functional form, residual plots offer a great insight on what possible functional form we may have missed.</p>\n",
    "<p>We should note that when having multiple models, it is only meaningful to compare the residuals of models with the same dependent variable. For example, comparing the residuals of a linear-linear model (with <span class=\"math inline\">\\(Y\\)</span>) and of a log-linear model (with <span class=\"math inline\">\\(\\log(Y)\\)</span>) is <strong>not</strong> meaningful as they have different value scales.</p>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "Transforming the dependent or the independent variables may help to alleviate some of the problems of the residuals:\n",
    "</p>\n",
    "<ul>\n",
    "<li>\n",
    "If nonlinearities are present in the residual plots - we must firstly account for them, and only after can we check, whether the errors have a constant variance.\n",
    "</li>\n",
    "<li>\n",
    "Transforming <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{Y}\\)</span></span> primarily aims to help with problems with the error terms (and may help with non-linearity);\n",
    "</li>\n",
    "<li>\n",
    "Transforming <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{X}\\)</span></span> primarily aims to help with correcting for non-linearity;\n",
    "</li>\n",
    "<li>\n",
    "Sometimes transforming <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{X}\\)</span></span> is enough to account for non-linearity and have normally distributed errors, while transforming <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{Y}\\)</span></span> may account for non-linearity but might make the errors non-normally distributed.\n",
    "</li>\n",
    "<li>\n",
    "Other times, transforming <span class=\"math inline\"><span class=\"math inline\">\\(\\mathbf{X}\\)</span></span> does not help account for the nonlinear relationship at all;\n",
    "</li>\n",
    "</ul>\n",
    "</div>\n",
    "<p>Remember that the Q-Q plot plots quantiles of the data versus quantiles of a distribution. If the observations come from a normal distribution we would expect the observed order statistics plotted against the expected (theoretical) order statistics to form an approximately straight line.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xmc1vP6x/HXVaSy5ZQ1KgcHhcKIdDqWOg7Zf7bo2IqWOSHLsZwskTWylkglNNaSkkSWZE2L9jiSItkSFXW0Xb8/Pvdt7qb7nrmnudeZ9/PxmMd939/16jvTXPPZzd0RERFJVrVsByAiIvlFiUNERMpFiUNERMpFiUNERMpFiUNERMpFiUNERMpFiUNERMpFiUNERMpFiUNERMpls2wHkA716tXzRo0aZTsMEZHc98svsHAhU9auXeLu2ydzSqVMHI0aNWLy5MnZDkNEJHd99x106wbDh8OBB2KffLIw2VNVVSUiUpW4w5Ah0LgxjB4Nd9wBEyeW6xKVssQhIiJxLFgAnTvD66/DX/8KAwfC3nuX+zIqcYiIVHbr18NDD8F++8EHH0C/fvDOO5uUNEAlDhGRyu3TT+Gii+D99+HYY+GRR6BhwwpdUiUOEZHKaM0auP12aNoU5s6FJ5+EMWMqnDRAiUNEJOOKiqBRI6hWLbwWFaX4BlOnQvPm0KMHnHwyzJkD554LZim5vBKHiEgGFRVBp06wcGHo4LRwYfickuSxahVcd11IGt99By++CM8/DzvumIKLF1PiEBHJoB49YOXKDbetXBm2V8h770GzZnDnnXD++aGUceqpFbxofEocIiIZ9NVX5dtephUrwkC+Vq1g9WoYNw4GDYLtttvkGMuixCEikkENGpRve6lefRWaNIGHH4bu3WHWLGjTpkLxJSNricPM9jazaTFfy82se4ljjjSzZTHH3JiteEVEUuG226B27Q231a4dtiftp5/gvPOgbVvYaqvQ1fa++2DLLVMaayJZG8fh7p8BzQDMrDrwDTAizqHvuvsJmYxNRCRd2rcPrz16hOqpBg1C0ohuL5U7DBsWqqaWLoUbbggX2mKLtMZcUq4MAGwNfOHuSU+yJSKSr9q3TzJRxPr2WygshJdegoMPDtOGNG2alvjKkittHO2AZxLsa2Fm083sVTNrkugCZtbJzCab2eQff/wxPVGKiGSaOwweDPvuC2PHQu/e8NFHWUsaAObuWbs5gJnVABYDTdz9+xL7tgHWu/uvZtYWeMDd9yrrmgUFBa5p1UUk7335ZRjk8cYb8Le/hUkJ9yrzV+AmMbMp7l6QzLG5UOI4DphaMmkAuPtyd/818n4MsLmZ1ct0gCJStaV9pHdJ69bBAw+ESQknToT+/eHtt9OWNMorF9o4ziZBNZWZ7QR87+5uZs0Jie6nTAYnIlVbdKR3dNBedKQ3bEI7RTLmzIGOHUN1VNu2YVLC3XZLw402XVZLHGZWG/g78GLMti5m1iXy8XRglplNBx4E2nm269ZEpEpJ20jvklavhl694MAD4fPPYejQsNBSjiUNyIE2jnRQG4eIpEq1aqF9uiSzsMxFSkyeHEoZM2ZAu3ahmmqHHVJ08eTkWxuHiEjOSulI75JWrYKrr4ZDD4UlS2DkSHjmmYwnjfJS4hARKUVKRnrH8847cMABcPfdobQxezacdFIFL5oZShwiIqVo3x4GDAjrH5mF1wEDKtAwvnw5dO0KRx4Z6rrefDNcsE6dVIadVrnQq0pEJKdt0kjveF55Bbp0gcWL4YorQmN4yeJMHlCJQ0Qk3ZYsgX/+E044AbbdFj74APr0ycukAUocIiLp4w7PPhumC3n+ebjpprCs66GHZjuyClHiEJEqKe2jwb/5Bk45Bc4+G3bfHaZMgZ49oUaNFN8o85Q4RKTSSpQc0rrutzs89hg0bhxW4+vTBz78EPbfPwUXzw1KHCJSKZRMEoWFiZND2kaDf/EFtG4dbnTwwTBzZmgEr169ghfOLRo5LiJ5r+R8UhC6zsb79dawYVhAKaWjwaOTEl5/PWy+OdxzD1x0UbhgntDIcRGpUuKVIBL9TRxddS+eTRoNPmsWHH44XHllWO97zhy4+OK8ShrlpcQhInnvq6+SPza6VGuFR4OvXg033wwHHQTz54epQkaOhPr1y3GR/KTEISJ5L1FJoeQf/dHkUOHR4B9/HNowevaEM8+EuXPD5ISVuJQRS4lDRPJeohJEly6Jk0P79rBgQWjTWLAgyaSxcmWokmrRAn75JUx7PnQo1Kta68tpyhERyXvRX/o9ehS3YURLFinz9tuhwXv+/JCR7roLttkmhTfIH0ocIlIppGw+qZKWLYN//zuMzdhzTxg/Ho44Ig03yh+qqhIRSeTll8NAvkGDQvKYPr3KJw1Q4hAR2diPP4apQk46CerWhYkToXfvvJ2UMNWUOEREotzh6afDpITDh8Mtt4RlXQuSGhdXZShxiEheSvkkhV9/DSeeGBpK9toLpk2DG26oFJMSppoax0Uk75ScYiQ6DxVsQgP5+vWhn+7VV4epQ+6/H7p1q3TzS6VS1kscZrbAzGaa2TQz22iCKQseNLN5ZjbDzA7KRpwikjtSNknh55/D0UeHpVybNw+TEl52mZJGGXKlxHGUuy9JsO84YK/I16FA/8iriFRRiaYYSXrqkbVr4b774MYbYYstQq+pCy+sMiO/KyrrJY4knAw86cFHQB0z2znbQYlI9lRoksIZM8LI76uvhn/8I0xK2KGDkkY55ELicOB1M5tiZp3i7K8PfB3zeVFkm4hUUZs0SeHvv4cSxsEHh6LJ88/DiBGwyy5pjbUyyoWqqpbuvtjMdgDGmdmn7j4hZn+8PwM2mjA5knQ6ATTYpLmRRSRflHuKkQ8/hI4dw2SE550H994bxmfIJsl6icPdF0defwBGAM1LHLII2C3m867A4jjXGeDuBe5esP3226crXBHJEUlNUvjbb9C9O7RsCb/+CmPGwBNPKGlUUFYTh5ltaWZbR98DxwCzShw2Cjgv0rvqMGCZu3+b4VBFJN+88Qbst19Yma+wEGbPhuOOy3ZUlUK2Sxw7Au+Z2XTgY+AVdx9rZl3MrEvkmDHAfGAe8BhQmJ1QRSTbkhr098svoVrq738Py7hOmAB9+8LWW2c42sorq20c7j4faBpn+yMx7x34VybjEpHck9Sgv5deCqWLH36Aa68NjeG1amUl3sos2yUOEZGklDro7/vvw0p8p54KO+4YVui74w4ljTRR4hCRnBNbJVWvXvhauDDekc7fFj4Vpj4fOTJ0rfr447AOuKRNLnTHFREBQsK47DL46afibbHvY+3GVzxKZ45jLOxzeBj9vc8+mQm0ilOJQ0SyqqgolCjM4J//TJwoooz1FNKP2TShFe8y6bwH4d13lTQySCUOEcmKeKWLsvyFzxjIRbTiPSbU/DtL7xjAKd0bpS1GiU+JQ0QyrmQPqbJUZy1XcQ896ckqanFV3ce558fzNb9UlqiqSkQyLl4PqUSaMo2JHMqdXMdoTuDgWnM58IELlDSySIlDRDIumenPt+B/3EoPJlPArvYNpzOMqxoOo9djO5V/sSZJKVVViUjGNWiQqHttcDjvM6RaR/Za/xlccAE79unDsD/9KXMBSqlU4hCRjGvbNn5N05b8yqM1L+U9a8Veu/0PXnsNHn8clDRyihKHiGRMtOtt//7gMYsjmMGDJ7zOrw33o9PvfbFu3WDWLDjmmOwFKwmpqkpEMiJRT6rtWEofv5ILRw+BvfcOYzJatsxKjJIclThEJCPi9aT6P4Yzh8acy1PcRg+YNk1JIw8ocYhI2sSOCo9tDN+R73iB0xnO6SxmFwqYzGMNb4WaNbMXrCRNiUNEUq6oCLbaKt4UIs75DGEu+3ICo7mGO2nOx3xeu1np64VLTikzcZjZHma2ReT9kWZ2qZnVSX9oIpKPCgtDwvjttw23N2QBr/EPhnAhs9iPpkynN9dQp+5mDBhQynrhknOSKXEMB9aZ2Z7AIGB34Om0RiUieamwMPSYimWspxsPMYv9aMGHFNKPI3iH/7I3AEuWKGnkm2R6Va1397Vmdipwv7s/ZGafpDswEckvbdrAm29uuG0f5jKQi2jJB7zKsXThEb6i4R/7GzZE8lAyJY41ZnY2cD4wOrJt8/SFJCL5pmTS2Iw1XMftTKMZ+/Ap5/IkbRmzQdKoUQO1a+SpZBLHhUAL4DZ3/9LMdgeGpjcsEckXJZPGgUxlEodwOz14iVNozByGci5QPFS8bl0YPFhVVPmqzMTh7nOAa4Cpkc9fuvud6Q5MRHJfYWFx0qjJKu7gWj6mOTvyPacwgnY8xw/sCEDXrmG0uLvaNfJdMr2qTgSmAWMjn5uZ2aiK3tjMdjOzt81srpnNNrPL4hxzpJktM7Npka8bK3pfEUmdRx8Nr3/lXabRjGu5iyFcQGPmMJJTgNAtd+hQePjhLAYqKZVM43hPoDkwHsDdp0WqqypqLXClu081s62BKWY2LlLCifWuu5+QgvuJSAq1aQO116/gTq7lXzzMlzSiDeN4kzZ/HNO6NbzxRhaDlLRIpo1jrbsvK7HN4x5ZDu7+rbtHq79WAHOB+hW9roikV2FhGAm++ZuvMpsmdKU/99Gd/ZilpFFFJJM4ZpnZOUB1M9vLzB4CPkhlEGbWCDgQmBhndwszm25mr5pZk1TeV0SSF00Yz/X/iSc4j1dpywq2piXvcwX3sZIt/zhWSaNySyZxXAI0AX4HngGWA91TFYCZbUUYZNjd3ZeX2D0VaOjuTYGHgJdKuU4nM5tsZpN//PHHVIUnIkCTJtC/v3MGzzOXfTmbZ7iFGziIqXxEi42OV9Ko3My9wrVOm35zs80JY0Nec/d7kzh+AVDg7ktKO66goMAnT56cmiBFqqiiIujQAVavhp1ZTD/+xam8xGQOpgODmckBcc+rWzf0mpL8YmZT3L0gmWMTNo6b2cuU0pbh7idtQmyx1zfCFCZzEyUNM9sJ+N7d3cyaE0pIP8U7VkRSp0kTmDMHwOnAYPpwJVvwO/+mN/dxOesS/OowgwceyGiokgWl9aq6J833bgmcC8w0s2mRbf8BGgC4+yPA6UBXM1sLrALaeTaLSCKVWFERnHtu8cp8uzOfAXSiDW/yDn/jIgYyj70Snr/ZZjBkiMZnVAUJE4e7v5POG7v7e8QOJY1/TF+gbzrjEJENR39XYx2X8BC30YN1VKcL/RlAJ7yUJtGuXTVOoyoprarqeXc/08xmEqfKyt3jV3CKSF7Zbjv45Zfwfl/mMIiOtOAjRnM8XenPInZLeK56T1VNpVVVRUdya/CdSCVU3I4Bm7Oaa7iLG+jFcrbhHIp4hrMprVJApYyqK2HZ092/jbwtdPeFsV9AYWbCE5FUq18/NGJHk0YBk5hMAb24keGcxr7M5RnOIVHSqFlTU4hUdcmM4/h7nG3HpToQEUmvJk1Cwli8OHyuxUru4mo+4jDq8hMnMZJzeIYlbB/3/NatQ8P5qlVqAK/qSmvj6EooWfzZzGbE7NoaeD/dgYlI6lSvDuvXF3/+G+8wkIvYi3kM4GL+zd0sZ9u459asCQMHKllIsdLaOJ4GXgXuAK6N2b7C3ZemNSoRSYn69YtLGABbs5y7uIauPMIX/JmjeZO3OTruuXXqwM8/ZyhQySultXEsc/cF7n42sAhYQ+hdtZWZNchUgCJSftF2jNik0ZZXmE0TOjGAe7iS/ZmZMGk0bqykIYmVOa26mXUjTK3+PRAt7DokmG9ARLLKSrRp1+NH7qc77XmamezHaQxnEs0TnvvUU6qWktIl0zjeHdjb3Zu4+/6RLyUNkRwTnb22mHMWzzKHxpzBC9xETw5mSsKk0bVraAdR0pCyJLOQ09dAyfU4RCSHlFz3exe+oT9dOYmXmUhzOjKI2ewX91yNx5DySiZxzAfGm9krhKnVAUhmNlsRSb+SpYyLGMg9XMXmrOEK+vAAl7Ge6hudV6sWrFyZsTClEkmmquorYBxQg9AVN/olIlkUbQCP+jNf8CateYxOTOFg9mcm93FF3KTRurWShmy6Mksc7n5zJgIRkeTFJoxqrKM799OLG1jD5lzMAAZyEfFGfu+yC3zzTebilMopmV5V2wNXE1YBrBnd7u7x+/GJSNqUHMjXhFkMpgPNmcQoTqQr/VlM/Y3OU7WUpFIyVVVFwKfA7sDNwAJgUhpjEpESttsulDKiSWNzVnMTPZnKQTRiAWfxLCczMm7SGDpUSUNSK5nG8bruPsjMLous0fGOmaV1rQ4RKVZyXMYhfMxgOrAfsxlKe7pzPz9RL+65WvZM0iGZEseayOu3Zna8mR0I7JrGmESEjcdl1GIl93AlH9KCbVnG8YzmXIYqaUjGJVPiuNXMtgWuBB4CtgEuT2tUIlVcyVLGUbzFY1zMHsynP124hrtYwTZxz9XiSpJuyfSqGh15uww4Kr3hiEhs0tiWX7ibf3MxA/mcPTmC8UzgiITnqpQhmZBMr6rHib90bIe0RCRSRZUsZZzIKPrTlZ34jru4mp705H/UinuuEoZkUjJVVaNj3tcETgUWJzhWRMqpdu2wOFLU9vzAg1xKO55jBvtzMiOZQkHcc6tVg3XrMhSoSESZjePuPjzmqwg4ExJMelNOZnasmX1mZvPM7No4+7cws+ci+yeaWaNU3FckFxQVhVJGcdJwzqGIOTTmVEZwPb0oYHLCpOGupCHZkUyJo6S9gAqvx2Fm1YF+hKVpFwGTzGyUu8+JOawj8LO772lm7YC7gLMqem+RbKtRA9asKf68K1/Tn66cwCt8yGF0ZBBzaRz3XC2wJNlWZonDzFaY2fLoK/AycE0K7t0cmOfu8919NfAscHKJY04Gnoi8Hwa0NitZEyySP6ID+aJJw1hPZx5hNk04ire5jPv5K+8lTBruShqSfcn0qkrXhIb1CVO2Ry0CDk10jLuvNbNlQF1gSZpiEkmbkn/y7MnnDOQijmAC42hDJwawgN3jnqtShuSSUhOHmdUC2sMff/5MBoZFSggVFa/kULJvSDLHhAPNOgGdABo00Mq2kjtKJozqrOUK7uVmbuJ3tqADg3icC4n/464eU5J7ElZVmdn+wFygFWF+qoXAP4D3zayOmd1awXsvAnaL+bwrG/fW+uMYM9sM2BZYGu9i7j7A3QvcvWD77bevYGgiFRdt/I51ANP5iMPozTWM5VgaM4fH6UC8pFGtmpKG5KbSShwPAhe7+7jYjWbWBpgFzK7gvScBe5nZ7sA3QDvgnBLHjALOBz4ETgfectd/Jcl9JWexrcHvXM+tXMudLOVPnMHzDON0VMqQfFRa4ti5ZNIAcPc3zGwNYTzHJou0WXQDXgOqA4PdfbaZ3QJMdvdRwCDgKTObRyhptKvIPUUyoWQp4zA+ZBAdacxcnuA8ruBellI37rlKGJIPSksc1cxsC3f/PXajmdUE1rh7hSdqdvcxwJgS226Mef8/4IyK3kckE7bbDn75pfjzlvzKrVzPpTzIInblOMYwluMSnq+kIfmitO64TwLDYwfdRd4/DzyVzqBE8o3ZhkmjDeOYyf505wEeppAmzE6YNNyVNCS/JEwc7n4rMBaYYGZLzGwJ8A4wzt17ZSpAkVwWHZcRVYefGUhHxnEMq6lBKyZwCX35lY17tXftqoQh+anU7rju3hfoa2ZbRz6vyEhUInmgZFvGKYzgYQrZnh+5g2sj3W1rxj1XCUPyWVJTjihhiGwoNmnswPc8xCWcyQt8QjOO5xU+4aC45ylhSGWQzAqAIhKjOGk45/Ikc9mXkxnJf7iN5nyspCGVnhKHSJJi2zMasJAxtOVJzmcu+9KU6dzBf1jL5hudV6eOkoZULslMcljbzG4ws8cin/cysxPSH5pIbqhdu7jXlLGeQvoxi/1oxbt04yFa8S6fsU/cczUpoVRGyZQ4Hgd+B1pEPi8CKjrdiEjOq19/w/Uy/sJnvMMR9KMbH3A4+zGLfnTD4/w32mUXlTKk8komcezh7r2BNQDuvopE8ySIVBJmsDgyc9pmrOEa7mQ6TWnCbM5nCMcyloU0inuuO3zzTeZiFcm0ZHpVrY7MkusAZrYHoQQiUinF9phqxicMoiMH8QnDOI1u9OV7dkp4rkoZUhUkU+K4iTAQcDczKwLeBK5Oa1QiWRJNGlvwP26lB5M4hF1YzGkM4wyGJUwamslWqpJk1hwfB/wfcAHwDFDg7uPTG5ZIZhUWFieNw3mfaTSjB7fzJOfRmDm8yGkJz23dWmt/S9WSsKrKzEp2Rv828trAzBq4+9T0hSWSObVrhwbwrVjB7fyHf9GPr2jAMbzGOI5JeJ5W5ZOqqrQ2jj6l7HPg6BTHIpIxRUXwz38Wfz6G1xhAJ3bjax7iEnpwG7+xVdxzGzeG2RVdjUYkjyVMHO5+VCYDEcmEkgljO5ZyL1dwAU8wl31oxbt8QMu459aqBSsrvJiASP4rs1dVZP2NQuCvhJLGu8AjkbUyRPJCyYQB8H8Mpx//oh5LuJUe3Mr1CSclVLWUSLFkuuM+CawAHop8PpuwHocWWJKcV1QE55+/YeP1TnxLX7pxGi8ylQM5lrFMp1nCa9SqpaQhEiuZxLG3uzeN+fy2mU1PV0AiqVBYCP37l9zqnM8T3Mfl1GIV13AnfbiSdaX8N2jdGt54I62hiuSdZBLHJ2Z2mLt/BGBmhwLvpzcskU1Xv37xqO+ohixgAJ04hnFMoBUXMZDP+UvCa6hqSiSxZAYAHgp8YGYLzGwB8CFwhJnNNLMZaY1OpJyaNNkwaVRjHZfwILPYjxZ8SCH9OJLxCZOGGQwdqqQhUppkShzHpj0KkRQoLIQ5c4o/78NcBnIRLfmAVzmWzjzK1zSIe6662IokL5mR4wuB5cC2QN3ol7svjOwrNzO728w+NbMZZjbCzOokOG5BpGQzzcwmb8q9pGqIbdPYjDX8h9uYRjP24VPO5UnaMiZu0mjdOkwVoqQhkrxkuuP2Ikw38gWRiQ6p+ADAccB17r7WzO4CrgOuSXDsUe6+pAL3kkqsqAg6dIDVq8Png5jCIDrSjOk8x5lcyoP8wI4bnadGb5FNl0xV1ZmEqdVXp+qm7v56zMePgNNTdW2p/OL1mKrJKm7iZq7iHn5gB05hBCM5ZaNzlTBEKi6ZxvFZQNyqpBTpALyaYJ8Dr5vZFDPrlMYYJMcVFcFWW4XG65JJoxUTmE5TruUuhnABjZmzUdIYOjRUSSlpiFRcMiWOOwhdcmcRsw6Hu59U2klm9gbEnYO6h7uPjBzTA1gLFCW4TEt3X2xmOwDjzOxTd5+Q4H6dgE4ADRrEbwCV/FNUBJ07w2+/bbxva5ZzB9fxLx5mPrvTmjd4i9YbHTd0KLRvn4FgRaoI8zIWETCz2cCjwExgfXS7u79ToRubnQ90AVq7e5kzAJlZT+BXd7+nrGMLCgp88mS1pee7+IP4gmN5lUfpzK4s4gEu43puZSVbbnScqqZEkmNmU9y9IJljkylxLHH3BysY0wbM7FhCY/gRiZKGmW0JVHP3FZH3xwC3pDIOyU2llTL+xE/cx+Wcx1PMpjGH8wETOSzudZQ0RNIjmTaOKWZ2h5m1MLODol8VvG9fYGtC9dM0M3sEwMx2MbMxkWN2BN6LTG/yMfCKu4+t4H0lBxUVQb16of3CLExGuHHScM7geeayL2fzDLdwAwcxNW7SqFkzVE8paYikRzIljgMjr7H/QyvUHdfd90ywfTHQNvJ+PtA03nFSeRQVwYUXwpo1iY/ZmcU8TCGnMJJJFNCGN5jJARsdt9VW8Mgjas8QSbcyE4fW5ZB0iTdz7YacDgymD1eyBb9zFXdzP93/mJRQiUIkO5IpcWBmxwNNoHixAndXe4NsssLC8Es/Ud+M3ZnPY1xMa95iPEdwEQP5gj2VLERyQJltHJH2h7OASwAjrMPRMM1xSSVTVASNGoU2jGrVQm+peEmjGuu4jPuZyf4cwiQ68whH8xZfsCddu8KKFUoaItmWTOP44e5+HvCzu98MtAB2S29Yku9KJop//hMWRmY2S1TKaMxs3qcl93M5b3MUjZnDADqz5VbVGDoUHn44Y+GLSCmSSRyrIq8rzWwXYA2we/pCknxXVASdOpWdKKI2ZzU3cAufcCB7Mo9zKOJEXuZ/dXdl6FCVMkRyTTJtHKMjs9feDUwl9Kh6LK1RSV4qKoIePYoTRjIKmMQgOnIAM3mas6nx8AM83XV7nk5fmCJSQcn0quoVeTvczEYDNd19WXrDknwRmyzMyi5dRNViJTdzE1dwL9+yMycxil27nsjDXdMbr4hUXMKqKjM7xMx2ivl8HvA80MvM/pSJ4CS3lbdKKuoIxjOdpvybexjIRbTabjZnDT1RbRgieaK0No5HgdUAZvY34E7gSWAZMCD9oUmu69EDVpY5y1ixbVhGf7ownqMwnLN3fIuthj7Kl0u3VRuGSB4praqqursvjbw/Cxjg7sMJVVbT0h+a5LqvvkruuOrV4R/rXmFg9c7suP5buOJK9rzlFp6pXTu9AYpIWpRW4qhuZtHE0hp4K2ZfUgMHpXKJdrGtVi28/qmMCsvateGFh39k7VnteYUT2Hnf7aj20Ydwzz1hp4jkpdISwDPAO2a2hNAl910AM9uTUF0lVUi0PSNaNbVwIWy+OdSoUbxsKxQ3kDds4Aw94Vn+euOlsGwZ9OwJ110XThCRvJawxOHutwFXAkOAv3rxwh3VCKPIpQqJ156xZg1svTU0bBgSRsOG8NRT4F8vYsEBJ/HXh8+BP/8Zpk6Fm25S0hCpJEqtcnL3j+Js+2/6wpFclag9Y+lSWLIk8mH9ehg4EJr8O2SVe++FSy8NjRwiUmkkM3JchESr8f6xfd68sHJS585w8MEwcyZcfrmShkglpMQhSbntto3bs2vXhtt7rYM+feCAA0KV1GOPwZtvwh57ZCdQEUk79Y6SpETHWfToEaqtGjSAfl1mcvxDHWHSJDjxxDDlbf362Q1URNJOJQ5JWvv2sGABrF/1OwvOv4njbzgobHj2WRg5UklDpIpQ4pC4So7ZKCqK7Jg4MbRh3HILtGsHc+bAWWeFblUiUiUocchGYuegcg+v3S/+jbnHXQEtWoRxGaNHh7639eplO1wRyTAlDtlIyTEbR/EWH62uXpu8AAAQwklEQVQ6gH3H3gddusDs2XD88dkLUESyKiuJw8x6mtk3ZjYt8tU2wXHHmtlnZjbPzK7NdJxVVXTMxrb8wgAu5i1as55qHMn4sAzfNttkNT4Rya5sljjuc/dmka8xJXeaWXWgH3Ac0Bg428waZzrIqqhBAziRUcymCR0YzF1czQHMYEHDI7IdmojkgFyuqmoOzHP3+e6+GngWODnLMVV+P/zA+J3bMYqTWUI9DmUi13IX1WrX4rbbsh2ciOSCbCaObmY2w8wGm9l2cfbXB76O+bwosk3SwR2GDoV996XR1BFMP70XpzWYzFQroGFDGDBA636LSJC2AYBm9gawU5xdPYD+QC/C+uW9gD5Ah5KXiHNuwjXmzKwT0AmgQaL5MSS+r78Ojd5jxsBhh8GgQTRt3Jh52Y5LRHJS2hKHu7dJ5jgzewwYHWfXImC3mM+7AotLud8AIisTFhQUJLmIaRW3fj08+ihccw2sWwf33w/duml+KREpVbZ6Ve0c8/FUYFacwyYBe5nZ7mZWA2gHjMpEfFXCf/8LRx0FhYVw6KEwaxZcdpmShoiUKVttHL3NbKaZzQCOAi4HMLNdzGwMgLuvBboBrwFzgefdfXaW4q081q6F3r2haVOYPh0GDYLXX4fdd892ZCKSJ7IyyaG7n5tg+2KgbcznMcBGXXVlE02fDh06hFlsTzkF+vWDXXbJdlQikmdyuTuupMrvv8MNN0BBASxaBC+8AC++qKQhIptE06pXdh9+CB07wty5cN55YVW+unWzHZWI5DGVOCqrX3+F7t2hZUv47Td49VV44gklDRGpMJU4KqNx48L0tgsWwL/+BXfcAVtvne2oRKSSUImjMvn559D4fcwxUKMGTJgAffsqaYhISilxVBYjRkDjxvDkk3DddaEHVatW2Y5KRCohVVXlu+++g0sugWHDoFkzeOUVOOigbEclIpWYShz5yj2ULho3hpdfhttvh48/VtIQkbRTiSMfLVwInTvDa6/B4YeH0d/77JPtqESkilCJI5+sXx9Ge++3H7z3Hjz0ELz7rpKGiGSUShz54rPPwkC+998PvaYefRQaNcp2VCJSBanEkevWrAnjMJo2hTlzYMgQGDs2JUmjqChcplq18FpUVOFLikgVoBJHLvvkk1DK+OQTOP30UDW1U7y1scqvqCiMEVy5MnxeuDB8Bq30JyKlU4kjF/3vf/Cf/8Ahh8DixTB8eJiYMEVJA6BHj+KkEbVyZdguIlIalThyzfvvh1LGZ5/BhRdCnz6wXbwl2Svmq6/Kt11EJEoljlyxYkUYyNeqVShxvPYaDB6clqQBkGhZdi3XLiJlUeLIBa+9FrrY9usXksesWaHnVBrddhvUrr3httq1w3YRkdIocWTT0qVw/vlw7LHht/Z778EDD8BWW1Xossn0lmrfHgYMgIYNwSy8DhighnERKZvaOLJl2LAw5fnSpaFF+vrroWbNCl+2PL2l2rdXohCR8lOJI9O+/RZOOw3OOAN23RUmTYJbb01J0gD1lhKR9FPiyBR3ePzxMCnhK6/AnXfCxIlhRtsUUm8pEUm3rFRVmdlzwN6Rj3WAX9x9o9+gZrYAWAGsA9a6e0HGgkylL78MkxKOGxd6TQ0cCH/5S1pu1aBBqJ6Kt11EJBWyUuJw97PcvVkkWQwHXizl8KMix+Zf0li3Dh58MPSY+vDD0Gtq/Pi0JQ1QbykRSb+sVlWZmQFnAs9kM460mDs3lC4uuwyOOAJmz4bCwtDVKY3UW0pE0i3bvapaAd+7++cJ9jvwupk58Ki7D8hcaJtozRro3RtuuSV0q33qqfBb2yxjIai3lIikU9oSh5m9AcSbXKmHu4+MvD+b0ksbLd19sZntAIwzs0/dfUKC+3UCOgE0yFaF/pQp0KEDzJgBZ54ZJiXcYYfsxCIikiZpSxzu3qa0/Wa2GfB/wMGlXGNx5PUHMxsBNAfiJo5IaWQAQEFBgW9i2Jtm1Sro2TPMK7XDDjBiBJxySkZDEBHJlGy2cbQBPnX3RfF2mtmWZrZ19D1wDDArg/ElZ8KEsFZG795wwQVhzQwlDRGpxLKZONpRoprKzHYxszGRjzsC75nZdOBj4BV3H5vhGBNbvjw0dh9xBKxdC2+8EbrZ1qmT7chERNIqa43j7n5BnG2LgbaR9/OBphkOKzljxkCXLrBoEVx+OfTqBVtume2oREQyItu9qvLLkiUhUQwdGkaAf/ABHHZYtqMSEckoTTmSDHd4/vmQLJ59Fm68EaZOVdIQkSpJJY6yLF4c2jJGjoSCgtCWccAB2Y5KRCRrVOJIxD00djduHBZauvvuMG2IkoaIVHFKHPHMnw9t2sDFF4fZa2fOhKuugs1SX0BLZtElEZFcosQRa906uO++MCnhpEnwyCPw1luw555puV100aWFC0MBJ7rokpKHiOQyJY6o2bOhZUu44go4+ugwkK9z57ROSqhFl0QkHylxrF4dJiQ88ED44gt4+ml4+eWwOl+aadElEclHVTtxTJoEBx8MN90Ep58eShlnn52xmWwTzcWoRZdEJJdVzcSxcmVo7D7sMPj5Zxg1KpQ0tt8+o2Fo0SURyUdVL3GMHx+61PbpAxddFNo2TjwxK6Fo0SURyUdVZwDgsmVw9dXhN/Mee4TeUkcdle2otOiSiOSdqlHiGD0amjQJA/quuiostJQDSUNEJB9V7sTx449wzjmhKmq77cLI77vv3rhhQUREklZ5E8czz4TpQoYNg5tvDsu6Nm+e7ahERPJe5WzjmDcvlDSaN4dBg8JIcBERSYnKWeJYvhzuvTesl6GkISKSUubu2Y4h5czsR2BhlsOoByzJcgybKl9jV9yZl6+xK+6NNXT3pAazVcrEkQvMbLK7F2Q7jk2Rr7Er7szL19gVd8VUzqoqERFJGyUOEREpFyWO9BmQ7QAqIF9jV9yZl6+xK+4KUBuHiIiUi0ocIiJSLkocKWJmz5nZtMjXAjObluC4BWY2M3Lc5EzHGY+Z9TSzb2Lib5vguGPN7DMzm2dm12Y6zjjx3G1mn5rZDDMbYWZ1EhyXE8+8rOdnZltEfo7mmdlEM2uU+Sg3imk3M3vbzOaa2WwzuyzOMUea2bKYn58bsxFrPGV97y14MPLMZ5jZQdmIs0RMe8c8y2lmttzMupc4JrvP3N31leIvoA9wY4J9C4B62Y6xREw9gavKOKY68AXwZ6AGMB1onOW4jwE2i7y/C7grV595Ms8PKAQeibxvBzyXAz8bOwMHRd5vDfw3TtxHAqOzHeumfO+BtsCrgAGHAROzHXOcn5vvCGMscuaZq8SRYmZmwJnAM9mOJcWaA/Pcfb67rwaeBU7OZkDu/rq7r418/AhI/3q/my6Z53cy8ETk/TCgdeTnKWvc/Vt3nxp5vwKYC9TPZkwpdjLwpAcfAXXMbOdsBxWjNfCFu2d7QPMGlDhSrxXwvbt/nmC/A6+b2RQz65TBuMrSLVJUH2xm28XZXx/4OubzInLrF0gHwl+O8eTCM0/m+f1xTCQhLgPqZiS6JESqzg4EJsbZ3cLMppvZq2bWJKOBla6s732u/1y3I/EfoVl75pVzksM0MbM3gJ3i7Orh7iMj78+m9NJGS3dfbGY7AOPM7FN3n5DqWEsqLXagP9CL8J+sF6GqrUPJS8Q5N+1d8pJ55mbWA1gLFCW4TFaeeQnJPL+sPONkmNlWwHCgu7svL7F7KqEq5ddI+9hLwF6ZjjGBsr73ufzMawAnAdfF2Z3VZ67EUQ7u3qa0/Wa2GfB/wMGlXGNx5PUHMxtBqMJI+y+xsmKPMrPHgNFxdi0Cdov5vCuwOAWhlSqJZ34+cALQ2iOVv3GukZVnXkIyzy96zKLIz9K2wNLMhJeYmW1OSBpF7v5iyf2xicTdx5jZw2ZWz92zPhdUEt/7rPxcJ+k4YKq7f19yR7afuaqqUqsN8Km7L4q308y2NLOto+8JjbuzMhhfXCXqdE8lfkyTgL3MbPfIX0LtgFGZiC8RMzsWuAY4yd1XJjgmV555Ms9vFHB+5P3pwFuJkmGmRNpYBgFz3f3eBMfsFG2LMbPmhN8rP2UuyviS/N6PAs6L9K46DFjm7t9mONREEtZeZPuZq8SRWhvVR5rZLsBAd28L7AiMiHy/NwOedvexGY9yY73NrBmhiL4A6Awbxu7ua82sG/AaoafHYHefna2AI/oCWxCqIAA+cvcuufjMEz0/M7sFmOzuowi/oJ8ys3mEkka7TMcZR0vgXGCmFXcx/w/QAMDdHyEkua5mthZYBbTLdsKLiPu9N7Mu8EfsYwg9q+YBK4ELsxTrBsysNvB3Iv8XI9ti487qM9fIcRERKRdVVYmISLkocYiISLkocYiISLkocYiISLkocYiISLkocUjOMrO6MbN/fmfFM/j+YmZzMhxLM4uZNdjMTrJNnCHYwoyt9eJs39bMnjSzLyJfRQmmf6mQ0v4tFmZKvirV95TKRYlDcpa7/+Tuzdy9GfAIcF/kfTNgfarvFxmtnUgzQn//aGyj3P3OFIcwCJjv7nu4+x6EsQVDUnwPyMy/RSoxJQ7JV9XN7DELa0S8bma1AMxsDzMbG5nU7l0z2yeyvaGZvRmZyPFNM2sQ2T7EzO41s7eBuyKjjQeb2SQz+8TMTo6M9L4FOCtS4jnLzC4ws76Ra+xoYT2Q6ZGvwyPbX4rEMdvKmFzRzPYkTFXTK2bzLUBTC+szHGlmo2OO72tmF0Te3xiJd5aZDYgZUTzezO4ys4/N7L9m1qqsf0uJmBI9yzMi95puZpmeukVygBKH5Ku9gH7u3gT4BTgtsn0AcIm7HwxcBTwc2d6XMH32AYTJEB+MudZfgDbufiVh0se33P0Q4CjgbmBz4EbC+hjN3P25ErE8CLzj7k2Bg4DoiPoOkTgKgEvNrLSZbhsD09x9XXRD5P0nwL5lPIu+7n6Iu+8H1CLM3RW1mbs3B7oDN0WmdC/t3xIr0bO8EfhH5N97UhmxSSWkKUckX33p7tEpMKYAjSzM4Ho48IIVL2OxReS1BWECSoCngN4x13oh5hf2McBJMfX8NYlMr1GKo4Hz4I9f9ssi2y81s1Mj73cjJLtE8wkZ8WdlTWY9jqPM7GqgNvAnQuJ6ObIvOinhFKBREtcKNy39Wb4PDDGz52OuL1WIEofkq99j3q8j/KVdDfgl0g5Slthf0r/FvDfgNHf/LPZgMzu0PMGZ2ZGESS9buPtKMxtPSEKJzAYONLNq7r4+co1qwAGEKbQbsGENQc3IMTUJJYECd//azHqWuE/0Oa2jfP/fEz7LyHxghwLHA9PMrJm7Z31SQ8kcVVVJpRGZavpLMzsD/lhPumlk9wcUTxrYHngvwWVeAy6JaSc4MLJ9BWHp1HjeBLpGjq9uZtsQpkT/OZI09iEsS1pa7PMI1VLXx2y+HnjT3b8CFgKNLaxLvi1hZTgoThJLIqWE00u7TxL/lmg8CZ+lme3h7hPd/UZgCRtOSy5VgBKHVDbtgY5mNp3wV3x0edZLgQvNbAZhttfLEpzfi9CmMcPMZlHcWP024Rf3NDM7q8Q5lxGqi2YSqoSaAGOBzSL360VY2rYsHQhTr88zsx8JySY6I+rXwPPADEIbzSeR7b8AjwEzCYv5TEriPqX9W2IlepZ3m9nMyPOZQFg/XaoQzY4rkoPMbG/ClN+XuPuYbMcjEkuJQ0REykVVVSIiUi5KHCIiUi5KHCIiUi5KHCIiUi5KHCIiUi5KHCIiUi5KHCIiUi7/D4iggk1WyyRpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "from matplotlib import pyplot as plt\n",
    "import scipy.stats as stats\n",
    "\n",
    "fig = sm.qqplot(FF5.resid.values, stats.t, distargs=(6,), fit=True, line=\"45\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Residual Heteroskedasticity\n",
    "\n",
    "\n",
    "<p>If <span class=\"math inline\">\\(\\mathbb{V}{\\rm ar} (\\epsilon_i | \\mathbf{X} ) = \\sigma^2_\\epsilon,\\ \\forall i = 1,..,N\\)</span>, we say that the residuals are <strong>homoskedastic</strong>. If this assumption is violated, we say that the residuals are <strong>heteroskedastic</strong> - that is, their variance is not constant throughout observations.</p>\n",
    "<div class=\"THEOREM\">\n",
    "<p>\n",
    "The consequences of heteroskedasticity are as follows:\n",
    "</p>\n",
    "<ul>\n",
    "<li>\n",
    "OLS parameters remain <strong>unbiased</strong>;\n",
    "</li>\n",
    "<li>\n",
    "OLS estimates are no longer efficient (i.e. they no longer have the smallest variance). The reason for this is that OLS gives equal weight to all observations in the data, when in fact, observation with larger error variance contain less information, compared to observations with smaller error variance;\n",
    "</li>\n",
    "<li>\n",
    "The variance estimate of the residuals <strong>is biased</strong>, and hence the standard errors are <strong>biased</strong>. This in turn leads to a bias in test statistics and confidence intervals.\n",
    "</li>\n",
    "<li>\n",
    "Because of standard error bias, we may fail to reject the null hypothesis whether <span class=\"math inline\"><span class=\"math inline\">\\(\\beta_i = 0\\)</span></span> in our estimated model, when the null hypothesis is actually false (i.e. making a Type II error).\n",
    "</li>\n",
    "</ul>\n",
    "<p>\n",
    "There are a few possible corrections to account for heteroskedasticity:\n",
    "</p>\n",
    "<ul>\n",
    "<li>\n",
    "Take logarithms of the data, this may be able to help <strong>linearize</strong> the data and in turn, the residuals;\n",
    "</li>\n",
    "<li>\n",
    "Apply a different estimation method. We will examine this later on, but one possibility is to use a <strong>Weighted Least Squares</strong> estimation method, which gives different observations different weights and allows to account for a non-constant variance;\n",
    "</li>\n",
    "<li>\n",
    "It is possible to correct the the biased standard errors for heteroskedasticity. This would leave the OLS estimates unchanged. <a href=\"https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors\">White’s heteroskedasticity-consistent standard errors</a> (or, robust standard errors) give a consistent variance estimator.\n",
    "</li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>There are a number of methods to test for the presence of heteroskedasticity. Some of the tests are:</p>\n",
    "<ul>\n",
    "<li><strong>Goldfeld–Quandt Test</strong>. It divides the dataset into two subsets. The subsets are specified so that the observations for which the explanatory variable takes the lowest values are in one subset, and the highest values - in the other subset. The subsets are not necessarily of equal size, nor do they contain all the observations between them. The test statistic used is the ratio of the mean square residual errors for the regressions on the two subsets. This test statistic corresponds to an F-test of equality of variances. The Goldfeld–Quandt test requires that data be <strong>ordered</strong> along a known explanatory variable, from lowest to highest.</li>\n",
    "</ul>\n",
    "<p>If the error structure depends on an unknown variable or an unobserved variable the Goldfeld–Quandt test provides little guidance. Also, error variance must be a monotonic function of the specified explanatory variable. For example, when faced with a quadratic function mapping the explanatory variable to error variance the Goldfeld–Quandt test may improperly accept the null hypothesis of homoskedastic errors.</p>\n",
    "<p>Unfortunately the Goldfeld–Quandt test is not very robust to specification errors. The Goldfeld–Quandt test detects non-homoskedastic errors but cannot distinguish between heteroskedastic error structure and an underlying specification problem such as an incorrect functional form or an omitted variable.</p>\n",
    "<ul>\n",
    "<li><p><strong>Breusch–Pagan Test</strong>. After estimating the linear regression <span class=\"math inline\">\\(Y = \\beta_0 + \\beta_1 X + \\epsilon\\)</span>, calculate the model residuals <span class=\"math inline\">\\(\\widehat{\\epsilon}_i\\)</span>. The OLS assumptions state that the residual variance does not depend on the independent variables <span class=\"math inline\">\\(\\mathbb{V}{\\rm ar} (\\epsilon_i | \\mathbf{X} ) = \\sigma^2_\\epsilon\\)</span>. If this assumptions is not true, then there may be a linear relationship between <span class=\"math inline\">\\(\\widehat{\\epsilon}_i^2\\)</span> and <span class=\"math inline\">\\(X_i\\)</span>. So, the Breush-Pagan test is the based on the following regression:\n",
    "<span class=\"math display\">\\[\n",
    "\\widehat{\\epsilon}_i^2 = \\gamma_0 + \\gamma_1 X_i + v_i\n",
    "\\]</span>\n",
    "The hypothesis tests is:\n",
    "\n",
    "$\\begin{aligned}\n",
    "H_0&: \\gamma_1 = 0 \\text{ (residuals are homoskedastic)}\\\\\n",
    "H_1&: \\gamma_1 \\neq 0 \\text{ (residuals are heteroskedastic)}\n",
    "\\end{aligned}$\n",
    "\n",
    "It is a chi-squared test, where the test statistic:\n",
    "$\n",
    "LM = N \\cdot R^2_{\\widehat{\\epsilon}}\n",
    "$\n",
    "is distributed as $\\chi^2_1$ under the null. Here $R^2_{\\widehat{\\epsilon}}$is the R-square of the squared residual regression. One weakness of the BP test is that it assumes that the heteroskedasticity is a <strong>linear</strong> relationship of the independent variables. If we fail to reject the null hypothesis, we still do not rule out the possibility of a non-linear relationship between the independent variables and the error variance.</p></li>\n",
    "<li><p><strong>White Test</strong> is more generic than the BP test as it allows the independent variables to have a nonlinear effect on the error variance. For example, a combination of linear, quadratic and cross-products of the independent variables. It is a more commonly used test for homoskedasticity. The test statistic is calculated the same way as in <strong>BP</strong> test:\n",
    "<span class=\"math display\">\\[\n",
    "LM = N \\cdot R^2_{\\widehat{\\epsilon}}\n",
    "\\]</span>\n",
    "the difference from <strong>BP</strong> is that the squared residual model, from which we calculate <span class=\"math inline\">\\(R^2_{\\widehat{\\epsilon}}\\)</span>, may be nonlinear. A shortcoming of the White test is that it can lose its power if the model has many exogenous variables.</p></li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4.736381826174274,\n",
       " 0.029531116461051936,\n",
       " 4.747868990115543,\n",
       " 0.02954045766211842)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statsmodels.stats.diagnostic as sm_diagnostic\n",
    "\n",
    "# Breusch–Pagan Test\n",
    "sm_diagnostic.het_breuschpagan(resid = FF5.resid.values, exog_het = sm.add_constant(FF5.resid.values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1133.0, 9.379420394983054e-247, 7.973561307442355e+29, 0.0)\n"
     ]
    }
   ],
   "source": [
    "# White Test\n",
    "print(sm_diagnostic.het_white(resid = FF5.resid.values, exog = sm.add_constant(FF5.resid.values)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Residual Autocorrelation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>If <span class=\"math inline\">\\(\\mathbb{C}{\\rm ov} (\\epsilon_i, \\epsilon_j) \\neq 0\\)</span> for some <span class=\"math inline\">\\(i \\neq j\\)</span>, then the errors are correlated. Autocorrelation is frequently encountered in time-series models.</p>\n",
    "\n",
    "<br>\n",
    "Assume that our model is defined as follows:\n",
    "</div>\n",
    "\n",
    "<p><span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "Y_t = \\beta_0 + \\beta_1 X_t + \\epsilon_t \\\\\n",
    "\\epsilon_t = \\rho \\epsilon_{t-1} + u_t,\\ |\\rho| &< 1,\\ u_t \\sim \\mathcal{N}(0, \\sigma^2)\n",
    "\\end{aligned}\n",
    "\\]</span>\n",
    "Then we say that the model has <strong>autocorrelated</strong>, or <strong>serially correlated</strong> errors.</p>\n",
    "<p>In this case, we have that:\n",
    "<span class=\"math display\">\\[\n",
    "\\mathbb{C}{\\rm ov}(\\epsilon_t, \\epsilon_{t-1}) = \\mathbb{C}{\\rm ov}(\\rho \\epsilon_{t-1} + u_t, \\epsilon_{t-1}) = \\rho \\mathbb{C}{\\rm ov}(\\epsilon_{t-1},\\epsilon_{t-1}) = \\rho \\sigma^2 \\neq 0\n",
    "\\]</span>\n",
    "Estimating the coefficients via OLS and ignoring the violation will still result in unbiased and consistent OLS estimators. However, the estimators are inefficient and the variance of the regression coefficients will be <strong>biased</strong>.</p>\n",
    "<p>On the other hand, autocorrelation in errors may be a result of a misspecified model.</p>\n",
    "\n",
    "\n",
    "<p>There are a number of tests for the presence of autocorrelation:</p>\n",
    "<ul>\n",
    "<li><p><strong>Durbin–Watson Test</strong> for the hypothesis:\n",
    "<span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "H_0&:\\text{the errors are serially uncorrelated}\\\\\n",
    "H_1&:\\text{the errors follow a first order autoregressive process (i.e. autocorrelation at lag 1)}\n",
    "\\end{aligned}\n",
    "\\]</span>\n",
    "The test statistic:\n",
    "<span class=\"math display\">\\[\n",
    "d = \\dfrac{\\sum_{i = 2}^N (\\widehat{\\epsilon}_i - \\widehat{\\epsilon}_{i-1})^2}{\\sum_{i = 1}^N \\widehat{\\epsilon}_i^2}\n",
    "\\]</span>\n",
    "The value of <span class=\"math inline\">\\(d\\)</span> always lies between 0 and 4. <span class=\"math inline\">\\(d = 2\\)</span> indicates no autocorrelation. If the Durbin–Watson statistic is not close to 2, there is evidence of a serial correlation.</p></li>\n",
    "<li><p><strong>Breusch-Godfrey Test</strong> is a more flexible test, covering autocorrelation of higher orders and applicable whether or not the regressors include lags of the dependent variable. Consider the following linear regression:\n",
    "<span class=\"math display\">\\[\n",
    "Y_i = \\beta_0 + \\beta_1 X_i + \\epsilon_i\n",
    "\\]</span>\n",
    "We then estimate the model via OLS and fit the following model on the residuals <span class=\"math inline\">\\(\\widehat{\\epsilon}_i\\)</span>:\n",
    "<span class=\"math display\">\\[\n",
    "\\widehat{\\epsilon}_i = \\alpha_0 + \\alpha_1 X_i + \\rho_1 \\widehat{\\epsilon}_{i - 1} + \\rho_2 \\widehat{\\epsilon}_{i - 2} + ... + \\rho_p \\widehat{\\epsilon}_{i - p} + u_t\n",
    "\\]</span>\n",
    "and calculate its <span class=\"math inline\">\\(R^2\\)</span> (R-squared), then testing the hypothesis:\n",
    "<span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "H_0&:\\rho_1 = \\rho_2 = ... = \\rho_p = 0\\\\\n",
    "H_1&:\\rho_j \\neq 0 \\text{ for some } j\n",
    "\\end{aligned}\n",
    "\\]</span>\n",
    "Under the null hypothesis the test statistic:\n",
    "<span class=\"math display\">\\[\n",
    "LM = (N-p)R^2 \\sim \\chi^2_p\n",
    "\\]</span></p></li>\n",
    "</ul>\n",
    "\n",
    "Note There is also the Ljung-Box Test for testing the null hypothesis of no autocorrelation of residuals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6737921626623802\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats.stattools import durbin_watson\n",
    "# Durbin–Watson Test\n",
    "print(durbin_watson(FF5.resid.values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(40.17074024060269, 1.892494342030728e-09, 20.67827145690064, 1.5173694149624436e-09)\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats.diagnostic import acorr_breusch_godfrey\n",
    "# Breusch-Godfrey Test\n",
    "print(acorr_breusch_godfrey(FF5, nlags = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Residual Normality Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>The normality requirement is necessary if we want to obtain the correct <span class=\"math inline\">\\(p\\)</span>-values and critical <span class=\"math inline\">\\(t\\)</span>-values when testing the hypothesis that <span class=\"math inline\">\\(H_0: \\beta_j = c\\)</span>, especially for significance testing, with <span class=\"math inline\">\\(c = 0\\)</span>. Assume that we want to test whether our residuals <span class=\"math inline\">\\(z_1,...,z_N\\)</span> come from a normal distribution. The hypothesis can be stated as:\n",
    "<span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "H_0:\\text{residuals follow a normal distribution}\\\\\n",
    "H_1:\\text{residuals do not follow a normal distribution}\n",
    "\\end{aligned}\n",
    "\\]</span></p>\n",
    "<p>There are a number of normality tests, like:</p>\n",
    "<ul>\n",
    "<li><p><strong>Anderson-Darling Test</strong>. The test statistic is calculated as:\n",
    "<span class=\"math display\">\\[\n",
    "A^2 = -N - \\sum_{i 1}^N \\dfrac{2i-1}{N}\\left[ \\log(F(z_{(i)}) + \\log\\left(1 - F(z_{(N+1-i)}) \\right)\\right]\n",
    "\\]</span>\n",
    "where <span class=\"math inline\">\\(z_{(i)}\\)</span> are the <strong>ordered data</strong> and <span class=\"math inline\">\\(F(\\cdot)\\)</span> is the cumulative distribution function (cdf) of the distribution being tested (for the univariate regression residuals - we are usually interested in testing for the normal distribution). The test statistic is compared against the critical values from the normal distribution. <a href=\"https://web.archive.org/web/20150630110326/http://instatmy.org.my/downloads/e-jurnal%202/3.pdf\">Empirical testing indicates</a> that the Anderson–Darling test is not quite as good as Shapiro-Wilk, but is better than other tests.</p></li>\n",
    "\n",
    "<br>\n",
    "<li><p><strong>Shapiro-Wilk Test</strong>. The test statistic is:\n",
    "<span class=\"math display\">\\[\n",
    "W = \\dfrac{\\left(\\sum_{i = 1}^N a_i z_{(i)} \\right)^2}{\\sum_{i = 1}^N (z_i - \\overline{z})^2}\n",
    "\\]</span>\n",
    "where <span class=\"math inline\">\\(z_{(i)}\\)</span> is the <span class=\"math inline\">\\(i\\)</span>-th <strong>smallest value in the sample</strong> (i.e. the data are ordered). <span class=\"math inline\">\\(a_i\\)</span> values are calculated using means, variances and covariances of <span class=\"math inline\">\\(z_{(i)}\\)</span>. <span class=\"math inline\">\\(W\\)</span> is compared against tabulated values of this statistic’s distribution. Small values of <span class=\"math inline\">\\(W\\)</span> will lead to the rejection of the null hypothesis. Monte Carlo simulation has found that Shapiro–Wilk has the best power for a given significance, followed closely by Anderson–Darling when comparing the Shapiro–Wilk, Kolmogorov–Smirnov, Lilliefors and Anderson–Darling tests.</p></li>\n",
    "\n",
    "<br>\n",
    "<li><p><strong>Kolmogorov-Smirnov Test</strong>. The test statistic is given by:\n",
    "<span class=\"math display\">\\[\n",
    "D = \\max\\{ D^+; D^-\\}\n",
    "\\]</span>\n",
    "where:\n",
    "<span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "D^+ = \\max_i \\left( \\dfrac{i}{N} - F(z_{(i)})\\right)\\\\\n",
    "D^- = \\max_i \\left( F(z_{(i)}) - \\dfrac{i - 1}{N} \\right)\n",
    "\\end{aligned}\n",
    "\\]</span>\n",
    "where <span class=\"math inline\">\\(F(\\cdot)\\)</span> is the theoretical cdf of the distribution being tested (for the univariate regression residuals - we are usually interested in testing for the normal distribution). The <strong>Lilliefors Test</strong> is based on the Komogorov-Smirnov Test as a special case of this for the normal distribution.. For the normal distribution case, the test statistic is compared against the critical values from a normal distribution in order to determine the <span class=\"math inline\">\\(p\\)</span>-value.</p></li>\n",
    "\n",
    "<br>\n",
    "<li><p><strong>Cramer–von Mises Test</strong> is an alternative to the Kolmogorov–Smirnov test. The test statistic:\n",
    "<span class=\"math display\">\\[\n",
    "W = N\\omega^2 = \\dfrac{1}{12N} + \\sum_{i = 1}^N \\left[ \\dfrac{2i-1}{2N} - F(z_{(i)}) \\right]^2\n",
    "\\]</span>\n",
    "If this value is larger than the tabulated value, then the hypothesis that the data came from the distribution <span class=\"math inline\">\\(F\\)</span> can be rejected.</p></li>\n",
    "\n",
    "<br>\n",
    "<li><p><strong>Jarque–Bera Test</strong> (valid for large samples). The statistic is calculated as:\n",
    "<span class=\"math display\">\\[\n",
    "JB = \\dfrac{N-k+1}{6} \\left(S^2 + \\dfrac{(C - 3)^2}{4}\\right)\n",
    "\\]</span>\n",
    "where\n",
    "<span class=\"math display\">\\[\n",
    "\\begin{aligned}\n",
    "S = \\dfrac{\\dfrac{1}{N}\\sum_{i = 1}^N (z_i - \\overline{z})^3}{\\left( \\dfrac{1}{N}\\sum_{i = 1}^N (z_i - \\overline{z})^2 \\right)^{3/2}}= \\dfrac{\\widehat{\\mu}_3}{\\widehat{\\sigma}^3}\\\\\n",
    "C = \\dfrac{\\dfrac{1}{N}\\sum_{i = 1}^N (z_i - \\overline{z})^4}{\\left( \\dfrac{1}{N}\\sum_{i = 1}^N (z_i - \\overline{z})^2 \\right)^{2}} = \\dfrac{\\widehat{\\mu}_4}{\\widehat{\\sigma}^4}\\\\\n",
    "\\end{aligned}\n",
    "\\]</span>\n",
    "<span class=\"math inline\">\\(N\\)</span> is the sample size, <span class=\"math inline\">\\(S\\)</span> is the skewness and <span class=\"math inline\">\\(C\\)</span> is kurtosis and <span class=\"math inline\">\\(k\\)</span> is the number of regressors (i.e. the number of different independent variables <span class=\"math inline\">\\(X\\)</span>, with <span class=\"math inline\">\\(k = 1\\)</span> outside a regression context). <br> If the data comes from a normal distribution, then the <span class=\"math inline\">\\(JB\\)</span> statistic has a chi-squared distribution with <strong>two degrees of freedom</strong>, <span class=\"math inline\">\\(\\chi^2_2\\)</span>.</p></li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1163.04560231688,\n",
       " 2.804505886896713e-253,\n",
       " 0.1707597112963196,\n",
       " 7.958348312142612)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from statsmodels.stats.stattools import jarque_bera\n",
    "jarque_bera(FF5.resid.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardized Residuals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>When we compare residuals for different observations, we want to take into account that their variances may be different. One way to account for this is to divide the residuals by an estimate the <strong>residuals standard deviation</strong>. This results in calculating the <strong>standardized</strong> residuals:\n",
    "<span class=\"math display\">\\[\n",
    "s_i = \\dfrac{\\widehat{\\epsilon_i}}{\\widehat{\\sigma}\\sqrt{1 - h_{ii}}}\n",
    "\\]</span>\n",
    "where <span class=\"math inline\">\\(h_{ii}\\)</span> is the <span class=\"math inline\">\\(i\\)</span>-th diagonal element of <span class=\"math inline\">\\(\\mathbf{H}\\)</span>. Standardized residuals are useful in detecting outliers. Generally, any observation with a standardized residual greater than 2 in absolute value should be examined more closely.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "data3 = pd.read_csv(\"S*Isf..Oil.Ref.Co..csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>&lt;TICKER&gt;</th>\n",
       "      <th>&lt;DTYYYYMMDD&gt;</th>\n",
       "      <th>&lt;FIRST&gt;</th>\n",
       "      <th>&lt;HIGH&gt;</th>\n",
       "      <th>&lt;LOW&gt;</th>\n",
       "      <th>&lt;CLOSE&gt;</th>\n",
       "      <th>&lt;VALUE&gt;</th>\n",
       "      <th>&lt;VOL&gt;</th>\n",
       "      <th>&lt;OPENINT&gt;</th>\n",
       "      <th>&lt;PER&gt;</th>\n",
       "      <th>&lt;OPEN&gt;</th>\n",
       "      <th>&lt;LAST&gt;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>S*Isf..Oil.Ref.Co.</td>\n",
       "      <td>20210102</td>\n",
       "      <td>12610.0</td>\n",
       "      <td>12800.0</td>\n",
       "      <td>12210.0</td>\n",
       "      <td>12230.0</td>\n",
       "      <td>2186795515120</td>\n",
       "      <td>178766967</td>\n",
       "      <td>25733</td>\n",
       "      <td>D</td>\n",
       "      <td>12850.0</td>\n",
       "      <td>12210.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>S*Isf..Oil.Ref.Co.</td>\n",
       "      <td>20201230</td>\n",
       "      <td>13150.0</td>\n",
       "      <td>13410.0</td>\n",
       "      <td>12660.0</td>\n",
       "      <td>12850.0</td>\n",
       "      <td>2972144134290</td>\n",
       "      <td>231210626</td>\n",
       "      <td>42179</td>\n",
       "      <td>D</td>\n",
       "      <td>13320.0</td>\n",
       "      <td>12660.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>S*Isf..Oil.Ref.Co.</td>\n",
       "      <td>20201229</td>\n",
       "      <td>14000.0</td>\n",
       "      <td>14180.0</td>\n",
       "      <td>13130.0</td>\n",
       "      <td>13320.0</td>\n",
       "      <td>4251547609120</td>\n",
       "      <td>319161794</td>\n",
       "      <td>54773</td>\n",
       "      <td>D</td>\n",
       "      <td>13820.0</td>\n",
       "      <td>13140.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>S*Isf..Oil.Ref.Co.</td>\n",
       "      <td>20201228</td>\n",
       "      <td>13640.0</td>\n",
       "      <td>13840.0</td>\n",
       "      <td>13250.0</td>\n",
       "      <td>13820.0</td>\n",
       "      <td>4642000687820</td>\n",
       "      <td>335877057</td>\n",
       "      <td>44026</td>\n",
       "      <td>D</td>\n",
       "      <td>13190.0</td>\n",
       "      <td>13840.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>S*Isf..Oil.Ref.Co.</td>\n",
       "      <td>20201227</td>\n",
       "      <td>12960.0</td>\n",
       "      <td>13880.0</td>\n",
       "      <td>12600.0</td>\n",
       "      <td>13190.0</td>\n",
       "      <td>4155853228050</td>\n",
       "      <td>315119174</td>\n",
       "      <td>63980</td>\n",
       "      <td>D</td>\n",
       "      <td>13220.0</td>\n",
       "      <td>13500.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             <TICKER>  <DTYYYYMMDD>  <FIRST>   <HIGH>    <LOW>  <CLOSE>  \\\n",
       "0  S*Isf..Oil.Ref.Co.      20210102  12610.0  12800.0  12210.0  12230.0   \n",
       "1  S*Isf..Oil.Ref.Co.      20201230  13150.0  13410.0  12660.0  12850.0   \n",
       "2  S*Isf..Oil.Ref.Co.      20201229  14000.0  14180.0  13130.0  13320.0   \n",
       "3  S*Isf..Oil.Ref.Co.      20201228  13640.0  13840.0  13250.0  13820.0   \n",
       "4  S*Isf..Oil.Ref.Co.      20201227  12960.0  13880.0  12600.0  13190.0   \n",
       "\n",
       "         <VALUE>      <VOL>  <OPENINT> <PER>   <OPEN>   <LAST>  \n",
       "0  2186795515120  178766967      25733     D  12850.0  12210.0  \n",
       "1  2972144134290  231210626      42179     D  13320.0  12660.0  \n",
       "2  4251547609120  319161794      54773     D  13820.0  13140.0  \n",
       "3  4642000687820  335877057      44026     D  13190.0  13840.0  \n",
       "4  4155853228050  315119174      63980     D  13220.0  13500.0  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data3.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}