notebook/responsibleaidashboard-house-decision-making.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 機械学習モデルで意思決定をサポートする\n",
    "Responsible AI Toolbox を用いて家の価格を向上させるための分析を行います。元データは [Housing Prices Dataset](https://www.kaggle.com/alphaepsilon/housing-prices-dataset) を用いています。列情報は [Data Description](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data) で確認できます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### アジェンダ\n",
    "0. 事前設定\n",
    "1. ライブラリのインポート\n",
    "2. データ準備\n",
    "3. モデル学習\n",
    "4. Responsible AI Dashboard 構築"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 事前設定\n",
    "- Jupyter Kernel :  `raitoolbox` を選択する。\n",
    "    - [setup.ipynb](../setup.ipynb) の手順に Python 環境のセットアップを完了しておくこと。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. ライブラリ\n",
    "データの前処理やモデル学習に必要な Python ライブラリをインポートします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from lightgbm import LGBMRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. データ準備\n",
    "住宅の価格に関するデータを Pandas DataFrame としてインポートします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data = pd.read_csv('../data/apartments-train.csv')\n",
    "all_data = all_data.drop(['Sold_HigherThan_Median','SalePrice'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. モデル構築"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### scikit-learn パイプラインの作成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "def split_label(dataset, target_feature):\n",
    "    X = dataset.drop([target_feature], axis=1)\n",
    "    y = dataset[[target_feature]]\n",
    "    return X, y\n",
    "\n",
    "def clean_data(X, y, target_feature):\n",
    "    features = X.columns.values.tolist()\n",
    "    classes = y[target_feature].unique().tolist()\n",
    "    pipe_cfg = {\n",
    "        'num_cols': X.dtypes[X.dtypes == 'int64'].index.values.tolist(),\n",
    "        'cat_cols': X.dtypes[X.dtypes == 'object'].index.values.tolist(),\n",
    "    }\n",
    "    num_pipe = Pipeline([\n",
    "        ('num_imputer', SimpleImputer(strategy='median'))#,\n",
    "        #('num_scaler', StandardScaler())\n",
    "    ])\n",
    "    cat_pipe = Pipeline([\n",
    "        ('cat_imputer', SimpleImputer(strategy='constant', fill_value='?')),\n",
    "        ('cat_encoder', OneHotEncoder(handle_unknown='ignore', sparse=False))\n",
    "    ])\n",
    "    feat_pipe = ColumnTransformer([\n",
    "        ('num_pipe', num_pipe, pipe_cfg['num_cols']),\n",
    "        ('cat_pipe', cat_pipe, pipe_cfg['cat_cols'])\n",
    "    ])\n",
    "    X = feat_pipe.fit_transform(X)\n",
    "    print(pipe_cfg['cat_cols'])\n",
    "    return X, feat_pipe, features, classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_feature = 'SalePriceK'\n",
    "categorical_features = []\n",
    "\n",
    "X, y = split_label(all_data, target_feature)\n",
    "\n",
    "# 学習データとテストデータに分割\n",
    "X_train_original, X_test_original, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=1234)\n",
    "\n",
    "X_train, feat_pipe, features, classes = clean_data(X_train_original, y_train, target_feature)\n",
    "y_train = y_train[target_feature].to_numpy()\n",
    "\n",
    "X_test = feat_pipe.transform(X_test_original)\n",
    "y_test = y_test[target_feature].to_numpy()\n",
    "\n",
    "train_data = X_train_original.copy()\n",
    "train_data[target_feature] = y_train\n",
    "\n",
    "test_data = X_test_original.copy()\n",
    "test_data[target_feature] = y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM を用いたモデル学習を実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = LGBMRegressor(n_estimators=5)\n",
    "model = clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Responsible AI Dashboard 構築"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from raiwidgets import ResponsibleAIDashboard\n",
    "from responsibleai import RAIInsights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "dashboard_pipeline = Pipeline(steps=[('preprocess', feat_pipe), ('model', model)])\n",
    "rai_insights = RAIInsights(dashboard_pipeline, train_data, test_data, target_feature, 'regression',\n",
    "                             categorical_features=categorical_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# モデル説明性 (InterpretML)\n",
    "rai_insights.explainer.add()\n",
    "# モデル誤差解析 (Error Analysis)\n",
    "rai_insights.error_analysis.add()\n",
    "# 反実仮想サンプルの生成 (DiCE)\n",
    "rai_insights.counterfactual.add(total_CFs=10, desired_range=[50, 1000])\n",
    "# 因果推論 (EconML)\n",
    "rai_insights.causal.add(treatment_features=['OverallCond', 'OverallQual', 'Fireplaces', 'GarageCars', 'ScreenPorch'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "インプットした情報をベースに計算をします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rai_insights.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Responsible AI Toolbox ダッシュボードを生成します。表示される URI にブラウザからアクセスします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ResponsibleAIDashboard(rai_insights, locale=\"ja\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "d37be6dceca4eca8a6b9a12f8b026da8e21bfbb5796587ea169621cd1ce7de94"
  },
  "kernelspec": {
   "display_name": "rai-shap",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}