examples/Example4-PROC_PHREG-SGF2021-Simple_and_Efficient_Bootstrap_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Framework for Simple and Efficient Bootstrap Validation in SAS®, with Examples\n",
    "\n",
    "\n",
    "## Example 4: PROC PHREG &nbsp;&bullet;&nbsp; Invited Paper &nbsp;&bullet;&nbsp;  SAS Global Forum 2021\n",
    "\n",
    "This notebook contains an executable version of the example in Appendix E of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup\n",
    "\n",
    "The code below downloads the example dataset for the tutorial \"Introduction to Survival Analysis in SAS\" at https://stats.idre.ucla.edu/sas/seminars/sas-survival/. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename whas500 \"%sysfunc(pathname(work))/analysis_data.sas7bdat\";\n",
    "proc http\n",
    "        url='https://stats.idre.ucla.edu/wp-content/uploads/2016/02/whas500.sas7bdat'\n",
    "        method='get'\n",
    "        out=whas500\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Train a Model using PROC PHREG\n",
    "\n",
    "See page 16-17 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC PHREG output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ods output concordance=model_association_table(\n",
    "    keep=estimate\n",
    "    rename=(estimate=original_model_c_statistic)\n",
    ");\n",
    "proc phreg data=analysis_data concordance;\n",
    "    class gender;\n",
    "    model lenfol*fstat(0) = gender age;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Train a Model using PROC PHREG (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Generate Bootstrap Samples\n",
    "\n",
    "See page 5 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC SURVEYSELECT output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc surveyselect\n",
    "        data=analysis_data\n",
    "        out=bootstrap_samples\n",
    "        seed=1354687\n",
    "        method=urs\n",
    "        outhits\n",
    "        rep=500\n",
    "        samprate=1\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Generate Bootstrap Samples (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Train Models in Each Bootstrap using PROC PHREG\n",
    "\n",
    "See page 17 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. (Depending on the version of the SAS Kernel being used by Jupyter, you may see either empty output or the SAS log.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* turn off all output;\n",
    "ods graphics off;\n",
    "ods exclude all;\n",
    "ods noresults;\n",
    "\n",
    "ods output concordance=bootstrap_association_table(\n",
    "    keep=replicate estimate\n",
    "    rename=(estimate=c_statistic_value)\n",
    ");\n",
    "proc phreg data=bootstrap_samples concordance;\n",
    "    by replicate;\n",
    "    class gender;\n",
    "    model lenfol*fstat(0) = gender age;\n",
    "    store coxmodel;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Train Models in Each Bootstrap using PROC PHREG (additional optional step)\n",
    "\n",
    "If the log for the previous cell did not appear, the code below can be used to view it. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n",
    "\n",
    "**Warning**: This log entry will be particularly long since each of the 500 iterations of PROC PHREG produces its own log entries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Test Bootstrap Models\n",
    "\n",
    "See pages 17-18 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. (Depending on the version of the SAS Kernel being used by Jupyter, you may see either empty output or the SAS log.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc plm restore=coxmodel;\n",
    "    score data=analysis_data out=scored_data(keep=replicate id lenfol fstat predicted);\n",
    "run;\n",
    "\n",
    "proc sql noprint;\n",
    "    select count(*)\n",
    "        into :totobs\n",
    "        from analysis_data;\n",
    "quit;\n",
    "\n",
    "data bootstrap_scores(keep=replicate nch ndh pairs c_statistic_value);\n",
    "    set scored_data(\n",
    "        keep=replicate id predicted lenfol fstat\n",
    "        where=(fstat=1)\n",
    "        rename=(id=idn_i predicted=y_i lenfol=x_i)\n",
    "    );\n",
    "    by replicate;\n",
    "    if first.replicate then do;\n",
    "        nch = 0;\n",
    "        ndh = 0;\n",
    "        pairs = 0;\n",
    "    end;\n",
    "    do i=1+((replicate-1)*&totobs) to replicate*&totobs;\n",
    "        set scored_data(\n",
    "            keep=id predicted lenfol\n",
    "            rename=(id=idn_j predicted=y_j lenfol=x_j)\n",
    "        ) point=i;\n",
    "        if idn_i NE idn_j and x_i < x_j then do;\n",
    "            if y_i > y_j then\n",
    "                nch + 1;\n",
    "            else if y_i EQ y_j then\n",
    "                nch + 0.5;\n",
    "            else if x_i < x_j then\n",
    "                ndh + 1;\n",
    "            pairs + 1;\n",
    "        end;\n",
    "    end;\n",
    "    if last.replicate then do;\n",
    "        c_statistic_value=nch/pairs;\n",
    "        output;\n",
    "    end;\n",
    "run;\n",
    "\n",
    "* turn output back on;\n",
    "ods results;\n",
    "ods select all;\n",
    "ods graphics on;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Test Bootstrap Models (additional optional step)\n",
    "\n",
    "If the log for the previous cell did not appear, the code below can be used to view it. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Estimate Optimism\n",
    "\n",
    "See page 8 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc sql;\n",
    "    create table model_optimism as\n",
    "        select\n",
    "            avg(A.c_statistic_value - B.c_statistic_value) as optimism\n",
    "        from\n",
    "            bootstrap_association_table as A\n",
    "            inner join\n",
    "            bootstrap_scores as B\n",
    "            on A.Replicate = B.Replicate\n",
    "    ;\n",
    "quit;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Adjust Performance with Optimism\n",
    "\n",
    "See page 9 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC PRINT output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* assemble C-statistic for original model, optimism, and corrected C-statistic into a 1x3 table;\n",
    "data corrected_model_evaluation;\n",
    "    set model_association_table;\n",
    "    set model_optimism;\n",
    "    \n",
    "    corrected_c_statistic = original_model_c_statistic - optimism;\n",
    "\n",
    "    label\n",
    "        original_model_c_statistic = 'Naive C-Statistic'\n",
    "        optimism = 'Optimism'\n",
    "        corrected_c_statistic = 'Optimism-Corrected C-Statistic'\n",
    "    ;\n",
    "    \n",
    "    keep original_model_c_statistic optimism corrected_c_statistic;\n",
    "run;\n",
    "\n",
    "* print final results;\n",
    "proc print\n",
    "        data=corrected_model_evaluation\n",
    "        noobs\n",
    "        label\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Adjust Performance with Optimism (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SAS",
   "language": "sas",
   "name": "sas"
  },
  "language_info": {
   "codemirror_mode": "sas",
   "file_extension": ".sas",
   "mimetype": "text/x-sas",
   "name": "sas"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}