3D-CNN_LUNGS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Determining when lesions in the lungs are cancerous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the United States, lung cancer strikes 225,000 people every year, and accounts for $12 billion in health care costs. Early detection is critical to give patients the best chance at recovery and survival.\n",
    "\n",
    "This project is crucial with regards to research in the field of medical biology."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will dramatically reduce the false positive rate that plagues the current detection technology, get patients earlier access to life-saving interventions, and give radiologists more **time** to spend with their patients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, you are given over a thousand low-dose CT images from high-risk patients in DICOM format. Each image contains a series with multiple axial slices of the chest cavity. Each image has a variable number of 2D slices, which can vary based on the machine taking the scan and patient.\n",
    "\n",
    "The DICOM files have a header that contains the necessary information about the patient id, as well as scan parameters such as the slice thickness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the modules and data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved - 0\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 100\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 200\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 300\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 400\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 500\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 600\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 700\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 800\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 900\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 1000\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 1100\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 1200\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 1300\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 1400\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Saved - 1500\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n",
      "Data is unlabeled\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import dicom\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import math\n",
    "\n",
    "dataDirectory = 'Lung_Cancer/stage1/stage1/'\n",
    "lungPatients = os.listdir(dataDirectory)\n",
    "labels = pd.read_csv('Lung_Cancer/stage1_labels/stage1_labels.csv', index_col=0)\n",
    "size = 50\n",
    "NoSlices = 20\n",
    "\n",
    "\n",
    "def chunks(l, n):\n",
    "    count = 0\n",
    "    for i in range(0, len(l), n):\n",
    "        if (count < NoSlices):\n",
    "            yield l[i:i + n]\n",
    "            count = count + 1\n",
    "\n",
    "\n",
    "def mean(l):\n",
    "    return sum(l) / len(l)\n",
    "\n",
    "\n",
    "def dataProcessing(patient, labels_df, size=50, noslices=20, visualize=False):\n",
    "    label = labels_df.get_value(patient, 'cancer')\n",
    "    path = dataDirectory + patient\n",
    "    slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]\n",
    "    slices.sort(key=lambda x: int(x.ImagePositionPatient[2]))\n",
    "\n",
    "    new_slices = []\n",
    "    slices = [cv2.resize(np.array(each_slice.pixel_array), (size, size)) for each_slice in slices]\n",
    "\n",
    "    chunk_sizes = math.floor(len(slices) / noslices)\n",
    "    for slice_chunk in chunks(slices, chunk_sizes):\n",
    "        slice_chunk = list(map(mean, zip(*slice_chunk)))\n",
    "        new_slices.append(slice_chunk)\n",
    "\n",
    "    if label == 1:\n",
    "        label = np.array([0, 1])\n",
    "    elif label == 0:\n",
    "        label = np.array([1, 0])\n",
    "    return np.array(new_slices), label\n",
    "\n",
    "\n",
    "imageData = []\n",
    "for num, patient in enumerate(lungPatients):\n",
    "    if num % 100 == 0:\n",
    "        print('Saved -', num)\n",
    "    try:\n",
    "        img_data, label = dataProcessing(patient, labels, size=size, noslices=NoSlices)\n",
    "        imageData.append([img_data, label,patient])\n",
    "    except KeyError as e:\n",
    "        print('Data is unlabeled')\n",
    "\n",
    "np.save('imageDataNew-{}-{}-{}.npy'.format(size, size, NoSlices), imageData)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Network and Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 completed out of 10 loss: 3.88795779023e+13\n",
      "Accuracy: 0.7\n",
      "Epoch 2 completed out of 10 loss: 1.24509060778e+13\n",
      "Accuracy: 0.6\n",
      "Epoch 3 completed out of 10 loss: 6.06095143603e+12\n",
      "Accuracy: 0.67\n",
      "Epoch 4 completed out of 10 loss: 3.88584119686e+12\n",
      "Accuracy: 0.71\n",
      "Epoch 5 completed out of 10 loss: 3.01922889866e+12\n",
      "Accuracy: 0.58\n",
      "Epoch 6 completed out of 10 loss: 1.83550548122e+12\n",
      "Accuracy: 0.48\n",
      "Epoch 7 completed out of 10 loss: 1.58098134931e+12\n",
      "Accuracy: 0.61\n",
      "Epoch 8 completed out of 10 loss: 1.02996348973e+12\n",
      "Accuracy: 0.67\n",
      "Epoch 9 completed out of 10 loss: 845322210816.0\n",
      "Accuracy: 0.52\n",
      "Epoch 10 completed out of 10 loss: 613924536640.0\n",
      "Accuracy: 0.6\n",
      "Final Accuracy: 0.61\n",
      "Patient:  dc1ecce5e7f8a4be9082cb5650fa62bd\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  dc5cd907d9de1ed0609832f5bf1fc6e2\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  dc66d11755fd073a59743d0df6b62ee2\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  dc9854bcdcc71b690d9806438009001d\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  dcb426dd025b609489c8f520d6d644b7\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  dcde02d4757bb845376fa6dbb0351df6\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  dcdf0b64b314e08e8f71f3bec9ecb080\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  dcf5fd36b9fff9183f63df286bf8eef9\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  dcf75f484b2d2712e5033ba61fd6e2a0\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  dd281294b34eb6deb76ef9f38169d50e\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  dd571c3949cdae0b59fc0542bb23e06a\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  dd5764803d51c71a27707d9db8c84aac\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  de04fbf5e6c2389f0d039398bdcda971\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  de4d3724030397e71d2ac2ab16df5fba\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  de635c85f320131ee743733bb04e65b9\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  de881c07adc8d53e52391fac066ccb9f\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  de9f65a7a70b73ce2ffef1e4a2613eee\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  df015da931ad5312ee7b24b201b67478\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  df1354de25723c9a55e1241d4c40ffe2\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  df54dc42705decd3f75ec8fd8040e76e\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  df75d5a21b4289e8df6e2d0e135ac48f\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  df761dd787bfc439890740ccce934f36\n",
      "Actual:  Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  df8614fd49a196123c5b88584dd5dd65\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e00832e96709eb85f8e0e608ca02c2b5\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e10c2b829c39d4a500c09caf04d461a1\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e127111e994be5f79bb0cea52c9d563e\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e129305f6d074d08cd2de0ebdfeaa576\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e1584618a0c72f124fe618e1ed9b3e55\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  e163325ccf00afde107c80dfce2bce80\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e188bdeea72bb41d980dc2556dc8aafa\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e1c92d3f85a37bd8bb963345b6d66e03\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e1e47812eecd80466cf7f5b0160de446\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e1f3a01e73d706b7e9c30c0a17a4c0b5\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e2a7eaebd0830061e77690aa48f11936\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e2b7fe7fbb002029640c0e65e3051888\n",
      "Actual:  Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  e2bcbfe1ab0f9ddc5d6234f819cd5df5\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e2ea2f046495909ff89e18e05f710fee\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e3034ac9c2799b9a9cee2111593d9853\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e3423505ef6b43f03c5d7bde52a5a78c\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e38789c5eabb3005bfb82a5298055ba0\n",
      "Actual:  Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  e3a9a6f8d21c6c459728066bcf18c615\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e3e518324e1a85b85f15d9127ed9ea89\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e414153d0e52f70cbe27c129911445a0\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e42815372aa308f5943847ad06f529de\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e43afa905c8e279f818b2d5104f6762b\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e4421d2d5318845c1cccbc6fa308a96e\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e4436b5914162ff7efea2bdfb71c19ae\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  e46973b13a7a6f421430d81fc1dda970\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e4a87107f94e4a8e32b735d18cef1137\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e4ff18b33b7110a64f497e177102f23d\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e537c91cdfa97d20a39df7ef04a52570\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e5438d842118e579a340a78f3c5775cc\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e54b574a7e7c650edc224cbdede9e675\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e56b9f25a47a42f4ae4085005c46109c\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e572e978c2b50aca781e6302937e5b13\n",
      "Actual:  Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  e58b78dc31d80a50285816f4ecd661e3\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e58cc57cab8a1738041b72b156fedc56\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e5c68cfa0f33540da3098800f0daae2c\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e5cf847e616cc2fe94816ffa547d2614\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e608c0e6cf3adf3c9939593a3c322ef7\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e6214ef879c6d01ae598161e50e23c0c\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e63f43056330bc418a11208aa3a9e7f0\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e659f6517c4df17e86d4d87181396ea6\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e67bc6cd24a71a486b626592d591a2da\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e6b3e750c6c7a70ca512d77defcfe615\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e6d4a747235bfcc1feac759571c8485c\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e6d8b2631843a24e6761f2723ea30788\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e6f4757b8f315f31559c5c256cb8dead\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e709901da9ba15a95d4a29906edc01dd\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e787e5fd289a9f1f6bba31569b7ad384\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e79f52e833ccca893509f0fdeeb26e9f\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e7adb2e4409683b9490e34b6b3604d9e\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  e7cb27a5362a7098e1437bfb1224d2dc\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e7d76f0723911280b64f0f83a4990c97\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e858263b89f0bb57597bcff325eaeecf\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e8be143b9f5e352f71043b24f79f5a17\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e8eb842ee04bbad407f85fe671f24d4f\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e92a2ed80510513497d5252b001cfa3e\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e977737394cee9abb19ad07310aae8eb\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  e9ccf1ce85c39779fafb9ec703c71555\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  ea7373271a2441b5864df2053c0f5c3e\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  eacb38abacf1214f3b456b6c9fa78697\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  ead64f9269f2200e1d439960a1e069b4\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  eaf753dc137e12fd06e96d27f3111043\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  eb008af181f3791fdce2376cf4773733\n",
      "Actual:  Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  eb8d5136918d6859ca3cc3abafe369ac\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  eba18d04b18084ef64be8f22bb7905ca\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  eba4bfb93928d424ff21b5be96b5c09b\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  ebd601d40a18634b100c92e7db39f585\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  ed0f3c1619b2becec76ba5df66e1ea56\n",
      "Actual:  Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  ed49b57854f5580658fb3510676e03dd\n",
      "Actual:  Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  ed83b655a1bbad40a782ad13cf27ce8f\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  eda58f4918c4b506cd156702bf8a56a3\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  edad1a7e85b5443e0ae9e654d2adbcba\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  edae2e1edd1217d0c9e20eff2a7b2dd8\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  edbf53a8478049de1494b213fdf942e6\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  ee71210fa398cbb080f6c537a503e806\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  ee88217bee233a3bfc971b450e3d8b85\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Patient:  ee984e8fba88691aac4992fbb14f6e97\n",
      "Actual:  No Cancer\n",
      "Predcited:  Cancer\n",
      "Patient:  ee9c580272cd02741df7299892602ac7\n",
      "Actual:  No Cancer\n",
      "Predcited:  No Cancer\n",
      "Predicted   0   1\n",
      "Actual           \n",
      "0          56  17\n",
      "1          18   9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ashwa\\Anaconda3\\envs\\tensorflow-gpu\\lib\\site-packages\\matplotlib\\collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  if self._edgecolors == str('face'):\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import tflearn\n",
    "from tflearn.layers.conv import conv_3d, max_pool_3d\n",
    "from tflearn.layers.core import input_data, dropout, fully_connected\n",
    "from tflearn.layers.estimator import regression\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "imageData = np.load('imageDataNew-50-50-20.npy')\n",
    "trainingData = imageData[0:800]\n",
    "validationData = imageData[-200:-100]\n",
    "\n",
    "x = tf.placeholder('float')\n",
    "y = tf.placeholder('float')\n",
    "size = 50\n",
    "keep_rate = 0.8\n",
    "NoSlices = 20\n",
    "\n",
    "\n",
    "def convolution3d(x, W):\n",
    "    return tf.nn.conv3d(x, W, strides=[1, 1, 1, 1, 1], padding='SAME')\n",
    "\n",
    "\n",
    "def maxpooling3d(x):\n",
    "    return tf.nn.max_pool3d(x, ksize=[1, 2, 2, 2, 1], strides=[1, 2, 2, 2, 1], padding='SAME')\n",
    "\n",
    "\n",
    "def cnn(x):\n",
    "    x = tf.reshape(x, shape=[-1, size, size, NoSlices, 1])\n",
    "    convolution1 = tf.nn.relu(\n",
    "        convolution3d(x, tf.Variable(tf.random_normal([3, 3, 3, 1, 32]))) + tf.Variable(tf.random_normal([32])))\n",
    "    convolution1 = maxpooling3d(convolution1)\n",
    "    convolution2 = tf.nn.relu(\n",
    "        convolution3d(convolution1, tf.Variable(tf.random_normal([3, 3, 3, 32, 64]))) + tf.Variable(\n",
    "            tf.random_normal([64])))\n",
    "    convolution2 = maxpooling3d(convolution2)\n",
    "    convolution3 = tf.nn.relu(\n",
    "        convolution3d(convolution2, tf.Variable(tf.random_normal([3, 3, 3, 64, 128]))) + tf.Variable(\n",
    "            tf.random_normal([128])))\n",
    "    convolution3 = maxpooling3d(convolution3)\n",
    "    convolution4 = tf.nn.relu(\n",
    "        convolution3d(convolution3, tf.Variable(tf.random_normal([3, 3, 3, 128, 256]))) + tf.Variable(\n",
    "            tf.random_normal([256])))\n",
    "    convolution4 = maxpooling3d(convolution4)\n",
    "    convolution5 = tf.nn.relu(\n",
    "        convolution3d(convolution4, tf.Variable(tf.random_normal([3, 3, 3, 256, 512]))) + tf.Variable(\n",
    "            tf.random_normal([512])))\n",
    "    convolution5 = maxpooling3d(convolution4)\n",
    "    fullyconnected = tf.reshape(convolution5, [-1, 1024])\n",
    "    fullyconnected = tf.nn.relu(\n",
    "        tf.matmul(fullyconnected, tf.Variable(tf.random_normal([1024, 1024]))) + tf.Variable(tf.random_normal([1024])))\n",
    "    fullyconnected = tf.nn.dropout(fullyconnected, keep_rate)\n",
    "    output = tf.matmul(fullyconnected, tf.Variable(tf.random_normal([1024, 2]))) + tf.Variable(tf.random_normal([2]))\n",
    "    return output\n",
    "\n",
    "\n",
    "def network(x):\n",
    "    prediction = cnn(x)\n",
    "    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y))\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(cost)\n",
    "    epochs = 10\n",
    "    with tf.Session() as session:\n",
    "        session.run(tf.global_variables_initializer())\n",
    "        for epoch in range(epochs):\n",
    "            epoch_loss = 0\n",
    "            for data in trainingData:\n",
    "                try:\n",
    "                    X = data[0]\n",
    "                    Y = data[1]\n",
    "                    _, c = session.run([optimizer, cost], feed_dict={x: X, y: Y})\n",
    "                    epoch_loss += c\n",
    "                except Exception as e:\n",
    "                    pass\n",
    "\n",
    "            correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))\n",
    "           # if tf.argmax(prediction, 1) == 0:\n",
    "            accuracy = tf.reduce_mean(tf.cast(correct, 'float'))\n",
    "            print('Epoch', epoch + 1, 'completed out of', epochs, 'loss:', epoch_loss)\n",
    "            # print('Correct:',correct.eval({x:[i[0] for i in validationData], y:[i[1] for i in validationData]}))\n",
    "            print('Accuracy:', accuracy.eval({x: [i[0] for i in validationData], y: [i[1] for i in validationData]}))\n",
    "        print('Final Accuracy:', accuracy.eval({x: [i[0] for i in validationData], y: [i[1] for i in validationData]}))\n",
    "        patients = []\n",
    "        actual = []\n",
    "        predicted = []\n",
    "\n",
    "        finalprediction = tf.argmax(prediction, 1)\n",
    "        actualprediction = tf.argmax(y, 1)\n",
    "        for i in range(len(validationData)):\n",
    "            patients.append(validationData[i][2])\n",
    "        for i in finalprediction.eval({x: [i[0] for i in validationData], y: [i[1] for i in validationData]}):\n",
    "            if(i==1):\n",
    "                predicted.append(\"Cancer\")\n",
    "            else:\n",
    "                predicted.append(\"No Cancer\")\n",
    "        for i in actualprediction.eval({x: [i[0] for i in validationData], y: [i[1] for i in validationData]}):\n",
    "            if(i==1):\n",
    "                actual.append(\"Cancer\")\n",
    "            else:\n",
    "                actual.append(\"No Cancer\")\n",
    "        for i in range(len(patients)):\n",
    "            print(\"Patient: \",patients[i])\n",
    "            print(\"Actual: \", actual[i])\n",
    "            print(\"Predcited: \", predicted[i])\n",
    "\n",
    "        from sklearn.metrics import confusion_matrix\n",
    "        y_actual = pd.Series(\n",
    "            (actualprediction.eval({x: [i[0] for i in validationData], y: [i[1] for i in validationData]})),\n",
    "            name='Actual')\n",
    "        y_predicted = pd.Series(\n",
    "            (finalprediction.eval({x: [i[0] for i in validationData], y: [i[1] for i in validationData]})),\n",
    "            name='Predicted')\n",
    "        df_confusion = pd.crosstab(y_actual, y_predicted)\n",
    "        print(df_confusion)\n",
    "\n",
    "        def plot_confusion_matrix(df_confusion, title='Confusion matrix', cmap=plt.cm.gray_r):\\\n",
    "            \n",
    "            plt.matshow(df_confusion, cmap=cmap)  # imshow  \n",
    "            # plt.title(title)\n",
    "            plt.colorbar()\n",
    "            tick_marks = np.arange(len(df_confusion.columns))\n",
    "            plt.xticks(tick_marks, df_confusion.columns, rotation=45)\n",
    "            plt.yticks(tick_marks, df_confusion.index)\n",
    "            # plt.tight_layout()\n",
    "            plt.ylabel(df_confusion.index.name)\n",
    "            plt.xlabel(df_confusion.columns.name)\n",
    "            plt.show()\n",
    "        plot_confusion_matrix(df_confusion)\n",
    "        # print(y_true,y_pred)\n",
    "        # print(confusion_matrix(y_true, y_pred))\n",
    "        # print(actualprediction.eval({x:[i[0] for i in validationData], y:[i[1] for i in validationData]}))\n",
    "        # print(finalprediction.eval({x:[i[0] for i in validationData], y:[i[1] for i in validationData]}))\n",
    "network(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict new patients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Patient:  ffe02fe7d2223743f7fb455dfaff3842\n",
      "Predcited:  No Cancer\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import tflearn\n",
    "from tflearn.layers.conv import conv_3d, max_pool_3d\n",
    "from tflearn.layers.core import input_data, dropout, fully_connected\n",
    "from tflearn.layers.estimator import regression\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "imageData = np.load('imageDataNew-50-50-20.npy')\n",
    "trainingData = imageData[0:800]\n",
    "testData = imageData[-1:]\n",
    "\n",
    "x = tf.placeholder('float')\n",
    "y = tf.placeholder('float')\n",
    "size = 50\n",
    "keep_rate = 0.8\n",
    "NoSlices = 20\n",
    "\n",
    "\n",
    "def convolution3d(x, W):\n",
    "    return tf.nn.conv3d(x, W, strides=[1, 1, 1, 1, 1], padding='SAME')\n",
    "\n",
    "\n",
    "def maxpooling3d(x):\n",
    "    return tf.nn.max_pool3d(x, ksize=[1, 2, 2, 2, 1], strides=[1, 2, 2, 2, 1], padding='SAME')\n",
    "\n",
    "\n",
    "def cnn(x):\n",
    "    x = tf.reshape(x, shape=[-1, size, size, NoSlices, 1])\n",
    "    convolution1 = tf.nn.relu(\n",
    "        convolution3d(x, tf.Variable(tf.random_normal([3, 3, 3, 1, 32]))) + tf.Variable(tf.random_normal([32])))\n",
    "    convolution1 = maxpooling3d(convolution1)\n",
    "    convolution2 = tf.nn.relu(\n",
    "        convolution3d(convolution1, tf.Variable(tf.random_normal([3, 3, 3, 32, 64]))) + tf.Variable(\n",
    "            tf.random_normal([64])))\n",
    "    convolution2 = maxpooling3d(convolution2)\n",
    "    convolution3 = tf.nn.relu(\n",
    "        convolution3d(convolution2, tf.Variable(tf.random_normal([3, 3, 3, 64, 128]))) + tf.Variable(\n",
    "            tf.random_normal([128])))\n",
    "    convolution3 = maxpooling3d(convolution3)\n",
    "    convolution4 = tf.nn.relu(\n",
    "        convolution3d(convolution3, tf.Variable(tf.random_normal([3, 3, 3, 128, 256]))) + tf.Variable(\n",
    "            tf.random_normal([256])))\n",
    "    convolution4 = maxpooling3d(convolution4)\n",
    "    convolution5 = tf.nn.relu(\n",
    "        convolution3d(convolution4, tf.Variable(tf.random_normal([3, 3, 3, 256, 512]))) + tf.Variable(\n",
    "            tf.random_normal([512])))\n",
    "    convolution5 = maxpooling3d(convolution4)\n",
    "    fullyconnected = tf.reshape(convolution5, [-1, 1024])\n",
    "    fullyconnected = tf.nn.relu(\n",
    "        tf.matmul(fullyconnected, tf.Variable(tf.random_normal([1024, 1024]))) + tf.Variable(tf.random_normal([1024])))\n",
    "    fullyconnected = tf.nn.dropout(fullyconnected, keep_rate)\n",
    "    output = tf.matmul(fullyconnected, tf.Variable(tf.random_normal([1024, 2]))) + tf.Variable(tf.random_normal([2]))\n",
    "    return output\n",
    "\n",
    "\n",
    "def network(x):\n",
    "    prediction = cnn(x)\n",
    "    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y))\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(cost)\n",
    "    epochs = 10\n",
    "    with tf.Session() as session:\n",
    "        session.run(tf.global_variables_initializer())\n",
    "        for epoch in range(epochs):\n",
    "            epoch_loss = 0\n",
    "            for data in trainingData:\n",
    "                try:\n",
    "                    X = data[0]\n",
    "                    Y = data[1]\n",
    "                    _, c = session.run([optimizer, cost], feed_dict={x: X, y: Y})\n",
    "                    epoch_loss += c\n",
    "                except Exception as e:\n",
    "                    pass\n",
    "\n",
    "        patients = []\n",
    "        actual = []\n",
    "        predicted = []\n",
    "\n",
    "        finalprediction = tf.argmax(prediction, 1)\n",
    "        actualprediction = tf.argmax(y, 1)\n",
    "        for i in range(len(testData)):\n",
    "            patients.append(testData[i][2])\n",
    "        for i in finalprediction.eval({x: [i[0] for i in testData], y: [i[1] for i in testData]}):\n",
    "            if(i==1):\n",
    "                predicted.append(\"Cancer\")\n",
    "            else:\n",
    "                predicted.append(\"No Cancer\")\n",
    "\n",
    "        for i in range(len(patients)):\n",
    "            print(\"Patient: \",patients[i])\n",
    "            print(\"Predcited: \", predicted[i])\n",
    "\n",
    "     \n",
    "network(x)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}