GPT_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GPT-2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Adminixtrator/gpt-2/blob/master/GPT_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u1z7jCKQ2Hri",
        "colab_type": "text"
      },
      "source": [
        "# Calling file from Repository\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LMrNoXGt2OSp",
        "colab_type": "code",
        "outputId": "cf2a4a44-9059-4420-9923-b3003968588d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "!git clone https://github.com/adminixtrator/gpt-2.git\n",
        "\n",
        "%cd gpt-2\n",
        "%ls"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'gpt-2'...\n",
            "remote: Enumerating objects: 19, done.\u001b[K\n",
            "remote: Counting objects:   5% (1/19)\u001b[K\rremote: Counting objects:  10% (2/19)\u001b[K\rremote: Counting objects:  15% (3/19)\u001b[K\rremote: Counting objects:  21% (4/19)\u001b[K\rremote: Counting objects:  26% (5/19)\u001b[K\rremote: Counting objects:  31% (6/19)\u001b[K\rremote: Counting objects:  36% (7/19)\u001b[K\rremote: Counting objects:  42% (8/19)\u001b[K\rremote: Counting objects:  47% (9/19)\u001b[K\rremote: Counting objects:  52% (10/19)\u001b[K\rremote: Counting objects:  57% (11/19)\u001b[K\rremote: Counting objects:  63% (12/19)\u001b[K\rremote: Counting objects:  68% (13/19)\u001b[K\rremote: Counting objects:  73% (14/19)\u001b[K\rremote: Counting objects:  78% (15/19)\u001b[K\rremote: Counting objects:  84% (16/19)\u001b[K\rremote: Counting objects:  89% (17/19)\u001b[K\rremote: Counting objects:  94% (18/19)\u001b[K\rremote: Counting objects: 100% (19/19)\u001b[K\rremote: Counting objects: 100% (19/19), done.\u001b[K\n",
            "remote: Compressing objects:   5% (1/17)\u001b[K\rremote: Compressing objects:  11% (2/17)\u001b[K\rremote: Compressing objects:  17% (3/17)\u001b[K\rremote: Compressing objects:  23% (4/17)\u001b[K\rremote: Compressing objects:  29% (5/17)\u001b[K\rremote: Compressing objects:  35% (6/17)\u001b[K\rremote: Compressing objects:  41% (7/17)\u001b[K\rremote: Compressing objects:  47% (8/17)\u001b[K\rremote: Compressing objects:  52% (9/17)\u001b[K\rremote: Compressing objects:  58% (10/17)\u001b[K\rremote: Compressing objects:  64% (11/17)\u001b[K\rremote: Compressing objects:  70% (12/17)\u001b[K\rremote: Compressing objects:  76% (13/17)\u001b[K\rremote: Compressing objects:  82% (14/17)\u001b[K\rremote: Compressing objects:  88% (15/17)\u001b[K\rremote: Compressing objects:  94% (16/17)\u001b[K\rremote: Compressing objects: 100% (17/17)\u001b[K\rremote: Compressing objects: 100% (17/17), done.\u001b[K\n",
            "remote: Total 19 (delta 2), reused 0 (delta 0), pack-reused 0\u001b[K\n",
            "Unpacking objects:   5% (1/19)   \rUnpacking objects:  10% (2/19)   \rUnpacking objects:  15% (3/19)   \rUnpacking objects:  21% (4/19)   \rUnpacking objects:  26% (5/19)   \rUnpacking objects:  31% (6/19)   \rUnpacking objects:  36% (7/19)   \rUnpacking objects:  42% (8/19)   \rUnpacking objects:  47% (9/19)   \rUnpacking objects:  52% (10/19)   \rUnpacking objects:  57% (11/19)   \rUnpacking objects:  63% (12/19)   \rUnpacking objects:  68% (13/19)   \rUnpacking objects:  73% (14/19)   \rUnpacking objects:  78% (15/19)   \rUnpacking objects:  84% (16/19)   \rUnpacking objects:  89% (17/19)   \rUnpacking objects:  94% (18/19)   \rUnpacking objects: 100% (19/19)   \rUnpacking objects: 100% (19/19), done.\n",
            "/content/gpt-2\n",
            "Dockerfile.cpu  domains.txt        LICENSE        README.md         \u001b[0m\u001b[01;34msrc\u001b[0m/\n",
            "Dockerfile.gpu  download_model.py  model_card.md  requirements.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K4KGursR0rdo",
        "colab_type": "text"
      },
      "source": [
        "# Using the gpt-2 model 345M"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UgycuUVz0llH",
        "colab_type": "code",
        "outputId": "efbf207a-c1f0-4913-c7e9-9f82223a56ef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "#Download the gpt-2 model 345M..\n",
        "!python3 download_model.py 345M"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rFetching checkpoint:   0%|                                              | 0.00/77.0 [00:00<?, ?it/s]\rFetching checkpoint: 1.00kit [00:00, 577kit/s]                                                      \n",
            "\rFetching encoder.json:   0%|                                           | 0.00/1.04M [00:00<?, ?it/s]\rFetching encoder.json: 1.04Mit [00:00, 34.2Mit/s]                                                   \n",
            "Fetching hparams.json: 1.00kit [00:00, 646kit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 1.42Git [00:29, 48.1Mit/s]                                 \n",
            "Fetching model.ckpt.index: 11.0kit [00:00, 6.46Mit/s]                                               \n",
            "Fetching model.ckpt.meta: 927kit [00:00, 30.3Mit/s]                                                 \n",
            "Fetching vocab.bpe: 457kit [00:00, 22.6Mit/s]                                                       \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wMeHjf3X1x_j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Encoding..\n",
        "!export PYTHONIOENCODING=UTF-8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Une1DBFK6k1-",
        "colab_type": "text"
      },
      "source": [
        "# Now to Implementing gpt-2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "51fzEHKv5sl5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Changing directory..\n",
        "import os\n",
        "\n",
        "os.chdir('src')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VQuw66yc668Z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Importing the necessary libraries..\n",
        "\n",
        "import json\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import model, sample, encoder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jkuOzFcyCqr1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Function to use the interaction model..\n",
        "\n",
        "def interact_model(model_name, seed, nsamples, batch_size, length, temperature, top_k, models_dir):\n",
        "  \n",
        "    models_dir = os.path.expanduser(os.path.expandvars(models_dir))\n",
        "    if batch_size is None:\n",
        "        batch_size = 1\n",
        "    assert nsamples % batch_size == 0\n",
        "\n",
        "    enc = encoder.get_encoder(model_name, models_dir)\n",
        "    hparams = model.default_hparams()\n",
        "    with open(os.path.join(models_dir, model_name, 'hparams.json')) as f:\n",
        "        hparams.override_from_dict(json.load(f))\n",
        "\n",
        "    if length is None:\n",
        "        length = hparams.n_ctx // 2\n",
        "    elif length > hparams.n_ctx:\n",
        "        raise ValueError(\"Can't get samples longer than window size: %s\" % hparams.n_ctx)\n",
        "\n",
        "    with tf.Session(graph=tf.Graph()) as sess:\n",
        "        context = tf.placeholder(tf.int32, [batch_size, None])\n",
        "        np.random.seed(seed)\n",
        "        tf.set_random_seed(seed)\n",
        "        output = sample.sample_sequence(hparams=hparams, length=length, context=context, batch_size=batch_size, temperature=temperature, top_k=top_k)\n",
        "\n",
        "        saver = tf.train.Saver(save_relative_paths=True)\n",
        "        ckpt = tf.train.latest_checkpoint(os.path.join(models_dir, model_name))\n",
        "        saver.restore(sess, ckpt)\n",
        "\n",
        "        while True:\n",
        "            raw_text = input(\"\\nModel prompt >>> \")\n",
        "            if raw_text == 'ADMIN_NIXTRATOR':\n",
        "              raw_text = False\n",
        "              break\n",
        "            while not raw_text:\n",
        "                print('\\nPrompt should not be empty!')\n",
        "                raw_text = input(\"\\nModel prompt >>> \")\n",
        "            context_tokens = enc.encode(raw_text)\n",
        "            generated = 0\n",
        "            for _ in range(nsamples // batch_size):\n",
        "                out = sess.run(output, feed_dict={\n",
        "                    context: [context_tokens for _ in range(batch_size)]\n",
        "                })[:, len(context_tokens):]\n",
        "                for i in range(batch_size):\n",
        "                    generated += 1\n",
        "                    text = enc.decode(out[i])\n",
        "                    print(\"=\" * 40 + \" SAMPLE \" + str(generated) + \" \" + \"=\" * 40)\n",
        "                    print(text)\n",
        "            print(\"=\" * 80)\n",
        "            \n",
        "            "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WekJLJV_HEs9",
        "colab_type": "text"
      },
      "source": [
        "# **Code Explanation**\n",
        "\n",
        "## **model_name**: \n",
        " This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights\n",
        "\n",
        "## **seed**: \n",
        "Integer seed for random number generators, fix seed to reproduce results\n",
        "\n",
        "## **nsamples**: \n",
        "This represents the number of sample texts generated in our output\n",
        "\n",
        "## **batch_size**: \n",
        "This only affects speed/memory. This must also divide nsamples\n",
        "\n",
        "*Note: To generate more than one sample, you need to change the values of both nsamples and batch_size and also have to keep them equal.*\n",
        "\n",
        "## **length**: \n",
        "It represents the number of tokens in the generated text. If the length is None, then the number of tokens is decided by model hyperparameters\n",
        "\n",
        "## **temperature**: \n",
        "This controls randomness in Boltzmann distribution. Lower temperature results in less random completions. As the temperature approaches zero, the model will become deterministic and repetitive. Higher temperature results in more random completions\n",
        "\n",
        "## **top_k**:  \n",
        "This parameter controls diversity. If the value of top_k is set to 1, this means that only 1 word is considered for each step (token). If top_k is set to 40, that means 40 words are considered at each step. 0 (default) is a special setting meaning no restrictions. top_k = 40 generally is a good value\n",
        "\n",
        "## **models_dir**: \n",
        "It represents the path to parent folder containing model subfolders (contains the <model_name> folder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pPLy_MvMIjJy",
        "colab_type": "text"
      },
      "source": [
        "# Results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n2ikkNd_Gl4t",
        "colab_type": "code",
        "outputId": "8ecf9f12-697d-44cb-ec49-887b6e401d99",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377
        }
      },
      "source": [
        "#Using the arguements above..\n",
        "\n",
        "interact_model('345M', None, 1, 1, 100, 1, 0, '/content/gpt-2/models')"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /content/gpt-2/models/345M/model.ckpt\n",
            "\n",
            "Model prompt >>> Yesterday was my birthday\n",
            "======================================== SAMPLE 1 ========================================\n",
            ".\n",
            "\n",
            "So, obviously, I've been living with this abandonment ever since I relapsed about five years ago, and my boyfriend (who was recovering from being on anti-depressants) once tried to fill in the blanks after my abandoned sister found out about the realtered pattern: \"If only I had somehow gotten pregnant when I was 7, 6, 5, 4…\" with cute fake prosthetic babies, all for the sake of making my life better! I knew getting a\n",
            "================================================================================\n",
            "\n",
            "Model prompt >>> He asked me out and\n",
            "======================================== SAMPLE 1 ========================================\n",
            " I said no!\", she tells Maryanne. \"For the next twelve years I was in a relationship with someone else. I was really content to remain single, keeping everything to myself. I think my anxiety has managed to overwhelm me with thoughts of one day never telling anyone and claiming I was married for the rest of my life. I never told my dad. When I went to museums I felt like 'someone has really intimidated me', I didn't want to be public about me, we just keep\n",
            "================================================================================\n",
            "\n",
            "Model prompt >>> I was up all night building this model\n",
            "======================================== SAMPLE 1 ========================================\n",
            " on foam and somehow to your advantage. Also this pore troll is well made! Anyways so beginning day of- course i knew i want something liveable so everyone can make sure they get thru the texture with placement alright so i made this as fast a wet paint can do the job lol. View this post on instagram! My friend was working at Starbucks earlier, as we were both busy, he helped me make this. I wanted something that looked nicer and more convenient by having a\n",
            "================================================================================\n",
            "\n",
            "Model prompt >>> ADMIN_NIXTRATOR\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2mX9jZR4umF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wCmM6nEe8JAp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}