notebooks/1.01_Overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #ffffff; color: #000000; padding: 10px;\">\n",
    "<img src=\"../media/img/kisz_logo.png\" width=\"192\" height=\"69\"> \n",
    "<h1> NLP Fundamentals\n",
    "<h2> Working with Embeddings\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #f6a800; color: #ffffff; padding: 10px;\">\n",
    "<h2> Part 1.1 - Overview\n",
    "</div>\n",
    "\n",
    "In this section we provide a comprehensive guide to understanding the code organization within the repository and offer general information to help participants navigate the project structure effectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #dd6108; color: #ffffff; padding: 10px;\">\n",
    "<h3>1. Code structure\n",
    "</div>\n",
    "\n",
    "Welcome! If you are reading this, we assume that everything went well installing everything needed for the workshop, and that you can see the folder structure. It should look like this:\n",
    "\n",
    "\n",
    "```\n",
    "├── artifacts                   <- Folder for storing artifacts\n",
    "|\n",
    "├── data                        <- Folder for containing the data\n",
    "│\n",
    "├── envs                        <- Folder for the environments\n",
    "│   └── environment.yml         <- Requirements for reproducing the coding environment\n",
    "│\n",
    "├── media                       <- Pictures and other media files\n",
    "│\n",
    "├── notebooks                   <- Jupyter Notebooks with the content for the workshop\n",
    "│\n",
    "├── references                  <- Recommended materials and references.\n",
    "│\n",
    "├── src                         <- Source code for this workshop.\n",
    "│   ├── __init__.py\n",
    "│   ├── data                    <- Scripts to download or generate data\n",
    "│   ├── embeddings              <- Scripts to download pretrained embeddings\n",
    "│   ├── normalizing             <- Scripts to tokenize text\n",
    "│   ├── plotting                <- Scripts for visualizing vectors\n",
    "│   ├── setup                   <- Scripts for setting everything up\n",
    "│   └── utils                   <- Utility scripts to support the workshop\n",
    "│\n",
    "├── tests                       <- Folder for pytest tests.\n",
    "│\n",
    "├── .gitignore                  <- Specifies files/directories ignored by Git\n",
    "├── CODE_OF_CONDUCT.md          <- Defines ommunity guidelines for good behaviour\n",
    "├── contributing.md             <- Guidelines for collaborating in the workshop\n",
    "├── instructions.md             <- Help For getting ready for the workshop\n",
    "├── Makefile                    <- For automatizing the processes\n",
    "├── Makefile.env                <-      of setting up the environment\n",
    "├── pytest.ini                  <- File for configuring the tests\n",
    "└── README.md                   <- Information about the repository\n",
    "```\n",
    "\n",
    "Take a few moments to familiarise yourself with the structure and feel free to ask anything if you have any questions. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #dd6108; color: #ffffff; padding: 10px;\">\n",
    "<h3>2. Data\n",
    "</div>\n",
    "\n",
    "To ensure a systematic approach to handling the data, we have established a well-defined folder structure. All the data for this workshop will be stored in the <kbd>data</kbd> folder, which contains four subfolders:\n",
    "\n",
    "> 1. **<kbd>raw</kbd> Subfolder**: Here you find the raw, unprocessed data in its original form. This includes text documents, datasets, or any other source material owned by us before any modifications or preprocessing. Use it if you want to try your own data in the workshop.\n",
    "> 2. **<kbd>interim</kbd> Subfolder**: The interim results, or partially processed data, will be stored in this folder. This provides a checkpoint for participants to inspect and understand the transformations applied so far.\n",
    "> 3. **<kbd>processed</kbd> Subfolder**: Here is where traditionally the final, cleaned, and preprocessed data that we want to use for train will reside, but in the context of this workshop, it will house the selected embeddings that we deem the best for our semantic search engine. These embeddings represent the refined input that our algorithms will utilize for semantic search tasks.\n",
    "> 4. **<kbd>external</kbd> Subfolder**: Sometimes, our models might benefit from external resources, such as pre-trained embeddings or additional datasets. The 'external' folder is designated for storing external resources, such as pre-trained embeddings or additional datasets. The data we will be working with is going to be almost exclusively external."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #dd6108; color: #ffffff; padding: 10px;\">\n",
    "<h3>3. Notebooks\n",
    "</div>\n",
    "\n",
    "In the 'Notebooks' folder, you can find a structured set of Jupyter notebooks that will serve as your guide throughout the workshop. These notebooks are categorized into four blocks, each focusing on a specific aspect of understanding and implementing embeddings in Natural Language Processing.\n",
    "\n",
    "> - **Part 1: Introduction**\n",
    ">   - *1.1 Overview*: Provides a comprehensive introduction to the workshop's main concepts and goals\n",
    ">   - *1.2 Metrics*: Explores common evaluation metrics used for comapring embeddings\n",
    ">   - *1.3 Visualization*: Demonstrates techniques for visualizing embeddings\n",
    "> - **Part 2**: Statistical Methods\n",
    ">   - *2.1 Text Normalization*: Covers techniques for normalizing text data, a crucial preprocessing step\n",
    ">   - *2.2 Bag of Words (BoW)*: Introduces the foundational concept of representing text using BoW\n",
    ">   - *2.3 TF-IDF*: Explores Term Frequency-Inverse Document Frequency as a method for text representation\n",
    "> - **Part 3**: Static Embeddings\n",
    ">   - *3.1 Word2Vec*: Dives into the Word2Vec model for generating word embeddings\n",
    ">   - *3.2 GloVe*: Explores the GloVe (Global Vectors for Word Representation) model\n",
    ">   - *3.3 FastText (optional)*: Optional exploration of FastText embeddings for a deeper understanding\n",
    ">   - *3.4 Doc2Vec (optional)*: Optional exploration of Doc2Vec for document-level embeddings\n",
    ">   - *3.5 Word2Vec from Scratch (optional)*: Optional hands-on experience in creating Word2Vec embeddings from scratch\n",
    "> - **Part 4**: Contextual Transformers\n",
    ">   - *4.1 Embeddings with BERT*: Explores embeddings generated by BERT and other transformer models\n",
    ">   - *4.2 SentenceTransformers*: Demonstrates the use of SentenceTransformers\n",
    ">   - *4.3 Indexing Embeddings with Faiss*: Discusses the process of indexing embeddings using Faiss for efficient retrieval."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #dd6108; color: #ffffff; padding: 10px;\">\n",
    "<h3>4. Source code\n",
    "</div>\n",
    "\n",
    "In the <kbd>src</kbd> (source code) folder, you can find a collection of scripts and tools designed to enhance the efficiency and automation of various processes and pipelines throughout the workshop. We encourage participants to take a close look at the code, understand its functionalities, and actively engage by suggesting changes and improvements.\n",
    "\n",
    "We invite participants to actively engage with the source code:\n",
    "> - **Explore**: Familiarize yourself with the existing codebase. Understand the structure, functions, and interactions within the scripts and tools\n",
    "> - **Experiment**: Feel free to experiment with the code. Make changes, tweak parameters, and observe the effects. This hands-on approach will deepen your understanding of the processes\n",
    "> - **Suggest Improvements**: If you identify areas for improvement or have suggestions to enhance the code's clarity, efficiency, or functionality, please share them. Your input is invaluable for continuous refinement\n",
    "\n",
    "#### How to Contribute:\n",
    "\n",
    "> - **Open an Issue**: If you come across any bugs, have questions, or wish to discuss broader improvements, feel free to open an issue. This provides a platform for community discussion and resolution\n",
    "> - **Pull Requests**: If you make changes or improvements, consider submitting a pull request. This collaborative approach allows us to benefit from collective expertise\n",
    "> - **Feedback Session**: We'll also hold a dedicated feedback session where participants can discuss their observations, suggestions, and any challenges faced while exploring the source code\n",
    "\n",
    "By actively participating in this aspect of the workshop, you not only gain hands-on experience with real-world code but also contribute to the collective learning experience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #dd6108; color: #ffffff; padding: 10px;\">\n",
    "<h3>5. Tests\n",
    "</div>\n",
    "\n",
    "To ensure the robustness and reliability of the source code in the <kbd>src</kbd> folder, we have incorporated a suite of tests using Pytest. These tests cover critical functionalities, helping to identify and address potential issues early in the development of the materials for the workshop.\n",
    "\n",
    "#### How to Contribute to Tests:\n",
    "\n",
    "> - **Adding New Tests**: If you identify areas that could benefit from additional testing, consider telling us about new test cases\n",
    "> - **Enhancing Existing Tests**: Feel free to tell us how to enhance or modify existing tests to improve coverage or better reflect the intended behavior\n",
    "> - **Reporting Issues**: If you encounter unexpected test failures or issues, please report them. Opening an issue allows for collaborative problem-solving\n",
    "\n",
    "By actively participating in the testing process, you contribute to the overall quality and reliability of this workshop codebase.\n",
    "\n",
    "You can run all tests with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Set the current working directory to the project's root\n",
    "os.chdir('..')\n",
    "\n",
    "# run the tests\n",
    "!pytest -v -p no:warnings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #dd6108; color: #ffffff; padding: 10px;\">\n",
    "<h3>6. Thank You and Enjoy!\n",
    "</div>\n",
    "\n",
    "A big **thank you** for jumping into this workshop! We're thrilled to have you on this learning adventure.\n",
    "\n",
    " As you navigate the world of embeddings, your active involvement is crucial. Feel free to dive into the notebboks and the source code, share your thoughts in discussions, and, most importantly, feel free to open issues with your observations, suggestions, or questions. Your participation not only enriches your learning but also plays a vital role in improving the workshop for everyone.\n",
    " \n",
    " So, thank you once again for being part of this journey. Enjoy the workshop!!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}