HW5.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/liuange/nlp23/blob/main/HW5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/github/dbamman/nlp23/blob/main/HW5/HW5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ],
      "metadata": {
        "id": "ABK6XyQnWmOI"
      },
      "id": "ABK6XyQnWmOI"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0a4f183a"
      },
      "source": [
        "# Homework 5: WordNet\n",
        "\n",
        "In this homework, you will be working with WordNet synsets and exploring methods to align new words (not in WordNet) with an existing synset."
      ],
      "id": "0a4f183a"
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "0adaa568",
        "outputId": "c127b073-0b5f-40e8-d7da-0cf4b8b99ea8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-03-22 06:19:13--  https://people.ischool.berkeley.edu/~dbamman/glove.6B.100d.100K.txt\n",
            "Resolving people.ischool.berkeley.edu (people.ischool.berkeley.edu)... 128.32.78.16\n",
            "Connecting to people.ischool.berkeley.edu (people.ischool.berkeley.edu)|128.32.78.16|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 85951834 (82M) [text/plain]\n",
            "Saving to: ‘glove.6B.100d.100K.txt.1’\n",
            "\n",
            "glove.6B.100d.100K. 100%[===================>]  81.97M  28.6MB/s    in 2.9s    \n",
            "\n",
            "2023-03-22 06:19:16 (28.6 MB/s) - ‘glove.6B.100d.100K.txt.1’ saved [85951834/85951834]\n",
            "\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: sentence_transformers in /usr/local/lib/python3.9/dist-packages (2.2.2)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (4.27.2)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (3.8.1)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (0.14.1+cu116)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (1.2.2)\n",
            "Requirement already satisfied: huggingface-hub>=0.4.0 in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (0.13.3)\n",
            "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (1.13.1+cu116)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (1.24.2)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (0.1.97)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (1.10.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from sentence_transformers) (4.65.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (3.10.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (4.5.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (2.27.1)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (23.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (6.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.9/dist-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.13.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (2022.10.31)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.9/dist-packages (from nltk->sentence_transformers) (1.1.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.9/dist-packages (from nltk->sentence_transformers) (8.1.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from scikit-learn->sentence_transformers) (3.1.0)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.9/dist-packages (from torchvision->sentence_transformers) (8.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (3.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2022.12.7)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (1.26.15)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2.0.12)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]   Package omw-1.4 is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "!wget https://people.ischool.berkeley.edu/~dbamman/glove.6B.100d.100K.txt \n",
        "!pip install sentence_transformers\n",
        "\n",
        "import nltk\n",
        "import math\n",
        "from nltk import word_tokenize\n",
        "from nltk.corpus import wordnet as wn\n",
        "import numpy as np\n",
        "from typing import Callable, List, Tuple, Dict\n",
        "from IPython.display import Markdown, display\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "nltk.download('wordnet')\n",
        "nltk.download('punkt')\n",
        "nltk.download('omw-1.4')\n",
        "\n",
        "def test_print(string):\n",
        "    display(Markdown(string))"
      ],
      "id": "0adaa568"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9c188261"
      },
      "source": [
        "## Preliminaries: WordNet in NLTK"
      ],
      "id": "9c188261"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dd0cba00"
      },
      "source": [
        "NLTK provides a great interface to the WordNet ontology.  Remember that core unit within WordNet is the **synset** (a category of near-synonyms).  A word (like \"blue\") can appear in many different synsets, each corresponding to a distinct *sense* of that word."
      ],
      "id": "dd0cba00"
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "852cf585",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "708009a3-cfc6-4d6c-e967-e5765ebd03a5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Synset('blue.n.01') blue color or pigment; resembling the color of the clear sky in the daytime\n",
            "Synset('blue.n.02') blue clothing\n",
            "Synset('blue.n.03') any organization or party whose uniforms or badges are blue\n",
            "Synset('blue_sky.n.01') the sky as viewed during daylight\n",
            "Synset('bluing.n.01') used to whiten laundry or hair or give it a bluish tinge\n",
            "Synset('amobarbital_sodium.n.01') the sodium salt of amobarbital that is used as a barbiturate; used as a sedative and a hypnotic\n",
            "Synset('blue.n.07') any of numerous small butterflies of the family Lycaenidae\n",
            "Synset('blue.v.01') turn blue\n",
            "Synset('blue.s.01') of the color intermediate between green and violet; having a color similar to that of a clear unclouded sky; - Helen Hunt Jackson\n",
            "Synset('blue.s.02') used to signify the Union forces in the American Civil War (who wore blue uniforms)\n",
            "Synset('gloomy.s.02') filled with melancholy and despondency\n",
            "Synset('blasphemous.s.02') characterized by profanity or cursing\n",
            "Synset('blue.s.05') suggestive of sexual impropriety\n",
            "Synset('aristocratic.s.01') belonging to or characteristic of the nobility or aristocracy\n",
            "Synset('blue.s.07') morally rigorous and strict\n",
            "Synset('blue.s.08') causing dejection\n"
          ]
        }
      ],
      "source": [
        "# get all of the synsets that a specific word belongs to; print their definitions\n",
        "synsets=wn.synsets('blue')\n",
        "for synset in synsets:\n",
        "    print(synset, synset.definition())"
      ],
      "id": "852cf585"
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "PMI-cNNrVoU9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "628fb1e5-e046-45a2-f140-d5fc53ee18de"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Synset('earnest.n.01') something of value given by one person to another to bind a contract\n",
            "Synset('earnest.s.01') characterized by a firm and humorless belief in the validity of your opinions\n",
            "Synset('dear.s.03') earnest\n",
            "Synset('businesslike.s.02') not distracted by anything unrelated to the goal\n"
          ]
        }
      ],
      "source": [
        "# get all of the synsets that a specific word belongs to; print their definitions\n",
        "synsets=wn.synsets('earnest')\n",
        "for synset in synsets:\n",
        "    print(synset, synset.definition())"
      ],
      "id": "PMI-cNNrVoU9"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "28f268c3"
      },
      "source": [
        "Any given synset will likewise contain multiple different words (all near-synonyms of each other)."
      ],
      "id": "28f268c3"
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "5f809838",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cb20db7c-9d14-41f8-dac7-2ef276ef66ac"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gloomy\n",
            "grim\n",
            "blue\n",
            "depressed\n",
            "dispirited\n",
            "down\n",
            "downcast\n",
            "downhearted\n",
            "down_in_the_mouth\n",
            "low\n",
            "low-spirited\n"
          ]
        }
      ],
      "source": [
        "# get all of the words/phrase in a given synset\n",
        "for lemma in wn.synset(\"gloomy.s.02\").lemmas():\n",
        "    print(lemma.name())"
      ],
      "id": "5f809838"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a7a2d1e5"
      },
      "source": [
        "Remember also that one of the powerful things about WordNet is that it places synsets within a hierarchical structure; a given synset has both **hypernyms** (other synsets that it is a subclass of) and **hyponyms** (other synsets that are subclasses of it)."
      ],
      "id": "a7a2d1e5"
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "53b8ff2f"
      },
      "outputs": [],
      "source": [
        "# Functions from http://www.nltk.org/howto/wordnet.html to get *all* of a synset's hyponym/hypernyms\n",
        "\n",
        "hypo=lambda s: s.hyponyms()\n",
        "hyper=lambda s: s.hypernyms()"
      ],
      "id": "53b8ff2f"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "033786e8"
      },
      "source": [
        "Find all of the synsets that are hyponyms of the target synset (descendents in the WordNet hierarchy)"
      ],
      "id": "033786e8"
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "15a5988f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cddae4c7-7da7-4872-f0ee-bca6f9a4bb29"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Synset('azure.n.01'),\n",
              " Synset('dark_blue.n.01'),\n",
              " Synset('greenish_blue.n.01'),\n",
              " Synset('powder_blue.n.01'),\n",
              " Synset('prussian_blue.n.02'),\n",
              " Synset('purplish_blue.n.01'),\n",
              " Synset('steel_blue.n.01'),\n",
              " Synset('ultramarine.n.02')]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "list(wn.synset(\"blue.n.01\").closure(hypo))"
      ],
      "id": "15a5988f"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84b305eb"
      },
      "source": [
        "Find all of the synsets that are hyperyms (ancestors up the tree) of the target synset"
      ],
      "id": "84b305eb"
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "2c22cece",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d1c2050c-033e-49eb-fca4-2ba9e084db80"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Synset('chromatic_color.n.01'),\n",
              " Synset('color.n.01'),\n",
              " Synset('visual_property.n.01'),\n",
              " Synset('property.n.02'),\n",
              " Synset('attribute.n.02'),\n",
              " Synset('abstraction.n.06'),\n",
              " Synset('entity.n.01')]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "list(wn.synset(\"blue.n.01\").closure(hyper))"
      ],
      "id": "2c22cece"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ac1001c"
      },
      "source": [
        "Here's how you can access all of the synsets in WordNet through NLTK (though note executing this may take a while, so it's commented out)."
      ],
      "id": "5ac1001c"
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "449d8b45",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "733e2f54-3eee-4d3a-8c25-de4962876685"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 Synset('able.a.01')\n",
            "1 Synset('unable.a.01')\n",
            "2 Synset('abaxial.a.01')\n",
            "3 Synset('adaxial.a.01')\n",
            "4 Synset('acroscopic.a.01')\n",
            "5 Synset('basiscopic.a.01')\n",
            "6 Synset('abducent.a.01')\n",
            "7 Synset('adducent.a.01')\n",
            "8 Synset('nascent.a.01')\n",
            "9 Synset('emergent.s.02')\n",
            "10 Synset('dissilient.s.01')\n",
            "11 Synset('parturient.s.02')\n"
          ]
        }
      ],
      "source": [
        "for idx, synset in enumerate(wn.all_synsets()):\n",
        "    print(idx, synset)\n",
        "    if (idx > 10): break"
      ],
      "id": "449d8b45"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0b4cc005"
      },
      "source": [
        "## Programming deliverables"
      ],
      "id": "0b4cc005"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MXniwcLlVoVC"
      },
      "source": [
        "### Motivation and setup"
      ],
      "id": "MXniwcLlVoVC"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cbe0f1a"
      },
      "source": [
        "WordNet is a great resource, but one of its downsides is *coverage* -- many of the words in our vocabulary aren't in WordNet, but could conceivably be placed within existing synsets within it.  Your task for this homework is to develop two methods to finding the closest synset for a given new word from Urban Dictionary."
      ],
      "id": "3cbe0f1a"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26301aaf"
      },
      "source": [
        "For the scope of this homework, we're only going to pretend that WordNet only has 12 different synsets within it, as stored in `target_synsets` below. (Though, feel free to use the `wn.all_synsets()` function above if you wanted to explore running it on all of WordNet)."
      ],
      "id": "26301aaf"
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "2f0c3485"
      },
      "outputs": [],
      "source": [
        "target_synsets=[\n",
        "    'spread.n.01', \n",
        "    'formidable.s.01', \n",
        "    'coziness.n.01', \n",
        "    'mutation.n.02', \n",
        "    'kernel.n.03', \n",
        "    'faineant.s.01', \n",
        "    'fund-raise.v.01', \n",
        "    'orientation.n.06', \n",
        "    'inappropriate.a.01', \n",
        "    'stranger.n.02', \n",
        "    'plausibility.n.01', \n",
        "    'sever.v.01'\n",
        "]"
      ],
      "id": "2f0c3485"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WuFHUEz1VoVD"
      },
      "source": [
        "The code below demonstrates how you can access the definitions in each synset:"
      ],
      "id": "WuFHUEz1VoVD"
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "81f2c025",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88a94839-4008-486d-df62-254ddf34a867"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Synset('spread.n.01')\n",
            "\tDefinition: process or result of distributing or extending over a wide expanse of space\n",
            "Synset('formidable.s.01')\n",
            "\tDefinition: extremely impressive in strength or excellence\n",
            "Synset('coziness.n.01')\n",
            "\tDefinition: a state of warm snug comfort\n",
            "Synset('mutation.n.02')\n",
            "\tDefinition: (genetics) any event that changes genetic structure; any alteration in the inherited nucleic acid sequence of the genotype of an organism\n",
            "Synset('kernel.n.03')\n",
            "\tDefinition: the choicest or most essential or most vital part of some idea or experience\n",
            "Synset('faineant.s.01')\n",
            "\tDefinition: disinclined to work or exertion\n",
            "Synset('fund-raise.v.01')\n",
            "\tDefinition: raise money for a cause or project\n",
            "Synset('orientation_course.n.01')\n",
            "\tDefinition: a course introducing a new situation or environment\n",
            "Synset('inappropriate.a.01')\n",
            "\tDefinition: not suitable for a particular occasion etc\n",
            "Synset('stranger.n.02')\n",
            "\tDefinition: an individual that one is not acquainted with\n",
            "Synset('plausibility.n.01')\n",
            "\tDefinition: apparent validity\n",
            "Synset('sever.v.01')\n",
            "\tDefinition: set or keep apart\n"
          ]
        }
      ],
      "source": [
        "for synset in target_synsets:\n",
        "    wn_synset=wn.synset(synset)\n",
        "    print(wn_synset)\n",
        "    print(\"\\tDefinition:\", wn_synset.definition())"
      ],
      "id": "81f2c025"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cff98e02"
      },
      "source": [
        "Here are the words that do not exist in WordNet now but that we want to add.  Each element of the tuple is: \n",
        "```\n",
        "(word, definition)\n",
        "```"
      ],
      "id": "cff98e02"
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "20eae7c7"
      },
      "outputs": [],
      "source": [
        "urban_dictionary_terms: List[Tuple[str, str]]=[\n",
        "    (\"Crowdfunding\", \"the practice of obtaining needed funding (as for a new business) by soliciting contributions from a large number of people especially from the online community\"), \n",
        "    (\"Hygge\", \"a cozy quality that makes a person feel content and comfortable\"), \n",
        "    (\"biohacking\", \"biological experimentation (as by gene editing or the use of drugs or implants) done to improve the qualities or capabilities of living organisms especially by individuals and groups working outside a traditional medical or scientific research environment\"), \n",
        "    (\"TL;DR\", \"a briefly expressed main point or key message that summarizes a longer discussion or explanation\"), \n",
        "    (\"Hellacious\", \"Exceptionally powerful or violent; remarkably good; extremely difficult; extraordinarily large\"), \n",
        "    (\"Unfriend\", \"To remove from one's list of friends (e.g. on a social networking website)\"), \n",
        "    (\"Infodemic\", \"A wide and rapid spread of misinformation through various media, namely social media\"),\n",
        "    (\"Onboarding\", \"The act or process of orienting and training a new employee\"), \n",
        "    (\"Truthiness\", \"something that seems true but isn’t backed up by evidence\"), \n",
        "    (\"Amotivational\", \"Relating to, or characterised by, a lack of motivation\"), \n",
        "    (\"NSFW\", \"Not Safe For Work. Used to describe Internet content generally inappropriate for the typical workplace, i.e., would not be acceptable in the presence of your boss and colleagues\"),\n",
        "    (\"Rando\", \"a person who is not known or recognizable or whose appearance (as in a conversation or narrative) seems unprompted or unwelcome\")\n",
        "]"
      ],
      "id": "20eae7c7"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "890ffff6"
      },
      "source": [
        "### Overview\n",
        "\n",
        "Your task here is to develop two different methods for finding the best matching synset:\n",
        "\n",
        "**Deliverable 1**. Find the WordNet synset with the highest cosine similarity between a) the average GloVe embeddings of its synset definition and b) the average GloVe embeddings of the new word definition.\n",
        "\n",
        "**Deliverable 2**. Find the WordNet synset with the highest cosine similarity between a) the *sentence embedding* of its synset definition and b) the sentence embedding of the new word definition.\n",
        "\n",
        "And evaluate them:\n",
        "\n",
        "**Deliverable 3**. Implement a function `accuracy` that assesses quality of the dictionaries you return from `method_one` and `method_two`."
      ],
      "id": "890ffff6"
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "<font color='red'><b>Please note:</b></font>\n",
        "\n",
        "<font color='red'><b>Do not change function signatures or return variable names</b>; if you believe it's necessary, let us know. \n",
        "\n",
        "<font color='red'><b>Put your solutions between `# BEGIN SOLUTION` and `# END SOLUTION`</b>.</font>"
      ],
      "metadata": {
        "id": "43KjMmuCS3ou"
      },
      "id": "43KjMmuCS3ou"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "afcfbfde"
      },
      "source": [
        "Here is some code for reading in GloVe embeddings:\n"
      ],
      "id": "afcfbfde"
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "6204372b"
      },
      "outputs": [],
      "source": [
        "def read_glove_vectors(filename: str):\n",
        "    vocab_map={}\n",
        "    embeddings=[]\n",
        "    with(open(filename, encoding=\"utf-8\")) as file:\n",
        "        for idx, line in enumerate(file):\n",
        "            cols=line.rstrip().split(\" \")\n",
        "            word=cols[0]\n",
        "            embedding=cols[1:]\n",
        "\n",
        "            embeddings.append(embedding)\n",
        "            vocab_map[word]=idx\n",
        "    \n",
        "    return vocab_map, np.array(embeddings, dtype=\"float\")"
      ],
      "id": "6204372b"
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "90c21c48"
      },
      "outputs": [],
      "source": [
        "glove_vocab_map, glove_embeddings=read_glove_vectors(\"glove.6B.100d.100K.txt\")"
      ],
      "id": "90c21c48"
    },
    {
      "cell_type": "code",
      "source": [
        "# vocab map is a dict with word and idx; use like below to get embedding array\n",
        "# glove_embeddings[glove_vocab_map[\"present\"]]"
      ],
      "metadata": {
        "id": "6_sOPhxCyLpT"
      },
      "id": "6_sOPhxCyLpT",
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f398eaec"
      },
      "source": [
        "Here is some code for loading the sentence transformer package:"
      ],
      "id": "f398eaec"
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "82797d7d"
      },
      "outputs": [],
      "source": [
        "sentence_model=SentenceTransformer('sentence-transformers/all-distilroberta-v1')"
      ],
      "id": "82797d7d"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y2lQks4EVoVH"
      },
      "source": [
        "Let's make sure it's working properly; the cell below should output this:\n",
        "```\n",
        "(768,)\n",
        "```"
      ],
      "id": "Y2lQks4EVoVH"
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "E37FP3NkVoVI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c3a289e-907e-4f96-d3fe-3140871d94b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(768,)\n"
          ]
        }
      ],
      "source": [
        "sentence_vector=sentence_model.encode(\"this is a sentence\")\n",
        "print(sentence_vector.shape)"
      ],
      "id": "E37FP3NkVoVI"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CEciw_iyhWZ3"
      },
      "source": [
        "Here's an implementation of cosine similarity that you will find useful."
      ],
      "id": "CEciw_iyhWZ3"
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "uQU8eoHKhJ4h"
      },
      "outputs": [],
      "source": [
        "def cosine_similarity(one, two):\n",
        "    return np.dot(one, two) / (np.linalg.norm(one) * np.linalg.norm(two))"
      ],
      "id": "uQU8eoHKhJ4h"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "102ce6a9"
      },
      "source": [
        "### Deliverable 1 (TODO). Implement the first method as `method_one` below.\n",
        "\n",
        "As mentioned above, you should: \n",
        "\n",
        "- a) Compute the average GloVe embedding of the Urban Dictionary word definition, and\n",
        "- b) Use cosine similarity to compare it with the average GloVe embedding of the synset definitions. \n",
        "- c) For each UD word, choose the definition that maximizes the cosine similarity with its definition. \n",
        "\n",
        "Your function should return a dictionary mapping each urban dictionary term to a WordNet synset ID, e.g.:\n",
        "\n",
        "```{\n",
        " \"adore\" : \"love.v.01\",\n",
        " \"dripping\" : \"stylish.a.01\"    \n",
        " }\n",
        "```"
      ],
      "id": "102ce6a9"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hm1BFvnYVoVJ"
      },
      "source": [
        "#### Q1 (a). calculate the average GloVe embedding\n",
        "\n",
        "Here are some things you need to do when calculating the average GloVe embedding of a sentence:\n",
        "\n",
        "- Use `nltk.word_tokenize()` to tokenize the sentence.\n",
        "- Treat everything as lowercase.\n",
        "- Skip any tokens which don't appear in the GloVe vocabulary.\n",
        "- Calculate the average value of the embedding vectors, which will be another vector of the same shape.\n",
        "\n",
        "Implement this in `get_average_embedding()` below."
      ],
      "id": "hm1BFvnYVoVJ"
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "4Ejh2j0nVoVJ"
      },
      "outputs": [],
      "source": [
        "def sentence_to_average_glove(sentence): \n",
        "    global glove_embeddings, glove_vocab_map\n",
        "    # BEGIN SOLUTION\n",
        "    tokens = [x.lower() for x in nltk.word_tokenize(sentence) if x.lower() in glove_vocab_map.keys()]\n",
        "    embs = [glove_embeddings[glove_vocab_map[tok]] for tok in tokens]\n",
        "    average_embedding = np.mean(embs, axis=0)\n",
        "    # END SOLUTION\n",
        "    return average_embedding"
      ],
      "id": "4Ejh2j0nVoVJ"
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "A7y4aeOMVoVJ"
      },
      "outputs": [],
      "source": [
        "def public_test_q1a(student_sol_func):\n",
        "    test_str='test sentence here'\n",
        "    q1a=np.array(\n",
        "        [-0.13161   ,  0.30526   ,  0.48708833, -0.01811933, -0.281338  ,\n",
        "        0.2051    , -0.09271333,  0.61335667, -0.86006667,  0.46980667,\n",
        "       -0.3530206 ,  0.34592667, -0.11837333,  0.174864  ,  0.227699  ,\n",
        "        0.022927  ,  0.082307  ,  0.05022667, -0.615689  ,  0.04992267,\n",
        "        0.43998   , -0.32244   ,  0.17315   ,  0.07476863,  0.24882667,\n",
        "        0.20515667,  0.04977933, -0.35719133,  0.046468  ,  0.2846    ,\n",
        "       -0.14525333,  0.17086367, -0.20106   , -0.16981597,  0.24622667,\n",
        "        0.307406  , -0.43141   , -0.0322    , -0.32304733, -0.22139667,\n",
        "       -0.69949667, -0.10295667,  0.58206667, -0.67504667,  0.18417   ,\n",
        "       -0.10986333,  0.59897667, -0.75551   , -0.25218   , -0.59849   ,\n",
        "        0.60331   , -0.4762    ,  0.19722333,  1.26916667, -0.39379333,\n",
        "       -1.79474333, -0.631325  , -0.11708333,  1.6112    ,  0.24849667,\n",
        "       -0.45793   ,  0.36410333, -0.16572   , -0.32961333,  0.45810667,\n",
        "        0.20734   ,  0.27776   ,  0.41938233, -0.49610333, -0.04512333,\n",
        "        0.10130303,  0.09548   ,  0.04292333, -0.24321   ,  0.18565   ,\n",
        "        0.56445   ,  0.05074167, -0.512398  , -0.69699   , -0.19803   ,\n",
        "        0.51716667, -0.28587633, -0.37449667, -0.37457667, -1.43945667,\n",
        "        0.01966433,  0.06207733, -0.20610567, -0.30307667, -0.03681667,\n",
        "       -0.23645   , -0.01583333, -0.07095133,  0.26235   ,  0.2177    ,\n",
        "        0.43546   ,  0.20054333, -0.28842667,  0.11428867,  0.16190333])\n",
        "    np.testing.assert_almost_equal(student_sol_func(test_str), q1a)\n",
        "    test_print('**q1a** passed sanity check!')"
      ],
      "id": "A7y4aeOMVoVJ"
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "CMRweBehVoVK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 46
        },
        "outputId": "371fa79c-4bc9-4f64-b2ce-944c145a99b2"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**q1a** passed sanity check!"
          },
          "metadata": {}
        }
      ],
      "source": [
        "public_test_q1a(sentence_to_average_glove)"
      ],
      "id": "CMRweBehVoVK"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKknHZ2OVoVK"
      },
      "source": [
        "#### Q1 (b). compare similarities\n",
        "\n",
        "Use cosine similarity to compare it with the average GloVe embedding of the synset definitions. \n",
        "\n",
        "- The input should be a) a sentence encoder, here `sentence_to_average_glove()` you just implemented (in q2 we'll be reusing this function, and pass a different encoder function there), b) an Urban Dictionary term, c) its definition, and d) your target WordNet synsets.\n",
        "- You will populate the dictionary `term_similarity_scores` provided; the key will be a WordNet synset and the value the similarity score.\n",
        "- You will need to sort the similairty scores, which will result in a list of key–value tuples.\n",
        "- Unless you pass `silent=True`, the entries with top 3 similarity scores will be printed for your reference."
      ],
      "id": "QKknHZ2OVoVK"
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "m8QcPUCeVoVK"
      },
      "outputs": [],
      "source": [
        "def get_term_similarity_scores(sentence_encoder, urban_dictionary_word, urban_dictionary_definition, target_synsets, silent=False):\n",
        "    global glove_embeddings, glove_vocab_map\n",
        "    term_similarity_scores={}\n",
        "    # BEGIN SOLUTION\n",
        "\n",
        "    # encode sentence from UD term definition --> avg embedding\n",
        "    \n",
        "    ud_dfn = urban_dictionary_definition #.split(\": \"))[1].strip()\n",
        "    # print(ud_dfn)\n",
        "    ud_emb = sentence_encoder(ud_dfn)\n",
        "\n",
        "    # get avg embeddings for target synset's definition\n",
        "    synset_embs = {}\n",
        "    for target in target_synsets:\n",
        "      wn_synset=wn.synset(target)\n",
        "      dfn = wn_synset.definition()\n",
        "        \n",
        "      synset_embs[target] = sentence_encoder(dfn)\n",
        "\n",
        "    # calculate distance betwn embeddings and save to term sim score under synset word as key\n",
        "    for synset, emb in synset_embs.items():\n",
        "      sim_score = cosine_similarity(emb, ud_emb)\n",
        "      term_similarity_scores[synset] = sim_score\n",
        "\n",
        "    # Sort similarity scores (key-value tuples) by value\n",
        "    scores = [x for x in term_similarity_scores.items()]\n",
        "    scores.sort(key=lambda a: a[1], reverse=True)\n",
        "    term_similarity_scores = scores\n",
        "\n",
        "    # END SOLUTION\n",
        "    if not silent:\n",
        "        for k, v in term_similarity_scores[:3]:\n",
        "            print(f\"UD word: {urban_dictionary_word} -- WordNet synset: {k}, similarity:{v}\")\n",
        "    return term_similarity_scores"
      ],
      "id": "m8QcPUCeVoVK"
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "FldVGn_YVoVL"
      },
      "outputs": [],
      "source": [
        "def public_test_q1b(student_sol_func):\n",
        "    q1b_test_term=('language', 'Language: A term in which if you ever curse around Steve, you will hear regularly.')\n",
        "    q1b_test_synsets=['language.n.01', 'natural.s.04'] \n",
        "    np.testing.assert_almost_equal(student_sol_func(sentence_to_average_glove,\n",
        "                                                    q1b_test_term[0], \n",
        "                                                    q1b_test_term[1],\n",
        "                                                    q1b_test_synsets, silent=True)[-1][-1], 0.8387250755283658)\n",
        "        \n",
        "    test_print('**q1b** passed sanity check!')"
      ],
      "id": "FldVGn_YVoVL"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1MHniZXVoVL"
      },
      "source": [
        "#### Sanity check for q1b"
      ],
      "id": "N1MHniZXVoVL"
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "KU_ek0auVoVM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 46
        },
        "outputId": "987cfdf9-b7dd-4c81-ddd2-1cee5ff1dbf5"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**q1b** passed sanity check!"
          },
          "metadata": {}
        }
      ],
      "source": [
        "public_test_q1b(get_term_similarity_scores)"
      ],
      "id": "KU_ek0auVoVM"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DpzB7yMHVoVM"
      },
      "source": [
        "#### Q1 (c). bringing it all together\n",
        "\n",
        "- For each UD word, choose the definition that maximizes the cosine similarity with its definition. \n",
        "\n",
        "- Your function should return a dictionary mapping each urban dictionary term to a WordNet synset ID, e.g.:\n",
        "\n",
        "```{\n",
        " \"adore\" : \"love.v.01\",\n",
        " \"dripping\" : \"stylish.a.01\"    \n",
        " }\n",
        "```"
      ],
      "id": "DpzB7yMHVoVM"
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "5737fbe2"
      },
      "outputs": [],
      "source": [
        "def method_one(sentence_encoder, \n",
        "               urban_dictionary_terms: List[Tuple[str, str]], \n",
        "               target_synsets: List[str]):\n",
        "    \"\"\"\n",
        "    Method 1: an algorithm based on GloVe embeddings that maps each urban dictionary term to a synset ID.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    sentence_encoder : \n",
        "        a function to encode a sentence into word embeddings\n",
        "        For q1, you should pass the function `sentence_to_average_glove` implemented in q1a here\n",
        "    urban_dictionary_terms : List[Tuple[str, str]]\n",
        "        a list of string 2-tuples where the first elements are words, second elements are definitions.\n",
        "    target_synsets : List[str]\n",
        "        a list of synset IDs that the words should be classified into.\n",
        "        You can call `wn.synset(\"<synset ID>\")` to get the synset object.\n",
        "        \n",
        "    Returns\n",
        "    --------\n",
        "    A dictionary mapping each urban dictionary term to a WordNet synet ID, e.g.\n",
        "    `{\"adore\" : \"love.v.01\", \"dripping\" : \"stylish.a.01\"}`\n",
        "\n",
        "    \"\"\"\n",
        "    best_matches={}\n",
        "    for word, definition in urban_dictionary_terms:\n",
        "        # BEGIN SOLUTION\n",
        "        best = get_term_similarity_scores(sentence_encoder, word, definition, target_synsets, silent=True)[0][0]\n",
        "        # print(best)\n",
        "        best_matches[word] = best\n",
        "        # END SOLUTION\n",
        "    return best_matches\n"
      ],
      "id": "5737fbe2"
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "b69452b2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ae9031c8-17ca-43c8-9976-855a7a16d929"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Crowdfunding': 'spread.n.01',\n",
              " 'Hygge': 'stranger.n.02',\n",
              " 'biohacking': 'kernel.n.03',\n",
              " 'TL;DR': 'stranger.n.02',\n",
              " 'Hellacious': 'formidable.s.01',\n",
              " 'Unfriend': 'stranger.n.02',\n",
              " 'Infodemic': 'spread.n.01',\n",
              " 'Onboarding': 'orientation.n.06',\n",
              " 'Truthiness': 'stranger.n.02',\n",
              " 'Amotivational': 'kernel.n.03',\n",
              " 'NSFW': 'stranger.n.02',\n",
              " 'Rando': 'stranger.n.02'}"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "method_one_results=method_one(sentence_to_average_glove, urban_dictionary_terms, target_synsets)\n",
        "method_one_results"
      ],
      "id": "b69452b2"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0b862967"
      },
      "source": [
        "### Deliverable 2 (TODO). Implement your second method as `method_two` below.\n",
        "\n",
        "In this function, you should compute the cosine similarity between the sentence embedding of the UD word definition and those of the synsets, then for each UD word, choose the synset with the highest cosine similarity. \n",
        "\n",
        "For consistency, use the sentence transformer model called `sentence-transformers/all-distilroberta-v1` stored in `sentence_model` from earlier.\n",
        "\n",
        "You *must* add all similarity scores to the list `all_term_similarity_scores` provided.\n",
        "\n",
        "Your function must also return a dictionary mapping each urban dictionary term to a WordNet synset ID, e.g.:\n",
        "\n",
        "```\n",
        "{\n",
        " \"adore\" : \"love.v.01\",\n",
        " \"dripping\" : \"stylish.a.01\"    \n",
        " }\n",
        "```"
      ],
      "id": "0b862967"
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Q2 (a). use sentence transformer to encode sentences"
      ],
      "metadata": {
        "id": "AXBetpKROx3L"
      },
      "id": "AXBetpKROx3L"
    },
    {
      "cell_type": "code",
      "source": [
        "# make sure you have `sentence_model`\n",
        "sentence_model=SentenceTransformer('sentence-transformers/all-distilroberta-v1')"
      ],
      "metadata": {
        "id": "zCqyilA0WQGw"
      },
      "id": "zCqyilA0WQGw",
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Use `sentence_model` to obtain the sentence embedding of the input sentence. You can do this with one line of code."
      ],
      "metadata": {
        "id": "pmmpKjKwXHNN"
      },
      "id": "pmmpKjKwXHNN"
    },
    {
      "cell_type": "code",
      "source": [
        "def sentence_to_sentence_embedding(sentence):\n",
        "    global sentence_model\n",
        "    # BEGIN SOLUTION\n",
        "    \n",
        "    return sentence_model.encode(sentence)\n",
        "\n",
        "    # Comment this line out before you start:\n",
        "    # return NotImplementedError \n",
        "    # END SOLUTION"
      ],
      "metadata": {
        "id": "1Fg4dXe5NdkV"
      },
      "id": "1Fg4dXe5NdkV",
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def public_test_q2a(student_sol_func):\n",
        "    test_str='test sentence here'\n",
        "    q2a=np.array([-1.43969897e-02, -3.58152948e-02,  7.82628136e-04,  6.85324892e-03, 5.31034134e-02, -1.05041144e-02,  3.62984999e-03,  3.53333056e-02, 5.01355790e-02, -4.29266617e-02, -4.22659479e-02, -7.64900371e-02, 4.23901081e-02, -5.56682684e-02, -4.78991754e-02,  8.80383980e-03, 3.55795096e-03, -3.28393700e-03,  3.95287480e-03,  1.80463120e-02, 7.05414824e-03,  5.80327772e-02,  4.47773710e-02, -4.21631448e-02, 7.39850774e-02,  1.79811213e-02,  7.04950839e-02, -4.73053604e-02, -4.15716507e-02, -9.59994365e-03,  1.89045668e-02,  2.20156256e-02, 1.04296235e-02,  4.42853756e-02, -1.09862518e-02,  1.55339660e-02, 2.17024330e-02, -4.06076433e-03,  2.82253250e-02, -1.43804150e-02, 6.81060879e-03,  2.42336132e-02, -1.20356837e-02, -2.86130030e-02, 5.95987029e-02,  8.22900385e-02, -2.92814523e-03, -2.80786771e-02, -1.55530889e-02, -1.26720862e-02,  5.18352538e-02, -9.45348665e-02, -7.13151123e-04,  1.45976376e-02,  2.42314935e-02,  7.27918756e-04, 1.15067856e-02, -3.24076670e-03, -1.01865055e-02, -2.41364352e-02, 1.49887558e-02, -2.51921508e-02, -5.90336472e-02,  1.79605652e-02, 4.63739224e-02, -2.56433915e-02, -1.97405703e-02,  2.92095672e-02, -3.46023068e-02, -2.02264003e-02, -1.32118966e-02, -1.35912253e-02, 2.53015254e-02, -5.09055592e-02,  3.18482565e-03, -3.80226485e-02, 3.16254012e-02, -7.34225884e-02, -1.46069124e-01, -9.40314960e-03, 1.08394744e-02,  4.54333872e-02, -5.17369248e-02,  3.76999713e-02, 2.26711687e-02,  4.00911309e-02,  2.55041961e-02,  5.65192022e-04, -2.15513241e-02,  2.19384879e-02, -3.45059782e-02,  4.55359109e-02, 4.56896704e-03, -2.33669560e-02, -3.02887280e-02,  6.08850420e-02, 2.45300177e-02,  2.40606125e-02,  2.72339229e-02,  6.88984478e-03, -4.30227630e-03, -7.75243575e-03, -1.88227482e-02,  7.43789896e-02, -9.96600464e-03,  6.63648024e-02, -6.23515360e-02, -7.00666606e-02, -2.60777902e-02, -5.30709652e-03,  1.86568033e-03,  4.86703739e-02, -1.44296940e-02,  1.09362872e-02, -2.21310705e-02,  1.04592685e-02, 1.24127809e-02, -3.27089690e-02,  1.95319913e-02, -2.89123747e-02, -5.01579279e-03,  1.30758854e-02,  4.36534435e-02,  1.51301390e-02, 3.28035615e-02,  1.91630777e-02,  3.04533802e-02, -1.89039763e-02, -1.84636805e-02,  1.00374024e-03, -2.56961808e-02, -4.05047648e-02, 5.04714297e-03, -3.24529260e-02,  1.29275182e-02,  6.61569429e-05, 2.02226080e-02,  2.19759587e-02,  1.99162010e-02,  1.09571014e-02, 4.45986092e-02,  1.74887069e-02, -5.32580987e-02,  1.72492728e-04, -5.38621619e-02,  1.20012527e-02, -2.01341659e-02,  1.04744546e-02, 1.34008070e-02, -3.57121192e-02,  3.88850905e-02, -5.42614348e-02, -1.90397184e-02,  2.95135677e-02, -3.42997424e-02, -4.99790721e-03, 8.60128850e-02,  2.37183720e-02,  2.86867414e-02, -1.79583840e-02, -7.38559617e-03,  5.38222939e-02,  4.61249128e-02,  7.04616010e-02, 1.41099785e-02, -4.96414315e-04,  9.29754507e-03, -3.82086150e-02, 1.39192184e-02,  2.26834547e-02,  1.01052104e-02,  9.58356913e-03, -2.27210857e-02, -2.94213556e-02, -7.82643352e-03, -1.66488234e-02, -9.44784377e-03,  3.56253237e-02,  1.13743758e-02,  3.67637873e-02, -1.71556808e-02,  1.45752802e-02, -7.41183162e-02, -1.02351904e-02, 4.72791940e-02,  9.87325143e-03,  4.21396792e-02, -3.39988689e-03, -4.57472429e-02, -5.81190810e-02, -2.32971762e-03,  3.74609232e-02, -3.66091728e-02, -2.70161405e-02, -3.99984568e-02, -1.26657430e-02, 3.62169091e-03,  1.37003260e-02, -1.19137971e-04,  3.54218693e-03, -2.26079952e-02, -3.18501964e-02,  5.62375076e-02, -2.67932564e-02, -1.21969115e-02, -3.08663696e-02, -6.37506843e-02, -1.23763066e-02, -3.85162383e-02, -1.60970073e-02, -1.88369378e-02,  5.61897494e-02, 1.52650988e-02, -7.10332319e-02, -4.49051186e-02,  4.17606756e-02, 4.91539724e-02,  1.21320277e-01,  2.65271459e-02,  2.31840499e-02, 1.15248952e-02,  2.47847717e-02, -2.54846532e-02,  2.77435537e-02, -2.07310710e-02,  4.05712193e-03,  1.01060709e-02,  1.10278511e-02, -2.04517208e-02, -8.31052568e-03, -8.33136216e-03,  4.86731622e-03, 1.38870850e-02, -1.80860292e-02,  6.39171004e-02, -3.62132750e-02, -1.53473103e-02,  2.90266182e-02,  1.01663947e-01, -6.53519644e-04, -6.04799576e-02,  1.92036871e-02,  1.27083687e-02, -8.96287244e-03, -4.05058414e-02, -5.20501323e-02,  4.02576253e-02,  3.52448854e-03, -2.20351247e-03, -2.05740128e-02,  3.82163152e-02, -2.97404267e-02, -3.58801670e-02,  3.58176492e-02,  2.51023937e-02,  2.89637502e-02, 3.27022783e-02,  2.23609824e-02,  5.98784201e-02,  6.23898134e-02, 1.07959574e-02,  5.99581189e-02, -2.52771825e-02, -3.60102393e-02, 3.81944813e-02,  1.31000252e-02, -3.57606262e-02,  1.95361394e-02, 3.43371592e-02, -3.59156504e-02,  4.64263707e-02,  7.47849140e-03, -2.85285488e-02,  2.93406332e-03,  6.56284466e-02,  1.29735116e-02, 1.24560520e-02, -5.08223148e-03, -1.02384610e-03, -4.41986658e-02, 1.62261128e-02,  1.92464534e-02,  1.16519704e-01,  2.90390477e-02, 9.47278365e-03, -4.93483655e-02, -1.72430475e-03,  3.76843326e-02, 6.50802180e-02, -3.66192758e-02, -2.50096843e-02, -4.26620021e-02, -1.34685393e-02,  5.50172776e-02, -2.76150219e-02,  2.52469238e-02, 3.18296216e-02, -3.14880908e-02, -1.40442792e-03, -6.13821484e-02, 3.07383598e-03, -2.92904209e-02,  1.14536854e-02, -3.03985961e-02, -6.07095212e-02,  6.29598051e-02,  4.26955670e-02,  4.85977158e-02, 3.74925360e-02,  4.74338420e-02,  4.36811596e-02,  7.18751410e-03, 2.66182031e-02, -1.75453406e-02, -5.01529798e-02,  2.43295953e-02, 6.73195114e-03, -8.34603608e-02, -1.65526681e-02,  2.05215290e-02, -1.32691357e-02, -6.70776516e-02, -4.31945808e-02,  2.31816266e-02, 1.09050423e-03,  2.27937270e-02, -1.91466529e-02, -9.33474803e-04, -3.29071912e-03,  5.57969064e-02, -4.72763628e-02,  8.58246163e-02, -2.87788399e-02, -5.85252829e-02,  3.27921589e-03, -2.85337213e-02, -5.56366751e-03, -4.36278582e-02, -3.53846438e-02, -3.15264873e-02, -1.54559398e-02, -5.99277439e-03,  2.26277858e-02,  2.21081451e-02, 3.32079083e-02,  5.43615222e-03, -2.63609700e-02, -2.17014011e-02, -1.56563744e-02, -5.15714958e-02, -1.18470462e-02,  9.13077407e-03, -3.29162180e-02, -4.87432145e-02, -1.53248115e-02, -2.59139016e-02, -5.03621586e-02, -3.27638933e-03,  1.30307525e-02,  1.59115363e-02, 3.17716822e-02, -1.08904252e-02,  3.35477921e-03,  4.80098538e-02, 4.10186350e-02, -4.31699045e-02,  1.78821647e-04,  7.64952749e-02, -2.16004103e-02, -7.06575140e-02, -4.08859830e-03,  5.81064820e-02, -6.29717857e-03,  7.91376680e-02, -2.37569329e-03, -2.12357566e-02, -5.21706827e-02,  3.16610979e-03,  2.22804025e-02, -4.81440965e-03, 9.12775099e-03, -1.45659773e-02, -2.43284404e-02,  2.55279113e-02, -1.48536442e-02, -3.03241257e-02,  3.34586538e-02, -2.24296972e-02, 3.02538127e-02,  4.30741794e-02,  1.01597176e-03, -2.10531484e-02, 2.86896043e-02, -6.38493225e-02, -2.05329736e-03, -1.82789490e-02, 1.55664505e-02,  9.86565053e-02,  6.35040738e-03, -2.68401410e-02, 6.70046881e-02,  5.00495397e-02, -3.30492780e-02,  3.69632058e-02, 9.45759937e-03,  2.34576706e-02, -2.00130064e-02, -4.30487208e-02, 1.74981970e-02, -3.20073813e-02,  2.88598910e-02,  8.17995071e-02, 4.89620084e-04, -4.20734249e-02,  3.48180123e-02,  5.11991279e-03, -1.73648857e-02,  1.03966352e-02,  3.10861529e-03, -1.72180422e-02, 5.10167284e-03,  2.48009302e-02, -5.66994138e-02, -7.50018563e-03, 1.01825185e-02,  3.93441087e-03,  2.16627177e-02, -2.06915103e-02, -4.18055169e-02, -1.45106604e-02, -1.27682080e-02,  9.98790935e-03, -3.73138972e-02,  7.35658268e-03, -5.62764667e-02,  4.28845584e-02, -1.20785451e-02, -2.26719789e-02,  5.04938252e-02,  3.59685905e-02, -8.81960690e-02, -1.59423053e-02,  4.72089276e-02, -3.27063128e-02, 4.45395522e-03, -3.42608895e-03,  7.66208544e-02, -1.07176946e-02, -1.02114361e-02,  2.96150129e-02, -8.01335424e-02,  4.82664369e-02, 1.24619063e-02, -5.94091602e-02,  8.46937392e-03,  3.32079385e-03, 2.49857269e-02, -1.14885652e-02, -7.53031448e-02, -3.56945992e-02, 3.20922285e-02,  2.93184631e-02, -2.13059857e-02,  7.78757385e-04, 3.75101455e-02, -7.08620995e-02, -1.06713437e-02, -1.99566446e-02, -3.40806693e-03, -6.10401202e-03, -2.98068654e-02, -1.89677011e-02, 1.95910651e-02, -2.74215173e-03,  6.01873687e-03, -4.17252770e-03, -2.19440833e-02, -3.11483443e-02, -5.90865836e-02, -5.50597208e-03, 4.07601334e-02,  1.52067132e-02,  2.86911968e-02, -2.95186434e-02, -1.66269727e-02,  7.85411987e-03,  5.38409241e-02,  1.36708999e-02, -2.32281592e-02, -9.39203426e-03, -1.21814879e-02, -3.63057144e-02, 8.95278354e-04, -5.63667668e-03,  1.91887808e-33, -4.33566533e-02, -4.42274734e-02,  1.86333358e-02,  4.24921960e-02,  1.13257607e-02, -2.67407764e-03, -4.58217897e-02, -5.72701218e-03, -2.79605594e-02, 4.46375944e-02, -6.25908673e-02,  3.28818220e-03,  2.22959835e-02, 3.74244042e-02,  4.57203537e-02,  1.89937577e-02, -3.54345255e-02, 2.64822971e-02,  1.45690255e-02, -3.92485857e-02,  4.19178531e-02, 3.90173234e-02,  3.45542072e-03,  3.05347256e-02, -6.39075530e-04, -1.67947542e-02, -2.83922497e-02, -3.94321941e-02,  1.54187549e-02, 4.23817858e-02, -2.45234016e-02, -7.61900172e-02, -5.63705526e-03, -9.18691009e-02, -5.36023825e-02,  4.43371981e-02,  4.22315635e-02, -3.44523080e-02,  3.90661620e-02, -5.89124151e-02, -2.61526443e-02, -2.07635760e-02,  1.10223796e-02,  4.35565077e-02,  7.57555151e-03, -2.78332755e-02,  3.89529876e-02,  4.31878679e-02, -1.92326345e-02, 1.31701231e-02,  5.30364476e-02,  4.43998016e-02,  3.05684954e-02, -2.07589958e-02, -1.02366246e-02,  6.86659589e-02, -6.39676526e-02, -4.87711802e-02,  8.70148912e-02,  1.04625914e-02, -2.50921659e-02, 4.18020450e-02,  1.15561457e-02, -4.14195610e-03, -3.87131199e-02, 3.52347605e-02,  4.59192209e-02,  2.35098451e-02,  3.33773680e-02, -2.84436215e-02,  1.21649001e-02, -9.07375943e-03, -8.08896348e-02, 4.91869524e-02,  1.24516524e-03, -1.02520417e-02,  6.17114594e-04, -1.95689369e-02,  7.67664984e-02,  3.83759253e-02, -1.94907077e-02, 4.78590429e-02,  1.73946582e-02, -1.17715588e-02, -8.49358924e-03, 8.47095996e-02, -6.47542579e-03,  6.88125715e-02, -9.90658160e-03, -2.78850831e-03,  3.16407233e-02,  3.69632877e-02, -5.23581170e-03, -6.05059438e-04, -3.62135172e-02,  9.22156963e-03, -2.12242045e-02, -4.31616716e-02, -2.58213263e-02, -8.72304384e-03,  6.15395745e-03, -2.94016227e-02, -1.55878393e-02,  1.09848185e-02, -4.14459258e-02, -3.33895050e-02, -5.50609119e-02, -6.33776560e-02,  2.05588173e-02, 1.21245971e-02,  5.87851601e-03,  7.67237991e-02, -1.63708124e-02, 3.24134976e-02, -2.57338639e-02,  1.22733647e-03, -9.45925415e-02, 7.58718047e-03, -8.23877566e-03,  8.21105465e-02, -6.74882531e-03, -6.96882373e-03, -3.43753137e-02, -4.91802022e-03, -1.49338543e-02, 2.07468867e-02, -6.28467202e-02, -2.62052026e-02,  7.27858022e-02, -4.30631153e-02, -3.11611369e-02, -1.94044467e-02, -8.72308016e-03, -2.93835416e-03,  1.72045082e-02,  8.22418369e-03,  2.78199501e-02, -1.37404520e-02, -2.70269578e-03, -5.78028802e-03,  5.91702014e-02, 8.28758776e-02,  4.05108556e-02,  8.50384589e-03,  4.26244438e-02, -1.49471089e-02, -3.68858501e-02, -2.95383446e-02, -3.12358830e-02, -3.95183936e-02, -2.26417929e-02, -3.15803513e-02, -1.15995742e-02, 3.71009484e-03, -1.69728752e-02,  3.41749452e-02, -7.19584152e-02, 7.16953054e-02, -3.14729027e-02,  3.83929387e-02, -3.83464759e-03, -1.10111488e-02, -1.89644080e-02,  2.59531336e-03, -3.37312147e-02, 3.88063886e-03,  3.57762240e-02, -5.93993589e-02, -3.35881971e-02, -5.47134764e-02, -5.87828048e-02,  2.23364085e-02,  4.12895270e-02, 4.21102112e-03,  5.01548871e-02,  1.46141611e-02, -1.60785783e-02, 5.77300135e-03, -3.51868495e-02, -1.82182658e-02,  8.88655055e-03, 3.33121940e-02, -2.53670122e-02,  3.28676659e-03,  7.61871366e-03, -1.45879644e-03, -3.47302225e-03,  4.92487587e-02, -5.55533059e-02, 7.93799758e-02, -3.27416360e-02, -3.92518155e-02,  3.62891778e-02, -1.44601474e-02, -2.82487813e-02, -2.82876007e-02,  6.23706505e-02, -1.22910319e-02, -1.53047586e-04,  3.62271890e-02,  4.88874204e-02, -4.92698178e-02, -3.61629785e-03, -5.03963567e-02, -3.29884999e-02, -3.31856944e-02, -4.99778017e-02, -3.85035314e-02,  4.63582315e-02, 4.76273410e-02, -2.65436415e-02, -1.74311179e-04, -4.35930565e-02, -1.52840372e-02, -1.01632727e-02,  3.76403667e-02,  4.97345328e-02, 2.29855012e-02, -2.10566702e-03, -1.36193829e-02,  9.54141468e-03, -2.36655977e-02,  7.74009451e-02, -7.66901346e-03,  1.48809124e-02, 4.60187905e-02, -1.06680663e-02, -2.11192295e-02, -3.06416098e-02, -7.30406046e-02, -5.52300289e-02, -3.10481731e-02,  8.47726478e-04, 7.79758382e-04,  3.68176624e-02,  4.00339290e-02,  2.02554967e-02, -1.57325473e-02,  2.30250582e-02,  2.21156999e-02,  4.23344672e-02, 2.67792940e-02,  5.36791161e-02, -3.57918330e-02, -1.78055055e-02, -5.42198494e-03, -5.47904484e-02, -4.75045741e-02, -1.97625849e-02, 2.13571619e-02,  3.82434092e-02, -1.73607767e-02,  4.52133827e-03, 8.48614797e-03, -7.99214020e-02, -2.59474106e-02,  2.81057004e-02, -8.50231387e-03,  2.01026630e-02,  2.83441357e-02,  2.13191882e-02, 3.70609835e-02, -2.01003812e-02,  2.27825530e-02, -3.87140103e-02, -6.00847974e-02,  3.00723668e-02, -1.39068738e-02, -1.29560912e-02, 1.14637606e-01,  4.56365151e-03, -2.91780233e-02, -1.05033480e-02])\n",
        "    np.testing.assert_almost_equal(student_sol_func(test_str), q2a)\n",
        "    test_print('**q2a** passed sanity check!')"
      ],
      "metadata": {
        "id": "IgVBT6tfQ9mA"
      },
      "id": "IgVBT6tfQ9mA",
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "public_test_q2a(sentence_to_sentence_embedding)"
      ],
      "metadata": {
        "id": "WKh8MkmsRSTS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 46
        },
        "outputId": "23f9f7ed-e401-49ae-edd3-ec2be8918e98"
      },
      "id": "WKh8MkmsRSTS",
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**q2a** passed sanity check!"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Q2 (b). method 2\n",
        "\n",
        "Like `method_one()` you implemented before. Here we do it but not with GloVe embeddings. Do what you did for Q1 (c), but use `sentence_to_sentence_embedding()`, **not** `sentence_to_average_glove()`.\n",
        "\n",
        "The thing your function must return is a dictionary mapping each urban dictionary term to a WordNet synset ID, e.g.:\n",
        "\n",
        "```\n",
        "{\n",
        " \"adore\" : \"love.v.01\",\n",
        " \"dripping\" : \"stylish.a.01\"    \n",
        " }\n",
        "```\n",
        "\n",
        "New to Q2: here, you *must* add appe d similarity scores to the list `all_term_similarity_scores` provided, and as shown in the template code, return that variable as well."
      ],
      "metadata": {
        "id": "C_POO10nP4rY"
      },
      "id": "C_POO10nP4rY"
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "7d2b70a3"
      },
      "outputs": [],
      "source": [
        "def method_two(sentence_encoder, urban_dictionary_terms: List[Tuple[str, str]], target_synsets: List[str]):\n",
        "    \"\"\"\n",
        "    Method 2: an algorithm based on sentence embeddings that maps each urban dictionary term to a synset ID.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    sentence_encoder : \n",
        "        a function to encode a sentence into word embeddings\n",
        "        You should pass the function `sentence_to_sentence_embedding` implemented in q2a here\n",
        "    urban_dictionary_terms : List[Tuple[str, str]]\n",
        "        a list of string 2-tuples where the first elements are words, second elements are definitions.\n",
        "    target_synsets : List[str]\n",
        "        a list of synset IDs that the words should be classified into.\n",
        "        You can call `wn.synset(\"<synset ID>\")` to get the synset object.\n",
        "    \n",
        "    Returns\n",
        "    --------\n",
        "    A dictionary mapping each urban dictionary term to a WordNet synet ID, e.g.\n",
        "    `{\"adore\" : \"love.v.01\", \"dripping\" : \"stylish.a.01\"}`\n",
        "    \n",
        "    \"\"\"\n",
        "    all_term_similarity_scores=[]\n",
        "    # BEGIN SOLUTION\n",
        "\n",
        "    # you should compute the cosine similarity \n",
        "    # between the sentence embedding of the UD word definition \n",
        "    # and those of the synsets, \n",
        "    # then for each UD word, choose the synset with the highest cosine similarity. \n",
        "\n",
        "    best_matches={}\n",
        "    for word, definition in urban_dictionary_terms:\n",
        "        scores = get_term_similarity_scores(sentence_encoder, word, definition, target_synsets, silent=True)\n",
        "        all_term_similarity_scores.append(scores)\n",
        "        best = scores[0][0]\n",
        "        best_matches[word] = best\n",
        "    # END SOLUTION\n",
        "    return best_matches, all_term_similarity_scores"
      ],
      "id": "7d2b70a3"
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "285b848a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "391e74a6-8a73-4a1e-ab37-0be581c756c5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Crowdfunding': 'fund-raise.v.01',\n",
              " 'Hygge': 'coziness.n.01',\n",
              " 'biohacking': 'mutation.n.02',\n",
              " 'TL;DR': 'kernel.n.03',\n",
              " 'Hellacious': 'formidable.s.01',\n",
              " 'Unfriend': 'stranger.n.02',\n",
              " 'Infodemic': 'spread.n.01',\n",
              " 'Onboarding': 'orientation.n.06',\n",
              " 'Truthiness': 'plausibility.n.01',\n",
              " 'Amotivational': 'faineant.s.01',\n",
              " 'NSFW': 'faineant.s.01',\n",
              " 'Rando': 'stranger.n.02'}"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "method_two_results, all_term_similarity_scores=method_two(sentence_to_sentence_embedding, urban_dictionary_terms, target_synsets)\n",
        "method_two_results"
      ],
      "id": "285b848a"
    },
    {
      "cell_type": "code",
      "source": [
        "def public_test_q2b(student_sim_scores):\n",
        "    np.testing.assert_almost_equal(student_sim_scores[-1][-5][-1], 0.12022592)\n",
        "    test_print('**q2b** passed sanity check!')"
      ],
      "metadata": {
        "id": "4wIeTMmzSHNh"
      },
      "id": "4wIeTMmzSHNh",
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "public_test_q2b(all_term_similarity_scores)"
      ],
      "metadata": {
        "id": "tOSb52srSp3Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 46
        },
        "outputId": "6a642fd1-c8a9-4fb6-c5a8-60af25d81cab"
      },
      "id": "tOSb52srSp3Y",
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**q2b** passed sanity check!"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5f329391"
      },
      "source": [
        "### Deliverable 3 (TODO): Define an evaluation metric (accuracy).  \n",
        "\n",
        "Throughout this semester we've stressed how critical evaluation is for any NLP method.  Implement a function `accuracy` that assesses quality of the dictionaries you return from `method_one` and `method_two`.  This accuracy function should return a single real number (the accuracy), and its input parameters are a prediction dict (the output of your model) and a truth dict (which you will need to create based on your own judgement). Make sure that the accuracies you calculate for the two methods match what you expect. \n"
      ],
      "id": "5f329391"
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "938b9fcd"
      },
      "outputs": [],
      "source": [
        "def accuracy(prediction: Dict[str, str], truth: Dict[str, str]) -> float:\n",
        "    # BEGIN SOLUTION    \n",
        "    '''\n",
        "    Inputs\n",
        "    ------\n",
        "    prediction: dict, model output of {UD: synset} pairs\n",
        "    truth: dict, gold labels of {UD: synset} pairs\n",
        "\n",
        "    Returns\n",
        "    ------\n",
        "    acc: float\n",
        "    '''\n",
        "    n_correct = 0\n",
        "    total = len(prediction.keys())\n",
        "    try: \n",
        "      check_keys = set(truth).intersection(set(prediction))\n",
        "      for k in check_keys:\n",
        "        if prediction[k] == truth[k]:\n",
        "          n_correct += 1\n",
        "    except KeyError:\n",
        "      print('None of the prediction keys are in the truth dict')\n",
        "    acc = n_correct / total\n",
        "    return acc\n",
        "    # END SOLUTION\n"
      ],
      "id": "938b9fcd"
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "766150ed",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b8ec2792-9b76-493f-d8f8-7d1dfe144591"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Crowdfunding': 'fund-raise.v.01',\n",
              " 'Hygge': 'coziness.n.01',\n",
              " 'biohacking': 'mutation.n.02',\n",
              " 'TL;DR': 'kernel.n.03',\n",
              " 'Hellacious': 'formidable.s.01',\n",
              " 'Unfriend': 'sever.v.01',\n",
              " 'Infodemic': 'spread.n.01',\n",
              " 'Onboarding': 'orientation.n.06',\n",
              " 'Truthiness': 'plausibility.n.01',\n",
              " 'Amotivational': 'faineant.s.01',\n",
              " 'NSFW': 'inappropriate.a.01',\n",
              " 'Rando': 'stranger.n.02'}"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ],
      "source": [
        "truth= {}\n",
        "pairs = [('Crowdfunding','fund-raise.v.01'),\n",
        " ('Hygge','coziness.n.01'),\n",
        " ('biohacking', 'mutation.n.02'),\n",
        " ('TL;DR','kernel.n.03'),\n",
        " ('Hellacious','formidable.s.01'),\n",
        " ('Unfriend','sever.v.01'),\n",
        " ('Infodemic','spread.n.01',),\n",
        " ('Onboarding','orientation.n.06'),\n",
        " ('Truthiness', 'plausibility.n.01'),\n",
        " ('Amotivational','faineant.s.01'),\n",
        " ('NSFW','inappropriate.a.01'),\n",
        " ('Rando', 'stranger.n.02')]\n",
        "for k,v in pairs:\n",
        "  truth[k] = v\n",
        "truth"
      ],
      "id": "766150ed"
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "9ea46c6b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1055dcb6-1ea7-447f-a846-0fbd49729cb5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.3333333333333333\n"
          ]
        }
      ],
      "source": [
        "print(accuracy(method_one_results, truth))"
      ],
      "id": "9ea46c6b"
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "c0135985",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b20b94d1-9434-4797-9fd9-684fedceeaae"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8333333333333334\n"
          ]
        }
      ],
      "source": [
        "print(accuracy(method_two_results, truth))"
      ],
      "id": "c0135985"
    },
    {
      "cell_type": "code",
      "source": [
        "def public_test_q3(student_sol_func):\n",
        "    test={\n",
        "        'NSFW': 'inappropriate.a.01',\n",
        "        'biohacking': 'mutation.n.02'\n",
        "    }\n",
        "    np.testing.assert_array_less(student_sol_func(method_one_results, test), \n",
        "                                 student_sol_func(method_two_results, test))\n",
        "    test_print('**q3** passed sanity check!')"
      ],
      "metadata": {
        "id": "eQRsw2ilVX1O"
      },
      "id": "eQRsw2ilVX1O",
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "public_test_q3(accuracy)"
      ],
      "metadata": {
        "id": "9kXkrSYvVusf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 46
        },
        "outputId": "d63bb03a-33c1-48c1-af9d-97a4714700cf"
      },
      "id": "9kXkrSYvVusf",
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "**q3** passed sanity check!"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eZv8eZJ8sfE"
      },
      "source": [
        "---\n",
        "\n",
        "**Congrats!** That's it for HW5.\n",
        "\n",
        "Submit your work to “HW5 (ipynb)” on Gradescope. Here are some recommended steps:\n",
        "\n",
        "- Restart your Colab runtime and re-run all your cells; make sure everything runs as expected.\n",
        "\n",
        "- Remember all your work MUST BE PLACED between `# BEGIN SOLUTION` and `# END SOLUTION`.\n",
        "\n",
        "- Download your Colab notebook as an `.ipynb file` (File → Download .ipynb)\n",
        "\n",
        "- Upload HW5.ipynb to Gradescope.\n"
      ],
      "id": "8eZv8eZJ8sfE"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djEZMMFT7j5G"
      },
      "source": [
        "### Optional\n",
        "\n",
        "Use the two methods you've defined to find the best-matching synset within the **entire** WordNet. Do the results make sense? Is one method consistently better than the other?\n",
        "\n",
        "Here's a reminder of how to iterate through all synsets:"
      ],
      "id": "djEZMMFT7j5G"
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "6M_gxGMo7g2T",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "55642051-066f-4e64-a4ee-ac8e60ba91de"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 Synset('able.a.01')\n",
            "1 Synset('unable.a.01')\n",
            "2 Synset('abaxial.a.01')\n",
            "3 Synset('adaxial.a.01')\n",
            "4 Synset('acroscopic.a.01')\n",
            "5 Synset('basiscopic.a.01')\n",
            "6 Synset('abducent.a.01')\n",
            "7 Synset('adducent.a.01')\n",
            "8 Synset('nascent.a.01')\n",
            "9 Synset('emergent.s.02')\n",
            "10 Synset('dissilient.s.01')\n",
            "11 Synset('parturient.s.02')\n"
          ]
        }
      ],
      "source": [
        "for idx, synset in enumerate(wn.all_synsets()):\n",
        "    print(idx, synset)\n",
        "    if (idx > 10): break"
      ],
      "id": "6M_gxGMo7g2T"
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "accelerator": "TPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}