notebooks/Stanza_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stanza Tutorial\n",
    "\n",
    "(C) 2023-2024 by [Damir Cavar](http://damir.cavar.me/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Version:** 1.1, January 2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Download:** This and various other Jupyter notebooks are available from my [GitHub repo](https://github.com/dcavar/python-tutorial-for-ipython)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prerequisites:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U stanza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To install [spaCy](https://spacy.io/) follow the instructions on the [Install spaCy page](https://spacy.io/usage)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U pip setuptools wheel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following installation of spaCy is ideal for my environment, i.e., using a GPU and CUDA 12.x. See the [spaCy homepage](https://spacy.io/usage) for detailed installation instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U 'spacy[cuda12x,transformers,lookups,ja]'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tutorial related to the [L645 Advanced Natural Language Processing](http://damir.cavar.me/l645/) course in Fall 2023 at Indiana University. The following tutorial assumes that you are using a newer distribution of [Python 3.x](https://python.org/) and [Stanza](https://stanfordnlp.github.io/stanza/) 1.5.1 or newer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook assumes that you have set up [Stanza](https://stanfordnlp.github.io/stanza/) on your computer with your [Python](https://python.org/) distribution. Follow the instructions on the [Stanza](https://stanfordnlp.github.io/stanza/) installation page to set up a working environment for the following code. The code will also require that you are online and that the specific language models can be downloaded and installed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the [Stanza](https://stanfordnlp.github.io/stanza/) module and [spaCy's Displacy](https://spacy.io/usage/visualizers) for visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\damir\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\transformers\\utils\\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n"
     ]
    }
   ],
   "source": [
    "import stanza\n",
    "from stanza.models.common.doc import Document\n",
    "from stanza.pipeline.core import Pipeline\n",
    "from spacy import displacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code will load the English language model for [Stanza](https://stanfordnlp.github.io/stanza/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "732e50b8c50944cf842caa7117ba9355",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.9.0.json:   0%|   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-11 19:12:37 INFO: Downloaded file to C:\\Users\\damir\\stanza_resources\\resources.json\n",
      "2024-11-11 19:12:37 INFO: Downloading default packages for language: de (German) ...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2d9af8948bd4e5ea8afeb3214e3c899",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://huggingface.co/stanfordnlp/stanza-de/resolve/v1.9.0/models/default.zip:   0%|          | 0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-11 19:13:10 INFO: Downloaded file to C:\\Users\\damir\\stanza_resources\\de\\default.zip\n",
      "2024-11-11 19:13:17 INFO: Finished downloading models and saved to C:\\Users\\damir\\stanza_resources\n"
     ]
    }
   ],
   "source": [
    "stanza.download('de')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can configure the [Stanza](https://stanfordnlp.github.io/stanza/) pipeline to contain all desired linguistic annotation modules. In this case we use:\n",
    "- tokenizer\n",
    "- multi-word-tokenizer\n",
    "- Part-of-Speech tagger\n",
    "- lemmatizer\n",
    "- dependency parser\n",
    "- constituent parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-11-11 19:13:17 WARNING: Can not find ner: ncbi_disease from official model list. Ignoring it.\n",
      "2024-11-11 19:13:17 WARNING: Can not find ner: ontonotes from official model list. Ignoring it.\n",
      "2024-11-11 19:13:18 INFO: Loading these models for language: de (German):\n",
      "===============================\n",
      "| Processor    | Package      |\n",
      "-------------------------------\n",
      "| tokenize     | gsd          |\n",
      "| mwt          | gsd          |\n",
      "| pos          | gsd_charlm   |\n",
      "| lemma        | gsd_nocharlm |\n",
      "| constituency | spmrl_charlm |\n",
      "| depparse     | gsd_charlm   |\n",
      "| sentiment    | sb10k_charlm |\n",
      "===============================\n",
      "\n",
      "2024-11-11 19:13:18 INFO: Using device: cpu\n",
      "2024-11-11 19:13:18 INFO: Loading: tokenize\n",
      "2024-11-11 19:13:18 INFO: Loading: mwt\n",
      "2024-11-11 19:13:18 INFO: Loading: pos\n",
      "2024-11-11 19:13:18 INFO: Loading: lemma\n",
      "2024-11-11 19:13:18 INFO: Loading: constituency\n",
      "2024-11-11 19:13:19 INFO: Loading: depparse\n",
      "2024-11-11 19:13:20 INFO: Loading: sentiment\n",
      "2024-11-11 19:13:20 INFO: Done loading processors!\n"
     ]
    }
   ],
   "source": [
    "nlp = stanza.Pipeline('de', processors='tokenize,mwt,pos,lemma,ner,depparse,constituency,sentiment', package={\"ner\": [\"ncbi_disease\", \"ontonotes\"]}, use_gpu=False, download_method=\"reuse_resources\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====== Sentence 1 tokens =======\n",
      "id: (1,)\ttext: Gummibärchen\n",
      "id: (2,)\ttext: habe\n",
      "id: (3,)\ttext: ich\n",
      "id: (4,)\ttext: grüne\n",
      "id: (5,)\ttext: noch\n",
      "id: (6,)\ttext: keine\n",
      "id: (7,)\ttext: gegessen\n",
      "id: (8,)\ttext: .\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Gummibärchen habe ich grüne noch keine gegessen.\")\n",
    "for i, sentence in enumerate(doc.sentences):\n",
    "    print(f'====== Sentence {i+1} tokens =======')\n",
    "    print(*[f'id: {token.id}\\ttext: {token.text}' for token in sentence.tokens], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word: Gummibärchen\tupos: NOUN\txpos: NN\tfeats: Case=Acc|Gender=Fem|Number=Plur\n",
      "word: habe\tupos: AUX\txpos: VAFIN\tfeats: Mood=Ind|Number=Sing|Person=1|Tense=Pres|VerbForm=Fin\n",
      "word: ich\tupos: PRON\txpos: PPER\tfeats: Case=Nom|Number=Sing|Person=1|PronType=Prs\n",
      "word: grüne\tupos: NOUN\txpos: NN\tfeats: Case=Acc|Gender=Fem|Number=Sing\n",
      "word: noch\tupos: ADV\txpos: ADV\tfeats: _\n",
      "word: keine\tupos: DET\txpos: PIS\tfeats: Case=Acc|Gender=Fem|Number=Sing|PronType=Neg\n",
      "word: gegessen\tupos: VERB\txpos: VVPP\tfeats: VerbForm=Part\n",
      "word: .\tupos: PUNCT\txpos: $.\tfeats: _\n"
     ]
    }
   ],
   "source": [
    "print(*[f'word: {word.text}\\tupos: {word.upos}\\txpos: {word.xpos}\\tfeats: {word.feats if word.feats else \"_\"}' for sent in doc.sentences for word in sent.words], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word: Gummibärchen \tlemma: Gummibärchen\n",
      "word: habe \tlemma: haben\n",
      "word: ich \tlemma: ich\n",
      "word: grüne \tlemma: grün\n",
      "word: noch \tlemma: noch\n",
      "word: keine \tlemma: kein\n",
      "word: gegessen \tlemma: essen\n",
      "word: . \tlemma: .\n"
     ]
    }
   ],
   "source": [
    "print(*[f'word: {word.text+\" \"}\\tlemma: {word.lemma}' for sent in doc.sentences for word in sent.words], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(ROOT (VROOT (S (NOUN Gummibärchen) (AUX habe) (PRON ich) (VP (NOUN grüne) (AP (ADV noch) (DET keine)) (VERB gegessen))) (PUNCT .)))\n"
     ]
    }
   ],
   "source": [
    "for sentence in doc.sentences:\n",
    "    print(sentence.constituency)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "entity: arthritis\ttype: DISEASE\n"
     ]
    }
   ],
   "source": [
    "print(*[f'entity: {ent.text}\\ttype: {ent.type}' for ent in doc.ents], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token: Gummibärchen\tner: None\n",
      "token: habe\tner: None\n",
      "token: ich\tner: None\n",
      "token: grüne\tner: None\n",
      "token: noch\tner: None\n",
      "token: keine\tner: None\n",
      "token: gegessen\tner: None\n",
      "token: .\tner: None\n"
     ]
    }
   ],
   "source": [
    "print(*[f'token: {token.text}\\tner: {token.ner}' for sent in doc.sentences for token in sent.tokens], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 -> 1\n"
     ]
    }
   ],
   "source": [
    "for i, sentence in enumerate(doc.sentences):\n",
    "    print(\"%d -> %d\" % (i, sentence.sentiment))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3996aec872e0402cbc93edad7bac8560",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.9.0.json:   0%|   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-09-25 14:20:36 INFO: Downloaded file to C:\\Users\\damir\\stanza_resources\\resources.json\n",
      "2024-09-25 14:20:36 INFO: Downloading default packages for language: multilingual (multilingual) ...\n",
      "2024-09-25 14:20:36 INFO: File exists: C:\\Users\\damir\\stanza_resources\\multilingual\\default.zip\n",
      "2024-09-25 14:20:36 INFO: Finished downloading models and saved to C:\\Users\\damir\\stanza_resources\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61ee1e2d6a0a4de6858be573533018ca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.9.0.json:   0%|   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-09-25 14:20:37 INFO: Downloaded file to C:\\Users\\damir\\stanza_resources\\resources.json\n",
      "2024-09-25 14:20:37 INFO: Downloading default packages for language: en (English) ...\n",
      "2024-09-25 14:20:38 INFO: File exists: C:\\Users\\damir\\stanza_resources\\en\\default.zip\n",
      "2024-09-25 14:20:41 INFO: Finished downloading models and saved to C:\\Users\\damir\\stanza_resources\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e0bf45f16a24a0d87c8af65d7d7f252",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.9.0.json:   0%|   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-09-25 14:20:41 INFO: Downloaded file to C:\\Users\\damir\\stanza_resources\\resources.json\n",
      "2024-09-25 14:20:41 INFO: Downloading default packages for language: de (German) ...\n",
      "2024-09-25 14:20:43 INFO: File exists: C:\\Users\\damir\\stanza_resources\\de\\default.zip\n",
      "2024-09-25 14:20:47 INFO: Finished downloading models and saved to C:\\Users\\damir\\stanza_resources\n"
     ]
    }
   ],
   "source": [
    "stanza.download(lang=\"multilingual\")\n",
    "stanza.download(lang=\"en\")\n",
    "# stanza.download(lang=\"fr\")\n",
    "stanza.download(lang=\"de\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-09-25 14:22:30 INFO: Checking for updates to resources.json in case models have been updated.  Note: this behavior can be turned off with download_method=None or download_method=DownloadMethod.REUSE_RESOURCES\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b528de766814f2fb2c2ef8909c0a99a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading https://raw.githubusercontent.com/stanfordnlp/stanza-resources/main/resources_1.9.0.json:   0%|   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-09-25 14:22:30 INFO: Downloaded file to C:\\Users\\damir\\stanza_resources\\resources.json\n",
      "2024-09-25 14:22:30 INFO: Loading these models for language: multilingual ():\n",
      "=======================\n",
      "| Processor | Package |\n",
      "-----------------------\n",
      "| langid    | ud      |\n",
      "=======================\n",
      "\n",
      "2024-09-25 14:22:30 INFO: Using device: cpu\n",
      "2024-09-25 14:22:30 INFO: Loading: langid\n",
      "2024-09-25 14:22:30 INFO: Done loading processors!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello world.\ten\n",
      "Hallo, Welt!\tit\n",
      "Ciao mondo!\tit\n",
      "Hola mundo!\tkmr\n"
     ]
    }
   ],
   "source": [
    "nlp = Pipeline(lang=\"multilingual\", processors=\"langid\")\n",
    "docs = [\"Hello world.\", \"Hallo, Welt!\", \"Ciao mondo!\", \"Hola mundo!\"]\n",
    "docs = [Document([], text=text) for text in docs]\n",
    "nlp(docs)\n",
    "print(\"\\n\".join(f\"{doc.text}\\t{doc.lang}\" for doc in docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing Dependency Parse Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I wrote the following function to convert the [Stanza](https://stanfordnlp.github.io/stanza/) dependency tree data structure to a [spaCy's Displacy](https://spacy.io/usage/visualizers) compatible data structure for the visualization of dependency trees using [spaCy's](https://spacy.io/) excellent visualizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_stanza_dep_displacy_manual(doc):\n",
    "    res = []\n",
    "    for x in doc.sentences:\n",
    "        words = []\n",
    "        arcs  = []\n",
    "        for w in x.words:\n",
    "            if w.head > 0:\n",
    "                head_text = x.words[w.head-1].text\n",
    "            else:\n",
    "                head_text = \"root\"\n",
    "            words.append({\"text\": w.text, \"tag\": w.upos})\n",
    "            if w.deprel == \"root\": continue\n",
    "            start = w.head-1\n",
    "            end = w.id-1\n",
    "            if start < end:\n",
    "                arcs.append({ \"start\":start, \"end\":end, \"label\": w.deprel, \"dir\":\"right\"})\n",
    "            else:\n",
    "                arcs.append({ \"start\":end, \"end\":start, \"label\": w.deprel, \"dir\":\"left\"})\n",
    "        res.append( { \"words\": words, \"arcs\": arcs } )\n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can generate an annotation object with [Stanza](https://stanfordnlp.github.io/stanza/) similarly to [spaCy's](https://spacy.io/) approach submitting a sentence or text segment to the NLP pipeline we specified above and assigned to the `nlp` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"Gummibärchen habe ich grüne noch keine gegessen.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now generate the [spaCy](https://spacy.io/)-compatible data format from the dependency tree to be able to visualize it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = get_stanza_dep_displacy_manual(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rendering can be achieved using the [Displacy](https://spacy.io/usage/visualizers) call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'dict' object has no attribute 'vocab'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[15], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43mdisplacy\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43mres\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstyle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdep\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# , manual=True, options={\"compact\":False, \"distance\":110})\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\damir\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\spacy\\displacy\\__init__.py:57\u001b[0m, in \u001b[0;36mrender\u001b[1;34m(docs, style, page, minify, jupyter, options, manual)\u001b[0m\n\u001b[0;32m     55\u001b[0m renderer_func, converter \u001b[38;5;241m=\u001b[39m factories[style]\n\u001b[0;32m     56\u001b[0m renderer \u001b[38;5;241m=\u001b[39m renderer_func(options\u001b[38;5;241m=\u001b[39moptions)\n\u001b[1;32m---> 57\u001b[0m parsed \u001b[38;5;241m=\u001b[39m [\u001b[43mconverter\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdoc\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m docs] \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m manual \u001b[38;5;28;01melse\u001b[39;00m docs  \u001b[38;5;66;03m# type: ignore\u001b[39;00m\n\u001b[0;32m     58\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m manual:\n\u001b[0;32m     59\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m docs:\n",
      "File \u001b[1;32mc:\\Users\\damir\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\spacy\\displacy\\__init__.py:137\u001b[0m, in \u001b[0;36mparse_deps\u001b[1;34m(orig_doc, options)\u001b[0m\n\u001b[0;32m    135\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(orig_doc, Span):\n\u001b[0;32m    136\u001b[0m     orig_doc \u001b[38;5;241m=\u001b[39m orig_doc\u001b[38;5;241m.\u001b[39mas_doc()\n\u001b[1;32m--> 137\u001b[0m doc \u001b[38;5;241m=\u001b[39m Doc(\u001b[43morig_doc\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvocab\u001b[49m)\u001b[38;5;241m.\u001b[39mfrom_bytes(\n\u001b[0;32m    138\u001b[0m     orig_doc\u001b[38;5;241m.\u001b[39mto_bytes(exclude\u001b[38;5;241m=\u001b[39m[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124muser_data\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124muser_hooks\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[0;32m    139\u001b[0m )\n\u001b[0;32m    140\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m doc\u001b[38;5;241m.\u001b[39mhas_annotation(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDEP\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[0;32m    141\u001b[0m     warnings\u001b[38;5;241m.\u001b[39mwarn(Warnings\u001b[38;5;241m.\u001b[39mW005)\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'dict' object has no attribute 'vocab'"
     ]
    }
   ],
   "source": [
    "displacy.render(res, style=\"dep\") # , manual=True, options={\"compact\":False, \"distance\":110})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Format - CoNLL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stanza.utils.conll import CoNLL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "CoNLL.write_doc2conll(doc, \"output.conllu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2023-2024 by [Damir Cavar](http://damir.cavar.me/) <<dcavar@iu.edu>>**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}