asr-customize-speechhints.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cdc3b68-b297-4e27-8fc4-67da496f8bb8",
   "metadata": {},
   "source": [
    "# Creating Grammars for Speech Hints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8940eb9-822a-4d3c-8fdd-23ba7fe78540",
   "metadata": {},
   "source": [
    "This tutorial walks you through the process of creating custom speech hint grammars for use in inverse text normalization.\n",
    "The primary application of speech hint grammars is to provide specific normalization post Automatic Speech Recognition (ASR).\n",
    "\n",
    "## Dependencies\n",
    "You need to download [speech hints grammars](https://github.com/nvidia-riva/speech-hints)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1d6936-a70d-4362-8540-34b2768d8ff0",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "This tutorial assumes that you are familiar with finite state acceptors and transducers. Familiarity with the [Pynini](https://www.openfst.org/twiki/bin/view/GRM/Pynini) library and Nemo's [Weighted Finite-State Transducers (WFST) tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/text_processing/WFST_Tutorial.ipynb) is assumed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6581ec4-b81e-4687-99ee-02d013fe7334",
   "metadata": {},
   "source": [
    "## Overview\n",
    "Functionally, speech hints should at the bare minimum have the following:\n",
    "1. A passthrough finite-state transducer (FST) that transduces the input text as is ($\\Sigma *$ - a set of all possible strings over the alphabet $\\Sigma$). This FST should have the longest distance/weight relative to the other FSTs.\n",
    "2. An FST per class of interest. FST for a class can import from other FSTs. However, the resultant FST when exported will be independent of FSTs it imports from.\n",
    "\n",
    "Grammars in speech hints are composed on the fly. Grammars can be standalone references to the FST or consist of FSTs in context with sentences. Grammars are denoted by \\\\$\\<FSTNAME\\>. For English, the following [grammars](https://github.com/nvidia-riva/speech-hints) are supported:\n",
    "1. `$OOV_NUMERIC_SEQUENCE`\n",
    "2. `$OOV_ALPHA_SEQUENCE`\n",
    "3. `$OOV_ALPHA_NUMERIC_SEQUENCE`\n",
    "4. `$FULLPHONENUM`\n",
    "5. `$POSTALCODE`\n",
    "6. `$OOV_CLASS_ORDINAL`\n",
    "7. `$OOV_CLASS_NUMERIC`\n",
    "8. `$PERCENT`\n",
    "9. `$TIME`\n",
    "10. `$MONEY`\n",
    "11. `$MONTH`\n",
    "12. `$DAY`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d341398-695b-4878-9bc2-8a8c79f011bd",
   "metadata": {},
   "source": [
    "## Using Existing Speech Hint Grammars in Python\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a3fd6abc-97fe-4c98-b7d2-de1e046f945a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from speech_hint import apply_hint\n",
    "import pynini\n",
    "from pynini.lib import pynutil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ddd25e28-0e33-4596-b512-2dd502a549ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1-800-555-4001'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Applying `$FULLPHONENUM` Grammar on the Input Text\n",
    "apply_hint(\"one eight hundred five five five four oh oh one\",\"$FULLPHONENUM\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "96dd7885-138a-4b69-b8fb-8008c3054d2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my phone number is 1-800-555-4001'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "apply_hint(\"my phone number is one eight hundred five five five four oh oh one\",\"$FULLPHONENUM\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7075d97c-b29f-4a03-9dd6-a79cb71486d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my phone number is 1-800-555-4001'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Specifying `$FULLPHONENUM` grammar in context\n",
    "apply_hint(\"my phone number is one eight hundred five five five four oh oh one\",\"my phone number is $FULLPHONENUM\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "71aa5028-6eb3-4fc4-b7b9-9fb0a2e2c98a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I think my phone number is 1-800-555-4001'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "apply_hint(\"I think my phone number is one eight hundred five five five four oh oh one\",\"my phone number is $FULLPHONENUM\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6289f415-ca24-44a2-b660-fe52630f93ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'my phone number is one eight hundred five five five four oh oh one'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Specifying `$FULLPHONENUM` Grammar in Context - Context Does Not Match\n",
    "apply_hint(\"my phone number is one eight hundred five five five four oh oh one\",\"my phone number is not $FULLPHONENUM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c86e9dfd-568b-410d-af3b-fcd11b5cee03",
   "metadata": {},
   "source": [
    "## Sample Grammar for Handling Alphabet Sequences\n",
    "Let's say we need to build a grammar to support the conversion of alphabet sequences to a single word ('i b m' -> 'ibm').  For a detailed implementation, refer to the [`oov_class_alpha_sequence.py` script](https://github.com/nvidia-riva/speech-hints/blob/main/en/oov_class_alpha_sequence.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8435adee-e22c-49f4-b1d1-7daed2e810fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to Apply FST\n",
    "def apply_fst(utterance, fst):\n",
    "    try:\n",
    "        return pynini.shortestpath(utterance @ fst).string().strip()\n",
    "    except pynini.FstOpError:\n",
    "        print(f\"Error: No valid output with given input: '{utterance}, {fst}'\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2c1d0792-a568-4ef0-979a-45d44dd0e76b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ibm'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from en.primitives import NEMO_ALPHA, NEMO_WHITE_SPACE\n",
    "character = NEMO_ALPHA\n",
    "word_fst = pynini.closure(character)\n",
    "sequence = character + pynini.closure(pynutil.delete(\" \") + character, 1)\n",
    "fst = sequence @ (word_fst)\n",
    "\n",
    "apply_fst('i b m', fst)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1abc58f8-14e3-4000-887c-d13091b19aad",
   "metadata": {},
   "source": [
    "To use the custom FST (grammar) in `speech_hints`, add it with a suitable name to `fst_dict` in `speech_hints.py`. You can then export the grammars as an FST Archive (`.far`) file using the `export_to_far.py` script."
   ]
  }
  
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}