tts-phoneme-distribution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d21f599",
   "metadata": {},
   "source": [
    "# Calculate and Plot the Distribution of Phonemes in a TTS Dataset\n",
    "\n",
    "In this tutorial, we will analyze the phonemes distribution of a ljspeech text corpus with a reference text corpus. The reference corpus is assumed to have the right phoneme distribution.\n",
    "\n",
    "Our analysis will comprise creating a bar graph of both reference and sample phoneme distribution. Secondly, we will also calculate the difference in ljspeech phoneme distribution compared to a reference distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e72905c1",
   "metadata": {},
   "source": [
    "Install all the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f855888",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install bokeh cmudict prettytable nemo_toolkit['all']\n",
    "!pip install protobuf==3.20.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e6ad9a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put all the imports here for ease of use, and cleanliness of code \n",
    "import os\n",
    "import pprint\n",
    "import json\n",
    "# import nemo g2p class for grapheme to phoneme\n",
    "from nemo_text_processing.g2p.modules import EnglishG2p\n",
    "\n",
    "# Bokeh\n",
    "from bokeh.io import output_notebook, show\n",
    "from bokeh.plotting import figure, output_file, show\n",
    "from bokeh.io import curdoc, export_png\n",
    "from bokeh.models import NumeralTickFormatter\n",
    "output_notebook()\n",
    "\n",
    "from prettytable import PrettyTable\n",
    "\n",
    "# Counter \n",
    "import collections\n",
    "from collections import Counter\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent =4, depth=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cdfc8c0",
   "metadata": {},
   "source": [
    "#### Get arpabet file\n",
    "\n",
    "Get the CMU grapheme to phoneme dictionary from [nemo](https://raw.githubusercontent.com/NVIDIA/NeMo/v1.12.0/scripts/tts_dataset_files/cmudict-0.7b_nv22.08)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d17e187",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/NVIDIA/NeMo/v1.12.0/scripts/tts_dataset_files/cmudict-0.7b_nv22.08"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1330c959",
   "metadata": {},
   "source": [
    "Create the g2p object. This object will be used throughout to convert grapheme to phoneme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "463951e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "g2p = EnglishG2p(phoneme_dict=\"cmudict-0.7b_nv22.08\", ignore_ambiguous_words=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "459f7ff2",
   "metadata": {},
   "source": [
    "# Reference distribution:\n",
    "\n",
    "We will use a reference distribution that not only covers _every_ phoneme in the English language (there are 44) but also exhibits the same _frequency distribution_ as English. We can therefore use this reference distribution to compare the phoneme distribution of other datasets. This enables the identification of gaps in data, such as the need for more samples with a particular phoneme."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "931481fb",
   "metadata": {},
   "source": [
    "We will use [NeMo grapheme to phoneme](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/text_processing/g2p/g2p.html) to convert words to phonemes.  \n",
    "List all the phonemes in g2p dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1c6756b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a list of the CMU phoneme symbols\n",
    "phonemes = {k: v[0] for k, v in g2p.phoneme_dict.items()}\n",
    "pp.pprint(list(g2p.phoneme_dict.items()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe68fd24",
   "metadata": {},
   "source": [
    "### Load reference distribution\n",
    "\n",
    "We will load the reference freq distribution and visualize them in this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6d3f434",
   "metadata": {},
   "outputs": [],
   "source": [
    "ref_corpus_file = \"text_files/phoneme-dist/reference_corpus.json\"\n",
    "with open(ref_corpus_file) as f:\n",
    "    freqs_ref = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7ad873",
   "metadata": {},
   "source": [
    "Lets take a peek into the reference distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bdd62c7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "freqs_ref =dict(sorted(freqs_ref.items()))\n",
    "\n",
    "xdata_ref = list(freqs_ref.keys())\n",
    "ydata_ref = list(freqs_ref.values())\n",
    "\n",
    "pp.pprint(freqs_ref)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b179a4f",
   "metadata": {},
   "source": [
    "Plot the reference frequency distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dea4040",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = figure(x_range=xdata_ref, y_range=(0, 5000), plot_height=300, plot_width=1200, title=\"Reference phoneme frequency distribution\",toolbar_location=None, tools=\"\")\n",
    "\n",
    "curdoc().theme = 'dark_minimal'\n",
    "\n",
    "# Render and show the vbar plot\n",
    "p.vbar(x=xdata_ref, top=ydata_ref)\n",
    "\n",
    "# axis titles \n",
    "p.xaxis.axis_label = \"Arpabet phonemes\"\n",
    "p.yaxis.axis_label = \"Frequency of occurrence\"\n",
    "\n",
    "# display adjustments \n",
    "p.title.text_font_size = '20px'\n",
    "p.xaxis.axis_label_text_font_size = '18px'\n",
    "p.yaxis.axis_label_text_font_size = '18px'\n",
    "p.xaxis.major_label_text_font_size = '6px'\n",
    "p.yaxis.major_label_text_font_size = '18px'\n",
    "\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dbe64c0",
   "metadata": {},
   "source": [
    "Now write a function to get the word frequencies of a text corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b36678a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_word_freq(filename):\n",
    "    \"\"\"\n",
    "        Function to find phoneme frequency of a corpus.\n",
    "        arg: filename: Text corpus filepath.\n",
    "        return: frequencies of every phoneme in the file.\n",
    "    \"\"\"\n",
    "    freqs = Counter()\n",
    "    with open(filename) as f: \n",
    "        for line in f: \n",
    "            for word in line.split(): \n",
    "                freqs.update(phonemes.get(word, [])) \n",
    "    return freqs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c962af3",
   "metadata": {},
   "source": [
    "### Sample corpus Phonemes\n",
    "\n",
    "We will take a look at the phoneme distribution of text from [ljspeech dataset](https://keithito.com/LJ-Speech-Dataset/). We dont need the LJspeech wav files for this tutorial. Therefore for simplicity, we can use the Ljspeech transcripts file from `text_files/phoneme-dist/Ljspeech_transcripts.csv` "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f821163",
   "metadata": {},
   "source": [
    "Remove wav filenames from `Ljspeech_transcripts.csv` to generate Ljspeech text corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d40da6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!awk -F\"|\" '{print $2}' text_files/phoneme-dist/Ljspeech_transcripts.csv > ljs_text_corpus.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d448a900",
   "metadata": {},
   "source": [
    "Load the Ljspeech text corpus file and create a frequency distribution of the phonemes.  \n",
    "We will use [NeMo grapheme to phoneme](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/text_processing/g2p/g2p.html) to convert words to cmu phonemes.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f35d008e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "in_file_am_eng = 'ljs_text_corpus.txt'\n",
    "freqs_ljs = get_word_freq(in_file_am_eng)\n",
    "\n",
    "# display frequencies\n",
    "freqs_ljs=dict(sorted(freqs_ljs.items()))\n",
    "\n",
    "xdata_ljs = list(freqs_ljs.keys())\n",
    "ydata_ljs = list(freqs_ljs.values())\n",
    "\n",
    "pp.pprint(freqs_ljs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3a25b43",
   "metadata": {},
   "source": [
    "Lets plot the frequency distribution of ljspeech corpus like we did for reference distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d718808",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = figure(x_range=xdata_ljs, y_range=(0, 70000), plot_height=300, plot_width=1200, title=\"Phoneme frequency distribution of ljspeech corpus\",toolbar_location=None, tools=\"\")\n",
    "\n",
    "curdoc().theme = 'dark_minimal'\n",
    "\n",
    "# Render and show the vbar plot\n",
    "p.vbar(x=xdata_ljs, top=ydata_ljs)\n",
    "\n",
    "# axis titles \n",
    "p.xaxis.axis_label = \"Arpabet phonemes\"\n",
    "p.yaxis.axis_label = \"Frequency of occurrence\"\n",
    "\n",
    "# display adjustments \n",
    "p.title.text_font_size = '20px'\n",
    "p.xaxis.axis_label_text_font_size = '18px'\n",
    "p.yaxis.axis_label_text_font_size = '18px'\n",
    "p.xaxis.major_label_text_font_size = '6px'\n",
    "p.yaxis.major_label_text_font_size = '18px'\n",
    "\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d771df6",
   "metadata": {},
   "source": [
    "### Plot distribution together\n",
    "\n",
    "we will need a canonical list of the CMU dict symbols and need to know what the length of that is compared to our corpus.  \n",
    "\n",
    "    CMU phones:  \n",
    "    ['AA0', 'AA1', 'AA2', 'AE0', 'AE1', 'AE2', 'AH0', 'AH1', 'AH2', 'AO0', 'AO1', 'AO2', 'AW0', 'AW1', 'AW2', 'AY0', 'AY1', 'AY2', 'B', 'CH', 'D', 'DH', 'EH0', 'EH1', 'EH2', 'ER0', 'ER1', 'ER2', 'EY0', 'EY1', 'EY2', 'F', 'G', 'HH', 'IH0', 'IH1', 'IH2', 'IY0', 'IY1', 'IY2', 'JH', 'K', 'L', 'M', 'N', 'NG', 'OW0', 'OW1', 'OW2', 'OY0', 'OY1', 'OY2', 'P', 'R', 'S', 'SH', 'T', 'TH', 'UH0', 'UH1', 'UH2', 'UW0', 'UW1', 'UW2', 'V', 'W', 'Y', 'Z', 'ZH']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54eb6c63",
   "metadata": {},
   "outputs": [],
   "source": [
    "arpa_list = g2p.phoneme_dict.values()\n",
    "cmu_symbols = set()\n",
    "for pronunciations in arpa_list:\n",
    "    for pronunciation in pronunciations:\n",
    "        for arpa in pronunciation:\n",
    "            cmu_symbols.add(arpa)\n",
    "cmu_symbols = list(cmu_symbols)\n",
    "cmu_symbols.sort()\n",
    "print(cmu_symbols)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af22f4f7",
   "metadata": {},
   "source": [
    "Combine two distribution together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b729651e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_combined = dict.fromkeys(cmu_symbols, 0)\n",
    "\n",
    "for key in cmu_symbols: \n",
    "    if key in freqs_ref: \n",
    "        data_combined[key] += freqs_ref[key]\n",
    "    if key in freqs_ljs: \n",
    "        data_combined[key] += freqs_ljs[key]\n",
    "\n",
    "xdata_combined = list(data_combined.keys())\n",
    "ydata_combined = list(data_combined.values())\n",
    "\n",
    "pp.pprint(data_combined)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ae847e5",
   "metadata": {},
   "source": [
    "### Phoneme comparison to a reference distribution \n",
    "\n",
    "For analysis we will now look at the two distributions together.\n",
    "\n",
    "First of all, we want to scale the reference distribution to the LJs distribution. Take the total phonemes in the Ljspeech distribution and total reference distribution phonemes and calculate a scaling factor then scale the reference distribution phoneme volume to match the Ljspeech distribution and then plot the reference phoneme distribution against the Ljspeech phoneme distribution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ac179b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the total phonemes in the reference phonemes \n",
    "ref_total = sum(freqs_ref.values())\n",
    "\n",
    "# get the total phonemes in the comparison dataset\n",
    "ljs_ph_total = sum(freqs_ljs.values())\n",
    "\n",
    "# The scale factor is: \n",
    "scale_ljs = ljs_ph_total / ref_total\n",
    "\n",
    "scaled_for_ljs = {} \n",
    "for phonemes in freqs_ref.items(): \n",
    "    scaled_for_ljs[phonemes[0]] = int(phonemes[1] * scale_ljs)\n",
    "    \n",
    "\n",
    "xscaled_for_ljs = list(scaled_for_ljs.keys())\n",
    "yscaled_for_ljs = list(scaled_for_ljs.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4ca50d6",
   "metadata": {},
   "source": [
    "plot two distributions together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ca861a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = figure(x_range=xdata_combined, y_range=(0, 65000), plot_height=300, plot_width=1200, title=\"Phoneme Ljspeech frequency distribution (purple) compared to reference distribution(white)\",toolbar_location=None, tools=\"\")\n",
    "\n",
    "curdoc().theme = 'dark_minimal'\n",
    "\n",
    "# Render and show the vbar plot - this should be a plot of the comparison dataset\n",
    "p.vbar(x=xdata_ljs, top=ydata_ljs, fill_alpha=0.9, fill_color='purple')\n",
    "\n",
    "# Render a vbar plot to show the reference distribution\n",
    "p.vbar(x=xscaled_for_ljs, top=yscaled_for_ljs, fill_alpha=0.9, fill_color='white', width=0.5)\n",
    "\n",
    "# axis titles \n",
    "p.xaxis.axis_label = \"Arpabet phonemes\"\n",
    "p.yaxis.axis_label = \"Frequency of occurrence\"\n",
    "\n",
    "# display adjustments \n",
    "p.title.text_font_size = '20px'\n",
    "p.xaxis.axis_label_text_font_size = '18px'\n",
    "p.yaxis.axis_label_text_font_size = '18px'\n",
    "p.xaxis.major_label_text_font_size = '6px'\n",
    "p.yaxis.major_label_text_font_size = '12px'\n",
    "p.yaxis.formatter = NumeralTickFormatter(format=\"0\")\n",
    "\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a23f06d1",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/nvidia-riva/tutorials/blob/main/imgs/riva-tts-phoneme-distribution-comparison.png?raw=true\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7b398e",
   "metadata": {},
   "source": [
    "### Calculate key differences between reference distribution and total phonemes\n",
    "\n",
    "Here we will check the difference in volume and percentage between two corpus for each phoneme. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee1f4763",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "diff_combined = {} \n",
    "for phonemes in data_combined.items():\n",
    "    if phonemes[0] in scaled_for_ljs : \n",
    "        difference = phonemes[1] - scaled_for_ljs[phonemes[0]]\n",
    "        min_ = min(phonemes[1], scaled_for_ljs[phonemes[0]])\n",
    "        max_ = max(phonemes[1], scaled_for_ljs[phonemes[0]])\n",
    "        percentage = round(((1 - (min_ / max_)) * 100), 2)\n",
    "        diff_combined[phonemes[0]] = (difference, percentage)       \n",
    "    else : \n",
    "        diff_combined[phonemes[0]] = (0,0)\n",
    "\n",
    "t = PrettyTable(['Phoneme', 'Difference in volume', 'Difference in percentage'])\n",
    "for phonemes in diff_combined.items() : \n",
    "    t.add_row([phonemes[0], phonemes[1][0], phonemes[1][1]])\n",
    "print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d7633a",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "In this tutorial, we saw how to analyze the phoneme distribution of our data. We can look for any phonemes that need more representation in the dataset using the techniques explained in this tutorial. This tutorial should be followed up by balancing the phoneme distribution of the data. It can be achieved by adding more sentences that contain phonemes with less coverage."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "virtualenv-riva-tutorials-py38",
   "language": "python",
   "name": "virtualenv-riva-tutorials-py38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}