Processing Greek corpora for the isopsehical riddle solver.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing Greek corpora for the riddle solver\n",
    "\n",
    "<div><br/><img src=\"delphic_sibyl.png\" style=\"width: 50%; border: 1px solid #ddd; padding: 5px\" alt=\"Michelangelo’s Delphic Sibyl, Sistine Chapel\" /></div>\n",
    "<div><center><i>Michelangel's Delphic Sibyl, Sistine Chapel</i></center><br/></div>\n",
    "\n",
    "[Pseudo-Sibylline](https://en.wikipedia.org/wiki/Sibylline_Oracles) oracles \n",
    "contain hexametric poems written in Ancient Greek. These *oracula* were\n",
    "mainly composed in 150BC - 700AD to twelve distinct extant books. They were\n",
    "circulating and quite famous among the Judaeo-Christian community at that time.\n",
    "\n",
    "They shouldn't, however, be too much confused with the earlier \n",
    "[Sibylline books](https://en.wikipedia.org/wiki/Sibylline_Books). Sibylline books\n",
    "contained religious ceremonial advices that were consulted by the selected\n",
    "priests and curators in the Roman state, when it was in deep political trouble.\n",
    "The collection of the original Sibylline books were destroyed by different\n",
    "accidental events and deliberate actions in history.\n",
    "\n",
    "Pseudo-Sibylline oracles, on the other hand, contain Jewish narrative of the\n",
    "human history contrasted to the Greek mythology and to the chronology of the\n",
    "other great ancient empires. Other intention of the material is to support\n",
    "evolving Christian doctrine and interpretation of the prophesies. Prophesies\n",
    "were mostly grounded on Jewish literature, but surprisingly some pagan world\n",
    "events also came to be interpreted as signs of the coming Messiah. Sibyl, as a\n",
    "woman prophetess, child of Noah in the Pseudo-Sibylline lore, has a unique\n",
    "character crossing over the common borders in several ancient religions and \n",
    "art.\n",
    "\n",
    "Good introductions to the Pseudo-Sibylline oracles can be found from these two\n",
    "books:\n",
    "\n",
    "1. Sibylline Oracles in [The Old Testament Pseudepigrapha, Volume \n",
    "   I](https://books.google.fi/books?id=TNdeolWctsQC) by J. J. Collins\n",
    "2. Part 1 in [The Book Three of the Sibylline Oracles and Its Social \n",
    "   Setting](https://books.google.fi/books?id=Zqh8ZQZqnWYC) by Rieuwerd Buitenwerf\n",
    "\n",
    "Some material in the Pseudo-Sibylline oracles contain cryptic puzzles, referring \n",
    "to persons, cities, countries, and epithets of God for example. These secretive \n",
    "references are often very general in nature, pointing only to the first letter \n",
    "of the subject and its numerical value. Solving them requires, not so much of \n",
    "mathematical or cryptographical skills in modern sense, but a proper knowledge \n",
    "of the context, both inner textual and historical context.\n",
    "\n",
    "Most of the alphanumeric riddles in the oracles can already been taken as solved\n",
    "by various researchers. Some of the riddles are still problematic and open for\n",
    "better proposals. Better yet, few of these open riddles are more complex and\n",
    "specific enough so that one may try to solve them by modern programmable tools.\n",
    "\n",
    "As an independent researcher not associated or affiliated by any organization,\n",
    "the sole motivation and purpose of mine in the chapters one and two is to\n",
    "provide a reusable and a testable method for processing and analysing ancient\n",
    "corpora, especially detecting alphanumeric patterns in text.\n",
    "\n",
    "## Natural language processing\n",
    "\n",
    "Programmatical approach to solve the riddles requires huge Greek text corpora.\n",
    "Bigger it is, the better. I will download and preprocess available open source\n",
    "Greek corpora, which is a quite daunting task for many reasons. Programming\n",
    "language of my choice is [Python](http://python.org) for it has plenty\n",
    "of good and stable open source libraries required for my work. Python is widely\n",
    "recognized in academic and scientific circles and well oriented to the research\n",
    "projects.\n",
    "\n",
    "I have left the most of the overly technical details of these chapters for the\n",
    "enthusiasts to read straight from the commented code in \n",
    "[functions.py](https://git.io/vAS2Z) script. By collecting the large part of the \n",
    "used procedures to the separate script maintains this document more concise too.\n",
    "\n",
    "In the end of the task of the first chapter, I'll have a word database\n",
    "containing hundreds of thousands of unique Greek words extracted from the\n",
    "naturally written language corpora. Then words can be further used in the riddle \n",
    "solver in the second chapter.\n",
    "\n",
    "Note that rather than just reading, this, and the following chapters can also be \n",
    "run interactively in your local [Jupyter notebook](https://jupyter.org/) \n",
    "installation, if you prefer. That means that you may test and verify the procedure \n",
    "or alter parameters and try solving the riddles with your own parameters.\n",
    "\n",
    "Your can download independent Jupyter notebooks for [processing corpora](https://git.io/vASwM), \n",
    "[solving riddles](https://git.io/vASrY), and [analysing results](https://).\n",
    "\n",
    "You may also run code directly from [Python shell](https://www.python.org/shell/) \n",
    "environment, no problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Required components\n",
    "\n",
    "The first sub task is to get a big raw ancient Greek text to operate with. I have\n",
    "implemented an importer interface with [tqdm](https://github.com/tqdm/tqdm) library \n",
    "to the [Perseus](http://www.perseus.tufts.edu/hopper/opensource/download) and the \n",
    "[First1KGreek](http://opengreekandlatin.github.io/First1KGreek/) open source data \n",
    "sources in this chapter.\n",
    "\n",
    "I'm using my own [Abnum](https://github.com/markomanninen/abnum3) library to remove \n",
    "accents from the Greek words, remove non-alphabetical characters from the corpora, \n",
    "as well as calculating the isopsephical value of the Greek words. \n",
    "[Greek_accentuation](https://github.com/jtauber/greek-accentuation) library is used \n",
    "to split words into syllables. This is required because the riddles of my closest \n",
    "interest contain specific information about the syllables of the words. \n",
    "[Pandas](http://pandas.pydata.org/) library is used as an API (application programming \n",
    "interface) to the collected database. [Plotly](https://plot.ly/) library and online \n",
    "infographic service are used for the visual presentation of the statistics.\n",
    "\n",
    "You can install these libraries by uncommenting and running the next install lines in \n",
    "the Jupyter notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "#!{sys.executable} -m pip install tqdm abnum\n",
    "#!{sys.executable} -m pip install pandas plotly\n",
    "#!{sys.executable} -m pip install greek_accentuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For your convenience, my environment is the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:05:27) [MSC v.1900 64 bit (AMD64)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Python %s\" % sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `Python 3.4+` is required for all examples to work properly. To find\n",
    "out other ways of installing PyPI maintained libraries, please consult:\n",
    "https://packaging.python.org/tutorials/installing-packages/\n",
    "\n",
    "#### Downloading corpora\n",
    "\n",
    "I'm going to use `Perseus` and `OpenGreekAndLatin` corpora for the study by\n",
    "combining them into a single raw text file and unique words database.\n",
    "\n",
    "The next code snippets will download hundreds of megabytes of Greek text to a\n",
    "local computer for quicker access. `tqdm` downloader requires a stable internet\n",
    "connection to work properly.\n",
    "\n",
    "One could also download source zip files via browser and place them to the same\n",
    "directory with the Jupyter notebook or where Python is optionally run in shell\n",
    "mode. Zip files must then be renamed as `perseus.zip` and `first1k.zip`.\n",
    "\n",
    "1\\. Download packed zip files from their GitHub repositories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading: https://github.com/PerseusDL/canonical-greekLit/archive/master.zip\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "70.8MB [00:19, 3.82MB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading: https://github.com/OpenGreekAndLatin/First1KGreek/archive/master.zip\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "192MB [00:56, 3.57MB/s] \n"
     ]
    }
   ],
   "source": [
    "from functions import download_with_indicator, perseus_zip_file, first1k_zip_file\n",
    "# download perseus files\n",
    "fs = \"https://github.com/PerseusDL/canonical-greekLit/archive/master.zip\"\n",
    "download_with_indicator(fs, perseus_zip_file)\n",
    "# download first1k files\n",
    "fs = \"https://github.com/OpenGreekAndLatin/First1KGreek/archive/master.zip\"\n",
    "download_with_indicator(fs, first1k_zip_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2\\. Unzip files to the corresponding directories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from functions import perseus_zip_dir, first1k_zip_dir, unzip\n",
    "# first argument is the zip source, second is the destination dir\n",
    "unzip(perseus_zip_file, perseus_zip_dir)\n",
    "unzip(first1k_zip_file, first1k_zip_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3\\. Copy only suitable Greek text xml files from `perseus_zip_dir` and\n",
    "`first1k_zip_dir` to the temporary work directories. Original repositories\n",
    "contain a lot of unnecessary files for the riddle solver which are skipped in\n",
    "this process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying greek_text_perseus_zip\\canonical-greekLit-master\\data -> greek_text_perseus_tmp\n",
      "Copying greek_text_first1k_zip\\First1KGreek-master\\data -> greek_text_first1k_tmp\n"
     ]
    }
   ],
   "source": [
    "from functions import copy_corpora, joinpaths, perseus_tmp_dir, first1k_tmp_dir\n",
    "# important files resides in the data directory of the repositories\n",
    "for item in [[joinpaths(perseus_zip_dir,\n",
    "              [\"canonical-greekLit-master\", \"data\"]), perseus_tmp_dir],\n",
    "             [joinpaths(first1k_zip_dir,\n",
    "              [\"First1KGreek-master\", \"data\"]), first1k_tmp_dir]]:\n",
    "    copy_corpora(*item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on if the files have been downloaded already, the output may differ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collecting files\n",
    "\n",
    "When the files has been downloaded and copied, it is time to read them to the\n",
    "RAM (Random-Access Memory). At this point file paths are collected to the\n",
    "`greek_corpora_x` variable that is used on later iterators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unidentified language tag found from the file.\n",
      "Please check and possibly add to the ignore list: greek_text_perseus_tmp\\tlg0007\\tlg091\\.DS_Store\n",
      "1720 files found\n"
     ]
    }
   ],
   "source": [
    "from functions import init_corpora, perseus_dir, first1k_dir\n",
    "# collect files and initialize data dictionary\n",
    "greek_corpora_x = init_corpora([[perseus_tmp_dir, perseus_dir], [first1k_tmp_dir, first1k_dir]])\n",
    "print(len(greek_corpora_x), \"files found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actual files found may differ by increasing over time, because Greek corpora\n",
    "repositories are constantly maintained and new texts are added by voluteer\n",
    "contributors.\n",
    "\n",
    "#### Processing files\n",
    "\n",
    "Next step is to extract Greek content from the downloaded and selected XML\n",
    "source files. Usually this task might take a lot of effort in NLP. Python \n",
    "[NLTK](https://www.nltk.org/) and [CLTK](https://github.com/cltk/cltk) libraries\n",
    "would be useful at this point, but in my case I'm only interested of Greek words,\n",
    "that is, text content that has a certain \n",
    "[Greek Unicode](https://en.wikipedia.org/wiki/Greek_alphabet#Greek_in_Unicode) \n",
    "letter block. Thus I'm able to simplify this part by removing all other characters \n",
    "from source files. Again, details can be found from the \n",
    "[functions.py](https://git.io/vAS2Z) script.\n",
    "\n",
    "Extracted content is saved to the `corpora/author/work` based directories.\n",
    "Simplified uncial conversion is also made at the same time so that the final\n",
    "data contains only plain uppercase words separated by spaces. Pretty much in a\n",
    "format written by the ancient Greeks, except they didn't have even spaces to\n",
    "denote individual words and phrases.\n",
    "\n",
    "<div><br/><img src=\"P47.png\" style=\"width: 75%; border: 0px solid #ddd; padding: 5px\" alt=\"Papyrus 47, Uncial Greek text without spaces\" /></div>\n",
    "<div><center><i>Papyrus 47, Uncial Greek text without spaces. Rev 13:17-</i></center><br/></div>\n",
    "\n",
    "This will take several minutes depending on if you have already run it once and\n",
    "have the previous temporary directories available. Old processed corpora files\n",
    "are removed first, then they are recreated by calling `process_greek_corpora`\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1720/1720 [01:43<00:00, 16.56it/s]\n"
     ]
    }
   ],
   "source": [
    "from functions import remove, all_greek_text_file, perseus_greek_text_file, \\\n",
    "                      first1k_greek_text_file, process_greek_corpora\n",
    "# remove old processed temporary files\n",
    "try:\n",
    "    remove(all_greek_text_file)\n",
    "    remove(perseus_greek_text_file)\n",
    "    remove(first1k_greek_text_file)\n",
    "except OSError:\n",
    "    pass\n",
    "# process and get greek corpora data to the RAM memory\n",
    "# one could use a filter to process only selected files here...\n",
    "#greek_corpora = process_greek_corpora(list(filter(lambda x: \"aristot.nic.eth_gk.xml\" in x['file'], greek_corpora_x)))\n",
    "greek_corpora = process_greek_corpora(greek_corpora_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics\n",
    "\n",
    "After the files have been downloaded and preprocessed, I'm going to output the\n",
    "size of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the all raw text: 354.58 MB\n",
      "Size of the perseus raw text: 110.71 MB\n",
      "Size of the first1k raw text: 243.87 MB\n"
     ]
    }
   ],
   "source": [
    "from functions import get_file_size\n",
    "\n",
    "print(\"Size of the all raw text: %s MB\" % get_file_size(all_greek_text_file))\n",
    "print(\"Size of the perseus raw text: %s MB\" % get_file_size(perseus_greek_text_file))\n",
    "print(\"Size of the first1k raw text: %s MB\" % get_file_size(first1k_greek_text_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, I will calculate other statistics of the saved text files to compare their\n",
    "content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Corpora: perseus_greek_text_files.txt\n",
      "Letters: 52943492\n",
      "Words in total: 10199139\n",
      "Unique words: 425306\n",
      "\n",
      "Corpora: first1k_greek_text_files.txt\n",
      "Letters: 116083862\n",
      "Words in total: 23549150\n",
      "Unique words: 684129\n",
      "\n",
      "Corpora: all_greek_text_files.txt\n",
      "Letters: 169027354\n",
      "Words in total: 33748289\n",
      "Unique words: 847734\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from functions import get_stats\n",
    "\n",
    "ccontent1, chars1, lwords1 = get_stats(perseus_greek_text_file)\n",
    "ccontent2, chars2, lwords2 = get_stats(first1k_greek_text_file)\n",
    "ccontent3, chars3, lwords3 = get_stats(all_greek_text_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Letter statistics\n",
    "\n",
    "I'm using `DataFrame` class from `Pandas` library to handle tabular data and\n",
    "show basic letter statistics for each corpora and combination of them. Native\n",
    "`Counter` class in Python is used to count unique elements in the given\n",
    "sequence. Sequence in this case is the raw Greek text stripped from all special\n",
    "characters and spaces, and elements are the letters of the Greek alphabet.\n",
    "\n",
    "This will take some time to process too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functions import Counter, DataFrame\n",
    "# perseus dataframe\n",
    "df = DataFrame([[k, v] for k, v in Counter(ccontent1).items()])\n",
    "df[2] = df[1].apply(lambda x: round(x*100/chars1, 2))\n",
    "a = df.sort_values(1, ascending=False)\n",
    "# first1k dataframe\n",
    "df = DataFrame([[k, v] for k, v in Counter(ccontent2).items()])\n",
    "df[2] = df[1].apply(lambda x: round(x*100/chars2, 2))\n",
    "b = df.sort_values(1, ascending=False)\n",
    "# perseus + first1k dataframe\n",
    "df = DataFrame([[k, v] for k, v in Counter(ccontent3).items()])\n",
    "df[2] = df[1].apply(lambda x: round(x*100/chars3, 2))\n",
    "c = df.sort_values(1, ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first column is the letter, the second column is the count of the letter,\n",
    "and the third column is the percentage of the letter contra all letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"display:inline\" border=\"1\" class=\"dataframe\">\n",
       "  <caption style='text-align:center'>Perseus_First1K</caption><thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Letter</th>\n",
       "      <th>Count</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Α</td>\n",
       "      <td>18587337</td>\n",
       "      <td>11.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ο</td>\n",
       "      <td>16419906</td>\n",
       "      <td>9.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ι</td>\n",
       "      <td>16345105</td>\n",
       "      <td>9.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ε</td>\n",
       "      <td>16011033</td>\n",
       "      <td>9.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ν</td>\n",
       "      <td>14627042</td>\n",
       "      <td>8.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Τ</td>\n",
       "      <td>13674580</td>\n",
       "      <td>8.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Σ</td>\n",
       "      <td>12270512</td>\n",
       "      <td>7.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Υ</td>\n",
       "      <td>8056291</td>\n",
       "      <td>4.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ρ</td>\n",
       "      <td>6413496</td>\n",
       "      <td>3.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Η</td>\n",
       "      <td>6399799</td>\n",
       "      <td>3.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Κ</td>\n",
       "      <td>5800995</td>\n",
       "      <td>3.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Π</td>\n",
       "      <td>5520079</td>\n",
       "      <td>3.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ω</td>\n",
       "      <td>5338802</td>\n",
       "      <td>3.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Μ</td>\n",
       "      <td>4950882</td>\n",
       "      <td>2.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Λ</td>\n",
       "      <td>4684636</td>\n",
       "      <td>2.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Δ</td>\n",
       "      <td>3883629</td>\n",
       "      <td>2.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Γ</td>\n",
       "      <td>2697096</td>\n",
       "      <td>1.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Θ</td>\n",
       "      <td>2235674</td>\n",
       "      <td>1.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Χ</td>\n",
       "      <td>1515785</td>\n",
       "      <td>0.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Φ</td>\n",
       "      <td>1398848</td>\n",
       "      <td>0.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Β</td>\n",
       "      <td>950600</td>\n",
       "      <td>0.56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ξ</td>\n",
       "      <td>624479</td>\n",
       "      <td>0.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ζ</td>\n",
       "      <td>376643</td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ψ</td>\n",
       "      <td>237690</td>\n",
       "      <td>0.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϛ</td>\n",
       "      <td>5484</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϝ</td>\n",
       "      <td>506</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϡ</td>\n",
       "      <td>298</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϟ</td>\n",
       "      <td>114</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϙ</td>\n",
       "      <td>13</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table style=\"display:inline\"><table style=\"display:inline\" border=\"1\" class=\"dataframe\">\n",
       "  <caption style='text-align:center'>First1K</caption><thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Letter</th>\n",
       "      <th>Count</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Α</td>\n",
       "      <td>12783018</td>\n",
       "      <td>11.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ο</td>\n",
       "      <td>11346903</td>\n",
       "      <td>9.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ι</td>\n",
       "      <td>11314346</td>\n",
       "      <td>9.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ε</td>\n",
       "      <td>10940578</td>\n",
       "      <td>9.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ν</td>\n",
       "      <td>9955758</td>\n",
       "      <td>8.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Τ</td>\n",
       "      <td>9626685</td>\n",
       "      <td>8.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Σ</td>\n",
       "      <td>8333217</td>\n",
       "      <td>7.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Υ</td>\n",
       "      <td>5577444</td>\n",
       "      <td>4.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Η</td>\n",
       "      <td>4455881</td>\n",
       "      <td>3.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ρ</td>\n",
       "      <td>4375304</td>\n",
       "      <td>3.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Κ</td>\n",
       "      <td>4009488</td>\n",
       "      <td>3.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Π</td>\n",
       "      <td>3700865</td>\n",
       "      <td>3.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ω</td>\n",
       "      <td>3688546</td>\n",
       "      <td>3.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Μ</td>\n",
       "      <td>3364161</td>\n",
       "      <td>2.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Λ</td>\n",
       "      <td>3098958</td>\n",
       "      <td>2.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Δ</td>\n",
       "      <td>2587249</td>\n",
       "      <td>2.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Γ</td>\n",
       "      <td>1899739</td>\n",
       "      <td>1.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Θ</td>\n",
       "      <td>1538553</td>\n",
       "      <td>1.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Χ</td>\n",
       "      <td>1025308</td>\n",
       "      <td>0.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Φ</td>\n",
       "      <td>952109</td>\n",
       "      <td>0.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Β</td>\n",
       "      <td>638605</td>\n",
       "      <td>0.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ξ</td>\n",
       "      <td>425018</td>\n",
       "      <td>0.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ζ</td>\n",
       "      <td>268518</td>\n",
       "      <td>0.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ψ</td>\n",
       "      <td>171568</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϛ</td>\n",
       "      <td>5469</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϡ</td>\n",
       "      <td>294</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϝ</td>\n",
       "      <td>156</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϟ</td>\n",
       "      <td>111</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϙ</td>\n",
       "      <td>13</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table style=\"display:inline\"><table style=\"display:inline\" border=\"1\" class=\"dataframe\">\n",
       "  <caption style='text-align:center'>Perseus</caption><thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Letter</th>\n",
       "      <th>Count</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Α</td>\n",
       "      <td>5804319</td>\n",
       "      <td>10.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ο</td>\n",
       "      <td>5073003</td>\n",
       "      <td>9.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ε</td>\n",
       "      <td>5070455</td>\n",
       "      <td>9.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ι</td>\n",
       "      <td>5030759</td>\n",
       "      <td>9.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ν</td>\n",
       "      <td>4671284</td>\n",
       "      <td>8.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Τ</td>\n",
       "      <td>4047895</td>\n",
       "      <td>7.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Σ</td>\n",
       "      <td>3937295</td>\n",
       "      <td>7.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Υ</td>\n",
       "      <td>2478847</td>\n",
       "      <td>4.68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ρ</td>\n",
       "      <td>2038192</td>\n",
       "      <td>3.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Η</td>\n",
       "      <td>1943918</td>\n",
       "      <td>3.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Π</td>\n",
       "      <td>1819214</td>\n",
       "      <td>3.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Κ</td>\n",
       "      <td>1791507</td>\n",
       "      <td>3.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ω</td>\n",
       "      <td>1650256</td>\n",
       "      <td>3.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Μ</td>\n",
       "      <td>1586721</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Λ</td>\n",
       "      <td>1585678</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Δ</td>\n",
       "      <td>1296380</td>\n",
       "      <td>2.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Γ</td>\n",
       "      <td>797357</td>\n",
       "      <td>1.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Θ</td>\n",
       "      <td>697121</td>\n",
       "      <td>1.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Χ</td>\n",
       "      <td>490477</td>\n",
       "      <td>0.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Φ</td>\n",
       "      <td>446739</td>\n",
       "      <td>0.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Β</td>\n",
       "      <td>311995</td>\n",
       "      <td>0.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ξ</td>\n",
       "      <td>199461</td>\n",
       "      <td>0.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ζ</td>\n",
       "      <td>108125</td>\n",
       "      <td>0.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ψ</td>\n",
       "      <td>66122</td>\n",
       "      <td>0.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϝ</td>\n",
       "      <td>350</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϛ</td>\n",
       "      <td>15</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϡ</td>\n",
       "      <td>4</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ϟ</td>\n",
       "      <td>3</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table style=\"display:inline\">"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from functions import display_side_by_side\n",
    "# show tables side by side to save some vertical space\n",
    "display_side_by_side(Perseus=a, First1K=b, Perseus_First1K=c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`First1K` corpora contains mathematical texts in Greek, which explains why the rarely used digamma (Ϛ = 6), qoppa (Ϟ/Ϙ = 90), and sampi(Ϡ = 900) letters are included on the table. You can find other interesting differences too, like the occurrence of E/T, K/Π, and M/Λ, which are probably explained by the difference of the included text genres in the corpora.\n",
    "\n",
    "#### Bar chart\n",
    "\n",
    "The next chart will show visually which are the most used letters and the least\n",
    "used letters in the available Ancient Greek corpora.\n",
    "\n",
    "<img src=\"stats.png\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vowels with `N`, `S`, and `T` consonants pops up as the most used letters. The least used letters are `Z`, `Chi`, and `Psi`.\n",
    "\n",
    "#### Optional live chart\n",
    "\n",
    "Uncomment the next part to output a new fresh graph from Plotly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from plotly.offline import init_notebook_mode\n",
    "#init_notebook_mode(connected=False)\n",
    "\n",
    "# for the fist time set plotly service credentials, then you can comment the next line\n",
    "#import plotly\n",
    "#plotly.tools.set_credentials_file(username='MarkoManninen', api_key='xyz')\n",
    "\n",
    "# use tables and graphs...\n",
    "#import plotly.tools as tls\n",
    "# embed plotly graphs\n",
    "#tls.embed(\"https://plot.ly/~MarkoManninen/8/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unique words database\n",
    "\n",
    "Now it is time to collect unique Greek words to the database and show certain\n",
    "specialties of the word statistics. I'm reusing data from the `greek_corpora`\n",
    "variable that is in the memory already. Running the next code will take a\n",
    "minute or two depending on the processor speed of your computer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functions import syllabify, Abnum, greek, vowels\n",
    "\n",
    "# greek abnum object for calculating isopsephical value\n",
    "g = Abnum(greek)\n",
    "\n",
    "# lets count unique words statistic from the parsed greek corpora rather than the plain text file\n",
    "# it would be pretty dauntful to find out occurence of the all 800000+ unique words from the text \n",
    "# file that is over 600 MB big!\n",
    "unique_word_stats = {}\n",
    "for item in greek_corpora:\n",
    "    for word, cnt in item['uwords'].items():\n",
    "        if word not in unique_word_stats:\n",
    "            unique_word_stats[word] = 0\n",
    "        unique_word_stats[word] += cnt\n",
    "\n",
    "# init dataframe\n",
    "df = DataFrame([[k, v] for k, v in unique_word_stats.items()])\n",
    "# add column for the occurrence percentage of the word\n",
    "df[2] = df[1].apply(lambda x: round(x*100/lwords3, 2))\n",
    "# add column for the length of the word\n",
    "df[3] = df[0].apply(lambda x: len(x))\n",
    "# add isopsephy column\n",
    "df[4] = df[0].apply(lambda x: g.value(x))\n",
    "# add syllabified column\n",
    "df[5] = df[0].apply(lambda x: syllabify(x))\n",
    "# add length of the syllables column\n",
    "df[6] = df[5].apply(lambda x: len(x))\n",
    "# count vowels in the word as a column\n",
    "df[7] = df[0].apply(lambda x: sum(list(x.count(c) for c in vowels)))\n",
    "# count consonants in the word as a column\n",
    "df[8] = df[0].apply(lambda x: len(x) - sum(list(x.count(c) for c in vowels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store database\n",
    "\n",
    "This is the single most important part of the chapter. I'm saving all\n",
    "simplified unique words as a CSV file that can be used as a database for the\n",
    "riddle solver. After this you may proceed to the \n",
    "[riddle solver](https://git.io/vASrY) Jupyter notebook document in interactive \n",
    "mode, if you prefer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functions import csv_file_name\n",
    "# save dataframe to CSV file\n",
    "df.to_csv(csv_file_name, header=False, index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Noteworth is that stored words are not stems or any base forms of the words but\n",
    "contain words in all possible inflected forms. Due to nature of machine\n",
    "processed texts, one should also be warned about corrupted words and other noise\n",
    "to occur in results. Programming tools are good for extracting interesting\n",
    "content and filtering data that would be impossible for a human to do because\n",
    "of its enormous size. But results still need verification and interpretation,\n",
    "also procedures can be fine tuned and developed in many ways.\n",
    "\n",
    "### Most repeated words\n",
    "\n",
    "For a confirmation of the succesful task, I will show the total number of the\n",
    "unique words, and five of the most repeated words in the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total records: 847734\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Word</th>\n",
       "      <th>Count</th>\n",
       "      <th>Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>ΚΑΙ</td>\n",
       "      <td>1820060</td>\n",
       "      <td>5.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΔΕ</td>\n",
       "      <td>802702</td>\n",
       "      <td>2.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΤΟ</td>\n",
       "      <td>681699</td>\n",
       "      <td>2.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΤΩΝ</td>\n",
       "      <td>495983</td>\n",
       "      <td>1.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Η</td>\n",
       "      <td>495212</td>\n",
       "      <td>1.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from functions import display_html\n",
    "# use to_html and index=False to hide index column and output table\n",
    "words = df.sort_values(1, ascending=False).head(n=5).iloc[:,0:3]\n",
    "words.columns = ['Word', 'Count', 'Percent']\n",
    "print(\"Total records: %s\" % len(df))\n",
    "display_html(words.to_html(index=False), raw=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "KAI...\n",
    "\n",
    "For a curiosity, let's also see the longest words in the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Word</th>\n",
       "      <th>Count</th>\n",
       "      <th>Length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>ΠΑΡΕΓΕΝΟΜΕΝΟΜΕΝΟΣΗΝΚΑΙΕΤΙΕΚΤΗΣΛΕΣΒΟΥΟΥΦΑΜΕΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΛΛΗΣΤΗΣΑΝΩΘΕΝΘΕΡΜΟΤΗΤΟΣΑΤΜΙΔΟΥΜΕΝΟΝΦΕΡΕΤΑΙ</td>\n",
       "      <td>1</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΕΜΟΥΟΙΑΠΕΦΕΥΓΑΧΕΙΡΑΣΛΥΠΗΣΑΣΜΕΝΟΥΔΕΝΑΟΥΔΕΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΠΥΡΟΒΡΟΜΟΛΕΥΚΕΡΕΒΙΝΘΟΑΚΑΝΘΙΔΟΜΙΚΡΙΤΡΙΑΔΥ</td>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΔΥΝΑΤΟΝΔΕΤΟΑΙΤΙΑΙΗΣΓΕΝΕΣΕΩΣΚΑΙΤΗΣΦΘΟΡΑΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΠΥΡΒΡΟΜΟΛΕΥΚΕΡΕΒΙΝΘΟΑΚΑΝΘΟΥΜΙΚΤΡΙΤΥΑΔΥ</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΕΝΝΕΑΚΑΙΕΙΚΟΣΙΚΑΙΕΠΤΑΚΟΣΙΟΠΛΑΣΙΑΚΙΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΚΑΙΙΚΕΛΗΧΡΥΣΗΑΦΡΟΔΙΤΗΚΑΙΟΙΣΕΚΟΣΜΗΣΕ</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΚΑΙΤΟΝΑΡΙΣΤΑΡΧΟΝΑΣΜΕΝΩΣΤΗΝΓΡΑΦΗΝΤΟΥ</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΑΡΣΕΝΙΚΩΝΟΝΟΜΑΤΩΝΣΤΟΙΧΕΙΑΕΣΤΙΠΕΝΤΕ</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΡΘΡΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΤΙΤΟΥΜΗΔΙΑΠΡΟΤΕΡΩΝΟΡΙΖΕΣΘΑΙΤΡΕΙΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΤΕΤΤΑΡΑΚΟΝΤΑΚΑΙΠΕΝΤΑΚΙΣΧΙΛΙΟΣΤΟΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΑΡΙΣΤΑΡΧΟΣΚΑΙΟΙΑΠΟΤΗΣΣΧΟΛΗΣΦΑΣΙΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΥΝΙΚΑΝΩΣΠΕΡΙΑΥΤΩΝΗΜΙΝΕΝΤΟΙΣΠΕΡΙ</td>\n",
       "      <td>1</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΗΔΙΚΗΜΕΝΟΝΔΕΑΠΕΡΡΙΜΜΕΝΟΝΠΕΡΙΟΡΑΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΡΘΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ</td>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΑΥΤΟΜΑΤΟΙΔΕΟΙΘΕΟΙΑΠΑΛΛΑΣΣΟΜΕΝΟΙ</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΣΠΕΡΜΑΓΟΡΑΙΟΛΕΚΙΘΟΛΑΧΑΝΟΠΩΛΙΔΕΣ</td>\n",
       "      <td>3</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΤΟΞΑΙΣΕΛΑΙΟΞΑΝΘΕΠΙΠΑΓΚΑΠΥΡΩΤΟΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from functions import HTML\n",
    "# load result to the temporary variable for later usage\n",
    "l = df.sort_values(3, ascending=False).head(n=20)\n",
    "l = l[[0, 1, 3]]\n",
    "l.columns = ['Word', 'Count', 'Length']\n",
    "# output table\n",
    "HTML(l.to_html(index=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about finding out, which words have the biggest isopsephical values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Word</th>\n",
       "      <th>Count</th>\n",
       "      <th>Isopsephy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>ΟΡΘΡΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>5186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΒΡΥΣΩΝΟΘΡΑΣΥΜΑΧΕΙΟΛΗΨΙΚΕΡΜΑΤΩΝ</td>\n",
       "      <td>2</td>\n",
       "      <td>5122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΡΘΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ</td>\n",
       "      <td>2</td>\n",
       "      <td>5086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΓΛΩΣΣΟΤΟΜΗΘΕΝΤΩΝΧΡΙΣΤΙΑΝΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>5056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΚΑΙΤΟΝΑΡΙΣΤΑΡΧΟΝΑΣΜΕΝΩΣΤΗΝΓΡΑΦΗΝΤΟΥ</td>\n",
       "      <td>1</td>\n",
       "      <td>4969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΑΡΣΕΝΙΚΩΝΟΝΟΜΑΤΩΝΣΤΟΙΧΕΙΑΕΣΤΙΠΕΝΤΕ</td>\n",
       "      <td>1</td>\n",
       "      <td>4768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΛΛΗΣΤΗΣΑΝΩΘΕΝΘΕΡΜΟΤΗΤΟΣΑΤΜΙΔΟΥΜΕΝΟΝΦΕΡΕΤΑΙ</td>\n",
       "      <td>1</td>\n",
       "      <td>4754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΕΠΙΣΚΟΠΩΚΩΝΣΤΑΝΤΙΝΟΥΠΟΛΕΩΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>4701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΚΩΔΩΝΟΦΑΛΑΡΑΧΡΩΜΕΝΟΥΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>4642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΕΜΟΥΟΙΑΠΕΦΕΥΓΑΧΕΙΡΑΣΛΥΠΗΣΑΣΜΕΝΟΥΔΕΝΑΟΥΔΕΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΔΥΝΑΤΟΝΔΕΤΟΑΙΤΙΑΙΗΣΓΕΝΕΣΕΩΣΚΑΙΤΗΣΦΘΟΡΑΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>4481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΤΩΟΡΘΩΕΚΑΣΤΑΘΕΩΡΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΣΥΝΥΠΟΧΩΡΟΥΝΤΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΠΡΩΤΟΣΑΝΘΡΩΠΩΝΥΠΟΔΕΙΞΑΣ</td>\n",
       "      <td>1</td>\n",
       "      <td>4340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΟΥΝΙΚΑΝΩΣΠΕΡΙΑΥΤΩΝΗΜΙΝΕΝΤΟΙΣΠΕΡΙ</td>\n",
       "      <td>1</td>\n",
       "      <td>4285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΩΡΙΣΜΕΝΩΝΠΡΟΣΩΠΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΑΡΙΣΤΑΡΧΟΣΚΑΙΟΙΑΠΟΤΗΣΣΧΟΛΗΣΦΑΣΙΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΤΟΥΤΟΥΣΛΕΓΟΝΤΕΣΩΣΠΡΟΣΤΗΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΨΥΧΟΓΟΝΙΜΩΤΑΤΩΝ</td>\n",
       "      <td>1</td>\n",
       "      <td>4194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ΚΙΧΛΕΠΙΚΟΣΣΥΦΟΦΑΤΤΟΠΕΡΙΣΤΕΡΑ</td>\n",
       "      <td>2</td>\n",
       "      <td>4187</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort by the isopsephy column\n",
    "m = df.sort_values(4, ascending=False).head(n=20)\n",
    "m = m[[0, 1, 4]]\n",
    "m.columns = ['Word', 'Count', 'Isopsephy']\n",
    "# output table\n",
    "HTML(m.to_html(index=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many percent of the whole word base, the least repeated words take:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "words repeating 1 time(s):  44.7 %\n",
      "words repeating 2 time(s):  16.05 %\n",
      "words repeating 3 time(s):  7.46 %\n",
      "words repeating 4 time(s):  4.85 %\n",
      "words repeating 5 time(s):  3.33 %\n",
      "words repeating 6 time(s):  2.5 %\n",
      "words repeating 7 time(s):  1.93 %\n",
      "words repeating 8 time(s):  1.57 %\n",
      "words repeating 9 time(s):  1.29 %\n",
      "words repeating 10 time(s):  1.12 %\n"
     ]
    }
   ],
   "source": [
    "# length of the words database\n",
    "le = len(df)\n",
    "# group words by occurrence and count grouped items, list the first 10 items\n",
    "for x, y in df.groupby([1, 2]).count()[:10].T.items():\n",
    "    print(\"words repeating %s time(s): \" % x[0], round(100*y[0]/le, 2), \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words that repeat 1-4 times fills over 73% of the whole text. Words repeating three times takes 16.5% of the words being the greatest repeatance factor.\n",
    "\n",
    "Finally, for cross checking the data processing algorithm, I want to know in which texts the longest words occur:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " + Aristophanes, Lysistrata (tlg0019.tlg007.perseus-grc2.xml) =>    \n",
      "\n",
      "   ----- ΣΠΕΡΜΑΓΟΡΑΙΟΛΕΚΙΘΟΛΑΧΑΝΟΠΩΛΙΔΕΣ (1) -----\n",
      "   ὦ ξύμμαχοι γυναῖκες ἐκθεῖτ ἔνδοθεν ὦ σπερμαγοραιολεκιθολαχανοπώλιδες ὦ σκοροδοπανδοκευτριαρτοπώλιδες\n",
      "\n",
      " + Aristophanes, Wasps (tlg0019.tlg004.perseus-grc1.xml) =>    \n",
      "\n",
      "   ----- ΟΡΘΡΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ (1) -----\n",
      "   ς ἀκούειν ἡδἔ εἰ καὶ νῦν ἐγὼ τὸν πατέρ ὅτι βούλομαι τούτων ἀπαλλαχθέντα τῶν ὀρθροφοιτοσυκοφαντοδικοταλαιπώρων τρόπων ζῆν βίον γενναῖον ὥσπερ Μόρυχος αἰτίαν ἔχω ταῦτα δρᾶν ξυνωμότης ὢν καὶ φρονῶν\n",
      "\n",
      " + Athenaeus, Deipnosophistae (tlg0008.tlg001.perseus-grc3.xml) =>    \n",
      "\n",
      "   ----- ΤΟΞΑΙΣΕΛΑΙΟΞΑΝΘΕΠΙΠΑΓΚΑΠΥΡΩΤΟΣ (1) -----\n",
      "   προλαμβάνοντες ὑστεροῦμεν τἀγαθοῦ Ξενοφῶντος δὲ τὸν Ἀγησίλαον μέθης μὲν ἀπέχεσθαι ὁμοίως ᾤετο χρῆναι καὶ μανίας σίτων δὲ ὑπερ\n",
      "\n",
      "   ----- ΠΥΡΒΡΟΜΟΛΕΥΚΕΡΕΒΙΝΘΟΑΚΑΝΘΟΥΜΙΚΤΡΙΤΥΑΔΥ (1) -----\n",
      "   νῦν δὲ πρότερόν γε τοῦ πόνου τὴν ἡδονὴν προλαμβάνοντες ὑστεροῦμεν τἀγαθοῦ Ξεν\n",
      "\n",
      " + Athenaeus, TheDeipnosophists (tlg0008.tlg001.perseus-grc4.xml) =>    \n",
      "\n",
      "   ----- ΠΥΡΟΒΡΟΜΟΛΕΥΚΕΡΕΒΙΝΘΟΑΚΑΝΘΙΔΟΜΙΚΡΙΤΡΙΑΔΥ (1) -----\n",
      "   Ζανὸς καλέοντι τρώγματ ἐπεί γ ἐπένειμεν ἐγκατακνακομιγὲς πεφρυγμένον πυροβρομολευκερεβινθοακανθιδομικριτριαδυ βρωματοπαντανάμικτον ἄμπυκι καριδίᾳ στ\n",
      "\n",
      " + Plato, Laws (tlg0059.tlg034.perseus-grc2.xml) =>    \n",
      "\n",
      "   ----- ΤΕΤΤΑΡΑΚΟΝΤΑΚΑΙΠΕΝΤΑΚΙΣΧΙΛΙΟΣΤΟΝ (1) -----\n",
      "   ἐκγόνων τὴν δὲ οὐσίαν αὐτοὺς κεκτῆσθαι ἄπαις δὲ ὅστις ἂν τοιαύταις συμφοραῖς περιπέσῃ τοὺς συγγενεῖς συνελθόντας μέχρι ἀνεψιῶν παίδων τοῦ πεφευγότος ἀμφοτέρωθεν πρός τε ἀνδρῶ\n",
      "\n",
      " + Plato, Republic (tlg0059.tlg030.perseus-grc2.xml) =>    \n",
      "\n",
      "   ----- ΕΝΝΕΑΚΑΙΕΙΚΟΣΙΚΑΙΕΠΤΑΚΟΣΙΟΠΛΑΣΙΑΚΙΣ (1) -----\n",
      "   αὶ τρίτην αὔξην δῆλον δὴ ἀπόστασιν ὅσην ἀφεστηκὼς γίγνεται δῆλον ἔφη τῷ γε λογιστικῷ οὐκοῦν ἐάν τις μεταστρέψας ἀληθείᾳ ἡδονῆς τὸν βασιλέα\n",
      "\n",
      " + AlexanderOfAphrodisias, InAristotelisMetaphysicaCommentaria (tlg0732.tlg004.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΟΥΝΙΚΑΝΩΣΠΕΡΙΑΥΤΩΝΗΜΙΝΕΝΤΟΙΣΠΕΡΙ (1) -----\n",
      "   ο τ ιστεύσομεν ρ Φ τεθεώρηται μὲν οὐνὶκανῶςπερὶαὐτῶνἡμῖνἐντοῖςπερὶ φύσεως ἰκαὶἱκανῶς φησί περὶτῶ ν ἀρχῶν τῶν φυσικῶν ἐν τοῖς περὶ φύσεως διήλθομεν λόγοις ἐν τῇ ἐπιγραφ\n",
      "\n",
      " + AlexanderOfAphrodisias, InAristotelisTopicorumLibrosOctoCommentaria (tlg0732.tlg006.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΟΤΙΤΟΥΜΗΔΙΑΠΡΟΤΕΡΩΝΟΡΙΖΕΣΘΑΙΤΡΕΙΣ (1) -----\n",
      "   Τοῦ δὲ μὴ ἐκπροτέρων τρεῖς εἰσι τρόποι Ὅτιτοῦμὴδιὰπροτέρωνὁρίζεσθαιτρεῖς εἰσι τρόποι πρῶτοςμὲν εἰ διὰ τοῦ ἀντικειμένου τὸ ἀντικείμενον ὥρισται ἅμ γὰρ τῇ φύσει τὰ ἀντικείμενα τὰ δὲ καθ αὑτὰ πρός τι οὐκ\n",
      "\n",
      " + ApolloniusDyscolus, DeAdverbiis (tlg0082.tlg002.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΠΑΡΕΓΕΝΟΜΕΝΟΜΕΝΟΣΗΝΚΑΙΕΤΙΕΚΤΗΣΛΕΣΒΟΥΟΥΦΑΜΕΝ (1) -----\n",
      "   Λεϲβο παρεγενόμην καὶ ἔτι οῦ φαμεν παρεγενομενομενοϲηνκαιετιεκτηϲλεϲβουουφαμεν Α εκ τηϲ Λεϲβου ἔτι οὐ οἴκῳ μένει με ἐν δὲ τῷ οἴκῳ μ\n",
      "\n",
      " + ApolloniusDyscolus, DeConstructione (tlg0082.tlg004.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΑΡΣΕΝΙΚΩΝΟΝΟΜΑΤΩΝΣΤΟΙΧΕΙΑΕΣΤΙΠΕΝΤΕ (1) -----\n",
      "   τ τὸ ᾶ τελικόν ἐϲτιν κτλ Τελικὰ ἀρϲενικῶνὸνομάτωνϲτοιχεῖάἐϲτιπέντε θηλυκῶνδὲ ὸκτώ ᾶη ωνξΒ ψ οὐδετέ ρων δὲ ἐ ῦ εραίαν Β Β ἀΔιὰ τ\n",
      "\n",
      "   ----- ΚΑΙΤΟΝΑΡΙΣΤΑΡΧΟΝΑΣΜΕΝΩΣΤΗΝΓΡΑΦΗΝΤΟΥ (1) -----\n",
      "   φαϲ δὲ καίτὸνἈρίϲταρχονἀϲμένωϲτὴνγραφὴντοῦ Δικαιάρχουπαραδέξαϲθαι ἐνγὰρἁπάϲαιϲ ν τὸ εὲῇ ἐν πατρίδι γαί ὑπολαβόντα τὸ ἑαυτῆϲ νοεὶϲθαι ἐκ τοῦ ἑῇ δέον πόλιν ψ λῶϲ μετολαμβ\n",
      "\n",
      "   ----- ΑΡΙΣΤΑΡΧΟΣΚΑΙΟΙΑΠΟΤΗΣΣΧΟΛΗΣΦΑΣΙΝ (1) -----\n",
      "   θϲτή εϲι Β καθότ Ϲ καθ ϲ ὁ Ἀρίϲταρχοϲκαὶοίἀπὸτῆϲϲχολῆϲφαϲιν οὶϲ οὐ ϲυγκαταθετέον ε φαϲίν οὐκ ὀρθῶϲ ο\n",
      "\n",
      " + Artemidorus, Onirocriticon (tlg0553.tlg001.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΑΥΤΟΜΑΤΟΙΔΕΟΙΘΕΟΙΑΠΑΛΛΑΣΣΟΜΕΝΟΙ (1) -----\n",
      "   ετεύειν τὸν θεόν συντρίβειν δέ θεῶν ἀγάλματα καὶ ἐκβάλλειν τῆς οἰκίας ἱδρυμένα ἔνδον ἢ καταρρίπτειν ναὸν τι τῶν οὐχ ὁσίων ἐν ναῷ ποιεῖν πονηρὸν ἂν εἴη πάσι καὶ περιστάσεις μεγάλας σημαίνει οἱ γὰρ ἐν μεγάλαις συμφοραῖ\n",
      "\n",
      " + JoannesPhiloponus, InAristotelisMeteorologicorumLibrumPrimumCommentarium (tlg4015.tlg005.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΔΥΝΑΤΟΝΔΕΤΟΑΙΤΙΑΙΗΣΓΕΝΕΣΕΩΣΚΑΙΤΗΣΦΘΟΡΑΣ (1) -----\n",
      "   λὴ ἀνάλογόν ἐστι γενέσει ἡ δὲ τοὔμπαλιν τῶν κουφοτέρων εἰς τὰ βαρότεραφθορᾷ δυνατὸνδὲτὸαἰτίαιῆςγενέσεωςκαὶτῆςφθορᾶς διὰ τὸ ἄρθρον μὴ καθολικῶς ἀκούειν πάσης γενέσεως καὶ φθορᾶς ἀλλὰ ὑετοῦ χιόν\n",
      "\n",
      "   ----- ΛΛΗΣΤΗΣΑΝΩΘΕΝΘΕΡΜΟΤΗΤΟΣΑΤΜΙΔΟΥΜΕΝΟΝΦΕΡΕΤΑΙ (1) -----\n",
      "   νῦν μενούσης ἀμεταβλήτου τὸ οὖν περὶ τὴν γῆν ὑγρόν φησίν ὑπὸ τῶν ἀκτίνων καὶ ὑπὸ τῆς ὰ λληςτῆςἄνωθενθερμότητοςἀτμιδούμενονφέρεται ἄνω πῶς μὲν ἡ ἐκ τῶν ἀκτίνων γίνεται θερμότης ἐδίδαξεν ὅτι ὁ ε ναπο λαμβαν\n",
      "\n",
      " + JoannesPhiloponus, InAristotetelisMeteorologicorumLibrumPrimumCommentarium (tlg4015.tlg005.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΔΥΝΑΤΟΝΔΕΤΟΑΙΤΙΑΙΗΣΓΕΝΕΣΕΩΣΚΑΙΤΗΣΦΘΟΡΑΣ (1) -----\n",
      "   μπαλιν τῶν κουφοτέρων εἰς τὰ βαρότεραφθορᾷ δυνατὸνδὲτὸαἰτίαιῆςγενέσεωςκαὶτῆςφθορᾶς διὰ τὸ ἄρθρον μὴ καθολικῶς ἀκούειν πάσης γενέσεως καὶ φθορᾶς ἀλλὰ ὑετοῦ χιόνος καὶ τῶν λοιπῶν τοῦτο δ\n",
      "\n",
      "   ----- ΛΛΗΣΤΗΣΑΝΩΘΕΝΘΕΡΜΟΤΗΤΟΣΑΤΜΙΔΟΥΜΕΝΟΝΦΕΡΕΤΑΙ (1) -----\n",
      "   περὶ τὴν γῆν ὑγρόν φησίν ὑπὸ τῶν ἀκτίνων καὶ ὑπὸ τῆς ὰ λληςτῆςἄνωθενθερμότητοςἀτμιδούμενονφέρεται ἄνω πῶς μὲν ἡ ἐκ τῶν ἀκτίνων γίνεται θερμότης ἐδίδαξεν ὅτι ὁ ε ναπο λαμβανόμενος μεταξὺ τῆς ἀνακλάσεως αὐτῶν\n",
      "\n",
      " + Libanius, Epistulae1-839 (tlg2200.tlg001.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΕΜΟΥΟΙΑΠΕΦΕΥΓΑΧΕΙΡΑΣΛΥΠΗΣΑΣΜΕΝΟΥΔΕΝΑΟΥΔΕΝ (1) -----\n",
      "   ὶ δάκρυα πρὸ τῶν λόγωνἀφεὶς ἐγὼ μόλις τὰς τῶν παθόντων ἐμοῦόιαπέφευγαχεῖραςλυπήσαςμὲνοὐδέναοὐδέν ἡνίκα ἐξῆν μικρο δὲ διασπασθείς καὶ προσετίθει φυγὴν ἀδελφοῦ καὶ γένους ὅλου πλάνην καὶ γῆν ἄσπορον σκευῶν ἁρπαγήν\n",
      "\n",
      "   ----- ΚΑΙΙΚΕΛΗΧΡΥΣΗΑΦΡΟΔΙΤΗΚΑΙΟΙΣΕΚΟΣΜΗΣΕ (1) -----\n",
      "   σθην τοῖς μέν οἕαν ἔφυσαν σοὶ δέ οἴαν ἔχεις Δήλῳ δή ποτε τοῖον καὶἰκέληχρυσῇἈφροδίτῃκαὶοἷςἐκόσμησε γυναῖκας Ὅμηρος πάντα ἂν δέξαιτο ἀναμιμνησκόμενος οὖν ὅπως\n",
      "\n",
      " + Libanius, OratioI (tlg2200.tlg00401.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΗΔΙΚΗΜΕΝΟΝΔΕΑΠΕΡΡΙΜΜΕΝΟΝΠΕΡΙΟΡΑΣ (1) -----\n",
      "   ως ἀλλ ὁ μὲν ἠπατηκὼς τρυφᾷ τὸν ἠδικημένονδὲἀπερριμμένονπεριορᾷς τοι αυ τα με ν προ ς το ε δος πο ρρωθεν α πο τη ς στοα ς διε μὲν\n",
      "\n",
      " + Suda, SuidaeLexicon (tlg9010.tlg001.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΣΠΕΡΜΑΓΟΡΑΙΟΛΕΚΙΘΟΛΑΧΑΝΟΠΩΛΙΔΕΣ (1) -----\n",
      "   Ὦ ϲπερμαγοραιολεκιθολαχανοπώλιδεϲ ὦ ϲκοροδοπανδοκευτριαρτοπώλιδεϲ οὐκ ἐξέλκετ οὐ παιήϲετ οὐκ\n",
      "\n",
      "   ----- ΟΡΘΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ (2) -----\n",
      "   Ὀρθοφοιτοϲυκοφαντοδικοταλαιπώρων Ἀριϲτοφάνηϲ ὁτιὴ βούλομαι τούτων ἀπαλλαχθέντα τῶν ὀρθοφοιτοϲυκοφαντοδικοταλαιπώρω\n",
      "   Ἀριϲτοφάνηϲ ὁτιὴ βούλομαι τούτων ἀπαλλαχθέντα τῶν ὀρθοφοιτοϲυκοφαντοδικοταλαιπώρων τρόπων ζῆν βίον γενναῖον ὥϲπερ Μόρυχοϲ αἰτίαν ἔχων ταῦτα δρᾶν\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from functions import search_words_from_corpora\n",
    "# using already instantiated l variable\n",
    "words = list(y[0] for x, y in l.T.items())\n",
    "search_words_from_corpora(words, [perseus_dir, first1k_dir], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a small explanation: [Aristophanes](https://en.wikipedia.org/wiki/Aristophanes) was a Greek comic playwright and a word expert of a kind. Mathematical texts are also filled with long compoud words for fractions for example.\n",
    "\n",
    "So thats all for the Greek corpora processing and basic statistics. One could further investigate the basic stats, categorize and compare individual texts as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " + Appian, TheCivilWars (tlg0551.tlg017.perseus-grc2.xml) =>    \n",
      "\n",
      "   ----- ΣΥΝΥΠΟΧΩΡΟΥΝΤΩΝ (1) -----\n",
      "   καὶ ἡ σύνταξις ἤδη παρελέλυτο ὀξύτερον ὑπεχώρουν καί τῶν ἐπιτεταγμένων σφίσι δευτέρων καὶ τρίτων συνυποχωρούντων μισγόμενοι πάντες ἀλλήλοις ἀκόσμως ἐθλίβοντο ὑπὸ σφῶν καὶ τῶν πολεμίων ἀπαύστως αὐτοῖς ἐπικειμένων\n",
      "\n",
      " + Aristophanes, Ecclesiazusae (tlg0019.tlg010.perseus-grc1.xml) =>    \n",
      "\n",
      "   ----- ΚΙΧΛΕΠΙΚΟΣΣΥΦΟΦΑΤΤΟΠΕΡΙΣΤΕΡΑ (1) -----\n",
      "   αχογαλεο κρανιολειψανοδριμυποτριμματο σιλφιοτυρομελιτοκατακεχυμενο κιχλεπικοσσυφοφαττοπεριστερα λεκτρυονοπτεκεφαλλιοκιγκλοπε λειολαγῳοσιραιοβαφητραγα νοπτερυγών\n",
      "\n",
      " + Aristophanes, Wasps (tlg0019.tlg004.perseus-grc1.xml) =>    \n",
      "\n",
      "   ----- ΟΡΘΡΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ (1) -----\n",
      "   ς ἀκούειν ἡδἔ εἰ καὶ νῦν ἐγὼ τὸν πατέρ ὅτι βούλομαι τούτων ἀπαλλαχθέντα τῶν ὀρθροφοιτοσυκοφαντοδικοταλαιπώρων τρόπων ζῆν βίον γενναῖον ὥσπερ Μόρυχος αἰτίαν ἔχω ταῦτα δρᾶν ξυνωμότης ὢν καὶ φρονῶν\n",
      "\n",
      " + Athenaeus, Deipnosophistae (tlg0008.tlg001.perseus-grc3.xml) =>    \n",
      "\n",
      "   ----- ΒΡΥΣΩΝΟΘΡΑΣΥΜΑΧΕΙΟΛΗΨΙΚΕΡΜΑΤΩΝ (1) -----\n",
      "   ας καὶ τὰ καρχήσια καὶ τὰ τούτοις ὅμοια ἕνα μὲν γὰρ εἶναι πυθμένα τὸν κατὰ τὸ κύτος συγκεχαλκευμένον ὅλῳ τῷ ἀγγείῳ ἕτερον δὲ τὸν πρόσθετον ἀπὸ ὀξέος ἀρχόμενον κατα\n",
      "\n",
      " + Athenaeus, TheDeipnosophists (tlg0008.tlg001.perseus-grc4.xml) =>    \n",
      "\n",
      "   ----- ΒΡΥΣΩΝΟΘΡΑΣΥΜΑΧΕΙΟΛΗΨΙΚΕΡΜΑΤΩΝ (1) -----\n",
      "   τις ὑποπλατωνικὸς Βρυσωνοθρασυμαχειοληψικερμάτων πληγεὶς ἀνάγκῃ ληψιλογομίσθῳ τέχνῃ\n",
      "\n",
      " + AlexanderOfAphrodisias, InAristotelisMetaphysicaCommentaria (tlg0732.tlg004.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΟΥΝΙΚΑΝΩΣΠΕΡΙΑΥΤΩΝΗΜΙΝΕΝΤΟΙΣΠΕΡΙ (1) -----\n",
      "   ο τ ιστεύσομεν ρ Φ τεθεώρηται μὲν οὐνὶκανῶςπερὶαὐτῶνἡμῖνἐντοῖςπερὶ φύσεως ἰκαὶἱκανῶς φησί περὶτῶ ν ἀρχῶν τῶν φυσικῶν ἐν τοῖς περὶ φύσεως διήλθομεν λόγοις ἐν τῇ ἐπιγραφ\n",
      "\n",
      " + ApolloniusDyscolus, DeConstructione (tlg0082.tlg004.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΑΡΣΕΝΙΚΩΝΟΝΟΜΑΤΩΝΣΤΟΙΧΕΙΑΕΣΤΙΠΕΝΤΕ (1) -----\n",
      "   τ τὸ ᾶ τελικόν ἐϲτιν κτλ Τελικὰ ἀρϲενικῶνὸνομάτωνϲτοιχεῖάἐϲτιπέντε θηλυκῶνδὲ ὸκτώ ᾶη ωνξΒ ψ οὐδετέ ρων δὲ ἐ ῦ εραίαν Β Β ἀΔιὰ τ\n",
      "\n",
      "   ----- ΚΑΙΤΟΝΑΡΙΣΤΑΡΧΟΝΑΣΜΕΝΩΣΤΗΝΓΡΑΦΗΝΤΟΥ (1) -----\n",
      "   φαϲ δὲ καίτὸνἈρίϲταρχονἀϲμένωϲτὴνγραφὴντοῦ Δικαιάρχουπαραδέξαϲθαι ἐνγὰρἁπάϲαιϲ ν τὸ εὲῇ ἐν πατρίδι γαί ὑπολαβόντα τὸ ἑαυτῆϲ νοεὶϲθαι ἐκ τοῦ ἑῇ δέον πόλιν ψ λῶϲ μετολαμβ\n",
      "\n",
      "   ----- ΑΡΙΣΤΑΡΧΟΣΚΑΙΟΙΑΠΟΤΗΣΣΧΟΛΗΣΦΑΣΙΝ (1) -----\n",
      "   θϲτή εϲι Β καθότ Ϲ καθ ϲ ὁ Ἀρίϲταρχοϲκαὶοίἀπὸτῆϲϲχολῆϲφαϲιν οὶϲ οὐ ϲυγκαταθετέον ε φαϲίν οὐκ ὀρθῶϲ ο\n",
      "\n",
      " + ApolloniusDyscolus, DePronominibus (tlg0082.tlg001.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΩΡΙΣΜΕΝΩΝΠΡΟΣΩΠΩΝ (1) -----\n",
      "   παγγέλλεται προῦφεϲτῶϲαν ὅ ἐϲτι πάλιν πρόϲωπον ὡριϲμένον ὀρθῶϲ ἄρα ὡριϲμένωνπροϲώπων παραϲτατικὴ ἡ ἀντωνυμία Φήϲει τιϲ τ\n",
      "\n",
      " + Aristotle, MagnaMoralia (tlg0086.tlg022.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΤΩΟΡΘΩΕΚΑΣΤΑΘΕΩΡΩΝ (1) -----\n",
      "   καὶ μὴ διεψεῦσθαι τῷ λόγῳ ἔστιν δὲ καὶ ὁ φρόνιμός τοιοῦτος ὁτῷ λόγῳ τῷὀρθῷἕκασταθεωρῶν πότερον δ ἐνδέχεταιτὸν φρόνιμον ἀκρατῆ εἶναι ἢ οὔ ἀπορήσειε γὰρ ἄν τις τὰ εἰρημένα ἐὰν δὲ πα ρακολουθήσωμεν τοῖς ἔμπροσθενεἰρ\n",
      "\n",
      " + ChroniconPaschale, ChroniconPaschale (tlg2371.tlg001.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΟΠΡΩΤΟΣΑΝΘΡΩΠΩΝΥΠΟΔΕΙΞΑΣ (1) -----\n",
      "   δείξας οὐρανοδρομεῖν όπρῶτοςἀνθρώπωνὑποδείξας ἀγγέλων καὶ ἀνθρώπων μίαν ὁδόν ὁ τὴν γῆν λαχὼν οἰκητηιριον καὶ τὸν οὐρανὸν νòν ἀθρόον διατρέχω\n",
      "\n",
      " + EvagriusScholasticus, HistoriaEcclesiastica (tlg2733.tlg001.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΓΛΩΣΣΟΤΟΜΗΘΕΝΤΩΝΧΡΙΣΤΙΑΝΩΝ (1) -----\n",
      "   Περὶ Ὀνωρίχου τοῦ Βανδίλων ἄρχοντος καὶ τῶν γλωσσοτομηθέντωνΧριστιανῶν παῤ αὐτοῦ ιε Περὶ Καβαώνου τοῦ Μαυρουσίου\n",
      "\n",
      "   ----- ΕΠΙΣΚΟΠΩΚΩΝΣΤΑΝΤΙΝΟΥΠΟΛΕΩΣ (1) -----\n",
      "   Ἐπιστολὴ ἤτοι δέησις ἀποσταλεῖσα Ἀκακίῳ ἐπισκόπῳΚωνσταντινουπόλεως παρὰ τῶν τῆς Ἀσίας ἐπισκόπων Ἀκακίῳ τῷ ἁγιωτάτῳ καὶ ὁσιωτάτῳ πατριάρχῃ τῆς κατὰ τὴν βασιλεύουσαν\n",
      "\n",
      " + JoannesPhiloponus, InAristotelisMeteorologicorumLibrumPrimumCommentarium (tlg4015.tlg005.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΔΥΝΑΤΟΝΔΕΤΟΑΙΤΙΑΙΗΣΓΕΝΕΣΕΩΣΚΑΙΤΗΣΦΘΟΡΑΣ (1) -----\n",
      "   λὴ ἀνάλογόν ἐστι γενέσει ἡ δὲ τοὔμπαλιν τῶν κουφοτέρων εἰς τὰ βαρότεραφθορᾷ δυνατὸνδὲτὸαἰτίαιῆςγενέσεωςκαὶτῆςφθορᾶς διὰ τὸ ἄρθρον μὴ καθολικῶς ἀκούειν πάσης γενέσεως καὶ φθορᾶς ἀλλὰ ὑετοῦ χιόν\n",
      "\n",
      "   ----- ΛΛΗΣΤΗΣΑΝΩΘΕΝΘΕΡΜΟΤΗΤΟΣΑΤΜΙΔΟΥΜΕΝΟΝΦΕΡΕΤΑΙ (1) -----\n",
      "   νῦν μενούσης ἀμεταβλήτου τὸ οὖν περὶ τὴν γῆν ὑγρόν φησίν ὑπὸ τῶν ἀκτίνων καὶ ὑπὸ τῆς ὰ λληςτῆςἄνωθενθερμότητοςἀτμιδούμενονφέρεται ἄνω πῶς μὲν ἡ ἐκ τῶν ἀκτίνων γίνεται θερμότης ἐδίδαξεν ὅτι ὁ ε ναπο λαμβαν\n",
      "\n",
      " + JoannesPhiloponus, InAristotetelisMeteorologicorumLibrumPrimumCommentarium (tlg4015.tlg005.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΔΥΝΑΤΟΝΔΕΤΟΑΙΤΙΑΙΗΣΓΕΝΕΣΕΩΣΚΑΙΤΗΣΦΘΟΡΑΣ (1) -----\n",
      "   μπαλιν τῶν κουφοτέρων εἰς τὰ βαρότεραφθορᾷ δυνατὸνδὲτὸαἰτίαιῆςγενέσεωςκαὶτῆςφθορᾶς διὰ τὸ ἄρθρον μὴ καθολικῶς ἀκούειν πάσης γενέσεως καὶ φθορᾶς ἀλλὰ ὑετοῦ χιόνος καὶ τῶν λοιπῶν τοῦτο δ\n",
      "\n",
      "   ----- ΛΛΗΣΤΗΣΑΝΩΘΕΝΘΕΡΜΟΤΗΤΟΣΑΤΜΙΔΟΥΜΕΝΟΝΦΕΡΕΤΑΙ (1) -----\n",
      "   περὶ τὴν γῆν ὑγρόν φησίν ὑπὸ τῶν ἀκτίνων καὶ ὑπὸ τῆς ὰ λληςτῆςἄνωθενθερμότητοςἀτμιδούμενονφέρεται ἄνω πῶς μὲν ἡ ἐκ τῶν ἀκτίνων γίνεται θερμότης ἐδίδαξεν ὅτι ὁ ε ναπο λαμβανόμενος μεταξὺ τῆς ἀνακλάσεως αὐτῶν\n",
      "\n",
      " + Libanius, Epistulae1-839 (tlg2200.tlg001.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΕΜΟΥΟΙΑΠΕΦΕΥΓΑΧΕΙΡΑΣΛΥΠΗΣΑΣΜΕΝΟΥΔΕΝΑΟΥΔΕΝ (1) -----\n",
      "   ὶ δάκρυα πρὸ τῶν λόγωνἀφεὶς ἐγὼ μόλις τὰς τῶν παθόντων ἐμοῦόιαπέφευγαχεῖραςλυπήσαςμὲνοὐδέναοὐδέν ἡνίκα ἐξῆν μικρο δὲ διασπασθείς καὶ προσετίθει φυγὴν ἀδελφοῦ καὶ γένους ὅλου πλάνην καὶ γῆν ἄσπορον σκευῶν ἁρπαγήν\n",
      "\n",
      " + PhiloJudaeus, DeVitaMosisLibI‑Ii (tlg0018.tlg022.opp-grc1.xml) =>    \n",
      "\n",
      "   ----- ΨΥΧΟΓΟΝΙΜΩΤΑΤΩΝ (1) -----\n",
      "   ς ἐφεὶς τῷ Μωυσέως ἀδελφῷ τὰς δ ἴσας ἐξ ἀέρος καὶ πυρὸς τῶν ψυχογονιμωτάτων μόνῳ Μωυσεῖ μίαν δὲ κοινὴν ἀμφοτέροις ἑβδόμην ἐπιτρέπει τρεῖς δὲ τὰς ἄλλας εἰς συμπλήρωσιν δεκάδος ἀνατίθησιν αὑτῷ\n",
      "\n",
      " + Porphyrius, VitaPythagorae (tlg2034.tlg002.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΤΟΥΤΟΥΣΛΕΓΟΝΤΕΣΩΣΠΡΟΣΤΗΝ (1) -----\n",
      "   καὶ αὐτὰ ταῦτα βουλόμενοι παραδοῦναι παρεγένοντο ἐπὶ τοὺςχαρακτῆρας τούτουςλέγοντεςὡςπρὸςτὴν πρώτην διδασκαλίαν στοιχεῖα εἶναι ὕστερον μέντοι διδάσκου σιν ὅτι οὐχ οὗτοι στοιχεῖά εἰσιν οἱ χαρακτῆρες ἀλλὰ διὰ\n",
      "\n",
      " + Suda, SuidaeLexicon (tlg9010.tlg001.1st1K-grc1.xml) =>    \n",
      "\n",
      "   ----- ΚΩΔΩΝΟΦΑΛΑΡΑΧΡΩΜΕΝΟΥΣ (1) -----\n",
      "   οὕτω ψοφοῦνταϲ ψοφοῦντεϲ Κωδωνοφαλαραχρωμένουϲ αὐτὰϲ Κώδων Σοφοκλῆϲ Τυρρηνικῆϲ Σοφοκλῆϲ\n",
      "\n",
      "   ----- ΟΡΘΟΦΟΙΤΟΣΥΚΟΦΑΝΤΟΔΙΚΟΤΑΛΑΙΠΩΡΩΝ (2) -----\n",
      "   Ὀρθοφοιτοϲυκοφαντοδικοταλαιπώρων Ἀριϲτοφάνηϲ ὁτιὴ βούλομαι τούτων ἀπαλλαχθέντα τῶν ὀρθοφοιτοϲυκοφαντοδικοταλαιπώρω\n",
      "   Ἀριϲτοφάνηϲ ὁτιὴ βούλομαι τούτων ἀπαλλαχθέντα τῶν ὀρθοφοιτοϲυκοφαντοδικοταλαιπώρων τρόπων ζῆν βίον γενναῖον ὥϲπερ Μόρυχοϲ αἰτίαν ἔχων ταῦτα δρᾶν\n",
      "\n"
     ]
    }
   ],
   "source": [
    "words = list(y[0] for x, y in m.T.items())\n",
    "search_words_from_corpora(words, [perseus_dir, first1k_dir], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The [MIT](http://choosealicense.com/licenses/mit/) License\n",
    "\n",
    "Copyright &copy; 2018 Marko Manninen"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}