docs/fecon235-00-README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## fecon235 docs :: Intro to notebooks\n",
    "\n",
    "Here we discuss the usage of the **fecon235** repository \n",
    "for the casual user interested in Jupyter notebooks\n",
    "for **financial economics**. As of 2018, the source code,\n",
    "*but not the notebook archive*,\n",
    "was spun-off to **fecon236** so we will discuss that\n",
    "Python package as well. See https://git.io/econ for FAQ\n",
    "and installation details.\n",
    "\n",
    "**Everything works cross-platform (Linux, Mac, Windows)\n",
    "with at least Python 2.7 or 3.4 -- preferably with Anaconda.** \n",
    "To see examples of code at work, please pick out a subject of interest\n",
    "from the `nb` [directory](https://git.io/235nb), and examine that notebook.\n",
    "\n",
    "\n",
    "### Curated wrapper over the Python and Jupyter ecosystems\n",
    "\n",
    "After the external dependencies are satisfied,\n",
    "import consists of an one-liner.\n",
    "The complexity of vast ecosystems working in the background\n",
    "is intentionally tucked away, but easy to dissect.\n",
    "For example, the financial economist is *not* exposed to\n",
    "the annoying differences between pandas Series\n",
    "and DataFrame, between numpy arrary and matrix types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fecon236 as fe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![fecon236 logo](https://git.io/fecon236-px200.png)](https://github.com/MathSci/fecon236)\n",
    "\n",
    "\n",
    "### Every notebook should state its dependencies exactly\n",
    "\n",
    "One of our goals is reproducibility of research and\n",
    "the replication of computational results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Dependencies:*\n",
    "\n",
    "- fecon236 repository, https://github.com/MathSci/fecon236\n",
    "     \n",
    "*CHANGE LOG*\n",
    "\n",
    "    2018-06-23  Major revision with changes from fecon236.\n",
    "    2015-12-30  Add inventory of lib procedures as Appendix 1.\n",
    "    2015-12-28  First version of README notebook in docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " !:  Code for this project straddles python27 and python3.\n",
      " ::  Python 2.7.14\n",
      " ::  IPython 5.7.0\n",
      " ::  jupyter_core 4.4.0\n",
      " ::  notebook 5.4.0\n",
      " ::  matplotlib 1.5.1\n",
      " ::  numpy 1.11.3\n",
      " ::  scipy 1.0.0\n",
      " ::  statsmodels 0.8.0\n",
      " ::  sympy 1.1.1\n",
      " ::  pandas 0.22.0\n",
      " ::  pandas_datareader 0.6.0\n",
      " ::  fecon236 10.6.7b70\n",
      " ::  Repository: fecon235 v5.18.0312 develop\n",
      " ::  Timestamp: 2018-06-23T22:36:45Z\n"
     ]
    }
   ],
   "source": [
    "#  PREAMBLE-p10.18.0623 :: Settings, https://git.io/236pa\n",
    "from __future__ import absolute_import, print_function, division\n",
    "fe.system.specs()\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "#       Use 0 to disable autoreload when a module is modified.\n",
    "#  NOTEBOOK DISPLAY OPTIONS...\n",
    "import pandas as pd\n",
    "pd.set_option('display.notebook_repr_html', False)\n",
    "#       Represent pandas DataFrames as text; not HTML representation.\n",
    "from IPython.display import HTML  # Useful for snippets from web.\n",
    "#  e.g. HTML('<iframe src=http://en.mobile.wikipedia.org/?useformat=mobile \\\n",
    "#            width=700 height=350></iframe>')\n",
    "from IPython.display import Image\n",
    "#  e.g. Image(filename='holt-winters-equations.png', embed=True)\n",
    "#                  url= Also works instead of filename.\n",
    "from IPython.display import YouTubeVideo\n",
    "#  e.g. YouTubeVideo('1j_HxD4iLn8', start='43', width=600, height=400)\n",
    "from IPython.core import page\n",
    "get_ipython().set_hook('show_in_pager', page.as_hook(page.display_page), 0)\n",
    "#  Or equivalently in config file: \"InteractiveShell.display_page = True\",\n",
    "#  which will display results in secondary notebook pager frame in a cell.\n",
    "%matplotlib inline\n",
    "#  Generate PLOTS inside notebook, \"inline\" generates static png,\n",
    "#  whereas \"notebook\" argument allows interactive zoom and resize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Preamble for settings\n",
    "\n",
    "The preamble contains the latest shortcuts for notebook commands, \n",
    "but more importantly, it lists the specific dependencies \n",
    "which makes research **reproducible**. The \"Repository:\" line \n",
    "should indicate the annotated tag associated with the last good state \n",
    "of repository at the time of execution.\n",
    "\n",
    "The \"Timestamp:\" will indicate the staleness of the data.\n",
    "When notebooks are re-executed the most current data \n",
    "will generally be downloaded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Internal queries and documentation\n",
    "\n",
    "Notebooks have a wonderful feature: **?** and **??** \n",
    "which give further information on variables, functions, \n",
    "classes, etc. And where to exactly look for the source.\n",
    "\n",
    "The second question mark gives more verbose answers. \n",
    "All our codes have detailed docstrings and comments. \n",
    "We strive to be self-documenting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mType:\u001b[0m        module\n",
       "\u001b[0;31mString form:\u001b[0m <module 'fecon236.util.system' from '/media/yaya/virt18g/virt/dbx/Dropbox/ipy/fecon236/fecon236/util/system.pyc'>\n",
       "\u001b[0;31mFile:\u001b[0m        /media/yaya/virt18g/virt/dbx/Dropbox/ipy/fecon236/fecon236/util/system.py\n",
       "\u001b[0;31mDocstring:\u001b[0m  \n",
       "_______________|  system.py :: system and date functions including specs.\n",
       "\n",
       "Code in this module should be compatible with both Python 2 and 3\n",
       "until 2019-01-01, thereafter only python3.\n",
       "\n",
       "For example, it is used in the preamble of fecon235 Jupyter notebooks.\n",
       "\n",
       "\n",
       "REFERENCES:\n",
       "- Compatible IDIOMS: http://python-future.org/compatible_idioms.html\n",
       "                     Nice presentation.\n",
       "\n",
       "- SIX module is exhaustive: https://pythonhosted.org/six/\n",
       "        Single file source: https://bitbucket.org/gutworth/six\n",
       "\n",
       "\n",
       "CHANGE LOG  For latest version, see https://git.io/fecon236\n",
       "2018-06-20  Update specs(), include version for statsmodels.\n",
       "2018-05-15  Include version(\"fecon236\") to specs.\n",
       "2018-04-25  Ignore \"raw_input\" < python3 flake8.\n",
       "2018-04-21  yi_0sys module from fecon235 renamed to system.\n",
       "                Major flake8 fixes. Move notebook preamble to docs.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#  What the heck is \"system\" mentioned in the preamble?\n",
    "fe.system?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting data\n",
    "\n",
    "Our project currently has free access to data on equities, \n",
    "government bonds, commodities, and futures -- as well as, \n",
    "a full range of economic statistics. The key is finding \n",
    "the string which will retrieve the desired time-series.\n",
    "\n",
    "#### Sample: Unemployment rate\n",
    "\n",
    "Let's go through an example. The function `get()` is \n",
    "designed as a hostess of specialized get-type functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Assign a name to a pandas DataFrame\n",
    "#  which will contain monthly unemployment rates:\n",
    "\n",
    "unem = fe.get(fe.m4unemp)\n",
    "#                m4 implies monthly frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mType:\u001b[0m        str\n",
       "\u001b[0;31mString form:\u001b[0m UNRATE\n",
       "\u001b[0;31mLength:\u001b[0m      6\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#  But does fe.m4unemp really represent?\n",
    "fe.m4unemp??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables for data\n",
    "\n",
    "So we see that fe.m4unemp is a variable holding a string \"UNRATE\". \n",
    "That string is the internal code used by [FRED](https://fred.stlouisfed.org),\n",
    "the database at the Federal Reserve Bank in St. Louis.\n",
    "Our variables are generally easier to remember, and mentions the frequency. \n",
    "\n",
    "If there is no special fecon variable, one can \n",
    "always `fe.get(\"symbol_string\")` to directly retrieve data\n",
    "from the primary vendors (see fecon236 `host` directory).\n",
    "\n",
    "Sometimes a variable for a data set may trigger a \n",
    "subroutine which post-processes the original data \n",
    "(e.g. see our [unified inflation](https://git.io/infl)), or brings offline \n",
    "data into memory (for example, our compressed CSV files may \n",
    "contain synthetic data, e.g. the euro exchange rate \n",
    "years prior to its official circulation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Illustrate slicing: 1997 <= unem <= 2007:\n",
    "unem07 = unem['1997':'2007']\n",
    "#  Verify below by Head and Tail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Y\n",
      "count  132.000000\n",
      "mean     4.908333\n",
      "std      0.646073\n",
      "min      3.800000\n",
      "25%      4.400000\n",
      "50%      4.700000\n",
      "75%      5.500000\n",
      "max      6.300000\n",
      "kurtosis  1.968487\n"
     ]
    }
   ],
   "source": [
    "#  Quick summary:\n",
    "fe.stat(unem07)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Y\n",
      "count  132.000000\n",
      "mean     4.908333\n",
      "std      0.646073\n",
      "min      3.800000\n",
      "25%      4.400000\n",
      "50%      4.700000\n",
      "75%      5.500000\n",
      "max      6.300000\n",
      "\n",
      " ::  Index on min:\n",
      "Y   2000-04-01\n",
      "dtype: datetime64[ns]\n",
      "\n",
      " ::  Index on max:\n",
      "Y   2003-06-01\n",
      "dtype: datetime64[ns]\n",
      "\n",
      " ::  Head:\n",
      "              Y\n",
      "T              \n",
      "1997-01-01  5.3\n",
      "1997-02-01  5.2\n",
      "1997-03-01  5.2\n",
      " ::  Tail:\n",
      "              Y\n",
      "T              \n",
      "2007-10-01  4.7\n",
      "2007-11-01  4.7\n",
      "2007-12-01  5.0\n",
      "\n",
      " ::  Correlation matrix:\n",
      "     Y\n",
      "Y  1.0\n"
     ]
    }
   ],
   "source": [
    "#  More verbose statistical summary:\n",
    "fe.stats(unem07)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The correlation matrix has only one entry above. \n",
    "This is because `fe.stats()` is designed to take \n",
    "a dataframe with multiple columns as argument.\n",
    "Let's see how the function is written \n",
    "and where we can find it in the filesystem.\n",
    "\n",
    "Indeed `fe.stats()` calls our `cormatrix()` to compute \n",
    "the correlation matrix. And one can go on \n",
    "further to query that function... eventually \n",
    "that query would reach a core numerical package \n",
    "such as numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mfe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstats\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mSource:\u001b[0m   \n",
       "\u001b[0;32mdef\u001b[0m \u001b[0mstats\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m'''VERBOSE statistics on dataframe; CORRELATIONS without regression.'''\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdescribe\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" ::  Index on min:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0midxmin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" ::  Index on max:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0midxmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" ::  Head:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" ::  Tail:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtail\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\" ::  Correlation matrix:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcormatrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mFile:\u001b[0m      /media/yaya/virt18g/virt/dbx/Dropbox/ipy/fecon236/fecon236/tool.py\n",
       "\u001b[0;31mType:\u001b[0m      function\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fe.stats??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the file location, we can see that `stats()`\n",
    "resides in the `tool` module in a Python package called `fecon236`\n",
    "which in turn lives in a project container also called `fecon236`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing from the data\n",
    "\n",
    "The analysis of data is at the heart of this project. \n",
    "To follow up on unemployment example, see https://git.io/fed \n",
    "which scores the Federal Reserve on their dual mandate. \n",
    "Visualization is provided by our plot tools, \n",
    "which as a by-product discredits the Phillips curve \n",
    "as adequate causal theory.\n",
    "\n",
    "Other computational tools will be covered in \n",
    "in the `docs` directory, for example, see how $\\LaTeX$\n",
    "is rendered in our notebooks while learning about how fecon236 uses\n",
    "[symbolic mathematics](https://github.com/MathSci/fecon236/blob/develop/docs/READ/fe-54_Symbolic_sympy.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions or bugs?\n",
    "\n",
    "- Chat with fellow users at Gitter: https://gitter.im/rsvp/fecon235\n",
    "\n",
    "- Chat with fellow developers at Gitter: https://gitter.im/MathSci/fecon236\n",
    "\n",
    "\n",
    "- Report issues regarding our Jupyter notebooks at https://github.com/rsvp/fecon235/issues\n",
    "\n",
    "- Report issues regarding our Python code at https://github.com/MathSci/fecon236/issues\n",
    "\n",
    "- Blame the lead developer: *Adriano* [rsvp.github.com](https://rsvp.github.com)\n",
    "\n",
    "\n",
    "- Please see https://git.io/econ for further orientation and installation details."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}