IntentClassification_RA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulating an IA Chatbot classifying intents of user utterances\n",
    "\n",
    "A chatbot is simulated. Recent deployments of chatbot are looking at “Chatbots helping Agents, Agents Helping Chatbots.”  This architecture requires a seamless hand-off between agents and humans - CoChat: Enabling bot and human collaboration for task completion. Proc. AAAI 18, 5301 5308.\n",
    "\n",
    "We use a widely used data set (ATIS) to simulate the scenario of identify the human-automation interaction - Characterit\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\n",
    "from sklearn.base import TransformerMixin\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ATIS dataset\n",
    "The ATIS data has the user entered text and the intent information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "# read file\n",
    "with open('resources/MSDialog-Intent.json', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "\n",
    "# parse file\n",
    "obj = json.loads(data)\n",
    "all_dialogs = obj['rasa_nlu_data']['common_examples']\n",
    "sentences=[]\n",
    "intents=[]\n",
    "for obj in all_dialogs:\n",
    "    sentences.append(obj['text'])\n",
    "    intents.append(obj['intent'])\n",
    "\n",
    "data_all = pd.DataFrame()\n",
    "data_all['Sentence']  = sentences\n",
    "data_all['Intent']  = intents\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "flight                        3666\n",
       "airfare                        423\n",
       "ground_service                 255\n",
       "airline                        157\n",
       "abbreviation                   147\n",
       "aircraft                        81\n",
       "flight_time                     54\n",
       "quantity                        51\n",
       "flight+airfare                  21\n",
       "distance                        20\n",
       "airport                         20\n",
       "city                            19\n",
       "ground_fare                     18\n",
       "capacity                        16\n",
       "flight_no                       12\n",
       "meal                             6\n",
       "restriction                      6\n",
       "airline+flight_no                2\n",
       "cheapest                         1\n",
       "ground_service+ground_fare       1\n",
       "airfare+flight_time              1\n",
       "aircraft+flight+flight_no        1\n",
       "Name: Intent, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_all.head(5)\n",
    "data_all.Intent.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use NLP using spacy package\n",
    "\n",
    "Create a tokenizer that takes and string and generates the tokens\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "import spacy\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "from spacy.lang.en import English\n",
    "\n",
    "# Create our list of stopwords\n",
    "nlp = spacy.load('C:/conda/anaconda3/lib/site-packages/en_core_web_sm/en_core_web_sm-2.2.5')\n",
    "stop_words = spacy.lang.en.stop_words.STOP_WORDS\n",
    "punctuations = string.punctuation\n",
    "\n",
    "#english parser\n",
    "parser = English()\n",
    "\n",
    "#tokenize the sentence\n",
    "def tokenize(sentence):\n",
    "   #get tokens\n",
    "    tokens = parser(sentence)\n",
    "\n",
    "    #convert to lower case and lemmatize the words\n",
    "    tokens = [ word.lemma_.lower().strip() for word in tokens ]\n",
    "\n",
    "    # remove stop words\n",
    "    tokens = [ word for word in tokens if word not in stop_words and word not in punctuations ]\n",
    "\n",
    "    # return list of tokens\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean the text \n",
    "Remove spaces, convert to lower case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transform the input text\n",
    "class TextXform(TransformerMixin):\n",
    "    def transform(self, X, **transform_params):\n",
    "        # Cleaning Text\n",
    "        return [clean_text(text) for text in X]\n",
    "\n",
    "    def fit(self, X, y=None, **fit_params):\n",
    "        return self\n",
    "\n",
    "    def get_params(self, deep=True):\n",
    "        return {}\n",
    "\n",
    "# Basic function to clean the text\n",
    "def clean_text(text):\n",
    "    # Removing spaces and converting text into lowercase\n",
    "    return text.strip().lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform the text into a vector\n",
    "1. Clean the text\n",
    "2. Tokenize the text\n",
    "3. Generate a vector space model - vector with word frequency\n",
    "\n",
    "Split the text into train and test data for training and cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_vector = CountVectorizer(tokenizer = tokenize, ngram_range=(1,1))\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = data_all[\"Sentence\"] # the features we want to analyze\n",
    "ylabels = data_all[\"Intent\"] # the labels, or answers, we want to test against\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, ylabels, test_size=0.3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the classifier\n",
    "Use logistic regression. Any classifier could be used. \n",
    "In this case, given that we are using bow, the performance of most classifiers will be similar\n",
    "Our objective is not to train the best classifier but to show case the identification of levels of automation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\conda\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.\n",
      "  \"this warning.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('cleaner', <__main__.TextXform object at 0x000001A1B37DBE88>),\n",
       "                ('vectorizer',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=<function tokenize at 0x000001A1AC66DAF8>,\n",
       "                                 vocabulary=None)),\n",
       "                ('classifier',\n",
       "                 LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                    fit_intercept=True, intercept_scaling=1,\n",
       "                                    l1_ratio=None, max_iter=100,\n",
       "                                    multi_class='warn', n_jobs=None,\n",
       "                                    penalty='l2', random_state=None,\n",
       "                                    solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                                    warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Logistic Regression Classifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "classifier = LogisticRegression(solver='lbfgs')\n",
    "\n",
    "# Create pipeline using Bag of Words\n",
    "pipe = Pipeline([(\"cleaner\", TextXform()),\n",
    "                 ('vectorizer', bow_vector),\n",
    "                 ('classifier', classifier)])\n",
    "\n",
    "# model generation\n",
    "pipe.fit(X_train,y_train)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the cross-validation test data\n",
    "Run it through the same pipeline - cleaning, tokenizing, generating vectors and predicting \n",
    "\n",
    "\n",
    "Get the confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from sklearn import metrics\n",
    "# Predicting with a test dataset\n",
    "predicted = pipe.predict(X_test)\n",
    "\n",
    "# Model Accuracy\n",
    "\n",
    "\n",
    "from sklearn.metrics import classification_report\n",
    "report = classification_report(y_test, predicted, output_dict=True)\n",
    "#print(report)\n",
    "f1_dict = dict()\n",
    "for key,val in report.items():\n",
    "    if isinstance(val,dict):\n",
    "        f1_dict[key] = val['f1-score']\n",
    "        \n",
    "\n",
    "        \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ATIS Test Data - Unseen data to evaluate LOA\n",
    "Get the test data\n",
    "\n",
    "Predict the intent\n",
    "\n",
    "Identify the performance of the intent classifier for each intent using the confusion matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the distribution of the test data and accuracy\n",
    "#df = pd.DataFrame('pred':predicted)\n",
    "\n",
    "with open('resources/MSDialog-Intent_test.json', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "\n",
    "# parse file\n",
    "obj = json.loads(data)\n",
    "all_dialogs = obj['rasa_nlu_data']['common_examples']\n",
    "sentences=[]\n",
    "intents=[]\n",
    "for obj in all_dialogs:\n",
    "    sentences.append(obj['text'])\n",
    "    intents.append(obj['intent'])\n",
    "\n",
    "data_test = pd.DataFrame()\n",
    "data_test['Sentence']  = sentences\n",
    "data_test['Intent']  = intents\n",
    "\n",
    "X_unseen = data_test[\"Sentence\"] # the features we want to analyze\n",
    "y_unseen = data_test[\"Intent\"] # the labels, or answers, we want to test against\n",
    "\n",
    "pred = pipe.predict(X_unseen)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print LOA Histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    predicted     score              LOA\n",
      "0      flight  0.956522  Full Automation\n",
      "1      flight  0.956522  Full Automation\n",
      "2      flight  0.956522  Full Automation\n",
      "3      flight  0.956522  Full Automation\n",
      "4      flight  0.956522  Full Automation\n",
      "..        ...       ...              ...\n",
      "888    flight  0.956522  Full Automation\n",
      "889    flight  0.956522  Full Automation\n",
      "890    flight  0.956522  Full Automation\n",
      "891    flight  0.956522  Full Automation\n",
      "892    flight  0.956522  Full Automation\n",
      "\n",
      "[893 rows x 3 columns]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEJCAYAAAB/pOvWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de1xUdeL/8dcEOJZlajFR5Je2stUkL9GWdoG05CJSRlZqSVlamVJZSw8Fw2wrXPURD7No1+1uFyMlQaLRipa10Cy2zcbFLoakZMOQoqIx3Ob3hz9npaNchMMgvZ+PR4+cz5wz8x44zPucM3POsXg8Hg8iIiKHOcHXAUREpPNROYiIiIHKQUREDFQOIiJioHIQEREDf18HaKuGhgb2799PQEAAFovF13FERI4LHo+H2tpaevTowQknGLcTjvty2L9/P99++62vY4iIHJcuuOACTjnlFMP4cV8OAQEBwMEX2K1bNx+nERE5PtTU1PDtt99630N/67gvh0O7krp164bVavVxGhGR48vRdsfrA2kRETFQOYiIiIHKQUREDFQOIiJioHIQEREDlYOIiBioHERExEDl8P/V1Nb5OoJ0Qlou5PfquD8Irr10C/Bn5H1pvo4hnUx+xmxfRxDxCW05iIiIgcpBREQMVA4iImKgchAREQOVg4iIGKgcRETEQOUgIiIGKgcRETEw7SC4d955h9dff917e8eOHVx//fVce+21pKWl4Xa7iYmJYebMmQAUFxeTkpLC/v37ueSSS5g3bx7+/jpGT0TEF0zbcrjpppvIzs4mOzubRYsWcdpppzF16lSSk5PJyMggLy8Ph8NBQUEBAElJSaSmprJmzRo8Hg+ZmZlmRRMRkWZ0yG6lxx57jJkzZ7J9+3ZCQkLo27cv/v7+xMXFYbfbKSsro7q6miFDhgAQHx+P3W7viGgiInIEpu+3KSwspLq6mpiYGHJzcwkMDPTeZ7PZcDqdlJeXNxoPDAzE6XS26nkcDkebcoaFhbVpfum6ioqKfB1BpMOZXg7Lly9n8uTJADQ0NGCxWLz3eTweLBbLUcdbIzQ0FKvV2j6hRQ6jFQfpitxud5Mr1abuVqqpqeHzzz9n5MiRAAQFBeFyubz3u1wubDabYbyiogKbzWZmNBERaYKp5fDNN99wzjnncNJJJwEwePBgSkpKKC0tpb6+ntzcXMLDwwkODsZqtXo337OzswkPDzczmoiINMHU3Urbt28nKCjIe9tqtTJ//nwSExNxu91EREQQHR0NwKJFi5gzZw5VVVUMHDiQhIQEM6OJiEgTLB6Px+PrEG1xaL9Ze3zmoIv9yG/pYj/SVTX33qkjpEVExEDlICIiBioHERExUDmIiIiBykFERAxUDiIiYqByEBERA5WDiIgYqBxERMRA5SAiIgYqBxERMVA5iIiIgcpBREQMVA4iImKgchAREQOVg4iIGKgcRETEQOUgIiIGppZDfn4+8fHxxMTE8MQTTwBQWFhIXFwckZGRpKene6ctLi4mPj6eqKgoUlJSqKurMzOaiIg0wbRy2L59O3PnziUjI4OcnBz++9//UlBQQHJyMhkZGeTl5eFwOCgoKAAgKSmJ1NRU1qxZg8fjITMz06xoIiLSDNPK4YMPPmD06NEEBQUREBBAeno6J554IiEhIfTt2xd/f3/i4uKw2+2UlZVRXV3NkCFDAIiPj8dut5sVTUREmuFv1gOXlpYSEBDAvffey86dO7n66qvp168fgYGB3mlsNhtOp5Py8vJG44GBgTidTrOiiYhIM0wrh/r6er744guWLVvGSSedxLRp0+jevTsWi8U7jcfjwWKx0NDQcMTx1nA4HG3KGxYW1qb5pesqKirydQSRDmdaOZx++ukMHz6cPn36AHDttddit9vx8/PzTuNyubDZbAQFBeFyubzjFRUV2Gy2Vj1faGgoVqu1fcKLHEYrDtIVud3uJleqTfvMYcSIEXzyySfs3buX+vp61q1bR3R0NCUlJZSWllJfX09ubi7h4eEEBwdjtVq9a2jZ2dmEh4ebFU1ERJph2pbD4MGDmTJlChMnTqS2tpYrrriCCRMmcO6555KYmIjb7SYiIoLo6GgAFi1axJw5c6iqqmLgwIEkJCSYFU1ERJph8Xg8Hl+HaItDm0btsVtp5H1p7ZRKuor8jNm+jiBiiubeO3WEtIiIGKgcRETEQOUgIiIGKgcRETFQOYiIiIHKQUREDFQOIiJioHIQEREDlYOIiBioHERExEDlICIiBioHERExUDmIiIiBykFERAxUDiIiYqByEBERA5WDiIgYqBxERMRA5SAiIgb+Zj74pEmT2LVrF/7+B5/m8ccf58cff+T555+nrq6O22+/nVtvvRWAwsJC0tLScLvdxMTEMHPmTDOjiYhIE0wrB4/Hw7Zt2/j444+95eB0Opk5cyZZWVl069aN8ePHc9lll3H22WeTnJzMsmXLOPPMM7nnnnsoKCggIiLCrHgiItIE08rhhx9+AODOO++ksrKSm2++mR49ejBs2DB69eoFQFRUFHa7nUsvvZSQkBD69u0LQFxcHHa7XeUgIuIjpn3msHfvXoYPH85zzz3HK6+8wvLly/npp58IDAz0TmOz2XA6nZSXlx9xXEREfMO0LYehQ4cydOhQ7+1x48aRlpbGtGnTvGMejweLxUJDQwMWi8Uw3hoOh6NNecPCwto0v3RdRUVFvo4g0uFMK4cvvviC2tpahg8fDhx8ww8ODsblcnmncblc2Gw2goKCjjjeGqGhoVit1vYJL3IYrThIV+R2u5tcqTZtt9K+fftYsGABbrebqqoq3n33XRYuXMj69evZtWsXv/76K2vXriU8PJzBgwdTUlJCaWkp9fX15ObmEh4eblY0ERFphmlbDiNGjOCrr75i7NixNDQ0MHHiRMLCwpg5cyYJCQnU1tYybtw4Bg0aBMD8+fNJTEzE7XYTERFBdHS0WdFERKQZFo/H4/F1iLY4tGnUHruVRt6X1k6ppKvIz5jt6wgipmjuvVNHSIuIiIHKQUREDFQOIiJioHIQEREDlYOIiBioHERExEDlICIiBioHERExUDmIiIiBykFERAxUDiIiYqByEBERgxaVw5Guyvb999+3exgREekcmiyHyspKKisrmTp1Knv27PHerqioYMaMGR2VUUREOliT13N4+OGH+fTTTwG47LLL/jeTvz9RUVHmJhMREZ9pshxefPFFAGbPnk1amq51ICLye9GiK8GlpaVRVlbGnj17OPzaQAMHDjQtmIiI+E6LyuGZZ57hxRdf5LTTTvOOWSwWPvroI9OCiYiI77SoHFatWsXatWs544wzzM4jIiKdQIu+ynrmmWceczH89a9/ZdasWQAUFxcTHx9PVFQUKSkp1NXVAfDTTz9x6623Eh0dzbRp09i/f/8xPZeIiLSPFpXD8OHDWbBgAUVFRWzevNn7X3PWr1/Pu+++672dlJREamoqa9aswePxkJmZCcC8efOYOHEidrud0NBQMjIyjvHliIhIe2jRbqWsrCwA7Ha7d6y5zxwqKytJT0/n3nvvZcuWLZSVlVFdXc2QIUMAiI+P55lnnuGmm27i888/57nnnvOO33bbbSQlJR3zixIRkbZpUTnk5+e3+oFTU1OZOXMmO3fuBKC8vJzAwEDv/YGBgTidTnbv3s3JJ5+Mv79/o/HWcjgcrZ7ncGFhYW2aX7quoqIiX0cQ6XAtKoeXX375iOOTJ08+4vg777zDmWeeyfDhw71bHQ0NDVgsFu80Ho8Hi8Xi/f/hfnu7JUJDQ7Fara2eT6Q5WnGQrsjtdje5Ut2icvj222+9/66pqeHzzz9n+PDhR50+Ly8Pl8vF9ddfz549ezhw4AAWiwWXy+WdpqKiApvNRp8+fdi3bx/19fX4+fnhcrmw2WwtiSUiIiZp8UFwh3M6naSkpBx1+sO3NLKysti4cSNpaWmMGTOGoqIiwsLCyM7OJjw8nICAAC655BLy8vKIi4tj1apVhIeHH+PLERGR9nBMp+w+44wzKCsra/V8ixYtIi0tjejoaA4cOEBCQgIAc+fOJTMzk9GjR/PFF1/w4IMPHkssERFpJ63+zMHj8eBwOBodLd2U+Ph44uPjAejfvz8rVqwwTBMcHMyyZcta9HgiImK+Vn/mAAcPinvkkUdMCSQiIr7Xqs8cysrKqKurIyQkxNRQIiLiWy0qh9LSUu677z7Ky8tpaGigd+/e/P3vf+e8884zO5+IiPhAiz6Qfvzxx5kyZQqff/45RUVFTJs2jXnz5pmdTUREfKRF5fDLL79www03eG/feOON7N6927RQIiLiWy0qh/r6eiorK723d+3aZVogERHxvRZ95nDbbbdxyy23EBMTg8ViIS8vj9tvv93sbCIi4iMt2nKIiIgAoLa2lq1bt+J0Ohk1apSpwURExHdatOUwa9Ysbr31VhISEnC73bz11lskJyfzj3/8w+x8IiLiAy3acti9e7f3VBdWq5U77rij0Un0RESka2nxB9KHX2OhoqICj8djWigREfGtFu1WuuOOOxg7dixXXXUVFouFwsJCnT5DRKQLa1E5jBs3jtDQUDZs2ICfnx933XUXF1xwgdnZRETER1pUDnDwjKr9+/c3M4uIiHQSx3Q9BxER6dpUDiIiYqByEBERA5WDiIgYmFoOixcvZvTo0cTGxnovNVpYWEhcXByRkZGkp6d7py0uLiY+Pp6oqChSUlKoq6szM5qIiDTBtHLYuHEjGzZsICcnh5UrV7Js2TK2bNlCcnIyGRkZ5OXl4XA4KCgoACApKYnU1FTWrFmDx+MhMzPTrGgiItIM08rh0ksv5bXXXsPf359ffvmF+vp69u7dS0hICH379sXf35+4uDjsdjtlZWVUV1czZMgQAOLj47Hb7WZFExGRZpi6WykgIIBnnnmG2NhYhg8fTnl5OYGBgd77bTYbTqfTMB4YGNjodB0iItKxWnwQ3LG6//77mTp1Kvfeey/btm3DYrF47/N4PFgsFhoaGo443hoOh6NNOcPCwto0v3RdRUVFvo4g0uFMK4etW7dSU1PDgAEDOPHEE4mMjMRut+Pn5+edxuVyYbPZCAoKanSW14qKCmw2W6ueLzQ0FKvV2m75RQ7RioN0RW63u8mVatN2K+3YsYM5c+ZQU1NDTU0NH330EePHj6ekpITS0lLq6+vJzc0lPDyc4OBgrFardw0tOzub8PBws6KJiEgzTNtyiIiIYNOmTYwdOxY/Pz8iIyOJjY2lT58+JCYm4na7iYiIIDo6GoBFixYxZ84cqqqqGDhwoPf6ESIi0vEsnuP8wgyHNo3aY7fSyPvS2imVdBX5GbN9HUHEFM29d+oIaRERMVA5iIiIgcpBREQMVA4iImKgchAREQOVg4iIGKgcRETEQOUgIiIGKgcRETFQOYiIiIHKQUREDFQOIiJioHIQEREDlYOIiBioHERExEDlICIiBioHERExUDmIiIiBykFERAxMLYdnn32W2NhYYmNjWbBgAQCFhYXExcURGRlJenq6d9ri4mLi4+OJiooiJSWFuro6M6OJiEgTTCuHwsJCPvnkE959911WrVrF5s2byc3NJTk5mYyMDPLy8nA4HBQUFACQlJREamoqa9aswePxkJmZaVY0ERFphmnlEBgYyKxZs+jWrRsBAQGcd955bNu2jZCQEPr27Yu/vz9xcXHY7XbKysqorq5myJAhAMTHx2O3282KJiIizTCtHPr16+d9s9+2bRvvv/8+FouFwMBA7zQ2mw2n00l5eXmj8cDAQJxOp1nRRESkGf5mP8F3333HPffcwyOPPIKfnx/btm3z3ufxeLBYLDQ0NGCxWAzjreFwONqUMywsrE3zS9dVVFTk6wgiHc7UcigqKuL+++8nOTmZ2NhYNm7ciMvl8t7vcrmw2WwEBQU1Gq+oqMBms7XquUJDQ7Fare2WXeQQrThIV+R2u5tcqTZtt9LOnTuZPn06ixYtIjY2FoDBgwdTUlJCaWkp9fX15ObmEh4eTnBwMFar1buGlp2dTXh4uFnRRESkGaZtObz44ou43W7mz5/vHRs/fjzz588nMTERt9tNREQE0dHRACxatIg5c+ZQVVXFwIEDSUhIMCuaiIg0w+LxeDy+DtEWhzaN2mO30sj70toplXQV+RmzfR1BxBTNvXfqCGkRETFQOYiIiIHKQUREDFQOIiJioHIQEREDlYOIiBioHERExEDlICIiBioHERExUDmIiIiBykGkk6utq/d1BOmEzF4uTL+eg4i0TYC/H4kv/dPXMaSTWXLn1aY+vrYcRETEQOUgIiIGKgcRETFQOYiIiIHKQUREDFQOIiJioHIQERED08uhqqqKMWPGsGPHDgAKCwuJi4sjMjKS9PR073TFxcXEx8cTFRVFSkoKdXV1ZkcTEZGjMLUcvvrqKyZMmMC2bdsAqK6uJjk5mYyMDPLy8nA4HBQUFACQlJREamoqa9aswePxkJmZaWY0ERFpgqnlkJmZydy5c7HZbABs2rSJkJAQ+vbti7+/P3FxcdjtdsrKyqiurmbIkCEAxMfHY7fbzYwmIiJNMPX0GU8++WSj2+Xl5QQGBnpv22w2nE6nYTwwMBCn02lmNBERaUKHnlupoaEBi8Xive3xeLBYLEcdbw2Hw9GmbGFhYW2aX7quoqIinz6/lk05GjOXzQ4th6CgIFwul/e2y+XCZrMZxisqKry7oloqNDQUq9XabllFDtGbs3RWbVk23W53kyvVHfpV1sGDB1NSUkJpaSn19fXk5uYSHh5OcHAwVqvV24LZ2dmEh4d3ZDQRETlMh245WK1W5s+fT2JiIm63m4iICKKjowFYtGgRc+bMoaqqioEDB5KQkNCR0URE5DAdUg75+fnefw8fPpycnBzDNP3792fFihUdEUdERJqhI6RFRMRA5SAiIgYqBxERMVA5iIiIgcpBREQMVA4iImKgchAREQOVg4iIGKgcRETEQOUgIiIGKgcRETFQOYiIiIHKQUREDFQOIiJioHIQEREDlYOIiBioHERExEDlICIiBioHEREx6FTlsHr1akaPHk1kZCRvvPGGr+OIiPxu+fs6wCFOp5P09HSysrLo1q0b48eP57LLLuP888/3dTQRkd+dTlMOhYWFDBs2jF69egEQFRWF3W5nxowZTc7n8XgAqKmpaXOG3id3b/NjSNfidrt9HQGAkwMsvo4gnUxbl81D75mH3kN/q9OUQ3l5OYGBgd7bNpuNTZs2NTtfbW0tAN9++22bMyTfEt7mx5CuxeFw+DoCAOMuPNnXEaSTaa9ls7a2lu7djSvGnaYcGhoasFj+t3bk8Xga3T6aHj16cMEFFxAQENCi6UVE5OB7bG1tLT169Dji/Z2mHIKCgvjiiy+8t10uFzabrdn5TjjhBE455RQzo4mIdElH2mI4pNN8W+nyyy9n/fr17Nq1i19//ZW1a9cSHq7dPCIivtBpthzOOOMMZs6cSUJCArW1tYwbN45Bgwb5OpaIyO+SxXO0j6pFROR3q9PsVhIRkc5D5SAiIgYqBxERMVA5iIiIgcrBh3bs2EFoaCjXX399o/927tx51HmWLFnCkiVLAPjjH/941OmWLVtGaGgoLperRVlmz55NWVlZ615AK2zfvp3k5GQAvv76a1JSUkx7Lmm93y6LUVFRzJ49m4qKimN6vMWLF/PRRx8d9f6UlBS+/vrrY43rtWXLFhISErjuuuuIjY0lJSWFAwcOtPlxj8Xhy3hX0Gm+yvp7ZbPZyM7ObvfHzcrK4pprrmHlypXce++9zU7/2WefMX369HbPcchPP/3E9u3bAbjooou46KKLTHsuOTaHL4sej4enn36a+++/nzfffLPVj/XAAw80ef+TTz55TBl/a+bMmTz11FMMHTqUhoYG5s2bx+LFi5k9e3a7PH5rHL6MdwUqh05q1qxZXHrppcTHxwMHtxK++eabFs27ZcsW9uzZw9SpU7n//vu5++67OeGEE9ixYwcJCQnk5+cDeLdArFYr5eXl3H333bzxxhuUlpby5JNP4na76d27N48//jghISFMmjSJCy+8kKKiItxuN3/+85957bXX2Lp1K3fccQd33HEHTqeT5ORk9u3bR3l5OTfccAMPPPAATzzxBDt27GDevHlER0fz7LPPsmzZMkpKSkhNTaWyspKTTjqJlJQUBg0axKxZszj55JPZvHkzTqeT6dOnc+ONN5rzwxYDi8VCYmIiV1xxBVu2bKF///4sXbqU999/n/r6eq688kqSkpKwWCy88sorvPXWW/j5+TFixAiSkpK8y29kZCQPPfSQdwtk+vTpXHPNNUyaNIkZM2Zw2WWX8be//Y2cnBz8/Py44oorSEpKYufOncyYMYN+/fpRXFzMaaedxuLFi70n5jykoqKC6upq4ODZEmbMmOHdAj7a39CSJUv46aef2Lp1K7t37+aWW25hypQpZGVl8c9//pNffvkFl8vFiBEjmDVrFhaL5agZp0yZQu/evenevTsVFRXeZXzu3Lkd+Nsyh3Yr+Vh5eXmjXUovvPBCmx9z5cqVREdHExoair+/P+vWrWty+rvvvhubzcbSpUvp0aMHDz30EI8++ig5OTmMHz+ehx56yDutx+NhxYoVREVF8cQTT/Dss8/yxhtv8NxzzwGQm5vLmDFjyMzMZPXq1bz66qvs2rWLOXPmEBoaavijSUpKYtKkSaxevZrZs2fzwAMPeM8W+fPPP/Pmm2/y/PPPs2DBgjb/XKR1unXrRkhICD/88AP/+te/cDgcrFixglWrVuF0OsnJyWHTpk28+eabrFixgpycHDZv3tzohHAffPABwcHBZGVl8eSTTzY6RQ5AQUEB+fn5rFy5knfffZfS0lKWL18OHFzJmTx5Mrm5ufTs2ZPVq1cbMs6ePZtp06YRGRnJo48+yubNmxkyZEizr83hcPDyyy+TlZXF22+/zebNmwEoKipi8eLF5Obm8tVXX/HBBx80mbGkpISFCxfy8ssvH3UZP15py8HH2nu3Um1tLatXr+all14CICYmhuXLlxMREdGi+bdt20bPnj29R6fHxMSQmprKvn37ALynNDnrrLMYPHgwJ554IsHBwezduxeAu+66iw0bNvDiiy/y3XffUVtby6+//nrE59q/fz8//vgjkZGRAAwZMoRTTz2VH374AYArrrgCi8XCBRdcQGVl5TH+RKQtLBYL3bt3Z/369WzatMm7Fl5dXc1ZZ51FRUUFI0aM8J7f7JVXXmk0/9ChQ3n66adxOp1cffXVhl2XGzZsIDY2lhNPPBGAG2+8kVWrVhEREcFpp53GhRdeCEC/fv3Ys2ePIV98fDyRkZGsX7+ewsJCZs2aRVxcXLOfaY0ZM8Z7wrmRI0eyYcMGevfuzTXXXMPpp58OwOjRo9mwYQNWq7XJjGeffXZrfqTHDZVDJ2WxWLznWT90WvKW+Pjjj9m3b5/3Ohi1tbX88ssv/Pzzz40eE6Curg5//8aLQENDg+ExPR4P9fX1AAQEBHjHfzsvwPz589m+fTtjxozh2muvpbCw8Kjniz/S+OHPZbVaAXS2XR+pqamhpKSE888/nw0bNnD77bczefJkAPbu3Yufnx8rVqxo9PtxOp3eN1GAc845h/fff59169bx8ccf89JLL5GXl+e9/0jLW11dHfC/3z9gWHbh4IrMe++9x/Tp0xk1ahSjRo0iISGBG264gZSUlCb/hvz8/BplOHT7SONNZWzqxHXHO+1W6qR69erF999/D8CHH37Y4vmysrJ44IEHyM/PJz8/n3Xr1hEWFsY777xDz549qaysZNeuXdTU1DTa3eTn50d9fT3nnnsulZWV3mtp5OXlcdZZZxn29R7Np59+yl133UVMTAwlJSU4nU7vH9mhP6hDTj75ZM4++2zWrl0LwH/+8x8qKiro169fi1+vmKOhoYElS5YwePBg/u///o9hw4aRnZ3N/v37qaurY/r06axZs4ZLLrmEgoIC7/jDDz/caLfS66+/zpIlS4iJiWHu3Lns2rWLqqoq7/3Dhg3jvffeo7q6mrq6OlauXMmwYcNalLFPnz689tprrF+/3jtWXFzMgAEDgKb/hj788ENqamrYs2cPH3/8MVdeeSUA69atY9++fbjdbt577z3Cw8NbnPFIy/jxTFsOndSECRN48MEHiYuLY9iwYY0uhHQ0FRUVfPbZZzz11FONxidPnsxjjz3Gfffdx5QpUxg3bhxBQUGNvjF09dVXc/fdd/PCCy+Qnp7OX/7yF3799VdOPfVU0tPTW5z7nnvu4ZFHHqF79+4EBQURGhrKjh07GDBgAPv27SMpKYlx48Z5p1+4cCGPPfYYS5YsISAggCVLltCtW7cWP5+0n0Off8HBchgwYABPP/00cHDXy5YtW7j55pupr6/nqquu4oYbbsBisXDbbbcxfvx4GhoaGDVqFJdffjk5OTkAjB07loceeoi4uDj8/PxISkqiZ8+e3uccMWIExcXF3HjjjdTV1XHllVdy22238fPPPzebt2fPnixdupSFCxcyZ84cAgIC+MMf/uDN3NTfkNVqZeLEiVRVVXHPPfdw/vnns2nTJvr06cPUqVPZvXs31113HVdddRVAizKed9553mV84cKFbfhNdA468Z6I/K4c+pZeYmJio/GsrCw2btzI/PnzfRGr09FuJRERMdCWg4iIGGjLQUREDFQOIiJioHIQEREDlYNIK+3YsYOhQ4ce8b5DpwoZNWoU1113HWPHjuWFF17wHth3SG1tLVdeeSVTpkzpiMgiraZyEGkne/fuZcKECd6jgnNycnj11Vf5+uuveeSRRxpN+8EHH9C/f38cDgdbt271UWKRo1M5iLSTt956iwEDBjBlyhTvqUVOPfVUFixY4D030eHTXnPNNYwePZpXX33VV5FFjkrlINJOvvzyS/70pz8Zxq1WK2FhYfz73/8G4Pvvv+fLL78kOjqasWPHkp2dze7duzs6rkiTVA4i7ehoJ0k8dBpyOLjVMGLECHr37s2gQYM4++yzyczM7KiIIi2ichBpJxdffDEbN240jO/fv5+vv/6aiy++mAMHDpCdnU1RUREjR45k5MiRuFwuXn/99VadfVfEbCoHkXYyceJEtm7dytKlSynCD08AAACvSURBVL3fTtqzZw+zZs3ikksuYdCgQaxevZpevXqxbt0675lzP/zwQw4cOIDdbvfxKxD5H52VVeQYHDhwwPB11uXLl/P222+zePFiRo8eTUBAABaLhTFjxnDnnXcCB3cpTZ48udF1A3r27MmkSZN45ZVXiIuL69DXIXI0OreSiIgYaLeSiIgYqBxERMRA5SAiIgYqBxERMVA5iIiIgcpBREQMVA4iImKgchAREYP/B7Xza6AzZ9NsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.DataFrame({'predicted': pred})\n",
    "df['score']= df['predicted'].apply(lambda x: f1_dict[x])\n",
    "df['LOA']=df['score'].apply(lambda x: 'Full Automation' if x >0.95 else 'Decision Support')\n",
    "print(df)\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set(style=\"whitegrid\")\n",
    "ax = sns.countplot(x=\"LOA\",data=df,  palette=\"Blues_d\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}