This project aims to automatically score student essays using NLP techniques. Here, the problem of automatic grading is approached as a regression problem.

This project uses the ASAP-AES dataset (https://www.kaggle.com/c/asap-aes/data) and builds a model to predict scores for essays written by Grade 7 to Grade 10 students. To do this, we'll provide the model with a description of many essays having various features. The underlying objective is to observe the effect of essay features (taken separately and in combination) on the score.

1. About the ASAP-AES Dataset
2. Some Important Files
3. Application Design
4. Results
5. Setup Instructions
6. Usage Details
7. Visualization
   • 7.1 Scores Across Essay Sets
   • 7.2 Resolved Human Ratings V/s Meta Features.png
   • 7.3 Resolved Human Ratings V/s Extracted Features.png
   • 7.4 Resolved Human Ratings V/s Readability Features.png

The dataset was made available through a competition held by The William and Flora Hewlett Foundation (Hewlett). There are sevral available data formats including TSV and excel. Each file has a number of columns. The ones that are important to ur discussion of the project are:

• essay_set: 1-8, an id for each set of essays
• essay: The ascii text of a student's response
• rater1_domain1: Rater 1's domain 1 score; all essays have this
• rater2_domain1: Rater 2's domain 1 score; all essays have this
• domain1_score: Resolved score between the raters; all essays have this

The essay_set indicates which set the essay belongs to. each set has a different writing prompt and different range of scores in which the human grader grades.

The essay column has all the essays in text format with some named entities anonymized as:

"PERSON", "ORGANIZATION", "LOCATION", "DATE", "TIME", "MONEY", "PERCENT"

Besides, the above score columns, ie. rater1_domain1, rater2_domain1, domain1_score there are scores for other domains as well. However, they are not present for each and every essay. These are ignored for simplicity. However, incorporating these scores in the model building process post preprocessing may give better results.

Table: ASAP-AES Dataset overview:

Table: Dataset Stats (set-wise):

This repository contains a couple of Jupyter notebooks, and a single python script.

Jupyter Notebooks:
    data_etl.ipynb                  to extract transform and load the ASAP-AES dataset
    regression_model.ipynb          for interactive model building
    
Script:
    main.py                         similar to regression_model.ipynb notebook, see note below

Directories:
    input/asap-aes                  for storing extracted ASAP-AES dataset
    output/training_set_rel3.pkl    a pre formed dataframe with additional features for quick modeling

The script file main.py makes use of loops for model training and evaluation across all possible feature combinations. This is different from the regression_model notebook, since there training is done on all features at once.

In a regression problem, we aim to predict the output of a continuous value, like in this case, the score for an essay. Although the grading is done in discrete steps within the range of 0-60, we can solve this problem through regression. Other alternative approaches may include classification, Neural Nets and Deap Learning.

First, we load the data into a data frame and compute various sets of features. These can be roughly categorized as:

meta_features = ['essay_length', 'avg_sentence_length', 'avg_word_length']
grammar_features = ['sentiment', 'noun_phrases', 'syntax_errors']
redability_features = ['readability_index', 'difficult_words']

1. Meta Features:
   1. Essay Length (number of words)
   2. Average Sentence Length
   3. Average Word Length
2. Grammar Features:
   1. Sentiment (+ve/-ve) *
   2. Noun Phrases Count
   3. Syntax Errors Count
3. Readability Features:
   1. Readability Index Score
   2. Difficult Words Count

These features are computed using TextBlob package and other method and then fed into a Sequential model with two densely connected hidden layers, and an output layer that returns a single, continuous value. This model is trained for 1000 epochs, and record the training and validation accuracy. Callbacks are provided for early returns in case of no further improvement is observed.

The training happens across all possible feature combinations given and results can be compared (use main.py script). These combinations sum up to 511.

Please see the report here. A summary is provided below:

1. Inclusion of essay_set in training feature set always improved the results. Without essay_set, QWK 24.
2. The feature set (sentiment) performed worst with QWK -0.00016. The only feature set to have a “chance” agreement.
3. Considering only single feature sets, (essay_length) performed best with QWK ~ 0.15, followed by (avg_sentence_length), (difficult_words), (noun_phrases), (syntax_errors), (readability_index).
4. Adding more features didn't always give better results (i.e. more accurate scores).

Please make sure that the dataset is downloaded and extracted to input/asap-aes folder.

1. Install and activate python3 virtual environment, see: https://docs.python.org/3/library/venv.html
2. Install required packages via pip install -r requirements.txt
3. Download additional library files (see notebooks for details, most likely this won't be an issue)
4. Start Jupyter Server, see: https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html
5. Run data_etl.ipynb first, followed by regression_model.ipynb

As of now, the program has no command line parameters that could be passes. Just use python3 main.py once output/training_set_rel3.pkl is created.