ParsiNLU is a comprehensive suit of high-level NLP tasks for Persian language. This suit contains 6 different key NLP tasks --- Reading Comprehension, Multiple-Choice Question-Answering, Textual Entailment, Sentiment Analysis, Query Paraphrasing and Machine Translation.

These challenges are collected by expert annotators native in Persian language and from various sources. If you'd like to see additional details on the dataset and how we collected, please check out our publication.

You can find the data under the data/ directory.

Alternatively, you can also access the data through the HuggingFace🤗 library. Here is an example:

from datasets import load_dataset

dataset = load_dataset("persiannlp/parsinlu_entailment")
for x in dataset['train']
  print(x)

Our models are deployed on HuggingFace's model hub. You can our list of models in this page.
Each model readme contains descriptions on how to use it.

First, make sure you have the data your data/ directory.

Set up your environment. You can use conda or virtualenv to create a Python environment. We have tested the code with Python3.7, but it should work on any version >= 3.5. Make sure to install all the necessary dependencies:

> python install -r requirements.txt

If you're using a GPU, make sure that you set the appropriate environmental variable. For example:

export CUDA_VISIBLE_DEVICES=YOUR_GPU_ID # for example, YOUR_GPU_ID can be 0 for using your first GPU; or 0,1 if you're using two GPUs

See the relevant section on how to train models for each task:

• Textual entailment
• Query Paraphrasing
• Reading Comprehension
• Multiple-choice QA
• Machine Translation
• Sentiment Analaysis

Textual Entailment is the task of deciding whether a whether two given questions are paraphrases of each other or not.

Here are several examples:

To reproduce our baselines (except mT5 models; see this), try train_and_evaluate_entailment_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

QQP is the task of detecting whether two given questions are paraphrases of each other or not.

Here are several examples:

To reproduce our numbers with all our baselines, try train_and_evaluate_qqp_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

In this task, the goal is to generate a response to question and its accompanying context paragraph. Here are several examples:

To reproduce our baselines (except mT5 models; see this, try train_and_evaluate_reading_comprehension_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

Here the task is to pick a correct answer among 3-5 given candidate answers. Here are several examples:

To reproduce our baselines, try train_and_evaluate_multiple_choice_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

Machine Translation of Persian/English is one of the few tasks that has received more work in the past few years. Unfortunately, most of the evaluation done for this task is often limited to few domains/datasets.
Here we have compiled a collection of high-quality resources for Persian machine translation. Specifically, here is our collection of evaluation sets:

• Quaran: BVased on the existing translations of Quran.
• Bible: Based on the existing translations of Bible.
• Mizan: Parallel corpora constructed from human translations of literary masterpieces.
• Global Voices:
• Quora queries: the translation instance extracted from our query paraphrasing task.

Here are several examples:

To downloading the data, take a look at the this and that directories.

All the baselines here use mT5. Take a look at this section for more description on that.

Our aspect-based sentiment analysis task includes three sub-tasks: 1) detecting the overall sentiment of a review/document, 2) extracting aspects toward which a sentiment is expressed, and 3) detecting the sentiment polarity of extracted aspects. Our annotation scheme is mainly inspired by the Sem-Eval 2014 Task 4, ABSA scheme, with minor adjustments. Sentiment scores are chosen from (very negative, negative, neutral, positive, very positive, mixed/borderline).

We have annotated documents from food & beverages (Digikala) and movie review (Tiwall) domains. We have predefined list of aspects for each domain. In the following, we have listed some examples from our dataset:

Domain	Review	Sentiment	(Aspect, Sentiment)
Food & beverages	خیلی خیلی کادوی جذابیه هم بسته بندی شیک هم شکلات خوشمزه و قلبی شکل خصوصا که پاکت هم داره	Very positive	(بسته بندی، خیلی مثبت) (طعم، مثبت)
Food & beverages	در شگفت انگیز به قیمت خیلی پایین خریدم ولی به نظرم ارزش نداره و طعم خاصی جز شکر نداره	Negative	(ارزش خرید، منفی) (طعم، منفی)
Movie review	در جشنواره متاسفانه نتونستم ببینم ولی دیشب در اکران فیلم های جشنواره فجر در پردیس چارسو موفق به دیدن فیلم شدم. چه فیلم خوبی از فضای بصری زیبا و چشم نواز، تا بازی فوق العاده حامد بهداد.....	Positive	(صحنه، مثبت) (بازی، خیلی مثبت)
Movie review	فیلمی بسیار ضعیف، علی الخصوص در زمینه ی تدوین و فیلم نامه پر از شعار زدگی، کلیشه و اغراق آمیز!!! واقعا خانم درخشنده توی این فیلم تنزل فاحشی پیدا کردن. بعد اصلا معلوم نیست اون زن دوم اون وسط چی میگه، از بس که شخصیت پردازی ضعیفه!	Very negative	(بازی، خیلی منفی) (داستان، خیلی منفی) (کارگردانی، خیلی منفی)
Movie review	فیلم از فضای نقد اجتماعی و سیاسی تهی است...یه قصه غیر قابل باور که هیجان خاصی نداشت...ریتم فیلم قابل قبول بود...الناز شاکردوست هم خیلی فراتر از انتظار بود...نمره 5 از 10	Mixed/borderline	(داستان، منفی) (بازی، خیلی مثبت)

To reproduce our baselines (except mT5 models; see this, try train_and_evaluate_sentiment_analysis_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

To use mT5 baselines you need the followings:

• Convert all the data into train/test/dev splits organized in TSV format (first column: input, 2nd column: output). For example, see the translation data. We have a script for this conversion.
• A Cloud storage: best to use Google cloud buckets to store your train/eval/dev data.
• A TPU: for the experiments in this work we used v3-8 TPUs. You can get such TPUs for free from the TFRC program (thanks, Google!!).
• A virtual machine to run the experiments: create one CPU machine, in the same zone/region as your TPU.
• Install T5 and run the file-tuning scripts on your virtual machine.
• After your obtain the predictions, use our evaluation scripts in this directory.

Note: If you don't have access to TPUs, you should be able to use the HuggingFace library to train mT5 on your GPU machine. We don't have the scripts for that, but you should be ale to follow the examples here.

I have GPU on my machine by n_gpu is shown as 0. Where is the problem? Check out this thread.

If you find this work useful please cite the following work:

@article{2020parsiglue,
    title={{ParsiNLU:} A Suite of Language Understanding Challenges for Persian},
    author={Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabadi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, Yadollah Yaghoobzadeh},
    journal={arXiv},
    year={2020}
}

If you use the translatin task, please cite the following work as well:

@article{kashefi2018mizan,
  title={MIZAN: a large persian-english parallel corpus},
  author={Kashefi, Omid},
  journal={arXiv preprint arXiv:1801.02107},
  year={2018}
}