Multi-task fine-tuning using prefix-prepend format in text-to-text transfer transformer for automatic question generation in Bahasa Indonesia

Abstract

Automatic question generation is one solution to help create test items that require a lot of time and effort. The state-of-the-art model for automatic question generation in Bahasa Indonesia, which uses idT5, has several drawbacks, including misuse of context, overuse of question words, and an answer target that must be exactly from the context, which makes it extractive. This research aims to address those problems by improving the performance of previous models through a new fine-tuning scheme. This research proposed a prefix-prepend format for fine-tuning with the idT5-base model on the Indonesian Stanford Question Answering Dataset (SQuAD) and the Typologically Diverse Question Answering (TyDiQA) dataset. We evaluate the model with Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and Bidirectional Encoder Representations from Transformers (BERT) similarity metrics. The results show that prefix-prepend fine-tuning improved the performance of the baseline model. Our best model achieved 0.1643 BLEU, 0.4099 ROUGE-L, and 0.7177 BERT similarity score on SQuAD, and 0.1941 BLEU, 0.4301 ROUGE-L, and 0.7291 BERT similarity score on TyDiQA. The human evaluation using the Content Validation Index (CVI) and a paired t-test indicated that the proposed model performed better than the baseline. While the proposed model addresses many of the baseline’s shortcomings, it still struggles to handle questions that require understanding complex relationships between entities. Future studies can explore improvements for this case using external knowledge or other models.