Closed-Domain Event Extraction Literature Review

Abstract

Executive Summary This work is a literature review on the task of closed-domain event extraction, a task within natural language processing (NLP) where the goal is to detect the presence of an event (an occurrence of an action or state change) and event-related information (known as arguments) and to return the event structure. In our review, we focus on recent state-of-the-art (SOTA) approaches which can be applied to unstructured English-language text. After the introduction, we begin by providing an overview of the terminology of closed-domain event extraction, including a breakdown of the constituent sub-tasks. Next, we discuss the dataset requirements for this task, and note the core datasets used in the literature. We begin our review of the literature by providing a typology of event extraction approaches based on the common differences between extant approaches. We provide a table summarising the main results in the literature and discuss standards of evaluation in the literature. We found that all reviewed SOTA approaches utilise a PLM, with the choice of model often dependent on whether a classification or generative approach is taken-no single approach is dominant. BERT and BART are popular choices of PLM in the literature. We argue that evaluation standards have been inconsistent, resulting in results which are difficult to compare. We then describe some of the details of the most performant models in our table. Finally, we discuss some of the main themes we have identified in our review. In particular, we outline some clear issues in the extant literature. We argue that there is a need for a new, open source dataset to act as the primary benchmark to facilitate more academic research, with high inter-annotator agreement (IAA) ensuring that good performance on the benchmark is meaningful. We also call for standardisation around pre-processing and evaluation of results to ensure the comparability of results. We note that low-resource performance has typically been under-explored. In our recommendations for future work, we argue that there is clear potential to assess the performance of modern, causal large language models (LLMs) such as the Llama or GPT model families. We argue that it there is clear potential to explore both in-context learning and fine-tuned approaches with these models. We also argue that these models have clear potential applications in the generation of synthetic data. We conclude by summarising our main points. 1