LLM Prompt Optimization: From Zero-Shot to Automatic Role-Playing Generation Using Ontology

Abstract

The performance of an LLM application is highly dependent on how the user commands the application through prompts. Previous researchers invented role-play prompting techniques to overcome these problems. Including roles relevant to the task can improve the accuracy of the responses generated by LLMs. Nevertheless, the assignment of roles in this study remained a human-driven process. This study addresses the research gap in previous studies by developing deep learning models to effectively predict relevant roles based on tasks. In addition, the impact of the ontology data as an enrichment of the training data on the model performance in two scenarios (with and without ontology) was examined. In the with-ontology setup, enrichment data are extracted from an occupational ontology that includes roles, skills, and abilities along with their corresponding definitions. Data that were restricted to four predefined occupations were incorporated into the benchmark datasets. The benchmark datasets consisted of questions used to assess the LLM performance aligned with predetermined occupations. In without-ontology settings, the models were trained using only the benchmark datasets. An experiment on three variants of recurrent-based and a graph-based model in both scenarios showed that these models exhibited stable convergence and performed best on ontology-enriched data. Notably, the recurrent neural network (RNN) exhibited significant performance gains, achieving improvements of 34% and 1.86% in the F1 score and receiver operating characteristic-area under the curve (ROC–AUC), respectively. Despite its great performance—92% F1 score and 97% ROC–AUC, the gated recurrent unit is computationally the most expensive model compared to long short-term memory, RNN, and graph convolutional network. Further research is required to explore alternative model architectures and expand the scope of their predictable roles.