Design and Evaluation of Large Language Model Approaches for Text2Cypher Generation in Domain-Specific Knowledge Graphs

Abstract

In recent years, the integration of Large Language Models (LLMs) with knowledge graphs has gained increasing attention as an approach to improve information retrieval and data accessibility. However, few studies have compared finetuned and prompt-based LLM approaches for generating Cypher queries from natural language inputs. This paper presents a comparative study conducted within BMW AG, focusing on the application of LLM-based Text2Cypher generation for querying a domain-specific knowledge graph designed to support virtual product development. Three approaches are evaluated: a baseline model fine-tuned for generic Cypher generation, a newly developed domain-specific fine-tuned model (KG-Finetune-LLM), and an embedding-based prompt engineering approach (KG-Embed-LLM) leveraging GPT-4. Results show that the KG-Embed-LLM achieves the highest retrieval accuracy (96.88%), outperforming both fine-tuned models. Error analysis further reveals that the primary challenge in industrial Text2Cypher generation lies in correctly interpreting domain-specific graph semantics, particularly relationship selection and graph traversal, rather than in generating syntactically valid Cypher queries. The findings highlight the importance of schema-aware grounding for domain-specific knowledge graph applications and identify key trade-offs between retrieval accuracy, deployment complexity, and reliance on external LLM services. Finally, the study outlines future research directions related to ontology complexity and robustness in enterprise knowledge graph environments.