A dynamic and self-corrective procedure with feedback-loop-based AI architecture with Large Language Models (LLMs) for near real-time validation and correction of generated queries and it is demonstrated that the proposed solution outperforms various training-free prompt injection defense methods.
Abstract
Text-to-Cypher generator systems translate natural language questions into Cypher queries, enabling intuitive interactions with graph databases such as Neo4j and Amazon Neptune. Despite recent advancements in LLM-based Cypher query generation, the vulnerabilities of the known methods—such as prompt injection attacks—are not discussed in detail. In this paper, we employ a robust Retrieval-Augmented Generation (RAG) architecture tailored specifically for text-to-Cypher tasks, leveraging dense vector retrieval to enhance query generation accuracy. We propose a dynamic and self-corrective procedure with feedback-loop-based AI architecture with Large Language Models (LLMs) for near real-time validation and correction of generated queries. We create a systematic procedure for generating datasets specifically designed to assess prompt injection robustness. Comprehensive evaluations are conducted using a diverse set of LLMs, including GPT-4o, DeepSeek R1, Claude 3.5 Sonnet and Qwen 2.5 Coder 32B Instruct. Our evaluation results indicate substantial improvements in resiliency against prompt injection attacks compared to various benchmarks. It is demonstrated that the proposed solution outperforms various training-free prompt injection defense methods.
Due to the surge in digital document creation, intelligent systems that can effectively retrieve accurate information from unstructured PDF content have become a necessity. Current keyword-based search techniques lack semantic meaning, while large language models (LLMs), when utilized individually, produce hallucinated results when they are not contextually aware of document content. In this paper, we introduce a novel real-time question answering system for PDF content using a lightweight approach to Retrieval-Augmented Generation (RAG). Our system combines a HuggingFace sentence transformer model for semantic embeddings with a FAISS vector similarity search for efficient retrieval, leveraging a Groq-hosted LLaMA 3.3 70B model for context-grounded answer generation. The system is built using LangChain for orchestration and Streamlit for web-based deployment, and requires no GPU, no fine-tuning, and no pre-indexed corpus. Evaluation across five complementary metrics—Exact Match, Token-level F1, BLEU, ROUGE, and semantic similarity—demonstrates strong alignment between generated and reference answers, with a peak semantic similarity of 0.8709 and a mean Token F1 of 0.4701, confirming the viability of this approach for real-world document intelligence.
Mukesh Lakshmi Sai Medikonda, Kalva Vishnu Teja, K. Greeshma et al.· International Conference Com...· 0 citations
Knowledge Graph (KG) retrieval is a promising augmentation to address knowledge gaps and hallucinations in LLMs. As KGs in practice are stored in graph databases (e.g., Wikidata, Freebase), accurate retrieval requires translating natural language questions into structured queries (query generation). A key challenge of query generation is Text-to-Cypher, which generates Cypher queries for property graphs (e.g., Neo4j), a paradigm increasingly adopted in industry for their scalable architectures and expressive schemas. However, compared to other query generation tasks such as Text-to-SQL or Text-to-SPARQL, Text-to-Cypher remains underexplored due to scarce public KGs and datasets. Existing datasets are small, domain-limited, and lack diversity, constraining LLM progress. To address this, we introduce CypherSmith, an instruction-tuning dataset over 12 × larger than prior public Text-to-Cypher datasets, spanning diverse domains to better support LLM fine-tuning. Our key distinction lies in fully leveraging open-source LLMs for large-scale synthetic data generation and introducing a novel likelihood-based filtering technique to ensure high-quality Text-to-Cypher data. Extensive experiments demonstrate the effectiveness of CypherSmith, achieving state-of-the-art LLM performance.
Zeyu Zhang, Kexuan Sun, Zheng Tang et al.· Annual Meeting of the Associ...· 0 citations
Large language models (LLMs) have demonstrated strong capabilities in structured query generation, making them a natural choice for Text-to-SPARQL, which translates natural language questions into executable SPARQL queries over knowledge graphs. However, their initial outputs remain unreliable: generated queries may be executable yet semantically misaligned with input questions, leading to incorrect retrieval. To address this issue, we propose Generator-Gate-Corrector (GGC), a framework for reliable LLM-based Text-to-SPARQL generation. GGC first uses a Generator to produce an initial query, then applies a Gate to predict whether correction is needed, and finally invokes a Corrector only for selected high-risk queries. This selective correction mechanism avoids unnecessary modifications and reduces the risk of degrading originally correct queries. Experiments on MCQA show that GGC improves query-level accuracy from 90.23\% to 98.33\% while reducing inference overhead by 45\% compared with correcting all generated queries. Ablation studies show that the Gate is robust across thresholds and that Corrector training data composition affects correction effectiveness and stability. Overall, the results demonstrate that selective correction enhances the accuracy, reliability, and efficiency of LLM-based text-to-SPARQL generation.
Ziyi Yang, Thanh-Son Nguyen, T. Nguyen et al.· 0 citations
Retrieval-augmented generation (RAG) effectively enhances the accuracy and timeliness of large language models (LLMs) by incorporating external knowledge retrieved from ex-ternal sources. However, with the increasing prevalence of LLM-generated content, exter-nal corpora used by RAG systems may become contaminated with LLM-generated texts. Such contamination compromises the reliability and quality of retrieved results, ultimately leading to a degradation in RAG performance, and raises concerns about the diminishing presence of human texts and the “Spiral of Silence” effect. A natural solution is to incorporate LLM text detectors into the RAG pipeline to filter out LLM-generated texts from the retrieved results. However, their effective use in RAG remains under-explored. In this paper, we explore the usage paradigms of LLM text detectors for RAG and highlight key limitations of off-the-shelf or directly fine-tuned detectors. To this end, we propose a RAG-aware data augmentation strategy that aligns detector training with realistic contamination patterns. Our approach synthesizes training data from both LLM and human texts under diverse generation modes. Experiments show that our method mit-igates performance degradation and improves the long-term stability of RAG systems.
Zhaoheng Huang, Yutao Zhu, Ji-Rong Wen et al.· Annual Meeting of the Associ...· 0 citations
Large Language Models tend to hallucinate when answering domain-specific ques tions from scientific documents without prior fine-tuning. Currently, methods such as Retrieval-Augmented Generation partially solve this problem but face different challenges: limited context knowledge, difference between sparse and dense retrieval, and retrieval noise. This paper presents an Advanced Multimodal Retrieval-Augmented Generation system that aims to solve those challenges and im prove the accuracy of information extraction. The proposed architecture introduces a multimodal ingestion pipeline that leverages an open-source Vision-Language Model (Qwen2-VL-2B-Instruct) to generate textual summaries of tables and fig ures. The retrieval phase integrates HNSW-based semantic search with GIN-based lexical search, unified through Reciprocal Rank Fusion and refined using Cross Encoder reranking to minimize retrieval noise. To ensure conversational coherence across multi-turn interactions, a Query Condenser module is employed. Evaluation is conducted by independently assessing the ingestion, retrieval and generation stages using the MMLongBench benchmark, a BeIR-format synthetic dataset and the DeepEval framework. Moreover, results demonstrate a 157% improvement in retrieval quality over a Naive-RAG baseline, with only 50 ms additional la tency, while Qwen2-VL-2B-Instruct achieved results comparable to cloud-based models in BERTScore. These findings validate that open-source optimized SLMs, paired with advanced retrieval strategies, can provide competitive performance for document understanding without relying on cloud-based models.
Retrieval augmented generation (RAG) combines language models with external corpora to support knowledge-intensive tasks, such as open-domain question answering. Standard RAG systems typically employ a fixed top-k retrieval strategy, retrieving the same number of passages regardless of query needs. This can lead to either insufficient evidence, or the inclusion of irrelevant contexts that lead to a degradation of generation performance. In this work, we conduct an empirical study of how irrelevant retrieved passages affect downstream generation, analyzing their impact across multiple standard generator models. Building on these insights, we propose a lightweight, context-size classification module that dynamically predicts how much context is required based on query-specific needs. We integrate this approach into a full RAG pipeline and demonstrate improved performance over several baselines.
Maya Iratni, M. Boughanem, T. Dkaki· Annual International ACM SIG...· 0 citations