The paper contributes three reusable deployment patterns: hybrid RAG evidence construction, multi-channel retrieval and reranking produce auditable FAQ candidates, and trace-driven RAG and reranker improvement, where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.
Abstract
Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.
Goal-oriented conversational systems must answer factual questions, understand visitor-provided information, and advance business objectives without becoming rigid questionnaires. This paper proposes a Symbolic-RAG-Generative architecture centered on the Goal-oriented Retrieval-Augmented Conversation Engine (GRACE). An instruction-constrained Business Goal Compiler transforms business intent into an immutable objective set, normalized priority vector, canonical questions, and initial state vector. At runtime, GRACE receives the complete conversation history, latest visitor message, current state, and grounded answer generated by a separate RAG component. It updates completion only from visitor-authored evidence and selects one contextually modulated follow-up. The core policy maximizes expected business progress subject to a minimum visitor-utility constraint. We formalize the state, monotonic transitions, source separation, question modulation, and constrained policy; present the reference architecture; and define an evaluation comprising 24 English real-estate and 10 Spanish professional-cleaning conversations, totaling 119 protocol-defined visitor turns. Across both domains, GRACE achieves 84.9% exact state-transition accuracy, 91.6% evidence precision, 89.6% evidence recall, 100% monotonicity, and 94.1% terminal-state accuracy. The evaluation establishes compelling symbolic-state performance across standard, multi-goal, RAG-detour, validation, refusal, and robustness scenarios.
Industrial support business processes often involve work outside core production activities, including record retrieval, spreadsheet checking, supplier communication, and follow-up of operational events. We examine these issues in a maintenance-support case where a low-code conversational Artificial Intelligence (AI) layer was connected to existing information and communication routines. Two agents were configured: ManuBot, for querying and updating maintenance-history data, and MailBot, for recurrent supplier-email handling. The empirical sequence covered baseline diagnosis, prototype testing and implementation-stage evaluation, drawing on workflow observations, user feedback, task comparisons and records from the implemented tools. The clearest measured changes were task-specific. MailBot reduced supplier-email preparation from about 12-15 min to 2-3 min per message. ManuBot reduced maintenance-data retrieval and querying time by approximately 50%. Users also reported easier access to historical malfunction records, better visibility of recurrent events, and more structured email routines. The case remained constrained by incomplete ERP (Enterprise Resources Planning) integration, data-structure quality, platform permissions and differences in user readiness. The evidence points to a task-specific use of low-code conversational AI: gains were observed when the agents were tied to specific records, supplier-email workflows and human validation points.
Paulo Peças, Diogo Pires, Diogo Jorge· International journal of mat...· 0 citations
Enterprise customer relationship management (CRM) systems function as decision-support environments where AI-generated recommendations can directly affect customer rights, refund eligibility, service level commitments, and privacy-sensitive decisions. Existing generative AI approaches for CRM, including standard retrieval-augmented generation (RAG), lack dedicated mechanisms for policy validation, risk-aware escalation, and decision auditability. This study proposes the Policy-Aligned Agentic Retrieval-Augmented Generation (PAL-CRM-RAG) framework for risk-aware intelligent decision support in enterprise CRM. The framework integrates ten processing layers encompassing CRM query understanding, risk-aware intent classification, hybrid BM25 and dense vector retrieval, reciprocal rank fusion, cross-encoder reranking, policy validation, agentic control, evidence-grounded generation, self-verification, and human escalation with audit logging. Using a synthetic CRM ticket corpus of 20,000 records across five issue categories and a constructed policy-knowledge corpus of 30 documents spanning six policy groups, PAL-CRM-RAG is evaluated against six baseline systems across retrieval quality, generation faithfulness, policy compliance, unsafe response rate, escalation accuracy, and response latency. Risk classification across three classes (Low, Medium, High) achieves an accuracy of 95.1% and a macro-F1 of 94.4%. PAL-CRM-RAG achieves a policy compliance rate of 100% and an escalation F1 of 1.000, compared with 0% compliance for the non-RAG baseline, and 90% compliance for the strongest retrieval-only baseline. An ablation study confirms that each architectural module contributes measurably, with removal of the policy validator reducing compliance to zero and removal of the risk classifier eliminating all escalation capability. These results demonstrate that policy alignment and risk-aware escalation can be operationalised within an Agentic RAG pipeline for enterprise CRM, advancing intelligent decision support theory and the practical governance of generative AI in customer service information systems.
Chitrapradha Ganesan· Journal of Intelligent Decis...· 0 citations
ASTRA, an agentic system for ticket resolution in which a central orchestrator coordinates three specialist information-gathering agents and drives a judge-orchestrator refinement loop to produce evidence-backed troubleshooting reports, is proposed.
Shashidhar Reddy Javaji, Mohamed Trabelsi, Jin Cao et al.· 0 citations
Comparative experiments conducted on four checkpoints from a single model family validate a concise and effective scaling trend: end-to-end detection correctness exhibits a complexity-differentiated scaling pattern, and the complexity metric derived from the constraint dependency graph can effectively quantify instance difficulty and the performance improvement potential of models.
Shuai Zhao, Fengmei Ni, Lichao Ma et al.· 0 citations
Results show that AGR achieves strong decision accuracy in triage and evidence verification, and produces more actionable and engineering-useful issue specifications than both raw feedback and a strong LLM baseline, while reducing unsupported details.
Zhiyao Wang, Jialong Li, Xiujing Guo et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
