This study presents a novel, manually annotated dataset of 121 business news articles related to five major steel companies, using the Cambridge Risk Taxonomy, and provides empirical evidence on the opportunities and limitations of LLMs for both analytical and narrative forms of automated risk assessment.
Abstract
The increasing complexity of global e-commerce supply chains underscores the need for automated, context-aware risk monitoring systems capable of interpreting large volumes of unstructured information. Although Large Language Models (LLMs) have shown strong performance across natural language processing tasks, their application to real-world supply chain risk detection remains limited. This study presents a novel, manually annotated dataset of 121 business news articles related to five major steel companies, using the Cambridge Risk Taxonomy. Leveraging this dataset, we evaluate two state-of-the-art LLMs in a multi-label risk classification task using few-shot prompting. The results demonstrate that LLMs can approximate human annotation, though challenges persist in detecting domain-specific risks such as Geopolitical threats and in avoiding label overgeneration. Beyond classification, we further assess the capacity of LLMs to generate managerial risk summaries. We show that summaries derived from model-predicted risks exhibit strong semantic alignment to summaries generated from human annotations, highlighting the potential of LLMs to support executive-level risk interpretation. Overall, this study contributes the first publicly available dataset of fine-grained, hierarchical risk annotations in an e-commerce supply chain context and provides empirical evidence on the opportunities and limitations of LLMs for both analytical and narrative forms of automated risk assessment.
In live-streaming e-commerce systems, massive volumes of user-generated danmaku constitute a critical yet underutilized source of business intelligence. However, transforming such unstructured, noisy, and highly context-dependent textual data into structured and actionable knowledge remains a fundamental challenge for enterprise information systems. To address this issue, this study proposes TaSC-LLM, an LLM-enabled topic recognition method for constructing interpretable topic measurements from unstructured user-generated content. The proposed framework integrates topic taxonomy construction and zero-shot classification into a unified semantic reasoning pipeline. Unlike conventional topic modeling or supervised classification approaches, TaSC-LLM leverages chain-of-thought reasoning, multi-stage taxonomy induction, sliding window context modeling, and self-consistency verification to eliminate reliance on predefined label spaces and annotated training data. This design allows the system to dynamically construct and update topic taxonomies while ensuring interpretability, robustness, and cross-scenario adaptability. Empirical evaluation on three large-scale live-streaming e-commerce danmaku datasets shows that TaSC-LLM achieves strong taxonomy coverage, classification accuracy, and agreement with expert annotations. The findings suggest that LLM-based reasoning can help convert unstructured user-generated text into interpretable topic measures for downstream empirical and managerial analysis. While the present evaluation is conducted offline, TaSC-LLM provides a methodological foundation for future business applications that can be further examined under multi-session, multi-platform, and deployment-oriented conditions.
Ensuring construction safety requires timely identification of latent risks embedded within unstructured documents such as inspection logs, incident reports, and supervisor notes. Traditional rule-based or statistical methods often struggle to extract such knowledge due to linguistic ambiguity, domain-specific expressions, and lack of contextual understanding. With the advent of large language models (LLMs), prompt engineering and fine-tuning have shown promise for structured information extraction, yet existing methods often treat them in isolation and fall short under low-resource or ambiguous conditions. In this work, we propose a hybrid framework that integrates prompt-guided attention with lightweight supervised fine-tuning to extract structured risk triples—comprising entities, attributes, and contexts—from heterogeneous construction texts. Our model first employs task-specific prompts to guide a pre-trained LLM towards relevant semantic spans, then applies a dual-objective training strategy combining masked language modeling and structured tuple loss for domain adaptation. We conduct extensive experiments on three datasets, including a real-world log corpus (ConRisk-Log), annotated incident reports (BuildSafe-Report), and a synthetically perturbed dataset (SynthRiskSim) designed to stresstest linguistic robustness. Results show that our method outperforms strong baselines such as BERT-NER, GPT-ZeroShot, and T5-FullFinetune by up to 9.7% in F1 score and maintains high extraction coverage and stability across diverse risk expressions. Ablation studies further demonstrate the complementary value of prompt design and fine-tuning. This work provides a scalable, accurate, and interpretable solution for real-time construction risk mining and contributes to safer engineering practices through advanced NLP techniques.
Jin-Fei Liu, Qun Luo, Dou-Dou Li et al.· International Conference on...· 0 citations
This work introduces RegNLI, a novel framework that formulates misbranding detection as a inference task between product claims and regulatory provisions, and builds a foundation for compliance-aware NLP systems and opens new directions for integrating formal reasoning with neural architectures in regulatory domains.
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.
Sil Hamilton, Albert Yu Sun, Oscar Romero et al.· 0 citations
This work introduces WildSEEK, a manually annotated dataset of 3k information-seeking queries from real user interactions, and an evaluation framework for LLM-generated responses, and finds that over a third of information-seeking queries are high-risk and more often analytical.
Tanise Ceron, Joachim Baumann, Elisa Bassignana et al.· 0 citations
This research focuses on enterprise profiling in scenarios where large volumes of diverse texts—such as registration records, annual reports, news articles, and bidding notices—are continuously generated. Instead of relying solely on a single data representation or classification model, we developed a comprehensive natural language processing (NLP) pipeline for extracting key information and identifying industries. The pipeline consists of several steps. First, we use a BERT-BiLSTM-CRF model to identify core e/nterprise entities. Then, we combine TF-IDF with BERT embeddings to create a hybrid feature scheme that captures both lexical cues and contextual semantics. To address the challenge of imbalanced industry labels, we apply SMOTE in the dense semantic space and pair it with Focal Loss to enhance learning for minority classes. Additionally, we introduce a Stacking strategy to integrate outputs from different models, making predictions more stable. Tests on a self-compiled dataset covering ten national economic sectors and about 50,000 enterprises show that our method achieves a macro-F1 score of 95.4%. It outperforms traditional machine learning baselines and single deep learning models, offering more reliable recognition for minority classes. These results suggest that our framework is well-suited for applications such as supply chain partner discovery, industrial mapping, and targeted investment promotion.
Xin-Yi Xu· Applied and Computational En...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
