Transformer-Based Topic Modeling and Sentiment Analysis for Multilingual Text Data
TL;DR
An integrated framework based on transformer architecture for topic modeling and sentiment analysis for Hindi and Italian social-media discourse, customer reviews and news corpus is introduced and it is suggested that there is clear benefit for morphologically complex text and mixed script text for using contextual embeddings and language-specific pretraining.
Abstract
Several new trends related to the rising volume of multilingual digital content have spurred a greater need for Natural Language Processing (NLP) systems capable of capturing the latent themes and understanding sentiment without stripping out the structure within a specific language. In this study, we introduce an integrated framework based on transformer architecture for topic modeling and sentiment analysis for Hindi and Italian social-media discourse, customer reviews and news corpus. The framework integrates language-aware representations in context, the sentiment classification system of supervised learning and the BERTopic-based topic discovery system. Hindi Processing focuses on Devanagari normalization, transliteration, mixed code and Hindi-English while Italian Processing focuses on inflectional, clitic morphology, colloquial and regionalization variation. In addition to classical baselines (Support Vector Machines (SVM), Long Short-Term Memory (LSTM) networks, and Latent Dirichlet Allocation (LDA)), MuRIL, UmBERTo, AlBERTo and LLaMAntino are studied. The performance is assessed descriptively in terms of accuracy, F1 Score, topic coherence (Cᵥ), and training behavior and out of distribution error analysis. On sentiment classification using Hindi, MuRIL achieved an accuracy of 0.89 and an F1-score of 0.88 which were better than the best reported sentiment classification in classical Hindi by 0.08 on both measures. UmBERTo's results for Italian were 0.90 accuracy and 0.89 F1-score, while LLaMAntino obtained the best reported sentiment scores of 0.91 and 0.90. For Hindi, the relative improvement over LDA was 53.7% and for Italian, it was 50.0% with respect to the topic coherence score measured by BERTopic. The findings suggest that there is clear benefit for morphologically complex text and mixed script text for using contextual embeddings and language-specific pretraining. However, there are still some limitations such as dialect-sensitive mistakes, spurious correlations, computational cost and low interpretability. Finally, given that error and fairness audits are not widely available, the study offers a more comprehensive grounding in error and fairness for the responsible deployment of multilingual topic and sentiment systems.