This work formalizes joint speech summarization and translation (JSumT), the generation of a succinct, faithful target-language summary directly from a long spoken document in a source language, and establishes a foundation for developing and evaluating multilingual systems capable of jointly interpreting, compressing, and translating long-form speech.
Abstract
As information increasingly traverses linguistic boundaries, users require concise cross-lingual representations of long-form content. Nevertheless, long-document summarization research remains text-centric, whereas multilingual speech research has largely prioritized translation, preserving source content rather than compressing it. We address this methodological gap by formalizing joint speech summarization and translation (JSumT): the generation of a succinct, faithful target-language summary directly from a long spoken document in a source language. We additionally introduce VoxSumm, the first multilingual and cross-lingual benchmark for this task, comprising 10,045 BBC article-summary pairs across 24 languages and encompassing approximately 703 hours of speech data. Our evaluation of representative speech-language models reveals pronounced variation across models and generation settings: Gemini3.1-Pro demonstrates the greatest consistency, summarization into English generally surpasses generation into non-English target languages, and translating an entire document before summarization compounds instruction-following failures. Through the release of VoxSumm, we establish a foundation for developing and evaluating multilingual systems capable of jointly interpreting, compressing, and translating long-form speech.
Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 \times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.
Yexing Du, Kaiyuan Liu, Youcheng Pan et al.· 0 citations
Cross-lingual communication remains constrained by translation tools that fail to preserve sentence-level meaning, par-ticularly for idiomatic phrasing, long or multi-clause sentences, and low-resource languages. This paper presents the design and evaluation of a browser-accessible multilingual translation plat-form built around a Transformer encoder–decoder network. A sin-gle shared model is fine-tuned across five languages – English, Hindi, French, Spanish, and German – using subword vocabu-lary construction, automatic source-language identification, and an attention-based explainability layer that exposes token-level align-ment through heatmap visualization. The system was evaluated on a held-out multilingual test set using BLEU, ROUGE-L, and accuracy metrics, achieving a macro-averaged translation accuracy of 92.7%, exceeding a 90% target and outperforming LSTM, GRU, statistical, and rule-based baselines by a wide margin. Average inference la-tency remained below 185 milliseconds per request, supporting in-teractive use. The results indicate that a moderately sized, shared self-attention architecture can deliver production-quality multilin-gual translation within the resource constraints of an academic de-ployment, while surfacing clear directions – low-resource language coverage, domain adaptation, and speech-based extension – for con-tinued development
Darshan Gowda D H and Dr. Kruti R· International Journal of Adv...· 0 citations
Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an evaluation blind spot concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce MORE, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) Unprecedented Scale: It covers 149 languages, making it the most linguistically diverse benchmark to date; (2) Structural Complexity: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) Data Authenticity: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark's effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.
Long Xu, Binghong Wu, Ting-Ting Yu et al.· 3 citations
ABSTRACT
DocuMind is a fully offline, privacy-preserving, multilingual Document Question Answering system built on the Retrieval-Augmented Generation (RAG) architecture. The system enables users to upload documents in any language and ask questions in their native language — including Kannada, Hindi, Tamil, Telugu, and English
— receiving accurate, contextually grounded answers from a locally running Large Language Model (LLM) with zero cloud dependency and zero recurring cost. The system employs Language-Agnostic BERT Sentence Embeddings (LaBSE), a multilingual model supporting 109 languages, to map all text into a shared 768-dimensional semantic vector space, enabling true cross-lingual retrieval without any translation step. Document embeddings are persisted in a local ChromaDB vector database and answer generation is performed by LLaMA
3.2 (3B parameters) running locally via the Ollama runtime. Evaluation using RAGAS-style metrics across five languages demonstrates Faithfulness scores of 0.77 to 0.87 and average query latency under 8 seconds on a standard consumer laptop with 8 GB RAM.
KEYWORDS Retrieval-Augmented Generation, Large Language Models, Multilingual NLP, LaBSE, ChromaDB, Ollama, LLaMA, Document Question Answering, Indic Languages, Offline AI, Natural Language Processing, Cross-lingual Retrieval.
Swetha M S, Dr. Sripalreddy· International Scientific Jou...· 0 citations
Extending automatic speech recognition (ASR) to low-resource African languages is constrained by the prohibitive demands of data collection at scale. A promising direction is to leverage the linguistic relatedness between a low-resource target language and languages previously seen by a model to reduce the volume of target-language data needed for effective adaptation. Although this approach has proven reliable for text-based models, its effectiveness in the speech domain remains contested. We employ a systematic controlled experimental design spanning six factors, two Africa-centric corpora, and four large ASR models, sequentially adapting on a related auxiliary language followed by the target to isolate whether linguistic relatedness reliably predicts cross-lingual transfer gains across these conditions. In every setting, pre-adaptation on related auxiliary languages yields no practically meaningful improvements once as little as one hour of target-language data is available, suggesting that relatedness alone may not reliably predict transfer gains in large multilingual ASR, or constitute an effective strategy for extending such models to low-resource languages.
A. Florian, C. Amol, Hope Kerubo Ombaba et al.· 0 citations
Telugu ranks among the most widely spoken language in South Asia, yet it remains conspicuously underrepresented in automatic summarization research. This neglect is not arbitrary; it reflects three genuine problems viz., Telugu’s agglutinative nature, syntactic representation, and lack of annotated corpora. To address these problems within a unified framework, this paper introduces MLOA-MA-ASeqNet architecture. The architectural core is MA-ASeqNet, a Multi-scale Attention and Adaptive Sequence-to-Sequence Network whose hierarchical encoder operates simultaneously at word, phrase and sentence level granularity. The optimization component, MLOA, is a Modified Lyrebird Optimization Algorithm that replaces manually configured, English-centric hyperparameter defaults with a principled population-based search. Experiments across three datasets – Telugu News NLP, Telugu Books, and TeSum; show consistent and statistically improvements over Seq2Seq, Transformer, T5 and Gemma baselines on ROUGE scores (ROUGE-L of 0.53). Qualitative analysis of the generated summaries was performed through a blind evaluation by five native Telugu speakers from diverse professional backgrounds. The summaries were evaluated on four parameters: fluency, adequacy, coherence, and readability. The proposed MLOA-MA-ASeqNet achieved the highest average score of 4.60 across fluency, adequacy, coherence and readability; surpassing all four baselines on every dimension. All pairwise differences were statistically significant according to the Wilcoxon signed-rank test with Bonferroni correction (p < 0.05 in all cases).
V. M, Kunal Chakma, Anupam Jamatia et al.· ACM Transactions on Asian an...· 0 citations