Multilingual Speech-to-Text and Automated Meeting Summarization Using NLP and Machine Learning

Abstract

Manually preparing minutes of a meeting is time-consuming and error-prone, especially in the multilingual case, such as that in India, where the same meeting can use Tamil, Telugu, Hindi, Kannada and English languages. In this paper we propose a pipeline to automate the preparation of meeting minutes by utilizing multilingual automatic speech recognition (Whisper Large-v3), GPU-based speaker diarization (Pyannote 3.1), large language model summarization and sentiment analysis (LLaMA-3.3-70B), two stage rule plus LLM based action item extractor, and retrieval augmented generation (RAG) based on FAISS for meeting question answering. In contrast to previous work which addressed transcription and summarization separately, our system integrates all stages through a single Flask/Angular web application and provides engineering metrics such as accuracy scores, latencies, GPU usage and costs which were mostly missing from earlier work on multilingual meeting summarization. On an evaluation with three recordings, our system delivers a WER of 5.34%, diarization accuracy of 91.3%, summarization BERTScore-F1 of 0.891, action item F1 of 0.877, RAG factual accuracy of 84.7% while taking less than 90 seconds for a 30 minute meeting on a single NVIDIA T4 GPU.