An Integrated STT-Based Multimodal Pre-Screening System for Korean Oral Presentations with an Analytics-Fitness Benchmark of Five Speech-to-Text Engines

Abstract

Oral presentations for investor relations, government funding, and technical review are typically prepared without objective, repeatable feedback on delivery. We present an integrated web-based pre-screening system for Korean oral presentations that combines automatic speech recognition (ASR), prosodic and disfluency analytics, presentation–material matching, and large language model (LLM)-based qualitative evaluation into a single automated pipeline. To ground the pipeline in a reliable transcription engine, we benchmark five commercial and open-source speech-to-text (STT) engines (Naver CLOVA Speech, Azure Speech, OpenAI Whisper, Google Speech-to-Text v2, and AssemblyAI) on a Korean presentation-domain evaluation set using character error rate (CER), timestamp quality, filler-word preservation, and speaker diarization. On 20 spontaneous Korean speech recordings with corrected reference normalization, CLOVA achieved the lowest mean CER (4.65%, SD 3.75), but CER differences among the five engines were not statistically significant (Friedman p = 0.219, Kendall’s W = 0.07). In contrast, filler preservation separated the engines into two sharply distinct clusters: preserving engines (CLOVA, Azure, Google: 84–93%) versus deleting engines (Whisper, AssemblyAI: 2–19%), with a large and highly significant effect (Friedman p < 10−6, Kendall’s W = 0.87). A complementary benchmark on 20 scripted professional voice-actor recordings reverses the CER ranking (Whisper best at 2.8%), confirming that engine rankings do not transfer across speech domains. CLOVA was adopted as the system’s default engine on the basis of this fitness profile: competitive CER combined with high filler preservation, fine-grained timestamps, and native diarization, rather than transcription accuracy alone. The deployed system converts an uploaded or live-recorded presentation into a sentence-level timeline; computes speaking rate, silence, filler-word, and per-speaker statistics; measures keyword coverage against uploaded slides; and generates prioritized improvement points and anticipated reviewer questions via an LLM. Three independent runs of the deployed system on the same recording produced identical deterministic outputs, including under an outage of the LLM service. A three-rater expert evaluation of nine generated reports rated evidence accuracy and anticipated-question quality highest and specificity lowest (grand mean 3.88/5), with low inter-rater agreement reflecting the small panel. The results provide a reproducible, domain-specific Korean ASR benchmark and a practical reference architecture for automated presentation assessment.