Skip to content
Preprint

Interpreting and Evaluating Dynamic-Rate Speech Codec Boundaries

Sep 2026 · 0 citations · 22 references
Computer Science

TL;DR

This work combines boundary interpretation and controlled reconstruction analysis by comparing predicted boundaries with linguistic and acoustic references and finds that boundary meaning depends on the underlying speech representation.

Abstract

Dynamic-frame-rate neural speech codecs replace a uniform frame grid with variable-duration tokens, making boundary placement part of the representation itself. Yet it is unclear what these boundaries encode and whether interpretable boundaries are also useful for neural speech reconstruction. This work combines boundary interpretation and controlled reconstruction analysis by comparing predicted boundaries with linguistic and acoustic references. We find that boundary meaning depends on the underlying speech representation. For the ASR-oriented SenseVoice and Whisper encoders, shallow layers emphasize phonetic, voicing, and acoustic transitions, whereas deeper layers shift toward syllable and subword structure. In a comparison of six dynamic-frame-rate algorithms and a uniform (fixed-frame-rate) baseline, higher-level linguistic alignment is associated with lower pooling distortion and better reconstruction from semantic tokens. Frame-rate-matched boundary tests make the distinction concrete: a syllable-derived partition improves over both Uniform and Similarity, while a phoneme-derived partition does not improve over Uniform. We infer that for semantically rich speech representations, useful codec boundaries are best understood as allocation decisions organized around the syllable scale.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs

LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries, is proposed.

Thanapat Trachu, Samuele Cornell, William Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

What Survives the Codec Shift: Pooled No-Vocals Residuals for Speech Deepfake Detection

The transition from vocoder-based to neural-codec speech synthesis makes generalization more difficult for speech deepfake detectors, particularly those relying on speech-oriented representations. It remains unclear which acoustic representations retain discriminative information when the generation mechanism changes....

Jia-Jun Xu, Meng-Lu Li, Xiao-Ping Zhang · 0 citations
Preprint Oct 2026

Acoustic gap placement in second-language read speech production

Speakers can differ not only in how much silence they produce but also in where interruptions fall, information that global pause counts obscure. We analyzed 115 publicly available Speech Accent Archive recordings of the same read passage: 57 speakers whose archive metadata listed Mandarin as their first language and a...

Peyman Jahanbin · 0 citations

A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition

Automatic speech recognition is typically trained assuming that the reference transcript is the only valid labeling of an utterance, yet even nominally verbatim transcripts contain localized differences in pronunciation, spelling, or lexical realization that the acoustics do not uniquely determine. Omni-temporal Classi...

Saurabh Kumar, Diptiman Mohanta, P. Ghosh · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.