This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. These conditions limit the value of optimizations designed for long-context or high-throughput language-model serving. Starting from LMT-60-0.6B, preliminary profiling suggests that vocabulary projection becomes a more important decode-time cost after GGUF quantization reduces the relative cost of Transformer blocks. We replace the original 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, migrate the embedding space, and adapt the model through embedding calibration followed by full supervised fine-tuning. On an OpenSubtitles2024 test set, LocalSubs achieves a 59.2% tie-excluded win rate against Google Translate under GPT-4o pairwise judging. Performance is strongest on short cues and declines as cue length increases. In a separate preliminary Apple M2 Metal profiling run, LocalSubs shows a 1.63x speedup over a 151k-vocabulary baseline. The code is available on https://github.com/aiden1020/localsubs .
The experiments show that the trade-off between inference efficiency and translation quality depends not only on the quantization format, but also on the choice of text chunking strategy, as well as on the choice of text chunking strategy.
Jim Zhao, Sohir Maskey, Koen Oostermeijer et al.· arXiv.org· 0 citations
Chunked ingestion, bounded queues, parallelized perception, temporal reordering, and a sentence-boundary state machine reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 latency from 2.919 to 2.130 seconds (27.03%) over the complete 9,872-example working subset.
Quantization, pruning, and parameter-efficient fine-tuning methods enabled models to match or outperform models with up to 18 times the parameters on SQuAD v2 while reducing storage as well as optimizer overhead and often improved task performance even when full fine-tuning failed.
Dominik Hildebrand, Benjamin Kiefer, Andreas Zell· Artificial Intelligence Revi...· 0 citations
This study presents the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks and three hardware backends, and identifies a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decod...
Guanyu Cai, Ruiming Tian, Lang Yang et al.· 0 citations
The two studies show that closed-loop GPU fine-tuning does not guarantee monotonic mobile gains, especially on harder classification tasks, and that multi-dataset, on-device measurement is needed to stress-test deployment objectives.
Saif U. Din, M. Hussain, R. Timofte et al.· 0 citations
DraftExpert is proposed, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference that improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.
Dengke Han· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
