Sparse matrix-dense matrix multiplication (SpMM) is a core operation in scientific computing and deep learning. On multi-GPU platforms, its scalability is limited by communication bottlenecks. To address this, we propose DistSpMM, a co-design framework integrating data layout, pipelining, and communication strategies. DistSpMM introduces HSDMA, a lightweight algorithm that reduces communication by optimizing the dense matrix allocation. DistSpMM features a topology-aware two-stage pipeline that manages the IB/NVLink bandwidth disparity to maximize the overlap of computation and communication. Finally, DistSpMM employs an adaptive selector that uses a performance model to dynamically choose the optimal communication granularity (coarse vs. fine-grained) based on data sparsity and network tier. Experiments on diverse real-world datasets demonstrate the superior performance of our method. It achieves average speedups of 1.6 × to 2.6 × in single-node multi-GPU environments and 4.0 × to 5.1 × in multi-node multi-GPU environments over the baseline. Compared to the best state-of-the-art implementations, our method delivers up to 2.0 × speedup.
Junyu Gu, Jue Wang, Zhikuang Xin et al.· ACM Transactions on Architec...· 0 citations
Large language model (LLM)-based approaches increasingly use molecular strings such as SMILES and SELFIES for molecular generation and property prediction. However, the statistical properties of molecular token distributions have not been systematically characterized. Here, we analyzed rank–frequency distributions of tokens derived by byte-pair encoding (BPE) across large molecular databases. BPE-derived tokens showed approximate Zipf-like rank–frequency scaling across the examined representations and chemical spaces, with fitted slopes moderately steeper than the canonical value of −1, paralleling a statistical pattern widely observed in natural language. Moreover, when BERT models were pretrained using BPE vocabularies with different rank–frequency slopes, the closeness of these slopes to the ideal Zipf value of −1 strongly correlated with performance on molecular property prediction tasks (Pearson r = 0.91, p < 0.001) and remained associated after adjustment for vocabulary size (partial r = 0.86, p < 0.001). Together, these findings show that molecular BPE vocabularies exhibit an approximate Zipf-like rank–frequency regularity and that slope closeness provides an empirical diagnostic for comparing vocabulary sizes within the examined SMILES/SELFIES BPE framework.
Anyu Liu, Chao Fang, Yuntao Li et al.· Journal of Physical Chemistr...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
