Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generat...
Jia-Han Li, Huang-Yuan Su, Sheng-Bang Tong et al.· 0 citations
OP-Mix (On-Policy Mix), a data mixing algorithm that operates across pretraining, midtraining, and instruction tuning, is introduced, suggesting a different view of language model training: not a sequence of distinct phases, but a single continuous process of learning from data.
Michael Y Hu, Apurva Gandhi, Kyunghyun Cho et al.· arXiv.org· 0 citations
The key insight is that fast behavioral fingerprints with occasional replay challenges are sufficient for auditing neural network training, and the method is applicable at scale with minimal overhead to the trainer, is cheap for the verifier, rejects bad training runs with amplifiable probability, and allows for exact...
Hou-Jun Liu, Pratyusha Sharma· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.