A JEPA-Inspired Span-Masked Framework for Language Representation Learning: Revisiting Cosine Similarity and VICReg Regularization

Abstract

Joint-Embedding Predictive Architectures (JEPAs) have recently emerged as a promising paradigm for self-supervised representation learning by predicting latent embeddings from partial observations rather than reconstructing raw inputs. Although JEPA has demonstrated considerable success in computer vision and large-scale language models, the design of lightweight JEPA-inspired representation learning frameworks for natural language remains insufficiently understood. This paper proposes a lightweight span-masked JEPA-inspired framework that formulates language representation learning as latent semantic prediction from partially observed contextual inputs. The framework predicts target embeddings generated by a momentum-updated encoder while avoiding direct token reconstruction, thereby emphasizing semantic abstraction over lexical recovery. To investigate the influence of objective function design, cosine similarity and Variance–Invariance–Covariance Regularization (VICReg) are systematically compared under identical architectural and training conditions, with and without auxiliary masked language modeling (MLM) supervision. The proposed framework is evaluated on one controlled dataset and six benchmark datasets, including AG News, SST-2, IMDb, TREC, DBPedia, and Yelp Polarity. Experimental results consistently demonstrate that the cosine-based JEPA objective without auxiliary MLM supervision provides the best balance between representation quality, computational efficiency, and downstream linear probing performance. Statistical significance analysis using the Friedman test and pairwise Wilcoxon signed-rank tests with Holm correction further confirms that the cosine-based objective significantly outperforms the remaining configurations, while no significant difference is observed between VICReg with and without MLM supervision. The findings indicate that latent semantic prediction alone is sufficient for learning robust language representations under the proposed lightweight framework, whereas auxiliary token-level reconstruction does not consistently improve downstream performance despite increasing computational cost. Overall, this work provides reproducible empirical evidence and practical guidance for objective function design in lightweight JEPA-inspired language representation learning, offering an efficient experimental foundation for future self-supervised language models.