Skip to content

A JEPA Recipe for Tabular Foundation Models

Sep 2026 · 0 citations · 33 references
Computer Science

TL;DR

A recipe under which the latent term of a joint-embedding predictive architecture (JEPA) collapsed in earlier runs and took the encoder with it to a constant map is reported, under which the value head reads the encoder field rather than the predictor, and the target is an exponential moving average difference.

Abstract

Tabular foundation models learn to predict cell values in context, whereas world-model self-supervision asks for prediction in representation space (LeCun, 2022; Assran et al., 2023). On a tabular foundation-model prior, the latent term of a joint-embedding predictive architecture (JEPA) collapsed in our earlier runs and took the encoder with it to a constant map. We report a recipe under which the latent term survives to convergence beside the value objective: the value head reads the encoder field rather than the predictor, and the target is an exponential moving average (EMA) difference. To bound its cost against the value-only arm, both arms train until a plateau rule stops them, with no fixed step budget. A fixed horizon had confounded a slowdown with a ceiling, since the value-only arm was still improving well past the usual budget. At convergence, in one run per arm, the JEPA arm trails the value-only arm across 147 real datasets, 32:70 wins to losses on classification (29:63 with one entry per dataset name) and 8:24 on regression, the margin small on classification and wider on regression, and the count leans the same way in each stratum and each benchmark. The JEPA arm (jepa) needs 1.42 times as many steps as the value-only arm (ds), and 1.66 times its wall-clock, to reach its plateau.

View source

Similar papers

Preprint Oct 2026

FTFM: Factorized Tabular Foundation Models

Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across...

Josif Grabocka · 0 citations
#artificial intelligence Preprint Oct 2026

Recurrent In-Context Learning for Tabular Foundation Models

Tabular foundation models adapt to new datasets through in-context learning (ICL), and, like most transformers, they tie computational depth to the number of independently parameterized layers. We ask whether the ICL stage of such a model needs a distinct set of parameters for every layer, or only depth. We introduce T...

Tejas Vaidhya, Gaurav Sahu, Irina Rish · 0 citations
#artificial intelligence Preprint Oct 2026

TICDA: Tabular In-Context Data Attribution

Tabular foundation models (TFMs) achieve strong predictive performance by conditioning on labeled demonstrations provided in context, without any parameter update. Yet how individual demonstrations shape a given prediction remains poorly understood. This gap matters in practice: the context is often assembled from what...

Yacine Benihaddadene, Milan Bhan, Eliot Dugelay et al. · 0 citations
#machine learning Preprint Sep 2026

Tabby: An Open Pretraining Recipe for Time Series Foundation Models

Tabby, a long context probabilistic time series foundation model, is released together with a complete and open recipe of how it was built, which achieves competitive zero-shot forecasting performance on GIFT-Eval and the out-of-distribution TIME benchmark.

Shi-Feng Xie, Bahaeddine Abdessalem, Ze-Hao Xiao et al. · 1 citation
#machine learning Preprint Sep 2026

Adaptive Latent Capacity for World Models

Adaptive LeWorldModel (ALeWM), a world model based on a joint-embedding predictive architecture (JEPA) that learns to concentrate predictive information in compact prefixes of a wide latent representation, consistently achieves higher mean success rates than tuned fixed-width LeWM, with lower planning capacity on avera...

Idan Achituve, Lior Dikstein, Idit Diamant et al. · 0 citations
#machine learning Preprint Oct 2026

One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models

Latent world models are routinely selected, checkpointed and ablated on one-step prediction loss, on the assumption that a model which predicts the next latent better is a better model. In model-based RL this assumption is known to be unreliable; we ask whether it also fails for latent world models, where the training...

Joshua Cheung, Zhang He · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.