KronSAE is proposed, a design that factorizes the latent space into heads and forms post-latent features as pairwise compositions of lower-dimensional pre-latents using mAND, a differentiable AND-like interaction that imposes a compositional co-activation prior while remaining compatible with standard SAE objectives and variants.
Vadim Kurochkin, Yaroslav Aksenov, Daniil Laptev et al.· 1 citation
Layer-Integrated Memory (LIMe) is introduced, a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from previous layers to improve perplexity per FLOP and yield strong gains on synthetic tasks while preserving higher value-vector entropy and token separability.
Large language models decode by projecting hidden states through a large vocabulary head at every step. This operation is computationally costly and forces all reasoning to be expressed in discrete tokens. We introduce Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized. Experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought. Our method achieves the highest pass@32 among all soft-thinking approaches, demonstrating that effective reasoning can be carried out in continuous space without discrete token generation.
N. Koriagin, Yaroslav Aksenov, George Bredis et al.· 0 citations
Qantara is presented, an end-to-end JEPA whose joint training objective pairs a Brownian-bridge interpolant between consecutive clean latents on the state axis with noise-to-data flow matching on the action axis, and these results move JEPA world models from single-paradigm planners to multi-paradigm controllers.
Ruslan Rakhimov, George Bredis, Yuriy Maksyuta et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
