Jul 2026· Proceedings of the Genetic and Evolutionary Computation Conference Companion· pp. 85-86· 1 citation· 24 references
TL;DR
It is suggested that unifying language model priors and learning string embedding space can overcome traditional barriers in universal BBO, paving the way for generalpurpose BBO algorithms.
Abstract
The pursuit of universal black-box optimization (BBO) algorithms is a longstanding goal. However, unlike domains such as language or vision, where scaling structured data has driven generalization, progress in offline BBO remains hindered by the lack of unified representations for heterogeneous numerical spaces. Thus, existing offline BBO approaches are constrained to single-task and fixed-dimensional settings, failing to achieve cross-domain universal optimization. Recent advances in language models (LMs) offer a promising path forward: their embeddings capture latent relationships in a unifying way, enabling universal optimization across different data types possible. In this paper, we discuss multiple potential approaches, including an end-to-end learning framework in the form of next-token prediction, as well as prioritizing the learning of latent spaces with strong representational capabilities. To validate the effectiveness of these methods, we collect offline BBO tasks and data from open-source academic works for training. Experiments demonstrate the universality and effectiveness of our proposed methods. Our findings suggest that unifying language model priors and learning string embedding space can overcome traditional barriers in universal BBO, paving the way for generalpurpose BBO algorithms. This paper for the Hot-off-the-Press track at GECCO 2026 summarizes the work Tan et al. Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings. In: Proceedings of the 42nd International Conference on Machine Learning (ICML), pp. 58499–58544, Vancouver, Canada, 2025.
Large Language Model (LLM)-driven AutoML agents have shown strong capabilities in constructing end-to-end machine learning pipelines. However, their effectiveness is limited by costly execution-based feedback, which can make the search for high-quality solutions inefficient under restricted computational budgets. We propose PACE (Pre-execution Admission via Code Embeddings), an online-adaptive admission control framework that improves budgeted sample efficiency by estimating candidate utility prior to execution from within-run execution history, without training a separate offline predictor. The core idea of PACE is to leverage latent structure in the solution space as a within-run admission signal. It projects candidate solutions into multi-view semantic embeddings, dynamically organizes executed candidates into clusters, and estimates new candidates by their proximity to historical elite regions. Moreover, PACE aggregates multi-view embeddings via an adaptive reweighting strategy that prioritizes views with higher discriminative power. This enables PACE to bias the agent toward high-potential regions under a limited computational budget while retaining exploration when local structure is weak. We demonstrate that PACE operates as a plug-and-play admission layer for AutoML agents, serving as either an execution gate or a search prior. In the tested settings, it improves the density of elite solutions found within fixed budgets without modifying the underlying agent architecture. Code, configurations, and prompt templates are publicly available at https://github.com/fendss/PACE.
Gangyi Zhao, Hebin Liang, Hongyao Tang et al.· Proceedings of the 32nd ACM...· 0 citations
This work advocates for Joint Online-Offline Fine-Tuning as a superior paradigm that breaks the convention of restricting offline data to SFT and online data to RFT, and provides the first comprehensive survey focusing specifically on the synchronization of data provenance.
Taihang Zhen, Guang Yang, Chenzhang Li et al.· 0 citations
TASSO, a new paradigm that efficiently preserves the latent space geometry while ensuring network plasticity, is introduced with two complementary techniques: subspace learning and geometry-aware knowledge distillation.
Changming Sun, Francesco Barbato, Matteo Caligiuri et al.· 0 citations
A practitioner's study of how to make distillation training efficient is presented, organised around two systems contributions, and a fused, chunked KL loss is introduced, making peak memory linear in the sequence length.
Bakbergen Ryskulov, Iker García-Ferrero, David Montero et al.· 0 citations
Sequential recommendation aims to predict users’ next items of interest based on their historical interactions. Recently, Large Language Models (LLMs) have shown strong potential in this field due to their powerful semantic understanding. However, existing methods face two core challenges: First, LLMs may generate non-existent recommendations due to hallucinations; Second, when utilizing LLM-generated embeddings for transfer to downstream recommendation tasks in different domains, the dimensional and structural differences across spaces exacerbate space misalignment. To address these challenges, we propose UniRec, a unified space learning framework that achieves deep alignment through a hierarchical space transformation path: language space → collaborative space → real space. This framework employs a two-stage learning mechanism: The first stage guides LLMs to align semantic and collaborative spaces through collaborative instruction fine-tuning, introduces a real-space constrained generation mechanism to reduce hallucination problems, and is optimized for large-scale applications; The second stage extracts core semantic principal components through singular value decomposition and freezes the representations, combines with gating networks to adaptively fuse semantic and domain-specific collaborative signals, achieving efficient cross-domain transfer. Experiments show that UniRec achieves over 27% average improvement on in-domain datasets and 24% on out-of-domain datasets, while maintaining low memory overhead.
Jiarui Yang, Junmei Feng, Yihan Zhang et al.· ACM Transactions on Informat...· 0 citations
This work argues that cross-modal alignment is implicitly captured in the information-compression trajectory, and proposes LLaVAFlow, an information-theoretic distillation framework that preserves alignment flow and enhances both downstream performance and generalization.
Muyao Yuan, Muyan Jiao, Jiangyong Ying et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
