Learning from Prior Experiments: Meta-learning Models of Workflow Performance
TL;DR
This study shows that accurate performance prediction for classical machine learning workflows can be achieved through meta-learning using readily available OpenML meta-data, and indicates that careful selection of regression models is more critical than increased representational complexity.
Abstract
Evaluating the performance of machine learning workflows is a major computational bottleneck in automated machine learning (AutoML), particularly for complex pipelines involving preprocessing, model selection, and hyperparameter optimization. This work aims to develop an efficient performance prediction framework that estimates the expected accuracy of candidate machine learning workflows on unseen datasets without requiring explicit model training. We formulate performance prediction as a meta-learning regression problem that leverages historical experimental results from the OpenML platform. Machine learning workflows are represented as structured pipelines and encoded using text-based vectorization techniques, including TF-IDF, count-based, and hashing vectorizers, as well as LLM BERT embeddings. These workflow descriptors are combined with dataset-level meta-features capturing basic statistical and structural properties. Several regression models are evaluated as meta-learners, including linear models, decision trees, random forests, Gaussian processes, and gradient-boosted decision trees. The approach is systematically evaluated on the OpenML-CC18 benchmark suite using cross-validation over more than 100,000 workflow-dataset evaluations. The proposed framework achieves strong predictive performance across a wide range of workflows and datasets. In particular, gradient-boosted decision tree regressors combined with standard TF-IDF representations of workflows consistently yield the best results, reaching an average coefficient of determination \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^2$$\end{document} of approximately 0.8 on unseen test data. While transformer-based MiniLM embeddings were evaluated, they did not consistently outperform sparse TF-IDF representations and incurred higher computational cost. Feature ablation studies indicate that restricting vocabulary size degrades performance, while extending representations with bigrams provides only marginal gains at substantially higher computational cost. The results demonstrate robust generalization across heterogeneous workflows and dataset characteristics. This study shows that accurate performance prediction for classical machine learning workflows can be achieved through meta-learning using readily available OpenML meta-data. The proposed approach enables rapid and computationally efficient estimation of workflow performance, making it well suited for accelerating AutoML search and model selection. The results indicate that careful selection of regression models is more critical than increased representational complexity, with simple and scalable workflow encodings yielding the most robust performance. Given its scalability and flexibility, the framework provides a strong foundation for future extensions incorporating richer dataset descriptors, larger meta-datasets, and more expressive embedding and regression models.