InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
TL;DR
By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis and generalizes effectively to complex, real-world scenarios.
Abstract
Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas, a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving $100\times$ speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis.
![Director of Machine Learning Insights [Part 4]](/uploads/articles/4d4046d553c83e57a3aa9af0665a2d2d482ad7c9.png)