The results suggest that persistent homology offers a useful quantitative lens on the compression--reconstruction tradeoff in PCNs, and that both model capacity and the recurrent, bidirectional dynamics of predictive coding inference shape when this tradeoff is resolved across layers.
Abstract
We study the topology of learned representations in predictive coding networks (PCNs), a neuro-inspired bidirectional architecture, using a quantitative layer-wise persistent homology analysis. We train well-performing PCNs on a synthetic classification dataset ($\geq 99.9\%$ test accuracy) and on MNIST ($\geq 95\%$ test accuracy), and measure how topological features change across layers for different architectures and activation functions. We find that smaller PCNs collapse connected components across layers earlier than larger models (Spearman $\unicode{x1D70C} \in [0.72, 0.79]$ across activations), with model size measured as the sum of hidden-layer widths. We also observe a strong negative correlation ($\unicode{x1D70C} = -0.58$) between the depth at which simplification occurs and reconstruction error; i.e., architectures that simplify later reconstruct better. Finally, a seed-level bootstrap comparison across architectures and activations shows that PCNs consistently collapse connected components later than matched MLPs, with an average difference of $3.6$ layers. These results suggest that persistent homology offers a useful quantitative lens on the compression--reconstruction tradeoff in PCNs, and that both model capacity and the recurrent, bidirectional dynamics of predictive coding inference shape when this tradeoff is resolved across layers.
Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8\% exceed (NLS>0.1) and 0.4\% are inactive. Pruning the lowest-activity 20--25\% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1\%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.
Graph Circuit Learning is introduced, a supervised, amortized framework that trains a GNN across multiple model--task pairs and applies it to unseen cases and preliminary results suggest that graph machine learning offers a natural and potentially powerful perspective on circuit localization.
Chester Tan, Moritz Lampert, Courtney Maynard et al.· 0 citations
Local learning methods offer an alternative to end-to-end backpropagation, but their unstructured local objectives can produce weak feature learning in deep networks. We study whether the order of hidden-state optimization can address this limitation. We propose a boundary-first inference schedule that partitions a model into chunks, first coordinates hidden states at chunk boundaries, and then refines representations within each chunk. We instantiate this schedule in predictive coding networks (PCNs), a local-learning framework in which hidden activities and prediction errors are explicitly exposed during inference. On CIFAR-10, the resulting boundary-first predictive-coding instantiation improves accuracy over standard predictive coding by $9.77\%$ under a standard parametrization and by $5.51\%$ under a $\mu$-parametrization. Diagnostic analyses further show more non-trivial early-layer updates, lower initial-to-final CKA, and more diverse layerwise gradients, consistent with stronger feature learning. These results support boundary-first, chunk-based inference as a practical design principle for predictive-coding training and motivate its study in broader local-learning systems.
Graph Neural Networks exhibit a puzzling numerical fragility under mixed-precision training, failing significantly more often than MLPs or CNNs. This failure is inherently tied to graph structure, with heterophilic graphs and high-degree nodes being particularly vulnerable. We identify the root cause as catastrophic cancellation during neighborhood aggregation. When neighboring node embeddings point in opposite directions, their sum collapses toward zero and amplifies floating-point errors by orders of magnitude. We formalize this through the cancellation ratio ?, proving that it is fundamentally governed by graph topology, including heterophily, node degree, and network depth. Consequently, we propose Aggregation-Aware Representation Learning (AARL) to learn numerically stable and cancellation-resistant representations without sacrificing expressiveness. Unlike naive approaches that enforce neighbor alignment and destroy discriminative power, AARL maintains representation diversity while ensuring numerically safe aggregation. Experiments on diverse benchmarks demonstrate that AARL substantially improves training stability under low precision while preserving or improving classification accuracy.
Jiawei Gu, Ziyue Qiao· Proceedings of the 32nd ACM...· 0 citations
This paper initiates a systematic approach to handling mathematical data structured as (truncated) infinite $q-series, or equivalently, infinite series of integers, and demonstrates that neural networks can reliably extract essential topological information, such as homology class and underlying graph structure, directly from the $q$-series coefficients.
Brandon Robinson, Shimal Harichurn, Fabian Ruehle et al.· 0 citations
This work uses a recently introduced neuro-symbolic framework, to embed a standard GNN into a Relational Bayesian Network (RBN), enabling maximum a-posteriori (MAP) inference that jointly optimises node labels to match the observed local homophily structure of a graph.
Raffaele Pojer, Andrea Passerini, Kim G. Larsen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
