Gabriel Mel: Beyond Spectra: Eigenvector Overlaps and Generalization in Machine Learning

Modern machine learning often interprets “loss geometry” through the Hessian spectrum, motivating ideas such as flat versus sharp directions. I will argue that this view is incomplete: learning is inherently a two-loss problem (train and test), so local geometry depends not only on eigenvalues but also on how the corresponding train and test eigendirections align. I will introduce a simple, universal fluctuation law showing how small training perturbations translate into test error, which reframes the usual intuition: flat directions of the training loss are not intrinsically good or bad -- their effect depends on whether they align with sensitive directions of the test loss. I will then turn to ridge regression, which provides a simple tractable model. In this setting, we obtain closed-form expressions that fully explain both domain shift and the phenomenon of multiple descent. The key takeaway is that generalization is controlled by how training variability projects onto test-sensitive directions -- captured precisely by train-test eigenspace alignment -- rather than by Hessian spectra alone. Finally, I will briefly discuss scalable methods for estimating these overlaps in large networks, and an application to a class imbalance in a trained ResNet model. Together, these results suggest a shift in perspective: spectra describe curvature, but overlaps determine how that curvature matters for generalization. This perspective provides both a conceptual and practical framework for analyzing modern learning systems.