Conventional wisdom attributes the mysterious generalization abilities of overparameterized neural networks to gradient descent (and its variants). The recent volume hypothesis challenges this view: it posits that these generalization abilities persist even when gradient descent is replaced by Guess & Check (G&C), i.e., by drawing weight settings until one that fits the training data is found. The validity of the volume hypothesis for wide and deep neural networks remains an open question. In this talk, I will theoretically investigate this question for matrix factorization---a common testbed in neural network theory. I will first show that generalization under G&C deteriorates with increasing width, establishing what is, to my knowledge, the first case where G&C is provably inferior to gradient descent. Conversely, I will show that generalization under G&C improves with increasing depth, revealing a stark contrast between wide and deep networks, which is validated empirically. These findings suggest that even in simple settings, there may not be a simple answer to the question of whether neural networks need gradient descent to generalize well.

I will discuss some connections between random matrix theory and machine learning, focusing on the spectrum of the hessian of the loss surface and touching on issues relating to neural collapse.

I will present recent advances in the asymptotic analysis of empirical risk minimization in overparameterized neural networks trained on synthetic data, focusing on two key extensions of classical models: depth and nonlinearity. The aim is to understand what different architectures can learn and to characterize their inductive biases. In the first part, I derive sharp asymptotic formulas for two-layer networks with quadratic activations and show that they exhibit a bias toward sparse representations, such as multi-index models. By mapping the learning dynamics to a convex matrix sensing problem with nuclear norm regularization, we obtain precise generalization thresholds and identify the fundamental limits of recovery. In the second part, I analyze deep architectures trained on hierarchical Gaussian targets and show that depth enables effective dimensionality reduction. This leads to significantly improved sample complexity compared to kernel or shallow methods. A key requirement is that the target function be compositional and robust to noise at each level. These results draw on tools from random matrix theory, convex optimization, and statistical physics, and help delineate the regimes in which deep learning offers provable advantages in high-dimensional settings.

Deep neural networks have become the cornerstone of modern machine learning, yet their multi-layer structure, nonlinearities, and intricate optimization processes pose considerable theoretical challenges. In this talk, I will review recent advances in random matrix analysis that shed new light on these complex ML models. Starting with the foundational case of linear regression, I will demonstrate how the proposed analysis extends naturally to shallow nonlinear and ultimately deep nonlinear network models. I will also discuss practical implications (e.g., compressing and/or designing "equivalent" NN models) that arise from these theoretical insights. The talk is based on a recent review paper https://arxiv.org/abs/2506.13139 joint with Michael W. Mahoney.

We investigate how entrywise application of a non-linear function to symmetric orthogonally invariant random matrix ensembles alters the spectral distribution. We treat also the multivariate case where we apply multivariate functions to entries of several orthogonally invariant matrices; where even correlations between matrices are allowed. We find that in all those cases a Gaussian equivalence principle holds, that is, the asymptotic effect of the non-linear function is the same as taking a linear combination of the involved matrices and an additional independent GOE. The ReLU-function in the case of one matrix and the max-function in the case of two matrices provide illustrative examples. This is joint work with Alexander Wendel.