Speaker
Abstract
Despite the remarkable empirical success of deep learning, its theoretical foundations remain incomplete. Even for the ostensibly simple setting of two-layer ReLU networks, a rigorous understanding of the optimization dynamics—and of why these dynamics lead to good generalisation—continues to be a central challenge in the theory of deep learning.
A full characterisation of the training dynamics for this architecture is still out of reach for arbitrary data distributions. Nevertheless, recent work has made progress under specific structural assumptions. Boursier et al. (NeurIPS 2022) showed that, when the training inputs are orthogonal, a network with a sufficiently large width m (specifically m ≳ 2n, where n denotes the number of training samples) converges, as the initialization scale tends to zero, to the minimum-norm interpolating solution.
In a milder over-parameterised regime (m ≳ log(n)), Dana et al. (NeurIPS 2025) proved that, regardless of the initialization scale, gradient descent drives the model to interpolate the training data. Their analysis, however, does not identify the implicit bias of the dynamics in this regime; which is conjectured to favour sparse interpolators when starting from small initializations.
Crucially, Boursier et al. (TMLR 2026) demonstrated that attaining the sparsest interpolator—a network that can be represented with just two neurons—requires the large over-parameterisation regime (m ≳ 2n). These works thus lead to the following natural question: what level of sparsity is enforced by small initializations in the mildly over-parameterised regimes?
Our contribution (NeurIPS 2026 Spotlight) answers this question. Precisely, we demonstrate that, even with intermediate over parameterisation (m ≳ log(n)), the training dynamics converge—again as the initialization scale vanishes—to a sparse interpolating solution. More precisely, we show that the optimisation process implements an incremental learning scheme: well-chosen non-zero neurons are added sequentially to the network until exact interpolation is achieved. This is the first work to rigorously establish such an incremental learning dynamics for ReLU networks. Moreover, our results illuminate a double-descent phenomenon: increasing the number of trainable parameters leads to progressively sparser solutions at convergence.
Joint work with: James Town, Etienne Boursier, and Matthias Englert
Bio
Ranko is a full professor in the Department of Computer Science at the University of Warwick, where he has served on the University Senate and as the Head of Department, and is the Deputy Head of the Theory and Foundations research theme. Ranko has been an Area Editor in Information Processing Letters, on Programme Committees of leading international conferences in theoretical computer science including ICALP and LICS, and an Area Chair of ICLR and NeurIPS.