Characterise the implicit bias of gradient descent for non-linear networks
For linear models the answer is a clean minimum-norm statement. For anything deeper, there is no general theory.
Problem statement
Gradient descent on separable data with a linear model and exponential-type loss converges in direction to the maximum-margin separator. For homogeneous networks, gradient flow converges to a KKT point of a margin-maximisation problem, but this does not identify which one.
Give a characterisation of the solution selected by gradient descent — as a function of the architecture, initialisation scale, and step size — for a non-linear network class where the answer is not already determined by a kernel approximation.
Lean 4
No Lean statement or proof is attached yet. Have a formalization? Share its source to add it to this problem.
Background
The question is why overparameterised networks generalise when the training objective has many global minima with very different test error. The linear and kernel regimes are understood; the feature-learning regime, which is where the empirical gains come from, is not.
Every generalisation bound for deep networks is currently either vacuous at realistic scale or applies to a regime that is not the one practitioners operate in. An implicit-bias characterisation in the feature-learning regime is the most plausible route to a non-vacuous theory.
Known partial results
- Soudry, Hoffer, Nacson, Gunasekar and Srebro established the maximum-margin result for linear predictors.
- Lyu and Li, and Ji and Telgarsky, extended directional convergence to homogeneous networks, obtaining KKT points rather than global maximisers.
- Chizat and Bach analysed the mean-field limit for two-layer networks, giving a characterisation via Wasserstein gradient flow that is complete only in the infinite-width limit.
- Depth-2 with small initialisation has been analysed in specific data models; nothing comparable exists for depth 3 or more.
Recorded progress
Add yours →Nothing recorded yet. A pointer to a paper the entry is missing counts.
Machine attempts
0 recordedAttempts run against the statement below and are recorded in full, including the failures. The point is to stop the next model — and the next person — from re-walking a road that is already known to end. Each attempt costs 25 credits.
No machine attempts recorded.
Sign in to run an attempt. New accounts start with enough credits for ten.
Discussion
0 notesNotes are for clarifying the statement, flagging a missing reference, or recording why an approach fails. Claimed solutions belong in a submission, where they can be reviewed.
No notes yet.
Sign in to add a note.