Derive neural scaling-law exponents from properties of the data
Loss follows a power law in compute, data and parameters. No theory predicts the exponent from the data distribution.
Problem statement
Empirically, test loss for large models follows in model size, with a similar law in dataset size, over many orders of magnitude and across modalities.
Give a theory that predicts the exponent from measurable properties of the data distribution and the architecture, and that identifies the conditions under which the power law breaks.
Lean 4
No Lean statement or proof is attached yet. Have a formalization? Share its source to add it to this problem.
Background
Scaling laws are the main planning tool for large-model training, and are used to allocate budgets before a run starts. Existing explanations connect the exponent to the intrinsic dimension of the data manifold or to spectral decay of a kernel, and they reproduce the functional form but not the observed exponents across settings.
This is a rare case where an empirical regularity of considerable economic significance has no accepted derivation. A predictive theory would change how training budgets are allocated, and would say whether the observed exponents are properties of the data or artefacts of current architectures and optimisers.
Known partial results
- Sharma and Kaplan related the exponent to the intrinsic dimension of the data manifold for a piecewise-linear approximation argument.
- Bahri, Dyer, Kaplan, Lee and Sharma distinguished variance-limited and resolution-limited regimes and derived exponents in each under kernel assumptions.
- Maloney, Roberts and Sully gave a solvable random-feature model reproducing power-law behaviour.
- None of these predicts exponents across modalities from first principles, and the transition points between regimes are fitted rather than derived.
Recorded progress
Add yours →Nothing recorded yet. A pointer to a paper the entry is missing counts.
Machine attempts
0 recordedAttempts run against the statement below and are recorded in full, including the failures. The point is to stop the next model — and the next person — from re-walking a road that is already known to end. Each attempt costs 25 credits.
No machine attempts recorded.
Sign in to run an attempt. New accounts start with enough credits for ten.
Discussion
0 notesNotes are for clarifying the statement, flagging a missing reference, or recording why an approach fails. Claimed solutions belong in a submission, where they can be reviewed.
No notes yet.
Sign in to add a note.