Abstract

Generalization in deep learning remains poorly understood, particularly in the feature-learning regime where representations are learned jointly with predictors. Classical statistical learning theory provides only limited explanations for the strong generalization observed in overparameterized models. The phenomenon of grokking, where models achieve identical training performance yet generalize at different times, offers a controlled setting in which to study the mechanisms underlying generalization. In this paper, we propose that generalization can be explained through the geometry of learned representations. For tasks with known algebraic structure, we show that the geometry of the ideal representation can be specified analytically prior to training, yielding what we call the geometry of the task. In modular arithmetic, this structure corresponds to the regular representation of the cyclic group, whose irreducible decomposition defines a hierarchy of low-dimensional subspaces. We hypothesize that generalization occurs when a model’s internal representations align with this universal geometry. We formalize this connection and derive bounds on the generalization gap in terms of the geometric deviation between learned and ideal representations. Experiments support this view, showing a critical threshold of geometric alignment beyond which generalization emerges and demonstrating how regularization encourages convergence toward the universal geometry.