Dear all / Bonjour à tous,

We are happy to invite you to the Predoc III evaluation of Tianyue H. Zhang on November 17th at 10 am (hybrid mode).

Vous êtes tous et toutes cordialement invité.es à assister à la présentation de projet du prédoc III de Tianyue H. Zhang, le 17 novembre à 10h00 (mode hybride).

Titre: Exploring Overlooked Characteristics in Machine Learning Optimization

Date: November 17th, 10 am

Location: Auditorium 1

Link: https://umontreal.zoom.us/j/84339939626?pwd=VJZdTBRyZ3XotWhTbIwo7Lu6zYYCTv.1

Jury

Abstract

Adam often outperforms stochastic gradient descent in training modern deep networks, yet the reasons are not fully understood. We examine characteristics of optimizers that are frequently overlooked in existing work, focusing on how the objective function and data structure influence training dynamics. This perspective allows us to understand better when and why adaptive optimizers like Adam provide an advantage in practice.

First, we investigate Adam's sensitivity to rotations of the parameter space. We show that Adam is sensitive to the choice of parameter basis: random rotations harm performance, and specific structured rotations preserve or enhance it. This demonstrates that rotation-invariant theoretical assumptions are insufficient to explain Adam's empirical advantages. We then examine the rotation-dependent assumptions in the literature and find that they fall short in describing Adam's behaviour across various rotation types. In contrast, we verify the orthogonality of the update as a promising indicator of Adam's basis sensitivity, suggesting it may be the key quantity for developing rotation-dependent theoretical frameworks.

In addition, previous work suggests that the performance gap between Adam and SGD is related to dataset class imbalance, with Adam showing an advantage when certain tokens appear far more frequently than others. However, this advantage is mainly observed in the large batch setting, and the gap nearly disappears with small batch training. We aim to understand whether the additional stochasticity counteracts this phenomenon. We start from studying a linear bigram next-token prediction model trained on data following the power law, and deriving the asymptotic behaviour of SGD in this setting.