Bonjour à tous,

Vous êtes tous et toutes cordialement invité.es à assister à la présentation de projet du prédoc III de Yipeng Zhang, le 15 décembre à 10h (mode hybride).

Titre : Self-Supervised Representation Learning and World Modeling on Naturally-Structured Data

Date: Lundi 15 décembre à 10h

Location: Auditorium 1, MILA, 2e étage

Jury

Résumé

Joint-embedding self-supervised learning (SSL) has become a central paradigm for learning visual representations without labels, typically by learning the relationship between semantically related data pairs. When these pairs are temporally related, the learned predictor naturally functions as a world model that anticipates future states. However,standard SSL objectives implicitly assume a fixed dataset and simple pairwise relationships. These assumptions often break down in practical scenarios involving naturally-structured data, where real-world dynamics introduce non-stationary data distributions and structured dependencies between data pairs. This report studies these challenges.

First, in a continual learning setting where the data distribution changesover time, we show that many existing methods fail to consolidate representations across time and struggle to adequately learn new data. We demonstrate that explicitly optimizing a consolidation objective, together with a separate embedding space to learn new data, improves performance. We also show that continual SSL models can benefit from naturally ordered data sequences.

Second, we study the predictive uncertainty in SSL, where each datum may correspond to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, such as successive video frames.We show that standard SSL objectives cannot learn this conditionalvuncertainty and propose AdaSSL, which adapts to different pairwise relationships and produces richer representations and more accurate latent world models.

Together, these contributions point toward self-supervised world modelsthat learn directly from naturally evolving data streams encountered inrealistic scenarios.