Bonjour à tous,

Vous êtes tous et toutes cordialement invité.es à assister à la présentation de projet du prédoc III de Hafez Ghaemi, le 16 décembre à 14h (mode hybride).

Titre : Predictive World Modeling from an Egocentric Perspective

Date: mardi 16 décembre à 14h

Location: Auditorium 1, MILA, 2e étage

Jury

Résumé

Embodied artificial intelligence requires agents capable of perceiving, reasoning, and acting within complex, stochastic environments. While generative world models have demonstrated the ability to simulate environmental dynamics, they are often computationally expensive and predisposed to modeling task-irrelevant noise. In this report, we advocate for latent predictive world models as a scalable alternative paradigm for embodied agents. Adopting an egocentric agent perspective, we structure our research around three fundamental pillars: (1) self-supervised world modeling from sequential interaction, (2) eliminating heuristics for view construction in self-supervised learning (SSL) via active foveation, and (3) endowing world models with a “theory of mind” for multi-agent environments.

First, we address the challenge of learning robust visual representations from sequential interaction. We introduce seq-JEPA, a self-supervised world model that processes sequences of action-observation pairs. Through architectural inductive biases, we demonstrate that seq-JEPA resolves the trade-off between representational invariance and equivariance in joint-embedding SSL. Furthermore, our model excels at tasks that inherently require aggregating sequential observations, such as path integration across action trajectories.

Second, we propose to transition self-supervised visual pretraining from passive observation to active foveation. Current methods rely on hand-crafted augmentations or random masking as static heuristics to construct “views”. We propose Energy-Guided Masking (EGM), a mechanism for actively sampling target views in SSL from input regions with higher uncertainty. This approach allows view construction to be driven directly by the SSL objective rather than external heuristics.

Finally, we extend the scope of predictive world modeling to multi-agent, social contexts. A single-agent world model fails to predict the behavior of other agents whose internal states and actions are inaccessible to the agent. To model the behavior of others, we adopt an approach that captures prediction uncertainty in joint-embedding SSL via a latent variable learned through a variational objective. By leveraging this latent variable to model observed behavior, our framework enables the agent to predict the behavioral trajectories of other agents, facilitating planning in multi-agent environments. Furthermore, we propose a mechanism for scaling this approach as new agents are introduced to the environment.