Dear all / Bonjour à tous,

We are happy to invite you to the Predoc III Evaluation of Yusong Wu on Friday, August 23rd at 1 pm (hybrid mode).

Vous êtes cordialement invité.e.s à la soutenance de thèse de Yusong Wu , le vendredi, 23 Août à 1pm (mode hybride).

Jury

Aishwarya Agrawal

Aaron Courville

Anna Huang

Pablo Samuel Castro

Abstract

While existing generative models are adept at creating expressive
compositions and accompaniments, they are not designed for live,
interactive scenarios that require real-time collaboration, adaptation, and
anticipation among multiple participants. Synchronous music interaction,
such as jamming, accompaniment, and improvisation, involves simultaneous
musical actions that necessitate a high degree of coordination and
collaborative creativity.

To address this gap, in the proposal, we aim to develop a framework of
building models for real-time synchronous music interaction. We first
explore the cognitive and physiological foundations of such interactions
and review the current state of music generation models and datasets.
Building on this background, we propose a formulation of the synchronous
music interaction problem as a multi-agent general-sum game. We then
outline a design approach that conceptualizes music interaction models as
generative agents, which are first pretrained using maximum likelihood
estimation and then fine-tuned with reinforcement learning.

We also present ReaLchords, an online generative model developed as part of
this research, which improvises chord accompaniments in real-time to a
user’s melody. Through experiments and listening tests, ReaLchords has
demonstrated effective adaptation to unfamiliar inputs, producing
harmonically and temporally coherent accompaniments. We will also present a
real-time interactive music system, ReaLJam, built upon ReaLchords,
showcasing how ReaLchords enables dynamic and responsive real-time music
interaction.

Finally, the proposal outlines ongoing and future work, including the
extension of ReaLchords to multi-agent co-adaptive settings, the
development of models that interact via audio inputs and outputs, and the
exploration of learning from human interactions and feedback.