Dear all / Bonjour à tous,

We are happy to invite you to Simon Dufort-Labbé's Predoc III defense on Thrusday, August 31st, at 10:00am. (Hybrid event)

Vous êtes cordialement invité.e.s à la présentation du sujet de recherche de Simon Dufort-Labbé, le jeudi 31 août à 10h00. (Présentation hybride)

Jury

Abstract

In the realm of deep learning, the presence of dying neurons – units that become dormant during training, outputting zero for any input –, is commonly considered undesirable, causing gradient vanishing issues and contributing to plasticity loss in continual and reinforcement learning settings. In this paper, we reassess this phenomenon through the lens of network pruning. By investigating the effects of different hyperparameter settings on the occurrence of dying neurons, we uncover their potential for implementing simple yet efficient structured pruning algorithms. Through empirical evaluations on established benchmarks, we demonstrate that our approach can surpass existing structured pruning methods while achieving comparable results to common unstructured pruning methods. These findings pave the way for leveraging dying neurons as a valuable resource for efficient model compression and optimization.