Simon Dufort-Labbé's Predoc III Presentation
Dear all / Bonjour à tous,
We are happy to invite you to Simon Dufort-Labbé's Predoc III defense on Thrusday, August 31st, at 10:00am. (Hybrid event)
Vous êtes cordialement invité.e.s à la présentation du sujet de recherche de Simon Dufort-Labbé, le jeudi 31 août à 10h00. (Présentation hybride)
Title: A Demon at Work: Leveraging Neuron Death for Efficient Neural Network Pruning
Date: August 31st, 2023 at 10:00am-12:00pm EST
Location: Auditorium 1 - 6650 rue Saint Urbain
Jury
Président | Courville, Aaron |
Directeur de recherche | Bacon, Pierre-Luc |
Membre | Lajoie, Guillaume |
Abstract
In the realm of deep learning, the presence of dying neurons – units that become dormant during training, outputting zero for any input –, is commonly considered undesirable, causing gradient vanishing issues and contributing to plasticity loss in continual and reinforcement learning settings. In this paper, we reassess this phenomenon through the lens of network pruning. By investigating the effects of different hyperparameter settings on the occurrence of dying neurons, we uncover their potential for implementing simple yet efficient structured pruning algorithms. Through empirical evaluations on established benchmarks, we demonstrate that our approach can surpass existing structured pruning methods while achieving comparable results to common unstructured pruning methods. These findings pave the way for leveraging dying neurons as a valuable resource for efficient model compression and optimization.