Passer au contenu

/ Department of Computer Science and Operations Research

Je donne

Rechercher

Navigation secondaire

Simon Dufort-Labbé's Predoc III Presentation

Dear all / Bonjour à tous,

We are happy to invite you to Simon Dufort-Labbé's Predoc III defense on Thrusday, August 31st, at 10:00am. (Hybrid event)

Vous êtes cordialement invité.e.s à la présentation du sujet de recherche de Simon Dufort-Labbé, le jeudi 31 août à 10h00. (Présentation hybride)


Title: A Demon at Work: Leveraging Neuron Death for Efficient Neural Network Pruning

Date: August 31st, 2023 at 10:00am-12:00pm EST

Location: Auditorium 1 - 6650 rue Saint Urbain

 

Jury

PrésidentCourville, Aaron
Directeur de rechercheBacon, Pierre-Luc
Membre
Lajoie, Guillaume

 

Abstract

In the realm of deep learning, the presence of dying neurons – units that become dormant during training, outputting zero for any input –, is commonly considered undesirable, causing gradient vanishing issues and contributing to plasticity loss in continual and reinforcement learning settings. In this paper, we reassess this phenomenon through the lens of network pruning. By investigating the effects of different hyperparameter settings on the occurrence of dying neurons, we uncover their potential for implementing simple yet efficient structured pruning algorithms. Through empirical evaluations on established benchmarks, we demonstrate that our approach can surpass existing structured pruning methods while achieving comparable results to common unstructured pruning methods. These findings pave the way for leveraging dying neurons as a valuable resource for efficient model compression and optimization.