Towards Principled Hyperparameter Optimization and Algorithm Selection

Par

Dravyansh Sharma

Northwestern University

Mercredi 20 août 2025, 14:00-16:00 EST, Salle 3195

Pavillon André-Aisenstadt, Université de Montréal, 2920 Chemin de la Tour

Abstract:Popular practical approaches for effective hyperparameter optimization include Bayesian optimization and bandit-based approaches come with limited theoretical guarantees. Bayesian optimization, often using Gaussian processes, is effective for expensive evaluations but struggles with high-dimensional spaces and the performance is highly sensitive to the choice of priors and internal parameters. Bandit-based approaches make additional assumptions that capture aspects specific to hyperparameter tuning, including fixed limiting values of arm rewards (Hyperband) or increasing pull-dependent rewards with diminishing returns (rising bandits). A major blind spot in effectively using these techniques is the lack of insights on how the algorithmic performance actually varies with the hyperparameter.

A recent line of theoretically grounded work elevates hyperparameter optimization and algorithm selection to a learning problem in its own right. A growing body of research over the past decade from the learning theory community has successfully analysed how to provably tune several fundamental algorithms including decision trees, linear regression, and very recently even deep learning. The new techniques apply naturally to both hyperparameter tuning and algorithm selection. Future research areas include integration of these structure-aware principled approaches with the currently used techniques, better optimization in high-dimensional and discrete spaces, and improving scalability in distributed settings.

The content of this talk is related to a UAI 2025 tutorial and an upcoming NeurIPS 2025 tutorial.

Bio: Dravyansh (Dravy) Sharma is an IDEAL postdoctoral researcher, hosted by Avrim Blum at TTIC and Aravindan Vijayaraghavan at Northwestern University. He obtained his PhD at Carnegie Mellon University, advised by Nina Balcan. His research interests include machine learning theory and algorithms, with a focus on provable hyperparameter tuning, adversarial robustness, and learning in the presence of rational agents. His work develops principled techniques for tuning fundamental machine learning algorithms to domain-specific data, including decision trees, linear regression, graph-based learning and, most recently, deep networks. He has published several papers at top ML venues, including NeurIPS, ICML, COLT, JMLR, AISTATS, UAI and AAAI, has multiple papers awarded with Oral presentations, won the Outstanding Student Paper Award at UAI 2024, and has interned with Google Research and Microsoft Research. He has presented a tutorial at UAI 2025, has an accepted tutorial at AutoML 2025 and an accepted joint tutorial at NeurIPS 2025.