The eighty-fifth UQSay seminar on UQ, DACE and related topics will take place online on Thursday afternoon, April 3, 2025.
2–3 PM — Bilel Bensaid (Toulouse School of Economics)
New insights in neural networks optimization: Lyapunov stability and splitting schemes
These recent years, a great number of algorithms have been developed to optimize neural networks parameters (p-GD, clipping GD, Momentum, RMSProp, Adam, ...) but they need an accurate tuning to be stable and efficient. To get rid of the long and experimental step of GridSearch, we are looking for adaptive optimizers that come with guarantees. By analysing the stability of these algorithms, a general methodology to adapt the learning rate is suggested (generalization of the Armijo rule) for any deep learning optimizers, relating "robust" optimizers to preserving discretization schemes. Convergence and complexity of these methods are discussed leading to acceleration results, promoting the use of adaptive learning rate strategies for Analytic and Recurrent Neural Networks.
Finally, this study is extended to the mini-batch setting, revealing the link between mini-batch optimization and splitting operator methods. In a nutshell, this work comes up with deep relations between neural network training and classical issues in the numerical analysis of differential equations. .
References:
- Deterministic Neural Networks Optimization from a Continuous and Energy Point of View, J. Scientific Computing 2023
- An Abstract Lyapunov Control Optimizer: Local Stabilization and Global Convergence, 2024
- Convergence of the Iterates for Momentum and RMSProp for Local Smooth Functions: Adaptation is the Key, 2024
Joint work with G. Poette (CEA DAM, CESTA - ENSEIRB-Matmeca) & R. Turpault (IMB - ENSEIRB-Matmeca).
Organizing committee: Pierre Barbillon (MIA-Paris), Julien Bect (L2S), Nicolas Bousquet (EDF R&D), Vincent Chabridon (EDF R&D), Amélie Fau (LMPS), Filippo Gatti (LMPS), Clément Gauchy (CEA), Bertrand Iooss (EDF R&D), Alexandre Janon (LMO), Sidonie Lefebvre (ONERA), Didier Lucor (LISN), Sébastien Petit (LNE), Emmanuel Vazquez (L2S), Xujia Zhu (L2S).
Coordinators: Sidonie Lefebvre (ONERA) & Xujia Zhu (L2S)
Practical details: the seminar will be held online using Microsoft Teams.
If you want to attend this seminar (or any of the forthcoming online UQSay seminars), and if you do not already have access to the UQSay group on Teams, simply send an email and you will be invited. Please specify which email address the invitation must be sent to (this has to be the address associated with your Teams account).
You will find the link to the seminar on the "General" UQSay channel on Teams, approximately 15 minutes before the beginning.
The technical side of things: you can use Teams either directly from your web browser or using the "fat client", which is available for most platforms (Windows, Linux, Mac, Android & iOS). We strongly recommend the latter option whenever possible. Please give it a try before the seminar to anticipate potential problems.
