The seventy-fourth UQSay seminar on UQ, DACE and related topics will take place online on Thursday afternoon, September 19, 2024.
3–4 PM — Alexander Terenin (Cornell University) — [slides]
Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent
The ability to deploy Gaussian-process-based decision-making systems such as Bayesian optimization at scale has traditionally been limited by computational costs arising from the need to solve large linear systems. The de facto standard for solving linear systems at scale is via the conjugate gradient algorithm - in particular, stochastic gradient descent is known to converge near-arbitrarily-slowly on quadratic objectives that correspond to Gaussian process models’ linear systems. In spite of this, we show that it produces solutions which have low test error, and quantify uncertainty in a manner that mirrors the true posterior. We develop a spectral characterization of the error caused by finite-time non-convergence, which we prove is small both near the data, and sufficiently far from the data. Stochastic gradient descent therefore only differs from the true posterior between these regions, demonstrating a form of implicit bias caused by benign non-convergence. We conclude by showing, empirically, that stochastic gradient descent achieves state-of-the-art performance on sufficiently large-scale regression tasks, and produces uncertainty estimates which match the performance of significantly more expensive baselines on large-scale Bayesian optimization.
References:
- Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent, NeurIPS, 2023 - [github sgd-gp]
- Stochastic Gradient Descent for Gaussian Processes Done Right, ICLR, 2024
- Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees, JMLR, 2024 - [github conjugate-gradient-sparse-gp]
Organizing committee: Pierre Barbillon (MIA-Paris), Julien Bect (L2S), Nicolas Bousquet (EDF R&D), Vincent Chabridon (EDF R&D), Amélie Fau (LMPS), Filippo Gatti (LMPS), Clément Gauchy (CEA), Bertrand Iooss (EDF R&D), Alexandre Janon (LMO), Sidonie Lefebvre (ONERA), Didier Lucor (LISN), Sébastien Petit (LNE), Emmanuel Vazquez (L2S), Xujia Zhu (L2S).
Coordinators: Sidonie Lefebvre (ONERA) & Xujia Zhu (L2S)
Practical details: the seminar will be held online using Microsoft Teams.
If you want to attend this seminar (or any of the forthcoming online UQSay seminars), and if you do not already have access to the UQSay group on Teams, simply send an email and you will be invited. Please specify which email address the invitation must be sent to (this has to be the address associated with your Teams account).
You will find the link to the seminar on the "General" UQSay channel on Teams, approximately 15 minutes before the beginning.
The technical side of things: you can use Teams either directly from your web browser or using the "fat client", which is available for most platforms (Windows, Linux, Mac, Android & iOS). We strongly recommend the latter option whenever possible. Please give it a try before the seminar to anticipate potential problems.
