The seventy-ninth UQSay seminar on UQ, DACE and related topics will take place online on Thursday afternoon, November 28, 2024.
2–3 PM — Sam Allen (ETH Zürich, Seminar of Statistics)
Evaluating forecasts using positive definite kernels
Probabilistic forecasts take the form of probability distributions over the set of possible outcomes. To assess the quality of a probabilistic forecast, proper scoring rules condense forecast performance into a single numerical value, with which competing forecasters can be ranked and compared. Many popular scoring rules can be expressed in terms of positive definite kernels. This allows us to leverage well-established results on kernels when evaluating probabilistic forecasts. In this work, we discuss two useful connections between kernel methods and scoring rules. Firstly, we demonstrate that well-known weighted scoring rules correspond to kernel scores. Weighted scoring rules allow users to emphasise particular outcomes during forecast evaluation, and by expressing these as kernel scores, we introduce a broad generalisation of existing weighted scores that can be applied to probabilistic forecasts on arbitrary outcome domains. This facilitates the introduction of weighted multivariate scoring rules, for example. Secondly, we demonstrate that by embedding probabilistic forecasts into a Reproducing Kernel Hilbert Space (RKHS), estimating the parameters of forecast models by optimising a kernel score often corresponds to a convex quadratic optimisation problem, which can therefore be solved robustly and efficiently. This encompasses common forecast combination methods, such as linear pooling, providing a means to optimally combine competing forecasts made on arbitrary outcome domains. To illustrate the practical benefit of these results, the two applications of kernel scores are applied in case studies on operational weather forecasts..
References:
- Evaluating Forecasts for High-Impact Events Using Transformed Kernel Scores, 2023
- Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts, 2024 - [R package]- [Python package]
Joint work with D. Ginsbourger (University of Bern) & J. Ziegel (ETH Zürich).
Organizing committee: Pierre Barbillon (MIA-Paris), Julien Bect (L2S), Nicolas Bousquet (EDF R&D), Vincent Chabridon (EDF R&D), Amélie Fau (LMPS), Filippo Gatti (LMPS), Clément Gauchy (CEA), Bertrand Iooss (EDF R&D), Alexandre Janon (LMO), Sidonie Lefebvre (ONERA), Didier Lucor (LISN), Sébastien Petit (LNE), Emmanuel Vazquez (L2S), Xujia Zhu (L2S).
Coordinators: Sidonie Lefebvre (ONERA) & Xujia Zhu (L2S)
Practical details: the seminar will be held online using Microsoft Teams.
If you want to attend this seminar (or any of the forthcoming online UQSay seminars), and if you do not already have access to the UQSay group on Teams, simply send an email and you will be invited. Please specify which email address the invitation must be sent to (this has to be the address associated with your Teams account).
You will find the link to the seminar on the "General" UQSay channel on Teams, approximately 15 minutes before the beginning.
The technical side of things: you can use Teams either directly from your web browser or using the "fat client", which is available for most platforms (Windows, Linux, Mac, Android & iOS). We strongly recommend the latter option whenever possible. Please give it a try before the seminar to anticipate potential problems.
