Teaching

Material for the two labs can be downloaded from here. Slides are here.

Optimization for large scale Machine Learning, M2 ENS 2022-2023 & 2023-2024

The goal of the class is to cover theoretical aspects and practical Python implementations of popular optimization algorithms in machine learning, with a focus on modern topics: huge scale models, automatic differentiation, deep learning, implicit bias, etc.

Notes for the class in their 2022 version are here and exercises are here

Schedule: From November 21st onwards: Tuesday 08 h 00, Friday 13 h 30 (room B1).

Validation: weekly theoretical homeworks, labs and paper presentation at the end of the class.

Syllabus:

• basics of convex analysis: convex sets, convex functions, strong convexity, smoothness, subdifferential, Fenchel-Legendre transform, infimal convolution, Moreau envelope

• gradient descent and subgradient descent, fixed point iterations, proximal point method (Lab 1)

• acceleration of first order methods: Nesterov and momentum

• algorithms for Deep Learning: stochastic gradient descent, ADAM, Adagrad (Lab 2)

• automatic differentiation

• second order algorithms: Newton and quasi-Newton methods

• duality in ML

• implicit regularization, Bregman geometry, mirror descent

• recent results in non convex optimization

• online learning

• other algorithms: Frank-Wolfe, primal-dual algorithms, variational inclusions, extragradient.

Resources:

• Introductory lectures on convex optimization: a basic course, Y. Nesterov, 2004. A reference book in optimization, updated in 2018: Lectures on Convex Optimization.

• First order methods in optimization, A. Beck, 2019.

• Convex optimization: algorithms and complexity, S. Bubeck, 2015. A short monograph (100 pages) covering basic topics.

OLISSIPO Winter school: dimensionality reduction (02/2023)

Convex optimization @Computation and Modelling summer school, WUST 2022

• intro slides

• exercises (updated every day)

• notebook 1 (Introduction to gradient descent on least squares)

• notebook 2 (Gradient descent on generic functions)

• notebook 3 (Non smooth optimization: the proximal gradient algorithm)

• notebook 4 (Alternatives to gradient descent)

Resources

My colleague Pierre Ablin and I have created a repository with some Python advice for our students: https://github.com/pierreablin/python-sessions.

Classes taught

Since 2019, I teach the Python for datascience class (42 h per year) in the X/HEC “Datascience for business” Master, using live coding inspired by the Software Carpentry workshops. I designed the course from scratch, collaborating with Joan Massich in 2019, Quentin Bertrand in 2020, and Hicham Janati in 2021.

Since 2020 I teach and handle practical sessions and data camps in Ecole Polytechnique's Executive education (70 h). Topics involved dimension reduction, clustering, scaling computations, visualization and datacamp. I designed 2 full python labs with Erwan Le Pennec on these topics.

From 2017 to 2019, my main teaching activity was the Optimization for datascience class of the Datascience Master, totalling 2*40 h including 4 h as lecturer. Amongst others, this involved refactoring of the practical sessions, tutoring of students during office hours, and partaking in the design of the final exam.

In 2016-2017, I was a TA at Télécom Paris, for

• Analysis and Probabilities (MDI 113/114, Bachelor, 10 h)

• Machine Learning and Data Mining (MDI 343, Executive Master, 20 h)

• Linear Models (SD 204, Master, 10 h)

• Practical Machine Learning (SD 207, Master, 10 h)

• Tools and applications for signals and images (SI 101, Bachelor, 6 h)