Teaching

Material for the two labs can be downloaded from here. Slides are here.

Optimization for large scale Machine Learning, M2 ENS 2022-2023 & 2023-2024

The goal of the class is to cover theoretical aspects and practical Python implementations of popular optimization algorithms in machine learning, with a focus on modern topics: huge scale models, automatic differentiation, deep learning, implicit bias, etc.

Notes for the class in their 2022 version are here and exercises are here

Schedule: From November 21st onwards: Tuesday 08 h 00, Friday 13 h 30 (room B1).

Validation: weekly theoretical homeworks, labs and paper presentation at the end of the class.

Syllabus:

  • basics of convex analysis: convex sets, convex functions, strong convexity, smoothness, subdifferential, Fenchel-Legendre transform, infimal convolution, Moreau envelope

  • gradient descent and subgradient descent, fixed point iterations, proximal point method (Lab 1)

  • acceleration of first order methods: Nesterov and momentum

  • algorithms for Deep Learning: stochastic gradient descent, ADAM, Adagrad (Lab 2)

  • automatic differentiation

  • second order algorithms: Newton and quasi-Newton methods

  • duality in ML

  • implicit regularization, Bregman geometry, mirror descent

  • recent results in non convex optimization

  • online learning

  • other algorithms: Frank-Wolfe, primal-dual algorithms, variational inclusions, extragradient.

Resources:

  • Introductory lectures on convex optimization: a basic course, Y. Nesterov, 2004. A reference book in optimization, updated in 2018: Lectures on Convex Optimization.

  • First order methods in optimization, A. Beck, 2019.

  • Convex optimization: algorithms and complexity, S. Bubeck, 2015. A short monograph (100 pages) covering basic topics.

OLISSIPO Winter school: dimensionality reduction (02/2023)

Convex optimization @Computation and Modelling summer school, WUST 2022

Resources

My colleague Pierre Ablin and I have created a repository with some Python advice for our students: https://github.com/pierreablin/python-sessions.

Classes taught

Since 2019, I teach the Python for datascience class (42 h per year) in the X/HEC “Datascience for business” Master, using live coding inspired by the Software Carpentry workshops. I designed the course from scratch, collaborating with Joan Massich in 2019, Quentin Bertrand in 2020, and Hicham Janati in 2021.

Since 2020 I teach and handle practical sessions and data camps in Ecole Polytechnique's Executive education (70 h). Topics involved dimension reduction, clustering, scaling computations, visualization and datacamp. I designed 2 full python labs with Erwan Le Pennec on these topics.

From 2017 to 2019, my main teaching activity was the Optimization for datascience class of the Datascience Master, totalling 2*40 h including 4 h as lecturer. Amongst others, this involved refactoring of the practical sessions, tutoring of students during office hours, and partaking in the design of the final exam.

In 2016-2017, I was a TA at Télécom Paris, for

  • Analysis and Probabilities (MDI 113/114, Bachelor, 10 h)

  • Machine Learning and Data Mining (MDI 343, Executive Master, 20 h)

  • Linear Models (SD 204, Master, 10 h)

  • Practical Machine Learning (SD 207, Master, 10 h)

  • Tools and applications for signals and images (SI 101, Bachelor, 6 h)