Jekyll2026-10-05T15:07:02+00:00/feed.xmlMathurin Massias’ webpageMathurin MassiasOn the link between maximizing the likelihood and minimizing the Kullback-Leibler divergence2025-10-27T22:00:00+00:002025-10-27T22:00:00+00:00/blog/kl_mle MathJax.Hub.Config({ TeX: { Macros: { pdata: "p_{\\mathrm{data}}", pdatahat: "\\hat{p}_{\\mathrm{data}}", data: ["x^{(#1)}", 1], KL: "\\operatorname{KL}" } } });

This blog post clarifies the links between the maximizing the likelihood and minimizing the Kullback-Leibler divergence between a parametric distribution \(p_\theta\) and the true distribution \(\pdata\).

TLDR: the negative log-likelihood is not the KL between \(\pdatahat\) and \(p_\theta\). Instead, it is an empirical estimate of the KL between \(\pdata\) and \(p_\theta\). Those two are not the same.

Setup

Suppose that you have access to i.i.d. samples \(\data{1}, \ldots, \data{n}\) from the true data distribution \(\pdata\), the latter admitting a density with respect to the Lebesgue measure. Define the empirical data distribution as \(\pdatahat := \frac{1}{n}\sum_{i=1}^n \delta_{\data{i}}\), where \(\delta\) is the Dirac distribution. Among a family of parametric distributions \(\{p_\theta\}_\theta\), a natural approach to find the one closest to \(\pdata\), or fitting the data the best, is to maximise the likelihood of the observed samples, i.e. to solve

\[\max_\theta \sum_{i=1}^n \log p_\theta(\data{i})\]

An incorrect interpretation

You can read online (e.g. here) that maximizing the likelihood amounts to making \(p_\theta\) close to \(\pdatahat\), because it amounts to solving \(\min_\theta \KL(\pdatahat, p_\theta)\).

The reasoning is as follows:

\[\begin{align} \KL(\pdatahat, p_\theta) &= \int \pdatahat(x) \log \frac{\pdatahat(x)}{p_\theta(x)} dx \\ &= \int \pdatahat(x) \log \pdatahat(x) dx - \int \pdatahat(x) \log p_\theta(x) dx \\ &= constant - \int \pdatahat(x) \log p_\theta(x) dx \\ &= constant - \frac{1}{n} \sum_{i=1}^n \log p_\theta(\data{i}) \end{align}\]

but these manipulations are not correct, as \(\pdatahat\) does not admit a density: \(\int \pdatahat(x) \log \pdatahat(x) dx\) is not defined (worse, the KL itself is not defined unless \(p_\theta\) is supported only on the \(\data{i}\)’s, which is unlikely).

The correct interpretation

The correct intepretation is: maximizing the likelihood amounts to minimizing an empirical estimate of the KL between \(\pdata\) (not \(\pdatahat\)) and \(p_\theta\). Indeed the latter is:

\[\begin{align} \KL(\pdata, p_\theta) &= \int \pdata(x) \log \frac{\pdata(x)}{p_\theta(x)} dx \\ &= \int \pdata(x) \log \pdata(x) dx - \int \pdata(x) \log p_\theta(x) dx \\ &= constant - \int \pdata(x) \log p_\theta(x) dx \end{align}\]

Up to a constant, an empirical estimate of the KL divergence is therefore:

\[\begin{align} \widehat{\KL}(\pdata, p_\theta) = -\int \pdatahat(x) \log p_\theta(x) dx \end{align}\]

which is equal to the negative likelihood.

]]>The Hutchinson trick2024-11-18T22:00:00+00:002024-11-18T22:00:00+00:00/blog/hutchinsonThe Hutchinson trick: a cheap way to evaluate the trace of a Jacobian, without computing the Jacobian itself!

$$ \DeclareMathOperator{\DIV}{div} \DeclareMathOperator{\tr}{tr} \DeclareMathOperator{\jac}{J} \newcommand{\Id}{\mathrm{Id}} \newcommand{\bbR}{\mathbb{R}} \newcommand{\bbE}{\mathbb{E}} \DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\diag}{diag} \newcommand{\norm}[1]{\Vert #1 \Vert} $$

Continuous normalizing flows (more in this blog post, in particular its Equation 6) heavily rely on the computation of the divergence of a network \(f: \bbR^d \to \bbR^d\), aka the trace of its Jacobian:

\[\begin{equation} \DIV f (x) = \tr \jac_f(x) \triangleq \sum_{i=1}^d \frac{\mathrm{d} f_i}{\mathrm{d} x_i} (x) \end{equation}\]

Computing this quantity would require evaluating the full Jacobian, which can be done at the prohibitive cost of \(d\) backpropagations (more details below). It turns out, the trace/divergence can be approximated reasonably well with a single call to backpropagation.

For that, let’s forget about Jacobians for a second and take a generic matrix \(A \in \bbR^{d \times d}\). The Hutchinson trick states that for any random variable \(z \in \bbR^d\) such that \(\bbE[zz^\top] = \Id_d\),

\[\begin{equation}\label{eq:hutchinson} \tr(A) = \bbE_z [z^\top A z] \end{equation}\]

It is typically used for \(z\) having iid entries of mean zero and variance, classically standard Gaussian or Rademacher. The proof is very simple:

\[\begin{align*} \bbE_z [z^\top A z] &= \bbE_z [\tr(z^\top A z)] \quad &\text{(a scalar equals its trace)} \\ &= \bbE_z [\tr (A z z^\top)] \quad &(\tr(MN) = \tr(NM)) \\ &= \tr \left(\bbE_z [A z z^\top]\right) \quad &\text{(trace and expectation commute)} \\ &= \tr \left(A \bbE_z [z z^\top] \right) \quad &\text{(linearity of expectation)} \\ &= \tr \left(A \Id_d \right) \quad &\text{(assumption on $z$ )} \\ &= \tr \left(A \right) \quad & \end{align*}\]

et voilà!

Why is it useful?

The numerical benefit is not obvious at first: if one has access to \(A \in \bbR^{d \times d}\), computing its trace costs \(\mathcal{O}(d)\), while the above formula requires \(\mathcal{O}(d^2)\) to compute \(z^\top A z\), not to mention multiple Monte-Carlo samples for the expectation. But there are cases in which one wants to compute the trace of a matrix, without having explicit access to said matrix!

The flagship example is the full Jacobian of a neural network \(f\). Explicitely computing it is out of reach: with backpropagation one can only compute Jacobian-vector products, aka \(\jac_f(x) v\) for \(v \in \bbR^d\). To compute the full Jacobian means computing \(\jac_f(x) e_i\) for all canonical vectors \(e_i\), hence calling backpropagation \(d\) times (though it can be done in a vectorized fashion). But, to compute the trace of the Jacobian, one can sample a single \(z\), and then approximate the expectation in \eqref{eq:hutchinson} by a (single) Monte-Carlo estimate:

\[\tr J_f(x) \approx z^\top \left(J_f(x) z \right)\]

where \(J_f(x) z\) is computed with a single backpropagation pass. This is what is done in Continuous normalizing flows, in the Ordinary Differential Equation required to evaluate the log likelihood loss. Very clever and elegant!

Is it a good estimator?

The Hutchinson estimator is a very elegant trick with a short proof, but it has a drawback: its variance is terrible.

Proposition 1: For standard Gaussian \(z\) the variance of the estimator \(z^\top A z\) is \(2 \norm{A}_F^2\).

Click to expand the proof Wlog assume that \(A\) is symmetric: we can do so because \(A\) has same trace as its symmetric part, and \(z^\top A z = z^\top (A^\top + A) z / 2 \). We can use that a Gaussian \(z \) is still Gaussian under orthogonal transform to assume \(A\) is diagonal. Then for $A = \Lambda = \diag(\lambda_1, \ldots, \lambda_d)$, \begin{align} \bbE[(z^\top \Lambda z)^2] &= \bbE [(\sum_i \lambda_i z_i^2)^2] \\ &= \bbE [(\sum_i \lambda_i^2 z_i^4 + \sum_{i \neq j} \lambda_i \lambda_j z_i^2 z_j^2)] \\ &= 3\sum_i \lambda_i^2 + \sum_{i \neq j} \lambda_i \lambda_j \\ &= 2\sum_i \lambda_i^2 + \sum_i \lambda_i^2 + \sum_{i \neq j} \lambda_i \lambda_j \\ &= 2\sum_i \lambda_i^2 + \sum_{i, j} \lambda_i \lambda_j \\ &= 2 \norm{A}^2_F + (\tr A)^2 \end{align} using that expectation of $z_i^4 = 3 \sigma^2 = 3$, and $z_i^2$ and $z_j^2$ are independent for $i \neq j$ (with expectation equal to 1) Since $\bbE[z^\top A z] = \tr A$, the final variance is $2 \norm{A}^2_F$.


This result is quite disappointing: the variance scales like \(\mathcal{O}(d^2)\), while the trace scales like \(d\). Can we improve this with other random variables? Yes, but not by much!

Proposition 2: For uniform sign \(z\) (aka Rademacher variables, taking values +1 or -1 with probability 1/2), the variance of the estimator is $2 \sum_{i \neq j} A_{ij}^2$. It is the law of \(z\) that minimizes the variance of the Hutchinson estimator.

Click to expand the proof Again assume that $A$ is symmetric wlog. Since $z_i^2 = 1$, \begin{align} z^\top A z - \bbE[z^\top A z] &= \sum_{i \neq j} A_{ij} z_i z_j \\ &= 2 \sum_{i < j} A_{ij} z_i z_j \end{align} The variables $z_i z_j$ $(i < j)$ are independent Rademacher variables, and so $\Var(z_i z_j) = 1$. The variance to compute is thus \begin{align} \Var \left(2 \sum_{i < j} A_{ij} z_i z_j\right) = 4 \sum_{i < j} A_{ij}^2 \Var(z_i z_j) = 2 \sum_{i \neq j} A_{ij}^2 \enspace. \end{align} For the minimal variance result: for any $z$ (not necessarily Rademacher) \begin{align} \bbE[(z^\top A z)^2] &= \bbE[\sum_{ijkl}z_i z_j z_k z_l A_{ij} A_{kl}] % &= \bbE[\sum_i A_{ii}^2 z_i^4 + \sum_{i, j} z_i^2 z_j^2 (A_{ii}^2 + 2 A_{ij}^2)] \end{align} Now the expectation is 0 as soon as one index is not equal to any of the others ($\bbE[z_i] = 0$ + independence). So we have four cases to consider:
  • $i=j=k=l$, yielding $\sum_i z_i^4A_{ii}^2$
  • $i=j$, $k=l$, $i \neq k$, yielding $\sum_{i \neq k} z_i^2 z_k^2 A_{ii} A_{kk}$
  • $i=k$, $j=l$, $i \neq j$, yielding $\sum_{i \neq j} z_i^2 z_j^2 A_{ij}^2$
  • $i=l$, $j=k$, $i \neq j$, yielding $\sum_{i \neq j} z_i^2 z_j^2 A_{ij}^2$ too
Hence by renaming $k$ into $j$ in the second case \begin{align} \bbE[(z^\top A z)^2] &= \bbE[\sum_i A_{ii}^2 z_i^4 + \sum_{i \neq j} z_i^2 z_j^2 (A_{ii} A_{jj} + 2 A_{ij}^2)] \\ &= \sum_i A_{ii}^2 \bbE[z_i^4] + \sum_{i \neq j} (A_{ii} A_{jj} + 2 A_{ij}^2) \end{align} since $\bbE[z_i^2 z_j^2] = \bbE[z_i^2] \bbE[z_j^2] = 1$ when $i \neq j$. To minimize $\bbE[(z^\top A z)^2]$ we should thus minimize $\bbE[z_i^4]$, which by Jensen inequality is lower bounded by $\bbE[z_i^2]^2 = 1$. The value 1 is achieved for $z_i$ being Rademacher.


In hindsight, this optimality result makes sense: if \(A\) is diagonal, then for any draw of a Rademacher \(z\), \(z^\top A z = \tr A\) ! With a small numerical experiment (entries of \(A \in \bbR^{20 \times 20}\) i.i.d. standard Gaussian) we can check that Rademacher gives a slightly smaller variance.

If we increase the diagonal entries of \(A\) (by 5 here), we see the variance for Gaussian \(z\) increase as predicted by Proposition 1, while the performance for Rademacher \(z\) is unaffected (coherent with Proposition 2)


Beyond trace: computing the full diagonal

As my colleague Antoine Gonon pointed to me, the Hutchinson trick can be used to compute all the entries of the diagonal:

\[\bbE_z [(Az) \odot z] = \diag(A)\]

The proof is very similar to the first one. This technique allows evaluating the diagonal of the Hessian of a neural network \(g: \bbR \to \bbR\), using Hessian-vector product (see this great blog post), which has applications in neural network pruning for example. See more in Section 9.8 of .

]]>Advice for presentations or how to avoid losing your audience2024-09-04T22:00:00+00:002024-09-04T22:00:00+00:00/presentation_adviceTLDR: Never underestimate how easily and quickly you can lose the audience, and how hard it is for them to catch up afterwards. Your slides should only be a visual support for what you say. Technicalities like equations with heavy notation, tables, complex figures are the enemy. Instead, use plenty of examples, figures, simple terms and explanations.

Content: choose the appropriate level of detail

First, select what you want to include in your presentation, and how you want to structure it.

  • Choose what you include in the presentation; you do not want to shoehorn all the technical content of your paper/report. The presentation is not the paper, it is here to highlight the most significant points only, to make people want to read the paper for more details. If you want, have backup slides (after the conclusion) with more details about some point, that you may use during the questions.
  • Contrary to a paper, slides do not need to be 100% rigorous, you can (and should!) be fuzzy, informal, have handwavy notation: all this, in order to gain clarity and concision.
  • When working on the presentation, ask yourself: what do I want somebody completely unaware of my research to remember at the end of the talk? What’s my big message? What are the points that really matter, and what is more of a technical detail (that should be omitted)? Instead of a “Questions?” slides, end with a recap slide that contains the 3/4 high-level takeaways.
  • Rule of thumb: use at most 10 lines per slide, and spend 1 min/slide. Going over these limits often means that you put too much; you will rush over the slides and lose everyone on slide 2.
  • Announcing your plan is usually not very interesting (introduction, state-of-the-art, presentation of method, results…). Instead, starting with a simplified motivating example allows to get the point quickly. Do not start by introducing all your notation.
  • “A mathematical presentation should content one proof and one joke, and the two should never be the same thing”. If you show proofs, just show an overview, or detail the key innovative step or whatever, but avoid writing down many lines of complex computations with heavy notation.
  • Avoid including results tables, they are really hard to parse. Most people that include them end up saying “and this table contains too much to read, but it shows that…”
  • Algorithms are also very hard to parse; the comments are usually more useful than the formulas themselves. Present them saying “those are the important steps, here is what they do” in layman terms; “the main difference with existing algorithms is…”, “the most costly part is this…” but do not ever go step by step through an algorithm

Pay attention to form

Beyond its raw content, the visual aspect of your slide is a powerful tool to convey information easily.

  • In your first slide, identify yourself and your lab/structure/school clearly, but also acknowledge the people you’ve worked with (joint work with…/internship under the supervision of…).
  • Avoid at all cost writing long complete sentences: the audience will read them instead of listening to you. Write only the important parts, to support your talk. The real content of the talk is your speech, not the slides. Somehow, if one can understand the slides just by reading them, it means you do not really provide additional value being there to introduce them.
  • Hierarchize content visually: use bold and boxes to show the audience the critical content of the slide, play with font size to distinguish the key part from the rest (typically, citations can be made smaller, enumerations too).
  • Make your slides as light as possible: do not number figures (do not even put them in a figure environment), do not number equations, remove the weird navigation bar that latex puts by default. Every useless information is a distraction from the interesting content.
  • Colors are very helpful; for example if you display a variation of a previous formula, colorize what has changed.
  • Use all the space in the slide (not by putting more content, but by putting enough space between each item). This is a pain in beamer but it’s mandatory, dull slides will lose your audience faster than anything else. If there is a single figure in your slide it should take the whole space, etc.

Spend time on figures

Those are very important, as a good figure conveys a lot of information quickly.

  • It’s often necessary to redo the figure specifically for your presentation (e.g. at least changing the ratio to better fit the slide). Editing the figure, previously saved as pdf, with inkscape or some other tool is often useful in that case.
  • Put xlabels, ylabels, and big enough. As an exercise, take one presentation of yours, project it on the screen on a slide with a figure, go to the back of the room and check if you’re able to parse it (you won’t).
  • Title and captions are often not needed.
  • A 1-line sentence to interpret the figure, below it (or in the slide title) is summarizes the message.

Presenting style

  • Rehearse: your slides should be the support for your speech; if when rehearsing you realize that you’re talking about something that needs visual support, it helps you improve your slides.
  • Start slow in the introduction/motivating example so that the people completely unfamiliar with your line of work at least get something out of it. We often underestimate how a significant fraction of the audience can be unfamiliar with our line of work. They’ll appreciate being walked through in the first 3/5 min.
  • Make breaks, emphasize the important parts (such as “if there is one think to remember from the talk it is this result/slide/figure…”)
  • Do some recaps when moving from one part to another, summarizing in simple terms: “so we’ve seen that the big problem when trying to do X is Y and now we’re going to see how to solve it”.
  • Give orders of magnitude, everything that helps grasp the problem (instead of vague sentences such as “this costs a lot”, “there are many parameters”, “the dimension is high”, “the algorithm takes a lot of time to run”, etc.)
  • Look at the audience rather than your computer or the slides. Having a pointer (its costs 10€) helps. Avoid remaining perfectly still for the whole presentation.

Small but useful stuff

  • Number your slides for easier communication (a member of the audience may note its number to ask a question later), but avoid the “3/n” format: if n is big this is bad for the audience morale. Similarly I find the progress bar on top of the slide, present in default templates, to be utterly depressing (my 2 cents).
  • Citations in slide are tricky: if the format is [ABC24] or [1] then they provide no info to the audience; I strongly recommend “Author et al (2024)”, and to put them in smaller font when they are not critical.
  • Beware of colors in graphs, many projectors are old and render them incorrectly.

Thanks to Quentin Bertrand for his input on this post!

]]>Advice for CNRS and INRIA recruitment2024-07-25T09:41:41+00:002024-07-25T09:41:41+00:00/cnrs_inria_adviceINRIA and CNRS “chargé de recherche” positions offer unique conditions of freedom to do first-grade research: lifelong contract, no teaching involved. The application process is challenging, but it’s definitely worth it! The goal of this guide is to make the application process as easy as possible.

It only reflects my particular opinion and limited experience of going through it, and helping people with it since 2022. Feedback is most welcome!

INRIA has issued a very useful guide: Advice for Applicants for a Young Graduate Scientist Position at INRIA.

The key points are:

  • the application requires lots of work and communication with the teams you apply to: start as soon as possible, no later than October
  • show 5 to 10 years ambition in your research project, depart from your PhD works
  • do not be too technical, neither in the research project, nor in the audition
  • keep in mind that most members of the jury are not expert in your research area
  • ask for a lot of feedback on every part of your application (project discussion and proofreading, audition rehearsals)

Most important advice: talk to many people

Those selection processes are very peculiar and very different from what you’ve been used to in your PhD and postdoc. You need to talk to as many people as possible! There are three categories of person to contact:

  • senior researchers having been involved in the process for several years. Talking to past members of the jury is priceless: they will tell you what the jury is looking for in candidates! Current members of the jury will rightfully decline speaking with you, as this is not ethical. Do not put them in an uncomfortable position by contacting them.
  • heads and members of the various teams you apply to. They should give you feedback on your application file, and organize rehearsals with you. If they don’t, it’s not a very good sign.
  • successful past candidates. They will tell you what worked and what didn’t for them. The yearly results are public; for example CNRS ones are available on FX Coudert’s website.

Gather as much advice as possible, even if they end up disagreeing: you’ll make your own choices when the time comes. Negative (but constructive) feedback is very precious: the auditions are very selective, and your project needs to be perfect. Negative feedback will help you improve a lot, but hearing negative stuff about projects that you care deeply about is unpleasant and takes its toll, so being surrounded by kind people is also very important.

Do not be afraid of contacting people, even if you don’t know them. Most people are happy to help. Worst can happen is people don’t answer your email. If they don’t, send at least two follow-up emails, waiting around 2 weeks between each. This is typical practice and will not be interpreted as pushy if you are polite in your mail (and concise! go to the point to save people’s time).

Have your PhD and postdoc advisors help you too; beware though that they are very familiar with your work and are not representative of a typical jury member.

Global timeline: start before October!

Laying out a full research project for 5/10 years requires several months of intensive work. The CNRS deadline is usually around the beginning of January (in 2024 it was more end of January, we’ll see if it’s a permanent change), the INRIA one is end of January/February. You must contact the potential hosting teams around October of the previous year, not after. The teams will ask you to come do a presentation to evaluate your compatibility and discuss a common project, and they will give feedback on your research project. All this takes time, so do not wait for December to do it. A team may be interested by your profile, but already have candidates.

If you read this soon enough, I encourage you to start writing a rough draft of a few pages as soon as you start your postdoc, and begin working on it research-wise right away: it’s much better if, the day of the audition, the jury sees that your project is grounded in concrete work, that you have already started working on. Starting as early as possible will make it easier to get publications on the topics of your research project, which helps tremendously.

CNRS/INRIA differences

Disclaimer: I know INRIA better.

  • research project format:
    • INRIA provides a template, your research project should be exactly 3 pages long. Your presentation of 3 selected works is also 3x1 page long, with a predefined structure for each work (description of work, personal contribution, originality and difficulty, impact, diffusion)
    • CNRS does not provide a template; the research project is, from the ones I’ve seen, 10 to 15 pages long, and the summary of past works is around 5 pages. I wrote the CNRS application file first and contracted it + adapted it to my integration team for INRIA.
  • affectation and integration into hosting team:
    • For CNRS the recruitment is national, and you suggest 2 or 3 teams for your integration (if you get accepted, CNRS decides where to send you, which may or may not be one of those teams). You mention the teams, each in a paragraph, at the end of your research project, and spend 30 seconds per team at the end of your oral presentation, to explain why they’d be a good match. You should still work on your integration and know what the people in the team are doing, and how you’d like to work with them, but the philosophy is, very roughly, that you should be able to lead your research in any lab, and that the surrounding people do not matter this much (my 2 cents from gathered feedbacks). This may differ between section 7 (computer science) and section 41 (maths).
    • INRIA is very different: you have one application file per INRIA center (there are 9 in France). Inside a center, you apply for a team (you have the possibility to not specify one, but it makes a bad impression: it means that there is no team for which you are a perfect fit). Your application file is thus very different from CNRS: the targeted team must appear frequently in it. In your whole research project, you must show what you will bring them, what they can help you with, why you are similar but complementary, who you’ll interact with, etc. Finally, for INRIA, since you apply in a team, its support is essential. Even more that for CNRS, you should contact the hosting team early.

Document 1: Summary of past research

Show that there is coherence in your past works, that you took initiatives, that all your works interact nicely in a grand scheme that defines your identity and uniqueness as a researcher. It should not be perceived as a sequence of unrelated works that were proposed by your PhD advisors. Do not be too technical, summarize in layman’s term. Show that you understood what were the big challenges of your field, and how you contributed to solving them.

Document 2: The research project

Writing the research project is the hardest (but also very interesting) part of the application. Ask at least 3 successful candidates from past years their research project, to see what worked.

The right balance: ambitious but realistic

Writing the research project is a very different exercise from what we’re used to in PhD/postdoc research statement. Understand that CNRS and INRIA chargé de recherche are lifelong positions. In your project, you must show that you have research ideas for your whole life, and not just for the next 3 years. The difficulty is that your project should:

  • be ambitious: show that you have a direction for roughly the next 10 years
  • be realistic: show that you know where to start, and why you have the tools to solve the problems you’ll consider.

Ideally, you give big directions in which you want to go, and descriptions of concrete ways with which you would start (even better: how you have already started). People usually divide their project into 3 linked axis; each axis should start with something simple (one research idea that you could start working on as soon as you get the position), and then aim for something more distant, that has the right to be less precise. The jury knows that research is hard to predict (you won’t probably do everything in your project), but they want to see that you have a plan.

You should avoid at all cost giving the impression that you’ll continue working exactly on what you did during your PhD. This would make the jury fear that once this line of work dries up, you’ll be left without ideas. You should show thematic mobility, while remaining coherent with your past research…

On the other hand, to avoid giving the impression that your project is just wishful thinking, show the jury that you have already started tackling some of the first steps in your research project (ideally you have already published it, but if not, it could be by contacting other researchers, by knowing the related literature, by organizing reading groups, etc). In general, if you have only worked on continuation of your PhD project, it sends the jury a very bad signal regarding your autonomy in terms of generating new ideas and directions.

Composition of the jury: speak to everybody

Your application will be read in details by 1 or 2 expert reviewers, that know your field very well. But, beware that the rest of the jury will be much less familiar with your line of work, and most probably only skim through the introduction. Having a look at the composition of the jury (it is public, for example here for Section 7 of CNRS and here for INRIA in 2024) is a good idea to realize the diversity of your audience - and to guess who your reviewer will be. The reviewer is usually the one that asks the first questions during the audition, so knowing their interests helps you prepare.

In short: be precise, for the specialists, but be understandable, for the rest of the jury. We tend to focus on the first part and ignore the second.

To speak to everybody: why is your problem important? which locks do you unlock? how is this different from what everybody does in the field? what are the current barriers in the field and why are you, with your experience, drive and ideas, the perfect person to lift them? You need to display maturity, global vision of your field: basically that you’re ready to be an independent, permanent researcher, no longer someone following a mentor. Show that what you’re working on has impact, that it’s not a small niche that you plan to exploit all your life: your work should unlock new research venues, be important to others too. Show that these questions are interesting for others, beyond you (using citations for example).

People not so familiar with your field should understand why you’re unique. As a personal example: all optimizers (me included) tend to mention that they want to develop faster algorithms, with better guarantees, and less data-hungry. But do some people look for slow algorithms that don’t work and need more data? This does not characterize you at all. Be more precise.

In your project: why are you legitimate for this? What have you accomplished that proves you’ll be able to do all this? Which tools do you master that others don’t?

Give illustrations, give orders of magnitude, do periodic recaps in your talk. Never underestimate how quickly someone, even expert, can lose track of what you say. It is not the time to be subtle: write stuff in bold for members of the jury that will cross-read your research project while you’re talking.

Do not write too much maths, it’s very hard to parse in a short presentation. Show that you are able to popularize your research, to speak to many. Technicality is the enemy here, you’re not writing a research article, not even an abstract.

The oral

Timing

Presentations are short (15 min for CNRS section 7, 20 min for INRIA). There is no room for approximation or hesitation, your presentation must be meticulously prepared.

The usual plan is:

  • 1 slide of career summary (place and topics for PhD and Postdocs)
  • introduction to research field through selected past works (preferably 2)
  • research project
  • for CNRS, 1 slide integration details (1 min)
  • finish with 1 slide of recap that you leave displayed after the presentation, summarizing who you are (with a catchy, easy to remember project name that summarizes everything)

The research project is the critical part of the presentation and should take around two-thirds of it. On the contrary, the summary of your career and past works basically only serves to justify why you are the ideal person to carry on this research project. Be quick on these parts, don’t brag too much. See it as a stepping stone for the real deal: your project.

Practical organization of the audition and its consequences for you

For CNRS: the jury gathers for 1 week, hears roughly 35 candidates (7/8 per day). Then the weekend passes, and they meet for 3 other days to rank the candidates. That is a lot, and draining. When they meet on day 6, one week has passed since they heard the first candidates. They do take notes, but your goal is to help them remember you.

They should think: “ah yes, X is the girl who wanted to do this, with those tools, and had done Z in the past”. To achieve this, it should be very clear for you who you are as a researcher (which problems you like to work on, what is your methodology to solve them, do you like theory, or practice, or both? Do you want applications? How do you summarize your project in one sentence?). Then, your goal is to communicate that to the jury.

You need to tell a story, that should be reflected in your research project’s name. This is a step back on your research that is probably new, but mandatory.

Adapted content

The composition of the jury is very important. Out of roughly 15 persons, maybe 2 or 3 only work in your domain. Even more than in the research project (that, I think, many members of the jury do not read), it is very important that you speak to everybody, at least in the introduction and in your first 2-3 slides. You should aim at getting questions from more than 5 persons in the audience, not just the 2-3 experts.

In order not to lose the other members of the jury:

  • your introduction slides should speak to them in general terms
  • having applied examples and drawings helps
  • periodically (especially, when moving from past research to project), remind the audience what you’ve said, and where you’re going, in layman’s term

Various stuff:

  • avoid being too technical (heavy notation, figures that are hard to parse). It’s already very hard to convey information in 15 minutes, be simple.
  • avoid long sentences and heavy slides
  • it’s advised to have 1 slide per minute. If you have more, you probably have too much content.
  • using clear color codes help (e.g. using the same color for a variable in an equation and the corresponding words in the associated sentence); you can also use repeatedly the same color code (green/orange) for short term/long term goals, etc
  • pretty slides matter. It is not uncommon to spend 2 full weeks on them. Dry slides are to avoid at all costs, they lose the jury’s attention. Use colors, illustrations, boxes around important content.
  • have backup technical slides if you want

Rhythm

You should know your presentation by heart, especially because you’re likely to be stressed in the beginning. Then hopefully after a few seconds you get into the presentation; knowing the intro by heart helps getting through this hard phase. Be prepared to speak exactly for the amount of time you’re allowed; if it’s too short you missed an opportunity to convey messages, if it’s too long you may get interrupted and appear unprepared.

Do not try to force as much content as possible; on the contrary, taking breaks to summarize what you just said (e.g. after your past works, and before diving into your project) allows jury members outside of your area of expertise to catch up. Remember, you can lose them more easily than you think.

Questions

The questions session lasts between 15 min (CNRS) and 20 (INRIA). Answer precisely and concisely. It’s very frustrating for the jury to not be able to ask all their questions for lack of time, because the candidate took too much time to answer some questions. It’s OK to say that you don’t know, don’t lie.

Some frequent questions:

  • “if you got funding for a PhD student when you arrive, what would they work on?”. Show that you have thought about it, and that you have a plan (in particular, you don’t have HDR, so you’ll need to take cosupervise the student: with which team member?).
  • “what is THE big problem that you’d want to solve in the next 10 years” this is clearly related to the big question “who are you as a researcher?” whose answer must be crystal clear for you
  • “what is the work that you’re the most proud of”: show that you are able to evaluate your own work, to take a step back on it. It also defines who you are: you like this work because of its practical impact, because it solved a theoretical question, etc.
  • questions about your publication strategy: why this conference vs this journal, open access vs paywalled journal, etc. Show that you are aware of the various things at stake (open access research, publication delays, different audience) and that you did not just follow blindly what your advisors said. Show that you are ready to choose where to publish by yourself.

Learn the many implicit rules

This is a major reason why you should discuss with people: there are many little unwritten things to know. For example, the year I applied to CNRS, in addition to section 7, I applied for section 41, but in the “mathematics” track. Little did I know that, as an optimizer, I belonged to the “mathematics and applications” track (I did not get auditioned in section 41 :).

For some projects, it’s important to show that you handle applications; for some it’s not, etc. You can only learn of all this by talking to experienced people.

For CNRS, know the different culture and expectations between Sections 6, 7 and 41.

For INRIA, know that the jury likes to see that your project has an applicative part, and a methodological part.

Show that you have been proactive in your preparation: instead of saying “this is something I plan on discussing with member of X of the team”, do it before the audition. If, when you arrive in the team, you plan to go to somebody to obtain data, contact them before the oral. This way you can show the jury that you’ve already done stuff.

Thanks to all the people that gave me advice when I was preparing: R. Gribonval, P. Goncalves, A. Gramfort, J. Salmon, L. Rosasco, S. Villa, F. Iutzeler, A. Garivier, J. Malick, R. Flamary, O. Cappé, I. Waldspurger, S. Barthelmé, N. Le Bihan and apologies to anyone I forgot.

]]>Finding equivalent for sequences: the alpha trick2024-07-21T22:00:00+00:002024-07-21T22:00:00+00:00/blog/alpha_trickThe alpha trick is a very cute and efficient trick to find equivalent to sequences defined recursively as \(x_{k+1} = f(x_k)\).

When trying to find an equivalent to the sequence defined by \(x_{k+1} = f(x_k)\), proceed as follows:

  • first show that \(x_k\) converges, and find a transformation of \(x_k\) into \(u_k\) (e.g. \(u_k = 1 / x_k\), etc) such that \(u_k \to 0\)
  • then do a limited development of the recursive equation satisfied by \(u_k\) around 0
  • the key part: raise the limited development equation to a power \(\alpha\) such that \(u_{k+1}^\alpha - u_k^\alpha\) converges to a non zero limit, say \(\ell\):
\[u^\alpha_{k + 1} - u^\alpha_{k} = \ell + o(1)\]
  • apply Cesaro to obtain that \(\frac{\sum_{k=1}^n u^\alpha_{k + 1} - u^\alpha_{k}}{n} \to \ell\), which means that \(u_{k}^\alpha \sim \ell k\).
  • convert this back to an equivalent in \(x_k\)

First example: gradient descent on the exponential

The best application I know of this trick is to study the iterates of gradient descent on \(F: x \mapsto \exp(-x)\). My motivation to to study this was that a minimizer does not exist (\(F > 0\) but \(F\) goes to 0 at infinity), so the classical convergence rates results in \(\mathcal{O}(1/k)\) do not apply. I wondered if we could say anything about the convergence speed anyways?

Consider gradient descent iterates on \(f\), started at 0. We easily see that such iterates will remain positive; when restricted to \(\mathbb{R}_+\), \(f\) is 1-smooth, so we take a stepsize of 1. Our iterates are thus:

\[x_{k+1} = x_k - \exp(-x_k) \, .\]

Writing \(u_k = f(x_k) = \exp(-x_k)\), one obtains from the above equation: \(u_{k+1} = u_k \exp(-u_k)\), with \(u_0 = 1 > 0\).

The sequence \((u_k)\) remains positive, hence \(\exp(-u_k) \leq 1\) and \(u_k\) decreases. Being in addition lower bounded, \(u_k\) converges; the limit must be a fixed point: it is 0. Hence, by doing a limited development of \(u_{k} \exp(-u_k)\) around 0:

\(\begin{align} u_{k+1} &= u_k (1 - u_k + o(u_k)) \\ u_{k+1}^\alpha &= u_k^\alpha (1 - u_k + o(u_k))^\alpha \\ &= u_k^\alpha (1 - \alpha u_k + o(u_k))\\ &= u_k^\alpha - \alpha u_k^{\alpha + 1} + o(u_k^{\alpha + 1}) \end{align}\) We see the power of the trick here: by raising the equation to the power \(\alpha\), we will be able to transform the \(-\alpha u_k^{\alpha + 1}\) into a constant! Indeed, picking \(\alpha=-1\) allows us to obtain \(u_{k+1}^{-1} - u_k^{-1} \to 1\). Hence \(\frac{\sum_0^n u_{k+1}^{-1} - u_k^{-1}}{n+1} \to 1\), and \(u_k \sim 1/k\).

So we retain the convergence rate in \(1/k\), which I find very nice!

One may wonder at which speed \(x_k\) goes to infinity; here \(u_k \to 0\) so it stays bounded away from 1 at infinity, and thus we can compose the equivalent by the log function: \(\log u_k \sim \log(1/k)\), and since \(\log u_k = -x_k\), the GD iterates \(x_k\) are equivalent to \(\log k\).

Second example: Nesterov’s sequence

Let \(t_{k+1} = \frac{1 + \sqrt{1 + 4 t_k^2}}{2}\), the infamous sequence appearing in Nesterov’s original accelerated algorithm. Let’s use the alpha trick to find an equivalent to \(t_k\). One has \(t_{k+1} \geq t_k +1/2\) so we cannot hope for convergence; consider \(u_k = 1/t_k\) instead:

\[u_{k+1} = \frac{2}{1 + \sqrt{1 + 4 / u_k^2}}\]

SInce \(t_k \to + \infty\), \(u_k \to 0\), and

\[\begin{align} u_{k+1} &= \frac{2}{1 + \sqrt{1 + 4/u_k^2}} \\ &= \frac{2 u_k}{u_k + 2 \sqrt{u_k^2/4 + 1}} \\ % &= \frac{2 u_k}{u_k + 2 (1 + u_k^2/8 + o(u_k^2))} \\ &= \frac{u_k}{1 + u_k/2 + o(u_k)} \\ &= u_k (1 - u_k/2 + o(u_k)) \\ u_{k+1}^\alpha &= u_k^\alpha (1 - \alpha u_k/2 + o(u_k)) \\ u_{k+1}^\alpha &= u_k^\alpha - \alpha u_k^{\alpha + 1}/2 + o(u_k^{\alpha + 1}) \end{align}\]

Hence picking \(\alpha = -1\), \(u_{k+1}^{-1} - u_k^{-1} \to 1/2\) and so \(u_k \sim 2/k\) and finally \(t_k \sim k/2\).

]]>