1 Introduction

Many problems arising in mathematical modeling and machine learning can be casted as an optimization problem, i.e.,  \[\inf f(x) \quad \text{subject to} \quad x \in \Omega ,\] where \(f: \mathcal{X}\to \mathbb{V}\) is a function with adequate “regularity” defined on a space \(\mathcal{X}\), \(\Omega \subseteq \mathcal{X}\) the domain of the problem and \(\mathbb{V}\) is a space where the notion of infimum is well-defined, such as \(\mathbb{V}= \mathbb{R}\). Depending on the context, we may be interested in proving the existence of a minimizer, i.e., a point \(x^\star \in \Omega\) that achieves this infimum. In this case, we say that \(f(x^\star)\) is the minimum of the function \(f\) over \(\Omega\).

1.1 Examples of optimization problem

This previous definition is quite vague. It requires some precisions, and to do so, we instantiate it on several important examples.

1.1.1 Toy examples

It is common in optimization to test algorithms on function from \(\mathbb{R}^{2}\) to \(\mathbb{R}\) for the single reason that it is easy to visualize their behavior using contour plots, and that their geometry are rich enough in comparaison to the univariate case. Figure 1.1 illustrates two functions:

1. The Rosenbrock function, also known as the “banana function”, is a function commonly used in optimization. It was first introduced by Rosenbrock (1960) as a way to test the performance of optimization algorithms. \[f(x,y) = (1 - x)^{2} + a (y - x^{2})^{2} .\] A typical value is \(a = 100\). Note that for \(a>0\), the global minimizer \((1,1)\) is unique. If \(a=0\), every \((1,y)\) is a minimizer; if \(a<0\), the function is unbounded below.

2. A (positive) quadratic form, defined as \[f(x,y) = (x-1)^{2} + (y-1)^{2}.\]

1.1.2 Ordinary Least Squares

One of the fundamental model in statistics is the ordinary least squares (OLS) problem. Given a matrix \(A \in \mathbb{R}^{n \times d}\) and a vector \(b \in \mathbb{R}^n\), the OLS problem seeks a vector \(x \in \mathbb{R}^d\) that minimizes the sum of squared residues: \[\min_{x \in \mathbb{R}^d} \frac{1}{2} |\!| Ax - b |\!|^2,\] where \(|\!| \cdot |\!|\) denotes the Euclidean norm. This formulation arises naturally in linear regression, where \(A\) represents the data matrix (with each row corresponding to an observation and each column to a feature), and \(b\) is the vector of observed responses, typically modeled as \(b=Ax_0+\varepsilon\), where \(x_0\) is unknown and \(\varepsilon\) represents noise or model error. The identity \(b=Ax_0\) is the noiseless special case.

The solution to the OLS problem, when \(A\) has full column rank, is given by the unique minimizer \[x^\star = (A^\top A)^{-1}A^\top b.\] If \(A\) does not have full rank, the set of minimizers is generally infinite, and the minimum-norm solution can be found using the Moore-Penrose pseudoinverse: \[x^\star = A^{\dagger} b.\]

1.1.3 (Empirical) Risk minimization

Given a function \(\phi: X \to Y\), a loss function \(\ell: Y \times Y \to \mathbb{R}\) and a probability measure \(\mathbb{P}\) on \(X \times Y\), the expected risk of the prediction \(\phi\) is defined as \[\mathcal{R}(\phi) = \mathbb{E}_{(x,y) \sim \mathbb{P}} [\ell(y, \phi(x))] .\]

Minimizing expected risk over a chosen class \(\mathcal F\) leads to the optimization problem \[\inf_{\phi \in \mathcal{F}} \mathcal{R}(\phi) ,\]

Here \(\mathcal F\) is a class of measurable predictors, and the loss is assumed measurable with well-defined expected risks, for example a nonnegative measurable loss. A minimizer, if it exists, is optimal within \(\mathcal F\). A Bayes predictor minimizes expected risk over all admissible measurable predictors; a restricted class need not contain one. Under the assumptions stated here, the infimum need not be attained.

In practice, in supervised learning, we don’t have the knowledge of the distribution \(\mathbb{P}\) but only to a training data \((x_{i}, y_{i}) \in X \times Y\). The empirical risk is defined as \[\hat{\mathcal{R}}(\phi) = \frac{1}{n} \sum_{i=1}^{n} \ell(y_{i}, \phi(x_{i})) .\]

Of particular interest is the structural risk minimization – a term that trace back to Vapnik and Chervonenkis (1974) – where in additional to the loss function, we consider a regularization function \(J: \mathcal{F} \to \mathbb{R}\) that represents an apriori, and an hyperparameter \(\lambda > 0\) and we seek to solve the minimization problem \[\inf_{\phi \in \mathcal{F}} \hat{\mathcal{R}}(\phi) + \lambda J(\phi) .\]

The topic of supervised learning with an emphasis on optimization is well reviewed in the textbook of Bach (2021) and further theoretical insights are developped for instance in (Shalev-Shwartz and Ben-David 2014; Mohri et al. 2012).

1.1.4 Portfolio optimization

Consider \(n\) economic assets whose returns are denoted \((a_{1},\dots,a_{n})\). We assume to know the expected returns generated \(\mu = (\mu_{1},\dots,\mu_{n})\) and the risk in/between the assets defined by the covariance matrix \(\Sigma \in \mathbb{R}^{n \times n}\) (assumed to be positive definite). The problem of portfolio optimization with minimal variance \(\text{Var}(\langle x,\,a\rangle)\) and expected return \(r\) is to find a \[\inf_{x \in \mathbb{R}^{n}} f(x) \overset{\mathrm{def.}}{=}\langle \Sigma x,\,x\rangle \quad \text{subject to} \quad \begin{cases} \mathbb{E}(\langle x,\,a\rangle) = \langle x,\,\mu\rangle = r \\ \langle x,\,\mathbf{1}\rangle = 1 . \end{cases}\] The foundational paper on modern portfolio theory was written by Markowitz (1952).

1.1.5 Optimal control

Given a dynamical system drive by the ordinary differential equation \[\dot x(t) = g(t, x(t), u(t)), \quad x(t_{0}) = x_{0} , \tag{1.1}\]

where

• \(I\) is an interval of \(\mathbb{R}\), \(U \subseteq \mathbb{R}^{n}\) open, \(V \subseteq \mathbb{R}^{m}\) open;

• \((t_{0}, x_{0}) \in I \times V\) are the initial conditions;

• \(g: I \times V \times U {\to\mathbb{R}^{m}}\) is \(C^{1}\).

For the remainder of this example, take \(t_0=0\) and let \(I\) be an open interval containing \(0\). Let \(\mathcal{U}\) consist of measurable controls \(u:I\to U\subseteq\mathbb{R}^n\) such that, on every compact interval \(J\subset I\), the values of \(u\) lie almost everywhere in some compact set \(K_J\subset U\). In particular, these controls belong to \(L_{\mathrm{loc}}^\infty(I,\mathbb{R}^n)\). The compact-range condition and \(g\in C^1\) give local integrable bounds and local Lipschitz bounds in the state variable. Thus each \(u\in\mathcal{U}\) has a unique maximal forward, locally absolutely continuous solution \(x_u:[0,t_e(u))\to V\), satisfying the differential equation almost everywhere and the initial condition at \(0\).

For \(T>0\) with \([0,T]\subset I\), let \(\mathcal{U}_T\) be the controls in \(\mathcal{U}\) whose solution exists and remains in \(V\) on \([0,T]\). Mere local boundedness in the ambient space, without suitable bounds inside \(U\), does not suffice for the preceding existence claim.

Given a cost function \(c: I \times V \times U {\to\mathbb{R}}\), assumed continuous so the cost integral is finite for each \(u\in\mathcal{U}_T\), the fixed-time optimal control problem is defined as \[\inf_{u \in \mathcal{U}} f(u) \overset{\mathrm{def.}}{=} \int_{0}^{T} c(t, x_{u}(t), u(t)) dt \quad \text{subject to} \quad u \in \mathcal{U}_{T} .\] Besides its natural applications in engineering (including aerospatial), optimal control is at the heart of reinforcement learning. For a comprehensive treatment of optimal control, see the lecture notes of Evans (1983) and for application to reinforcement learning, see (Recht 2019).

1.1.6 Optimal transport

Let \(X,Y\) two metric spaces, \(\mu\) a Borel measure on \(X\) and \(T: X \to Y\) a measurable map. We recall that the push-forward of \(\mu\) by \(T\) is the measure \(T_{\sharp}\mu\) defined by the change-of-variable formula \[\int_{Y} \phi(y) dT_{\sharp}\mu(y) = \int_{X} \phi(T(x)) d \mu(x) ,\] for all measurable and bounded \(\phi: Y \to R\).

For a given probability measures \(\mu\) and \(\nu\), and a cost function \(c: X \times Y \to \overline{\mathbb{R}}\), the Monge problem of optimal transportation is defined as \[\inf_{T: X \to Y} f(T) = \int_{X} c(x,T(x)) d \mu(x) \quad \text{subject to} \quad T_{\sharp}\mu = \nu . \tag{1.2}\]

The optimization is over measurable maps and may be infinite-dimensional, but need not be when the spaces are finite. For maps into a Euclidean space, the push-forward constraint is generally nonconvex, whereas convexity of the objective depends on the cost. For arbitrary metric spaces, convexity of maps requires additional linear structure; these claims are not universal. For more information, see (Villani 2009) for theoretical and (Peyré and Cuturi 2018) for computational applications (including machine learning).

1.2 Basic facts on optimization

Suppose we have an constrained minimization problem on \(\Omega \subseteq \mathbb{R}^{d}\) as follows \[\inf_{x \in \Omega} f(x) .\] Such problem may not have a solution. For instance, if \(\Omega = \mathbb{R}\), then \(f(x) = x\) is not lower-bounded, and the infimum is \(-\infty\). But a function may be lower-bounded, for instance \(f(x) = 1/x\) on \(\Omega = \mathbb{R}_{>0}\), but the infimum is never achieved.

The following theorem shows that if \(f\) is continuous, and that we consider a non-empty compact – that is bounded and closed – subset of \(\mathbb{R}^{d}\), then the infimum is a minimum.

Unfortunately, this argument cannot work anymore for unconstrained minimization when \(\Omega = \mathbb{R}^{d}\). The trick is to replace the compactness assumption on \(\Omega\) by the (0-)coercivity of \(f\), i.e.,  \[\lim_{|\!| x |\!| \to +\infty} f(x) = +\infty .\] In this case, we have the following result

It is possible to generalize this result by asking merely that \(f\) is lower semicontinuous. These two results (theorem 1.1, theorem 1.2) can be generalized in (infinite dimensional) Hilbert spaces with some additional hypotheses. See the references for more insights in this setting.

Let us finish this section by introducing our usual suspect all through these lecture notes.

2 Basic facts in convex analysis

Despite the fact that many problems encountered in statistics and machine learning are not convex, convexity is an important to study in order to understand what is the generic behaviour of an optimization procedure.

2.1 Convex set

A set \(C \subseteq \mathcal{X}\) is convex if every segment of \(C\) is contained in \(C\). It can formalized as the following definition and illustrated in figure 2.2.

One can observe that it is possible to replace \(\lambda \in [0,1]\) by \(\lambda \in (0,1)\) without any difference (see Exercise 1). In most of these lecture notes, the set \(\mathcal{X}\) will the Euclidean space \(\mathbb{R}^{p}\), but remark that this notion only requires that \(\mathcal{X}\) is a vector space over the real numbers \(\mathbb{R}\), possibly of infinite dimension.

The following proposition reviews some important algebraic properties of convex sets.

2.2 Convex functions

2.2.1 Definition

The intuitive definition of a convex function is a function such that “the line joining \(f(x)\) and \(f(y)\) lies above the graph of \(f\) between \(x\) and \(y\)”. Formally, this definition makes sense using the definition of the epigraph of \(f\).

Equipped with this notion, we can define convex functions using the definition of convex sets (definition 2.1) applied to the epigraph (definition 2.2).

It is possible to make explicit this definition as follow: \[\forall x,y \in \mathop{\mathrm{dom}}f,\, \forall \lambda \in [0,1], \qquad f(\lambda x + (1-\lambda) y) \leq \lambda f(x) + (1-\lambda) f(y) . \tag{2.1}\]

Following the convention introduced in (Hiriart-Urruty and Lemarechal 1996), we will denote the set of all convex functions of \(\mathbb{R}^{p}\) as \(\mathop{\mathrm{Conv}}\mathbb{R}^{p}\), and the set of closed – that is, with closed epigraph – convex functions of \(\mathbb{R}^{p}\) as \(\overline{\mathop{\mathrm{Conv}}}\mathbb{R}^{p}\).

2.2.2 Local and global minima of a convex function

For a (un)constrained minimization problem on a convex set \(C \subseteq \mathop{\mathrm{dom}}f\) with a function \(f : C \to \mathbb{R}\), \[\min_{x \in C} f(x) , \tag{2.2}\]

we say that \(x^\star\) is

• a local solution of (2.2), if there exists a neighborhood \(\mathcal{O}\) of \(x^\star\) such that for all \(x \in C \cap \mathcal{O}\), \(f(x) \geq f(x^\star)\) ;

• a (global) solution if for all \(x \in C\), \(f(x) \geq f(x^\star)\).

A fundamental result is that every local solution of (2.2) is global solution, and the set of (global) solutions is a convex set.

2.2.3 Differentiable properties of a convex function

A differentiable function is convex if, and only if, it is above all its tangents, see figure 2.4.

A similar criterion can be derived for strictly convex function¹.

For an unconstrained local minimizer in an open domain, \(\nabla f(x^\star)=0\), and (2.3) directly gives global optimality. For minimization over a convex set \(C\), local minimality instead implies \[\langle \nabla f(x^\star),\,x-x^\star\rangle\geq0\qquad\forall x\in C,\] by taking feasible one-sided derivatives along segments. Together with (2.3), this also proves global optimality. The gradient need not vanish at a constrained minimizer: for \(f(x)=x\) on \([0,1]\), the minimizer is \(0\) and \(f'(0)=1\).

2.3 Strongly convex functions

Throughout this section, \(\|\cdot\|\) denotes the Euclidean norm. In particular, the equivalence with subtraction of \(\frac\mu2\|x\|^2\) and the Hessian criterion below use this norm.

A strongly convex function satisfies a stronger statement of the Jensen’s inequality.

Another way to define strongly convex function is to say that \(f\) is \(\mu\)-strongly convex if the function \[x \mapsto f(x) - \frac{\mu}{2} |\!| x |\!|^{2}\] is a convex function.

A finite-valued strongly convex function \(f:\mathbb{R}^d\to\mathbb{R}\) has a unique minimizer. More generally, the conclusion holds for a proper (never \(-\infty\) and finite at some point) lower semicontinuous strongly convex function. Strong convexity alone does not imply attainment for an arbitrary extended-valued function; for example, \(f(x)=x^2\) on \(x>0\) and \(f(x)=+\infty\) otherwise has infimum \(0\) but no minimizer.

For the finite-valued full-domain case, set \(h(x)=f(x)-\frac\mu2\|x\|^2\). This is finite-valued and convex, so it is continuous and has a supporting affine function at \(0\): for some \(p\in\mathbb{R}^d\), \(h(x)\geq h(0)+\langle p,\,x\rangle\). Consequently, \[f(x)\geq f(0)+\langle p,\,x\rangle+\frac\mu2\|x\|^2\longrightarrow+\infty \quad\text{as }\|x\|\longrightarrow+\infty.\] Thus \(f\) is continuous and coercive, and theorem 1.2 ensures existence. Strong convexity implies strict convexity on the effective domain, so two distinct minimizers are impossible. Strict convexity alone gives at most one minimizer, rather than existence.

For a differentiable function on a convex domain, strong convexity is equivalent to a uniform quadratic lower bound above every tangent, with the same modulus \(\mu>0\) for all pairs of points, as in the following proposition. At the point of tangency the bound is an equality. This is a key property to defined lower bounds on \(F\).

Note that if \(f\) is \(C^{2}\), \(f\) is \(\mu\)-strongly convex if, and only if, its hessian \(\nabla^{2} f \geq \mu \mathop{\mathrm{Id}}\).

An easy consequence is that it is possible to establish a lower bound on the values based on a lower bound on the distance between points: \[f(x) \geq f(x^{\star}) + \frac{\mu}{2}\|x - x^{\star} \|^{2} , \tag{2.4}\]

where \(x^{\star}\) is the minimizer of \(f\).

2.4 Lipschitz continuity of the gradient

Here \(\| \cdot \|_{*}\) is the dual norm of \(\|\cdot\|\).

2.4.1 Quadratic upper-bound

\(L\)-smoothness is important in optimization because it ensures a quadratic upper bound of the function.

This result has a strong implication on the minima of \(f\).

2.4.2 Co-coercivity of the gradient of a \(L\)-smooth function

Convex \(L\)-smooth functions are co-coercive. This result is sometimes coined Baillon–Haddad theorem (Baillon and Haddad 1977), but one shall note that the original contributionis much more general than its application to the gradient.

It turns out that smoothness, co-coercivity and upper quadratic boundness are equivalent properties for full-domain convex function.

For the special case of the \(\ell^{2}\) norm, we have:

When \(f\) is both \(L\)-smooth and \(\mu\)-strongly convex, we have the following “strenghened” co-coercivity.

2.5 Exercises

References