Proof. Lipschitz continuity implies continuity, so a minimizer \(x^\star\) exists on the compact cube; a grid minimizer exists because the grid is finite and nonempty. For each coordinate, define \[z_i=\varepsilon\,\operatorname{sign}(x_i^\star) \left\lfloor\frac{|x_i^\star|}{\varepsilon}\right\rfloor, \qquad i=1,\dots,d,\] with \(\operatorname{sign}(0)=0\). Since \(|x_i^\star|\leq 1\), we have \(z\in G_\varepsilon^d\) and \(|z_i-x_i^\star|<\varepsilon\) for every \(i\). Consequently, \(|\!| z-x^\star |\!|\leq\sqrt{d}\,\varepsilon\). By global optimality of \(x^\star\), grid optimality of \(\hat x\), and Lipschitz continuity, \[0\leq f(\hat x)-f(x^\star) \leq f(z)-f(x^\star) \leq L|\!| z-x^\star |\!| \leq L\sqrt{d}\,\varepsilon.\] ◻

Proof. Let \(u \in \mathbb{R}^{d}\) such that \(|\!| u |\!| = 1\). Consider the function \(\phi_{u}: \mathbb{R}\to \mathbb{R}\) defined by \[\phi_{u}(t) = f(\bar x + t u).\] The function \(\phi_u\) is differentiable at \(0\) and its derivative reads \(\phi_{u}'(0) = \langle \nabla f(\bar x),\,u\rangle\). Denoting \(\theta\) the angle between \(\nabla f(\bar x)\) and \(u\), we have that \(\phi_{u}'(0) = |\!| \nabla f(\bar x) |\!| \cos \theta\). Hence, minimizing \(\langle \nabla f(\bar x),\,u\rangle\) is equivalent to finding the minimum of \(\cos \theta\), that is \(\theta = (2k+1) \pi\) for some \(k \in \mathbb{Z}\). Thus, we have \[u^{\star} = - \frac{\nabla f(\bar x)}{|\!| \nabla f(\bar x) |\!|} \quad \text{ and } \quad \langle \nabla f(\bar x),\,u^\star\rangle = - |\!| \nabla f(\bar x) |\!|.\] ◻

Proof. Using the expression of the gradient and (GD), we have for any minimizer \(x^{\star}\) \[\begin{align*} x^{(t+1)} &= x^{(t)} - \frac{\eta}{n} A^{\top}(A x^{(t)} - y) \\ &= x^{(t)} - \eta H (x^{(t)} - x^{\star}) . \end{align*}\] Substracting \(x^{\star}\) from both side gives \[x^{(t+1)} - x^{\star} = (I - \eta H)(x^{(t)} - x^{\star}) .\] This identity is linear recursion, and unrolling it leads to the result. ◻

Proof. Since there is a unique minimizer \(x^{\star}\), we have \(\mu > 0\). Observe that (using a spectral norm bound) \[\langle x^{(0)} - x^{\star} , (I - \eta H)^{2t} (x^{(0)} - x^{\star}) \rangle \leq \left( \max_{\lambda \in [\mu,L]} | 1 - \eta \lambda | \right)^{2t} \| x^{(0)} - x^{\star} \|_{2}^{2} .\]

Observe that \(\max_{\lambda \in [\mu,L]} | 1 - \eta \lambda |\) is minimized for \(\eta = \frac{2}{\mu + L}\), and value is \(\frac{\kappa - 1}{\kappa + 1} \in [0,1)\). Indeed, \[\max_{\lambda \in [\mu,L]} | 1 - \eta \lambda | = \max \left( | 1 - \eta \mu |, | 1 - \eta L | \right) .\]

If \(\mu=L\), this maximum is \(|1-\eta L|\), minimized at \(\eta=1/L\). For the following equivalences, assume \(\mu<L\).

Hence, \[\min_{\eta > 0} \max_{\lambda \in [\mu,L]} | 1 - \eta \lambda | = \min_{\eta > 0} \max \left( | 1 - \eta \mu |, | 1 - \eta L | \right) .\] Moreover, we have the following set of equivalence \[\begin{align*} | 1 - \eta L | &\leq | 1 - \eta \mu | \\ (1 - \eta L)^{2} &\leq (1 - \eta \mu)^{2} \\ \eta L^{2} - 2L &\leq \eta \mu^{2} - 2 \mu \\ \eta &\leq \frac{2}{L + \mu} . \end{align*}\] This minima is achieved when the two curve \(| 1 - \eta L |\) and \(| 1 - \eta \mu |\) intersect. ◻

Proof. Observe that \(f\) has an exact Taylor expansion of order 2. Indeed, for any \(x \in \mathbb{R}^{p}\) and any minimizer \(x^{\star}\): \[\begin{align} f(x) - f(x^{\star}) &= \langle \nabla f(x^{\star}), x - x^{\star} \rangle + \frac{1}{2} \langle x - x^{\star}, H (x - x^{\star}) \rangle \nonumber \\ &= \frac{1}{2} \langle x - x^{\star}, H (x - x^{\star}) \rangle . \tag{1.3} \end{align}\] Using the (exact) Taylor expansion (1.3) of \(f\), we have \[f(x^{(t)}) - f(x^\star) = \frac{1}{2} \langle x^{(t)} - x^\star, H (x^{(t)} - x^{\star}) \rangle .\] Using (1.2), we have \[f(x^{(t)}) - f(x^\star) = \frac{1}{2} \langle (I - \eta H)^{t} (x^{(0)} - x^{\star}) , H (I - \eta H)^{t} (x^{(0)} - x^{\star}) \rangle .\] Since \((I - \eta H)\) is symmetric, we have \[f(x^{(t)}) - f(x^\star) = \frac{1}{2} \langle x^{(0)} - x^{\star} , (I - \eta H)^{2t} H (x^{(0)} - x^{\star}) \rangle .\] Using the fact that for a symmetric positive semidefinite matrix \(B\), we have \(\langle B w,w\rangle\leq\|B\|_{sp}\|w\|^2\), with \(B=(I-\eta H)^{2t}\) and \(w=H^{1/2}(x^{(0)}-x^\star)\), and that \(B\) commutes with \(H\), we have \[f(x^{(t)}) - f(x^\star) \leq \left\| (I - \eta H)^{2t} \right\| \frac{1}{2} \langle x^{(0)} - x^{\star} , H (x^{(0)} - x^{\star}) \rangle .\] Using again (1.3), we have \[f(x^{(t)}) - f(x^\star) \leq \left\| (I - \eta H)^{2t} \right\| (f(x^{(0)}) - f(x^\star)) .\]

Case \(\mu > 0\). Now, we have according to the previous section, the following linear rate. \[f(x^{(t)}) - f(x^\star) \leq \left( 1 - \frac{1}{\kappa} \right)^{2t} (f(x^{(0)}) - f(x^\star)) .\]

Case \(\mu = 0\). Let’s study the eigenvalues of¹ \((I - \eta H)^{2t} H\). We keep our analysis restricted to \(0<\eta \leq \frac{1}{L}\). \[\begin{align*} | \lambda (1 - \eta \lambda)^{2t} | &\leq \lambda \exp(-\eta \lambda)^{2t} \\ &= \lambda \exp(-2t\eta\lambda) \\ &= \frac{1}{2t\eta} 2t\eta\lambda \exp(-2t\eta\lambda) & \text{insert the term } 2t\eta \\ &\leq \frac{1}{2t\eta} \sup_{\rho \geq 0} \rho \exp(-\rho) & \text{crude bound} \\ &= \frac{1}{2et\eta} & \text{maximum at } \rho = 1 \\ &\leq \frac{1}{4t\eta} & e > 2 . \end{align*}\] Hence, we obtain the claimed \(O(1/t)\) rate. ◻

Proof. (i) The value sequence is (strictly)decreasing (unless it reaches a critical point). We recall that the oracle GD chose the step-size \(\eta^{(t)}\) as \[\begin{gathered}\phi^{(t)}(\eta) \overset{\mathrm{def.}}{=}f(x^{(t)}-\eta\nabla f(x^{(t)})),\qquad \eta^{(t)}\in\underset{\eta\geq 0}{\mathop{\mathrm{argmin}}}\;\phi^{(t)}(\eta) . \end{gathered}\]

If \(\nabla f(x^{(t)})\neq 0\), coercivity implies \(\phi^{(t)}(\eta)\to+\infty\) as \(\eta\to+\infty\), so the oracle step exists. If the gradient is zero, the line-search objective is constant and the iterate stays unchanged.

Hence, evaluating \(\phi^{(t)}\) at \(\eta^{(t)}\) and 0 leads to

\[f(x^{(t+1)}) = \phi^{(t)}(\eta^{(t)}) {\,\leq\,} \phi^{(t)}(0) = f(x^{(t)}) .\] Suppose that \(f(x^{(t+1)}) = f(x^{(t)})\). Thus, 0 is a minimizer of \(\phi^{(t)}\) and therefore its one-sided first-order condition reads \(\frac{d}{d \eta}\phi^{(t)}(0) \geq 0\). Given \(\eta \geq 0\), we have \[\frac{d}{d \eta}\phi^{(t)}(\eta) = \langle -\nabla f(x^{(t)}),\,\nabla f (x^{(t)} - \eta \nabla f(x^{(t)}))\rangle .\] Evaluating this expression at \(\eta = 0\) gives us \(- |\!| \nabla f(x^{(t)}) |\!|^2\geq 0\), hence \(\nabla f(x^{(t)}) = 0\) that means \(x^{(t)}\) is a critical point.

(ii) The iterates sequence is converging (up to a subsequence). Consider the set \(S = \left\{ x \in \mathbb{R}^{d} \;:\; f(x) \leq f(x^{(0)}) \right\}\). Remark that:

• \(f\) is continuous, thus \(S\) is closed;

• \(f\) is coercive, thus \(S\) is bounded.

Hence, \(S\) is compact. Observe that for all \(t \geq 0\), \(x^{(t)} \in S\). Using Bolzano–Weierstrass theorem, we get that the sequence \((x^{(t)})\) has a convergent subsequence towards some \(x^{*}\). Let us denote it by \((x^{(\psi(t))})_{t \geq 0}\) where \(\psi : \mathbb{N}\to \mathbb{N}\) is increasing.

The value sequence is decreasing and bounded below, so it converges to \(\ell\). By continuity, \(\ell=f(x^*)\).

(iii) The limit of the iterates sequence is a critical point. Suppose that \(x^{*}\) is not a critical point. Using (i), we now that \(x^{**}\) defined as \(x^{**} = x^{*} - \eta^{*} \nabla f(x^{*}) \neq x^{*}\), where \(\eta^{*} > 0\) is the oracle step, is such that \(f(x^{**}) < f(x^{*})\). Consider the sequence \(y^{(t)} = x^{(\psi(t))}\) converging to \(x^{*}\). Remark that, using the continuity of the map \(z \mapsto z - \eta^{*} \nabla f(z)\) for a contiously differentiable function \(f\), we have \[\lim_{t \to +\infty} \left( y^{(t)} - \eta^{*} \nabla f(y^{(t)}) \right) = x^{*} - \eta^{*} \nabla f(x^{*}) = x^{**} . \tag{1.4}\]

By definition of \(\phi^{(t)}\), we also have for all \(t \geq 0\) that \[\phi^{(\psi(t))}(\eta^{*}) \geq \phi^{(\psi(t))}(\eta^{(\psi(t))}) \geq \ell=f(x^{*}).\] Hence, \[f(y^{(t)} - \eta^{*} \nabla f(y^{(t)})) \geq f(x^{*})\] Using (1.4), we get that \(f(x^{**}) \geq f(x^{*})\), that is a contradiction. In conclusion, \(x^{*}\) is a critical point.

The same argument applies to any convergent subsequence. ◻

Proof. See Exercise 2. ◻

Proof. (i) Bound on one step of gradient descent. For \(x \in \mathbb{R}^{d}\), let \(y = x - \eta \nabla f(x)\). Using proposition A.2, we have that \[\begin{align*} f(y) &\leq f(x) + \langle \nabla f(x),\,y - x\rangle + \frac{L}{2} |\!| y - x |\!|^{2} \\ &=f(x) + \langle \nabla f(x),\,-\eta \nabla f(x)\rangle + \frac{L}{2} |\!| \eta \nabla f(x) |\!|^{2} \\ &=f(x) - \eta |\!| \nabla f(x) |\!|^{2} + \frac{L \eta^{2}}{2} |\!| \nabla f(x) |\!|^{2} \\ &=f(x) - \eta(1 - \frac{L\eta}{2}) |\!| \nabla f(x) |\!|^{2} . \end{align*}\] To ensure that \(f(y) \leq f(x)\), we need to require that \(\eta(1 - \frac{L\eta}{2}) \geq 0\) that is \(0\leq\eta\leq\frac{2}{L}\). We can parameterize \(\eta\) as \(\eta = \frac{2 \alpha}{L}\) with \(\alpha \in (0,1)\). Hence, we get the descent \[f(x) - f(y) \geq \frac{2 \alpha(1-\alpha)}{L} |\!| \nabla f(x) |\!|^{2} . \tag{1.5}\]

(ii) Convergence of 1st-order optimality. Applying (1.5) to \(x = x^{(t)}\) and \(y = x^{(t+1)}\), we get that \[f(x^{(t)}) - f(x^{(t+1)}) \geq \frac{2 \alpha(1-\alpha)}{L} |\!| \nabla f(x^{(t)}) |\!|^{2} . \tag{1.6}\]

Summing inequality (1.6) for \(t = 0 \dots T\), we obtain \[\begin{equation*} \sum_{t=0}^{T} f(x^{(t)}) - f(x^{(t+1)}) \geq \frac{2 \alpha(1-\alpha)}{L} \sum_{t=0}^{T }|\!| \nabla f(x^{(t)}) |\!|^{2} . \end{equation*}\] Observing that the l.h.s. telescopes, we have \[\frac{2 \alpha(1-\alpha)}{L} \sum_{t=0}^{T }|\!| \nabla f(x^{(t)}) |\!|^{2} \leq f(x^{(0)}) - f(x^{(T+1)}) \leq f(x^{(0)}) - \bar f , \tag{1.7}\]

by definition of \(\bar f\). Since (1.7) is true for all \(T \in \mathbb{N}\), we get \(\nabla f(x^{(t)})\) converges towards 0.

(iii) Rate of convergence. Using the fact that for all \(t = 0 \dots T\), we have \[\min_{0\leq s\leq T}|\!| \nabla f(x^{(s)}) |\!|\leq|\!| \nabla f(x^{(t)}) |\!|\] for each \(0\leq t\leq T\), we have from (1.7) that \[\frac{2 \alpha(1-\alpha)}{L} (T+1) \min_{0 \leq t \leq T} |\!| \nabla f(x^{(t)}) |\!|^{2} \leq f(x^{(0)}) - \bar f ,\] Hence, the claimed result. ◻

Proof. Let \(x^{\star}\) a minimizer of \(f\).

If \(f(x^{(0)})=f(x^\star)\), the iterates are stationary and the bounds are understood as zero. Otherwise \(r^{(0)}>0\). In the following divisions, assume \(\delta^{(t+1)}>0\); if an iterate is a minimizer, all later iterates stay there and the claimed bounds are immediate.

We consider \(r^{(t)} = |\!| x^{(t)} - x^{\star} |\!|\) the lack of optimality². We have \[\begin{align*} (r^{(t+1)})^{2} &= |\!| x^{(t)} - x^{\star} - \eta \nabla f(x^{(t)}) |\!|^{2} \\ &= (r^{(t)})^{2} - 2\eta \langle \nabla f(x^{(t)}),\,x^{(t)} - x^{\star}\rangle + \eta^{2} |\!| \nabla f(x^{(t)}) |\!|^{2} \end{align*}\] Using the co-coercivity of the gradient (proposition A.3), we have that \[\begin{equation*} \langle \nabla f(x^{(t)}) - \underbrace{\nabla f(x^{\star})}_{=0},\,x^{(t)} - x^{\star}\rangle \geq \frac{1}{L} |\!| \nabla f(x^{(t)}) - \underbrace{\nabla f(x^{\star})}_{=0} |\!|^{2} . \end{equation*}\] Thus, \[(r^{(t+1)})^{2} \leq (r^{(t)})^{2} - \eta(\frac{2}{L} - \eta) |\!| \nabla f(x^{(t)}) |\!|^{2} .\] In particular, we get that \(r^{(t)} \leq r^{(0)}\) for all \(t \geq 0\).

Using the smoothness of \(f\), proposition A.2 gives us the bound \[\begin{align*} f(x^{(t+1)}) &\leq f(x^{(t)}) + \langle \nabla f(x^{(t)}),\,x^{(t+1)} - x^{(t)}\rangle + \frac{L}{2} |\!| x^{(t+1)} - x^{(t)} |\!|^{2} \end{align*}\] Using the gradient descent update, we have that \[\begin{equation*} f(x^{(t+1)}) \leq f(x^{(t)}) - \eta(1 - \frac{L\eta}{2}) |\!| \nabla f(x^{(t)}) |\!|^{2} . \end{equation*}\]

Now, let \(\delta^{(t)} = f(x^{(t)}) - f(x^{\star})\). Using the differential characterization of convexity (proposition A.1), we have \[\delta^{(t)} \leq \langle \nabla f(x^{(t)}),\,x^{(t)} - x^{\star}\rangle .\] Using Cauchy–Schwarz inequality, we have \[\delta^{(t)} \leq |\!| \nabla f(x^{(t)}) |\!| |\!| x^{(t)} - x^{\star} |\!| = r^{(t)} |\!| \nabla f(x^{(t)}) |\!| \leq r^{(0)} |\!| \nabla f(x^{(t)}) |\!| .\]

Hence, we have \[\delta^{(t+1)} \leq \delta^{(t)} - \frac{q}{(r^{(0)})^{2}} (\delta^{(t)})^{2} ,\] where \(q = \eta(1 - \frac{L\eta}{2})\). Multiplying both side by \(\frac{1}{\delta^{(t)}\delta^{(t+1)}} > 0\), we get that \[\frac{1}{\delta^{(t)}} \leq \frac{1}{\delta^{(t+1)}} - \frac{q}{(r^{(0)})^{2}} \frac{\delta^{(t)}}{\delta^{(t+1)}} .\] Since \((f(x^{(t)}))\) is nonincreasing, we have \(\frac{\delta^{(t)}}{\delta^{(t+1)}} \geq 1\), hence \[\frac{1}{\delta^{(t+1)}} \geq \frac{1}{\delta^{(t)}} + \frac{q}{(r^{(0)})^{2}} .\] Summing \(T\) inequalities of this type, we obtain \[\frac{1}{\delta^{(T)}} \geq \frac{1}{\delta^{(0)}} + T \frac{q}{(r^{(0)})^{2}} .\] We conclude using a bit of computation: \[\begin{equation*} \delta^{(T)} \leq \left(\frac{1}{\delta^{(0)}} + T \frac{q}{(r^{(0)})^{2}} \right)^{-1} = \left( \frac{(r^{(0)})^{2} + T q \delta^{(0)}}{\delta^{(0)}(r^{(0)})^{2}} \right)^{-1} = \frac{\delta^{(0)}(r^{(0)})^{2}}{(r^{(0)})^{2} + T q \delta^{(0)}} \end{equation*}\] Replacing \(q\), \(r^{(0)}\) and \(\delta^{(0)}\) by their expression gives the result.

Maximizing the descent \(\eta(2 - L\eta)\) gives the optimal step-size \(\eta = \frac{1}{L}\). Injecting this value, we get that \[\delta^{(T)} \leq \frac{2L\delta^{(0)}(r^{(0)})^{2}}{2L(r^{(0)})^{2} + T\delta^{(0)}} .\] Using the smoothness (proposition A.2), we have \[f(x^{(0)}) \leq f^{\star} + \langle \nabla f(x^{\star}),\,x^{(0)} - x^{\star}\rangle + \frac{L}{2} (r^{(0)})^{2} = f^{\star} + \frac{L}{2} (r^{(0)})^{2}\] Thus, \[2L (r^{(0)})^{2} + T \delta^{(0)} \geq (4 + T) \delta^{(0)} ,\] which is the claimed result.

Finally, the nonincreasing distances to any minimizer make the iterates bounded. A convergent subsequence exists, and the value bound and continuity show that its limit \(\bar x\) is a minimizer. Since \(\|x^{(t)}-\bar x\|\) is nonincreasing and tends to zero along that subsequence, the whole sequence converges to \(\bar x\). ◻

Proof. Let \(r^{(t)} = |\!| x^{(t)} - x^{\star} |\!|\). We start like the proof of proposition 1.7: \[\begin{align*} (r^{(t+1)})^{2} &= |\!| x^{(t)} - x^{\star} - \eta \nabla f(x^{(t)}) |\!|^{2} \\ &= (r^{(t)})^{2} - 2\eta \langle \nabla f(x^{(t)}),\,x^{(t)} - x^{\star}\rangle + \eta^{2} |\!| \nabla f(x^{(t)}) |\!|^{2} \end{align*}\] Using proposition A.5, we have \[\langle \nabla f(x^{(t)}) - \nabla f(x^{\star}),\,x^{(t)} - x^{\star}\rangle \geq \frac{\mu L }{\mu + L} |\!| x^{(t)}-x^{\star} |\!|^{2} + \frac{1}{\mu + L} |\!| \nabla f(x^{(t)}) - \nabla f(x^{\star}) |\!|^{2} ,\] that is \[\langle \nabla f(x^{(t)}) - \nabla f(x^{\star}),\,x^{(t)} - x^{\star}\rangle \geq \frac{\mu L }{\mu + L} (r^{(t)})^{2} + \frac{1}{\mu + L}|\!| \nabla f(x^{(t)}) |\!|^{2} .\] Hence, \[\begin{align*} (r^{(t+1)})^{2} &\leq \left( 1 - \frac{2 \eta \mu L}{\mu + L} \right) (r^{(t)})^{2} + \eta \left( \eta - \frac{2}{\mu + L} \right) |\!| \nabla f(x^{(t)}) |\!|^{2} . \end{align*}\] Since \(\eta - \frac{2}{\mu + L} \leq 0\), we can drop it and we obtain the recursion \((r^{(t+1)})^{2} \leq (1-q) (r^{(t)})^{2}\) with \(q = \frac{2 \eta \mu L}{\mu + L}\), that proves the first claim (linear rate).

Let \(\eta = \frac{2}{\mu + L}\). We have \[1 - q = 1 - \frac{4\mu L}{(\mu+L)^{2}} = \frac{(L - \mu)^{2}}{(L+\mu)^{2}} = \left( \frac{K_{f} - 1}{K_{f} + 1} \right)^{2}.\] The last inequality is yet another use of proposition A.2. ◻

Proof. We drop the exponents \(L,\mu\) in the definition of \(f_{k} = f_{k}^{L,\mu}\). The function \(f_{k}\) being a quadratic form, it is \(C^{\infty}\) and its partial derivatives read \[\frac{\partial^2 f_{k}}{\partial x_{i} \partial x_{j}}(x) = \mu 1_{ \{ i=j \} } + \frac{L - \mu}{4} \begin{cases} 2 & \text{if } i = j \leq k \\ - 1 & \text{if } j = i-1 \text{ and } 1 < i \leq k \\ - 1 & \text{if } j = i+1 \text{ and } 1 \leq i < k \\ 0 & \text{otherwise.} \end{cases}\] Thus, the Hessian matrix is given (for any \(x \in \mathbb{R}^{d}\)) by \[\nabla^{2} f_{k}(x) = \mu \mathop{\mathrm{Id}}_{d} + \frac{L-\mu}{4} L_{k} ,\] where \(L_{k}\) is a (thresholded) discrete Laplacian operator with Dirichlet boundary conditions that is tridiagonal \[L_{k} = \left( \begin{array}{ccccc|c} 2 & -1 & 0 & & & 0_{k,d-k} \\ -1 & 2 & -1 & & & \\ & -1 & \ddots & \ddots & & \\ & & \ddots & \ddots & -1 & \\ & & & -1 & 2 & \\ \hline \multicolumn{5}{c|}{0_{d-k,k}} & 0_{d-k,d-k} \end{array} \right) .\] Observe that we have (since \(f_{k}\) is a quadratic form) \[\begin{align*} f_{k}(x) &= \frac{1}{2} \langle \nabla^{2} f_{k}(x)x,\,x\rangle - \frac{L-\mu}{4} x_{1} + \frac{L-\mu}{8} . \end{align*}\] Note that:

1. The Hessian is definite (resp. semi-definite) positive if \(\mu>0\) (resp. \(\mu = 0\)). Indeed, \[\begin{align*} \langle \nabla^{2} f_{k}(x)h,\,h\rangle &= \mu |\!| h |\!|^{2} + \frac{L-\mu}{4} \langle L_{k} h,\,h\rangle . \end{align*}\] Since \(\langle L_{k} h,\,h\rangle = h_{1}^{2} + \sum_{i=1}^{k-1} (h_{i+1} - h_{i})^{2} + h_{k}^{2} \geq 0\) for any \(h\), the result follows depending on the value of \(\mu\).

2. Since \((a-b)^{2} \leq 2a^{2} + 2b^{2}\), we have \[\begin{aligned} &h_{1}^{2} + \sum_{i=1}^{k-1} (h_{i+1} - h_{i})^{2} + h_{k}^{2}\\ &\leq h_{1}^{2} + \sum_{i=1}^{k-1} ( 2 h_{i+1}^{2} + 2 h_{i}^{2} ) + h_{k}^{2}\\ &\leq 4 \sum_{i=1}^{k} h_{i}^{2} \leq 4 \sum_{i=1}^{d} h_{i}^{2} = 4 |\!| h |\!|^{2}. \end{aligned}\] Hence, \[\langle \nabla^{2} f_{k}(x)h,\,h\rangle \leq \mu |\!| h |\!|^{2} + (L - \mu) |\!| h |\!|^{2} = L |\!| h |\!|^{2} .\]

Thus, we have \(\mu \mathop{\mathrm{Id}}\preceq \nabla^{2} f_{k}(x) \preceq L \mathop{\mathrm{Id}}\).

Let us characterize a solution \(x^{k,\star}\), unique if \(\mu>0\) or \(k=d\) of the minimization of \(f_{k}\) over \(\mathbb{R}^{d}\). We aim to solve \(\nabla f_{k}(x^{k,\star}) = 0\) to find a critical point (which will be a minimum since we just proved that the Hessian is at least semidefinite positive), that is \[\mu x^{k,\star} + \frac{L-\mu}{4} L_{k} x^{k,\star} - \frac{L-\mu}{4} e_{1} = 0 .\] Projecting this relation on each coordinate \(2 \leq i \leq k-1\), we get that \[-x_{i-1}^{k,\star} + 2 x_{i}^{k,\star} - x_{i+1}^{k,\star} = - \frac{4 \mu}{L-\mu} x_{i}^{k,\star},\] which leads to \[x_{i}^{k,\star} = \frac{1}{2} \frac{L-\mu}{L+\mu} (x_{i+1}^{k,\star} + x_{i-1}^{k,\star}) .\]

For the following two boundary equations, assume \(k\geq 2\). If \(k=1\), the single equation gives \(x_1^{1,\star}=(L-\mu)/(2(L+\mu))\). The boundary notation below covers both cases.

Similarly, we have

\[x_{1}^{k,\star} = \frac{1}{2} \frac{L-\mu}{L+\mu} (x_{2}^{k,\star} + 1) \quad \text{and} \quad x_{k}^{k,\star} = \frac{1}{2} \frac{L-\mu}{L+\mu} x_{k-1}^{k,\star} .\] Consider \(y_{0}, \dots, y_{k+1}\) defined by \(y_{i} = x_{i}^{k,\star}\) for \(1 \leq i \leq k\) and \(y_{0} = 1\) and \(y_{k+1} = 0\). We have the relation \[y_{i} = \alpha (y_{i+1} + y_{i-1}) \quad \text{where} \quad \alpha = \frac{1}{2} \frac{L-\mu}{L+\mu} > 0 .\] We can rewrite it as the second-order linear recursion \(y_{i+2} - \alpha^{-1} y_{i+1} + y_{i} = 0\). The associated trinom is \(P = X^{2} - \alpha^{-1} X + 1 \in \mathbb{R}[X]\) whose discriminant is given by \[\Delta = (-\alpha^{-1})^{2} - 4 = 16 \frac{L\mu}{(L - \mu)^{2}} .\] We distinguish two cases:

1. If \(\mu = 0\), then the unique root is given by \(r = 1\).

2. If \(\mu > 0\), then the roots are given by \[\begin{align*} r &= \frac{1}{2}(\alpha^{-1} - \sqrt{\Delta}) = \frac{\sqrt{\frac{L}{\mu}} - 1}{\sqrt{\frac{L}{\mu}} + 1} = \frac{\sqrt{K_{f}} - 1}{\sqrt{K_{f}} + 1}\\ s &= \frac{1}{2}(\alpha^{-1} + \sqrt{\Delta}) = \frac{\sqrt{K_{f}} + 1}{\sqrt{K_{f}} - 1} = \frac{1}{r}. \end{align*}\]

Case \(\mu=0\). We have the affine relation \(y_{i} = (a + b i) r\) with constraints \(y_{0} = a = 1\) and \(y_{k+1} = a + b (k+1) = 0\). In turn, we have \(y_{i} = 1 - \frac{i}{k+1}\) and thus \[\begin{equation*} x_{i}^{k,\star} = \begin{cases} 1 - \frac{i}{k+1} & \text{if } 1 \leq i \leq k \\ 0 & \text{otherwise.} \end{cases} \end{equation*}\] The associated optimal value is given by \[\begin{aligned} f_{k}(x^{k,\star}) &= \frac{L}{8} \left( \left( -\frac{1}{k+1} \right)^2 + \sum_{i=1}^{k-1} \frac{1}{(k+1)^{2}} + \left(1 - \frac{k}{k+1}\right)^{2} \right)\\ & = \frac{L}{8} \frac{k+1}{(k+1)^{2}} = \frac{L}{8} \frac{1}{k+1}. \end{aligned}\]

Case \(\mu>0\). The solution can be written as \[y_{i} = a r^{i} + b s^{i} \quad \text{with} \quad \begin{cases} a + b &= 1 \\ a r^{k+1} + b s^{k+1} &= 0. \end{cases}\] Thus, we have \(b = 1-a\), hence \(\frac{a}{a-1} = s^{2(k+1)} > 0\), and in turn we have \[a = \frac{s^{2(k+1)}}{s^{2(k+1)} - 1} \quad \text{and} \quad b = \frac{1}{1 - s^{2(k+1)}} .\] Hence, \[y_{i} = \frac{s^{2(k+1)}}{s^{2(k+1)} - 1} s^{-i} + \frac{1}{1 - s^{2(k+1)}} s^{i} .\] ◻

Proof of Theorem 1.1. We restrict our attention the the case where \(x_{0} = 0\) w.l.o.g. Indeed, if \(x_{0} \neq 0\), we can set \(x \mapsto \tilde f(x) = f(x+x_{0})\) and the following proof carry on. Let \(k=t\), so that \(2k+1\leq d\), and set \(f=f_{2k+1}^{L,0}\) on \(\mathbb{R}^d\).

If \(t=0\), then \(f(0)-f(x^\star)=L/16\) and \(\|x^\star\|^2=1/4\), which gives the claimed bound. In what follows, assume \(t=k\geq 1\).

Remark that \[f(x^{(t)}) = f_{2k+1}^{L,0}(x^{(t)}) = f_{t}^{L,0}(x^{(t)}) \geq f_{t}^{\star} .\] Using Lemma 1.2, we have on one hand that \(f_{t}^{\star} = \frac{L}{8(t+1)}\), and then \[\begin{gathered}f(x^{(k)}) - f(x^{\star}) \geq \frac{L}{8(k+1)} - \frac{L}{16(k+1)} = \frac{L}{16(k+1)} .\end{gathered}\] On the other hand, \[\begin{align*} &|\!| x^{2k+1,\star} - x_{0} |\!|^{2} = |\!| x^{2k+1,\star} |\!|^{2} = \sum_{i=1}^{2k+1} (x^{2k+1,\star})_{i}^{2} \\ &= \sum_{i=1}^{2k+1} \left( 1 - \frac{i}{2(k+1)} \right)^{2} \\ &= \sum_{i=1}^{2k+1} 1 - \frac{2}{2(k+1)} \sum_{i=1}^{2k+1} i + \frac{1}{4(k+1)^{2}} \sum_{i=1}^{2k+1} i^{2} \\ &= (2k+1) - \frac{1}{k+1} \frac{2(k+1)(2k+1)}{2} + \frac{1}{4(k+1)^{2}}\frac{2(k+1)(4k+3)(2k+1)}{6} \\ &= \frac{1}{3} \frac{(2k+1)(4k+3)}{4(k+1)} \\ &\leq \frac{2k+1}{3} \leq \frac{2}{3} (k+1) . \end{align*}\] Thus, \[\frac{f(x^{(t)}) - f(x^{\star})}{|\!| x^{2k+1,\star} - x_{0} |\!|^{2}} \geq \frac{\frac{L}{16(k+1)}}{\frac{2}{3} (k+1)} = \frac{3L}{32(k+1)^{2}} ,\] that proves (1.9). ◻

Proof of Theorem 1.2. The proof follows the same strategy as before, but we start with a bound on the iterates instead of the objective function. Assume that \(x^{(0)} = 0\), otherwise let \(\tilde f = f(\cdot + x_{0})\). Let \(k=\lfloor(d-1)/2\rfloor\), \(D=2k+1\leq d\), and \(f=f_D^{L,\mu}\) on \(\mathbb{R}^d\). We rewrite the coordinate of \(x^{2k+1,\star}\) as \[x_{i}^{2k+1,\star} = \frac{s^{4(k+1)}}{s^{4(k+1)} - 1} s^{-i} + \frac{1}{1 - s^{4(k+1)}} s^{i} = s^{-i} \left( 1 - \frac{s^{2i} - 1}{s^{4(k+1)} - 1} \right) .\]

On one hand, we have: \[|\!| x^{(0)} - x^{2k+1,\star} |\!|^{2} = \sum_{i=1}^{2k+1} (x_{i}^{2k+1,\star})^{2} = \sum_{i=1}^{2k+1} s^{-2i} \left( 1 - \frac{s^{2i} - 1}{s^{4(k+1)} - 1} \right)^{2} \leq \sum_{i=1}^{2k+1} s^{-2i} ,\] where we used that for all \(1 \leq i \leq 2k+1\), we have \[0 \leq 1 - \frac{s^{2i} - 1}{s^{4(k+1)} - 1} \leq 1 .\] Bounding the tail of the geometric sums, we obtain \[|\!| x^{(0)} - x^{2k+1,\star} |\!|^{2} \leq 2 \sum_{i=1}^{k+1} s^{-2i} . \tag{1.13}\]

On the other hand, observe that for \(0\leq t\leq k\), the coordinates of \(x^{(t)}\) after the first \(t\) are zero. Let \(a=\log s>0\). We rewrite the coordinate of \(x^{D,\star}\) as \[x_i^{D,\star}=\frac{\sinh((D+1-i)a)}{\sinh((D+1)a)}, \qquad 1\leq i\leq D.\] Thus, setting \(m=D-t\), we have \[|\!| x^{(t)}-x^{D,\star} |\!|^2 \geq\frac{\sum_{j=1}^m\sinh^2(ja)}{\sinh^2((D+1)a)}, \qquad |\!| x^{(0)}-x^{D,\star} |\!|^2 =\frac{\sum_{j=1}^D\sinh^2(ja)}{\sinh^2((D+1)a)}.\] We now compare these two sums. For each \(1\leq j\leq D\), let \(u=\lceil jm/D\rceil\). Since \(D\geq 2t+1\), we have \(D/m<2\), \(1\leq u\leq m\), \(u\leq j\), and \(j-u\leq t\); each value of \(u\) appears at most twice. Concavity of \(v\mapsto1-e^{-2av}\), which is zero at \(v=0\), gives \[\frac{\sinh(ja)}{\sinh(ua)} =e^{(j-u)a}\frac{1-e^{-2ja}}{1-e^{-2ua}} \leq e^{ta}\frac{j}{u}\leq 2s^t.\] Hence, \[\sum_{j=1}^D\sinh^2(ja)\leq 8s^{2t}\sum_{u=1}^m\sinh^2(ua).\] Combining it with the preceding expressions, we have \[|\!| x^{(t)}-x^{D,\star} |\!|^2 \geq\frac18s^{-2t}|\!| x^{(0)}-x^{D,\star} |\!|^2 =\frac18\left(\frac{\sqrt{K_f}-1}{\sqrt{K_f}+1}\right)^{2t} |\!| x^{(0)}-x^{D,\star} |\!|^2,\] proving (1.11). The value bound (1.12) is obtained by applying (A.2) to this bound. ◻

Proof. Let \(z\) a minimizer of \(f\). By convexity, we have \[\frac{d}{d\tau}\frac12|\!| x(\tau)-z |\!|^2 =-\langle \nabla f(x(\tau)),\,x(\tau)-z\rangle \leq -(f(x(\tau))-f(z))\leq 0.\] The continuous vector field gives a local solution. Its distance to \(z\) is nonincreasing, so it stays bounded and extends to all \(\tau\geq 0\). Monotonicity of \(\nabla f\) makes the distance between two solutions with the same starting point nonincreasing, so the solution is unique. Moreover, \[\frac{d}{d\tau}f(x(\tau))=-|\!| \nabla f(x(\tau)) |\!|^2\leq 0.\] Integrating the first inequality and using this monotonicity, we obtain \[\tau(f(x(\tau))-f(z)) \leq\int_0^\tau(f(x(s))-f(z))\,ds \leq\frac12|\!| x^{(0)}-z |\!|^2.\] Hence, the claimed value bound holds for any minimizer \(z\). Since the trajectory is bounded, it has a limit point \(\bar x\) along a sequence of times tending to infinity. The value bound and continuity show that \(\bar x\) is a minimizer. Its distance to the trajectory is nonincreasing and tends to zero along that sequence, so \(x(\tau)\) converges to \(\bar x\). Taking \(z=\bar x=x^\star\) gives the displayed bound. ◻

Proof. Observe that for \(j \in \{1, \dots, K \}\), we have \[\frac{\partial \left( \sum_{k=1}^{K} e^{x_{k}} \right)^{-1}}{\partial x_{j}} = - \frac{1}{\left( \sum_{k=1}^{K} e^{x_{k}} \right)^{2}} \frac{\partial \left( \sum_{k=1}^{K} e^{x_{k}} \right)}{\partial x_{j}} = - \frac{e^{x_{j}}}{\left( \sum_{k=1}^{K} e^{x_{k}} \right)^{2}} = - \frac{1}{\sum_{k=1}^{K} e^{x_{k}}} \sigma_{j}(x).\]

Now, take \(i,j \in \{ 1,\dots, K \}\). Then, two cases occurs:

If \(i \neq j\). We have \[\frac{\partial \sigma_{i}}{\partial x_{j}}(x) = e^{x_{i}} \frac{\partial \left( \sum_{k=1}^{K} e^{x_{k}} \right)^{-1}}{\partial x_{j}} = - \sigma_{i}(x) \sigma_{j}(x) .\]

If \(i = j\). \[\frac{\partial \sigma_{j}}{\partial x_{j}}(x) = e^{x_{j}} \frac{\partial \left( \sum_{k=1}^{K} e^{x_{k}} \right)^{-1}}{\partial x_{j}} + e^{x_{j}} \left( \sum_{k=1}^{K} e^{x_{k}} \right)^{-1} = \sigma_{j}(x) (1 - \sigma_{j}(x)) .\] ◻

Proof. Let \(k \in \{1,\dots,K\}\). Then, \[\frac{\partial \psi_{i}(x)}{\partial x_{k}} = - \frac{1}{\sigma_{i}(x)} \frac{\partial \sigma_{i}(x)}{\partial x_{k}} = \begin{cases} \sigma_{k}(x) & \text{if } i \neq k \\ \sigma_{k}(x) - 1 & \text{if } i = k. \end{cases}\]

Differentiating \(\nabla\psi_i(x)=\sigma(x)-e_i\) gives \(\nabla^2\psi_i(x)=\mathop{\mathrm{Jac}}_{\sigma}(x)\). ◻

Proof. Using the preceding lemma, we have for any \(v\in\mathbb{R}^K\) \[v^\top\nabla^2\psi_k(x)v =\sum_{j=1}^K\sigma_j(x)v_j^2 -\left(\sum_{j=1}^K\sigma_j(x)v_j\right)^2 =\sum_{j=1}^K\sigma_j(x)(v_j-\bar v)^2\geq 0,\] where \(\bar v=\sum_{j=1}^K\sigma_j(x)v_j\), since \(\sum_j\sigma_j(x)=1\). Hence, \(\psi_k\) is convex. ◻

Proof. Answer of exercise 2

Proof by contradiction. Suppose that \(x^{(t)}\) does not converge towards \(x^{\star}\), i.e.,  \[\exists \varepsilon> 0, \forall T \in \mathbb{N}, \exists t \geq T, \quad |\!| x^{(t)} - x^{\star} |\!| > \varepsilon.\] Using boundedness from step \((ii)\), extract a convergent subsequence staying at distance at least \(\varepsilon\) from \(x^\star\). By step \((iii)\), its limit is a critical point distinct from \(x^\star\). Conclude that you have two critical points. ◻