---
title: Slides/Notes for optimization courses
tldr: Lecture slides and notes on optimization in finite dimensions
author: Samuel Vaiter
date: 2023-11-07
last-updated: 2023-11-08
status: in progress
tags:
  - optimization
---

Most of the notes here deal with optimization in a finite dimensional space, and **are incomplete**.
The content is available under the [CC BY 4.0 license](https://creativecommons.org/licenses/by/4.0/).

## Introduction

- [Introduction to optimization](https://files.samuelvaiter.com/notes/slides-notes-optimization/intro_optim_S/intro_optim_S.pdf) ([src](intro_optim_S/intro_optim_S.tex))
- [Existence of a minimizer](https://files.samuelvaiter.com/notes/slides-notes-optimization/existence_minimizer_S/existence_minimizer_S.pdf) ([src](existence_minimizer_S/existence_minimizer_S.tex))
- [Gradient and Hessian](https://files.samuelvaiter.com/notes/slides-notes-optimization/gradient_hessian_S/gradient_hessian_S.pdf) ([src](gradient_hessian_S/gradient_hessian_S.tex))

## Basic concepts in convex analysis

- [Convex set](https://files.samuelvaiter.com/notes/slides-notes-optimization/convex_set_S/convex_set_S.pdf) ([src](convex_set_S/convex_set_S.tex))
- [Convex function](https://files.samuelvaiter.com/notes/slides-notes-optimization/convex_function_S/convex_function_S.pdf) ([src](convex_function_S/convex_function_S.tex))
- [Local minima of convex functions are global](https://files.samuelvaiter.com/notes/slides-notes-optimization/local_minima_convex_function_S/local_minima_convex_function_S.pdf) ([src](local_minima_convex_function_S/local_minima_convex_function_S.tex))
- [Strongly convex function](https://files.samuelvaiter.com/notes/slides-notes-optimization/strongly_convex_function_S/strongly_convex_function_S.pdf) ([src](strongly_convex_function_S/strongly_convex_function_S.tex))
- [Lipschitz continuous gradient](https://files.samuelvaiter.com/notes/slides-notes-optimization/lipschitz_gradient_S/lipschitz_gradient_S.pdf) ([src](lipschitz_gradient_S/lipschitz_gradient_S.tex))
- [Smooth and strongly convex function](https://files.samuelvaiter.com/notes/slides-notes-optimization/smooth_strongly_convex_function_S/smooth_strongly_convex_function_S.pdf) ([src](smooth_strongly_convex_function_S/smooth_strongly_convex_function_S.tex))

## Descent methods

- [Blackbox query](https://files.samuelvaiter.com/notes/slides-notes-optimization/blackbox_query_S/blackbox_query_S.pdf) ([src](blackbox_query_S/blackbox_query_S.tex))
- [Descent direction](https://files.samuelvaiter.com/notes/slides-notes-optimization/descent_direction_S/descent_direction_S.pdf) ([src](descent_direction_S/descent_direction_S.tex))
- [Gradient descent 101](https://files.samuelvaiter.com/notes/slides-notes-optimization/gradient_descent_101_S/gradient_descent_101_S.pdf) ([src](gradient_descent_101_S/gradient_descent_101_S.tex))
- [Means of convergence](https://files.samuelvaiter.com/notes/slides-notes-optimization/mean_of_convergence_S/mean_of_convergence_S.pdf) ([src](mean_of_convergence_S/mean_of_convergence_S.tex))
- [Gradient descent on OLS](https://files.samuelvaiter.com/notes/slides-notes-optimization/gd_on_ols_S/gd_on_ols_S.pdf) ([src](gd_on_ols_S/gd_on_ols_S.tex))

## Rate of convergence for various classes of functions

- [Gradient descent on coercive non-convex functions](https://files.samuelvaiter.com/notes/slides-notes-optimization/gradient_descent_coercive_noncvx_S/gradient_descent_coercive_noncvx_S.pdf) ([src](gradient_descent_coercive_noncvx_S/gradient_descent_coercive_noncvx_S.tex))
- [Gradient descent on smooth non-convex functions](https://files.samuelvaiter.com/notes/slides-notes-optimization/gradient_descent_smooth_noncvx_S/gradient_descent_smooth_noncvx_S.pdf) ([src](gradient_descent_smooth_noncvx_S/gradient_descent_smooth_noncvx_S.tex))
- [Gradient descent on smooth convex functions](https://files.samuelvaiter.com/notes/slides-notes-optimization/gradient_descent_smooth_cvx_S/gradient_descent_smooth_cvx_S.pdf) ([src](gradient_descent_smooth_cvx_S/gradient_descent_smooth_cvx_S.tex))
- [Gradient descent on smooth strongly convex functions](https://files.samuelvaiter.com/notes/slides-notes-optimization/gradient_descent_smooth_strongly_cvx_S/gradient_descent_smooth_strongly_cvx_S.pdf) ([src](gradient_descent_smooth_strongly_cvx_S/gradient_descent_smooth_strongly_cvx_S.tex))

## Automatic differentiation (and related topics)

- [Finite differences](https://files.samuelvaiter.com/notes/slides-notes-optimization/finite_difference_S/finite_difference_S.pdf) ([src](finite_difference_S/finite_difference_S.tex))
- [Complex step differentiation](https://files.samuelvaiter.com/notes/slides-notes-optimization/complex_step_approximation_S/complex_step_approximation_S.pdf) ([src](complex_step_approximation_S/complex_step_approximation_S.tex))