#+title: Slides/Notes for optimization courses
#+subtitle: Lecture slides and notes on optimization in finite dimensions
#+date: 2023-11-07
#+last_updated: 2023-11-08
#+status: abandoned
#+ailvl: 0
#+keywords: optimization

Most of the notes here deal with optimization in a finite dimensional space, and *are incomplete*.
The content is available under the [[https://creativecommons.org/licenses/by/4.0/][CC BY 4.0 license]].

* Introduction
- [[media:notes/slides-notes-optimization/intro_optim_S/intro_optim_S.pdf][Introduction to optimization]] ([[file:intro_optim_S/intro_optim_S.tex][src]])
- [[media:notes/slides-notes-optimization/existence_minimizer_S/existence_minimizer_S.pdf][Existence of a minimizer]] ([[file:existence_minimizer_S/existence_minimizer_S.tex][src]])
- [[media:notes/slides-notes-optimization/gradient_hessian_S/gradient_hessian_S.pdf][Gradient and Hessian]] ([[file:gradient_hessian_S/gradient_hessian_S.tex][src]])

* Basic concepts in convex analysis
- [[media:notes/slides-notes-optimization/convex_set_S/convex_set_S.pdf][Convex set]] ([[file:convex_set_S/convex_set_S.tex][src]])
- [[media:notes/slides-notes-optimization/convex_function_S/convex_function_S.pdf][Convex function]] ([[file:convex_function_S/convex_function_S.tex][src]])
- [[media:notes/slides-notes-optimization/local_minima_convex_function_S/local_minima_convex_function_S.pdf][Local minima of convex functions are global]] ([[file:local_minima_convex_function_S/local_minima_convex_function_S.tex][src]])
- [[media:notes/slides-notes-optimization/strongly_convex_function_S/strongly_convex_function_S.pdf][Strongly convex function]] ([[file:strongly_convex_function_S/strongly_convex_function_S.tex][src]])
- [[media:notes/slides-notes-optimization/lipschitz_gradient_S/lipschitz_gradient_S.pdf][Lipschitz continuous gradient]] ([[file:lipschitz_gradient_S/lipschitz_gradient_S.tex][src]])
- [[media:notes/slides-notes-optimization/smooth_strongly_convex_function_S/smooth_strongly_convex_function_S.pdf][Smooth and strongly convex function]] ([[file:smooth_strongly_convex_function_S/smooth_strongly_convex_function_S.tex][src]])

* Descent methods
- [[media:notes/slides-notes-optimization/blackbox_query_S/blackbox_query_S.pdf][Blackbox query]] ([[file:blackbox_query_S/blackbox_query_S.tex][src]])
- [[media:notes/slides-notes-optimization/descent_direction_S/descent_direction_S.pdf][Descent direction]] ([[file:descent_direction_S/descent_direction_S.tex][src]])
- [[media:notes/slides-notes-optimization/gradient_descent_101_S/gradient_descent_101_S.pdf][Gradient descent 101]] ([[file:gradient_descent_101_S/gradient_descent_101_S.tex][src]])
- [[media:notes/slides-notes-optimization/mean_of_convergence_S/mean_of_convergence_S.pdf][Means of convergence]] ([[file:mean_of_convergence_S/mean_of_convergence_S.tex][src]])
- [[media:notes/slides-notes-optimization/gd_on_ols_S/gd_on_ols_S.pdf][Gradient descent on OLS]] ([[file:gd_on_ols_S/gd_on_ols_S.tex][src]])

* Rate of convergence for various classes of functions
- [[media:notes/slides-notes-optimization/gradient_descent_coercive_noncvx_S/gradient_descent_coercive_noncvx_S.pdf][Gradient descent on coercive non-convex functions]] ([[file:gradient_descent_coercive_noncvx_S/gradient_descent_coercive_noncvx_S.tex][src]])
- [[media:notes/slides-notes-optimization/gradient_descent_smooth_noncvx_S/gradient_descent_smooth_noncvx_S.pdf][Gradient descent on smooth non-convex functions]] ([[file:gradient_descent_smooth_noncvx_S/gradient_descent_smooth_noncvx_S.tex][src]])
- [[media:notes/slides-notes-optimization/gradient_descent_smooth_cvx_S/gradient_descent_smooth_cvx_S.pdf][Gradient descent on smooth convex functions]] ([[file:gradient_descent_smooth_cvx_S/gradient_descent_smooth_cvx_S.tex][src]])
- [[media:notes/slides-notes-optimization/gradient_descent_smooth_strongly_cvx_S/gradient_descent_smooth_strongly_cvx_S.pdf][Gradient descent on smooth strongly convex functions]] ([[file:gradient_descent_smooth_strongly_cvx_S/gradient_descent_smooth_strongly_cvx_S.tex][src]])

* Automatic differentiation (and related topics)
- [[media:notes/slides-notes-optimization/finite_difference_S/finite_difference_S.pdf][Finite differences]] ([[file:finite_difference_S/finite_difference_S.tex][src]])
- [[media:notes/slides-notes-optimization/complex_step_approximation_S/complex_step_approximation_S.pdf][Complex step differentiation]] ([[file:complex_step_approximation_S/complex_step_approximation_S.tex][src]])