Slides/Notes for optimization courses

by Samuel Vaiter · modified: 2023-11-08

Most of the notes here deal with optimization in a finite dimensional space, and are incomplete. The content is available under the CC BY 4.0 license.

Introduction

• Introduction to optimization (src)
• Existence of a minimizer (src)
• Gradient and Hessian (src)

Basic concepts in convex analysis

• Convex set (src)
• Convex function (src
• Local minima of convex functions are global (src)
• Strongly convex function (src)
• Lipschitz continuous gradient (src)
• Smooth and strongly convex function (src)

Descent methods

• Blackbox query (src)
• Descent direction (src)
• Gradient descent 101 (src)
• Means of convergence (src)
• Gradient descent on OLS (src)

Rate of convergence for various classes of functions

• Gradient descent on coercive non-convex functions (src)
• Gradient descent on smooth non-convex functions (src)
• Gradient descent on smooth convex functions (src)
• Gradient descent on smooth strongly convex functions (src)

Automatic differentiation (and related topics)

• Finite differences (src)
• Complex step differentiation (src)