Download or read book The Complexity of Optimization Beyond Convexity written by Yair Menachem Carmon and published by . This book was released on 2020 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Gradient descent variants are the workhorse of modern machine learning and large-scale optimization more broadly, where objective functions are often non-convex. Could there be better general-purpose optimization methods than gradient descent, or is it in some sense unimprovable? This thesis addresses this question from the perspective of the worst-case oracle complexity of finding near-stationary points (i.e., points with small gradient norm) of smooth and possibly non-convex functions. On the negative side, we prove a lower bound showing that gradient descent is unimprovable for a natural class of problems. We further prove the worst-case optimality of stochastic gradient descent, recursive variance reduction, cubic regularization of Newton's method and high-order tensor methods, in each case under the set of assumptions for which the method was designed. To prove our lower bounds we extend theory of information-based oracle complexity to the realm of non-convex optimization. On the positive side, we use classical techniques from optimization (namely Nesterov momentum and Krylov subspace methods) to accelerate gradient descent in a large subclass of non-convex problems with higher-order smoothness. Furthermore, we show how recently proposed variance reduction techniques can further improve stochastic gradient descent when stochastic Hessian-vector products available.