Lecture 1 | Convex Optimization I (Stanford)

First, finish lectures by Professor Gilbert Strang. [LINK] To my memory, session notes of CS229 is good enough for understanding SVM and gaussian distributions. Also watch youtube videos. [LINK] [LINK] If you just want to use the libraries, you can stop here. If you want to know more, read chapters 1-3 of nonlinear programming by Professor Dimitri Bertsekas before convex optimization. [LINK] Then, you can try to finish EE364 and watch the videos. [LINK] [VIDEO] If you want to roll your own algorithms, you have to know some optimization tools. [LINK] And there is some statistics knowledge you have to fill in. I used these: [LINK] [LINK] R is used in the courses.

A practical math topic I think all computer science (and engineering, really) students should take is convex optimization. Often a problem in engineering or computer science can be phrased as minimizing a cost function over some data and/or variable constraints. Take SVM for example, or sparse signal denoising, or template matching. And far too often, people will apply "advanced state of the art techniques" to solve these problems, e.g. neural networks of some exotic kind, graphical models, etc. And while these are great techniques, they need not be applied to convex problems of a reasonable size. In the examples I listed, I can almost guarantee that a knowledge of how to phrase the model as a convex optimization problem and throw it into an interior point solver is going to be much more effective than running stochastic gradient descent on some user-defined model with way too many (or too little) parameters. Because when you say a problem is convex, what you are saying is that it can be solved globally in polynomial time (usually on the order of ~20 least-squares problems of size corresponding to number of variables and constraints). There is a sophisticated convergence theory as well. And if you understand the mathematical theory, you get to use this mature technology to solve exactly a whole class of problems you wouldn't think you could. It's also worth mentioning that a lot of very hard non-convex problems can be approximated and solved well using these methods, and this

This is my favorite way to think about it as well. I think I learned it first in one of Steven Boyds astounding convex optimization lectures [VIDEO]

These lectures are fantastic after you've mastered the basics of linear algebra [VIDEO] (convex optimization, by a very experienced and often funny lecturer)