Yann LeCun on Unsupervised Learning

This is a recent Yann LeCun talk at CMU. Toward the end he discusses recent breakthroughs using GANs (Generative Adversarial Networks, see also Ian Goodfellow here and here).

LeCun tells an anecdote about the discovery of backpropagation. The first implementation of the algorithm didn't work, probably because of a bug in the program. But they convinced themselves that the reason for the failure was that the network could easily get caught in a local minimum which prevents further improvement. It turns out that this is very improbable in high dimensional spaces, which is part of the reason behind the great success of deep learning. As I wrote here:

In the limit of high dimensionality a critical point is overwhelmingly likely to be a saddlepoint (have at least one negative eigenvalue). This means that even though the surface is not strictly convex the optimization is tractable.

This (free version!) new textbook on deep learning by Goodfellow, Bengio, and Courville looks very good. See also Michael Nielsen's book.

If I were a young person I would be working in this area (perhaps with an eye toward applications in genomics, or perhaps working directly on the big problem of AGI). I hope after I retire they will let me hang out at one of the good AI places like Google Brain or Deep Mind :-)