Deep Learning as a branch of Statistical Physics

Via Jess Riedel, an excellent talk by Naftali Tishby given recently at the Perimeter Institute.

The first 15 minutes is a very nice summary of the history of neural nets, with an emphasis on the connection to statistical physics. In the large network (i.e., thermodynamic) limit, one observes phase transition behavior -- sharp transitions in performance, and also a kind of typicality (concentration of measure) that allows for general statements that are independent of some detailed features.

Unfortunately I don't know how to embed video from Perimeter so you'll have to click here to see the talk.

An earlier post on this work: Information Theory of Deep Neural Nets: "Information Bottleneck"

Title and Abstract:

The Information Theory of Deep Neural Networks: The statistical physics aspects

The surprising success of learning with deep neural networks poses two fundamental challenges: understanding why these networks work so well and what this success tells us about the nature of intelligence and our biological brain. Our recent Information Theory of Deep Learning shows that large deep networks achieve the optimal tradeoff between training size and accuracy, and that this optimality is achieved through the noise in the learning process.

In this talk, I will focus on the statistical physics aspects of our theory and the interaction between the stochastic dynamics of the training algorithm (Stochastic Gradient Descent) and the phase structure of the Information Bottleneck problem. Specifically, I will describe the connections between the phase transition and the final location and representation of the hidden layers, and the role of these phase transitions in determining the weights of the network.

About Tishby:

Naftali (Tali) Tishby נפתלי תשבי

Physicist, professor of computer science and computational neuroscientist
The Ruth and Stan Flinkman professor of Brain Research
Benin school of Engineering and Computer Science
Edmond and Lilly Safra Center for Brain Sciences (ELSC)
Hebrew University of Jerusalem, 96906 Israel

I work at the interfaces between computer science, physics, and biology which provide some of the most challenging problems in today’s science and technology. We focus on organizing computational principles that govern information processing in biology, at all levels. To this end, we employ and develop methods that stem from statistical physics, information theory and computational learning theory, to analyze biological data and develop biologically inspired algorithms that can account for the observed performance of biological systems. We hope to find simple yet powerful computational mechanisms that may characterize evolved and adaptive systems, from the molecular level to the whole computational brain and interacting populations.

Economics, ant farmers and free will theorems

Wow, it's been almost a week since my last post. I was pretty busy at the Perimeter Institute -- my brain is occupied trying to digest all the new things I've learned :-)

The conference had its origins in an essay Can science help solve the economic crisis? written by several of the organizers. The essay received responses from several prominent thinkers, and led to an exchange between Stanford economist Paul Romer and mathematician turned quant Eric Weinstein (see further down the page at the link above). Romer agreed to debate Weinstein, but then failed to show up! [Note added: in a private communication to me, Paul Romer writes that there was a misunderstanding and he never agreed to participate in a debate.]

The meeting went full speed from 9 AM until late at night for four days. The attendees were a mix of theoretical physicists (with particle physics and complexity theory, especially the Santa Fe Institute, well represented), practitioners from Wall St. (often from a math or physics background), academic economists (Barkley Rosser, Richard Freeman, etc.) and evolutionary biologists. See here for video recordings of talks. There were many good talks but I especially recommend Doyne Farmer's for the physicist's complex systems view of economics.

Too much ground was covered for a short summary, but one area of interest for almost all attendees was the possibility of going beyond the neoclassical paradigm via simulations of interacting agents. Examples of such simulations were discussed in several of the talks, including by Santa Fe "ant farmers" Bruce Sawhill and Jim Herriot, who used agents to model air traffic for the ambitious DayJet startup (see here and here).

The technology is certainly there to do some massive, quasi-realistic simulations of an actual economy (as opposed to an isolated financial market or sub-economy). The main problem, which wasn't given as much attention as I might have liked, is that the results are limited by the quality of the individual agents, who must have at least rudimentary learning capabilities to be realistic. Hence one is led full circle back to problems related to AI, machine learning and even psychology or cognitive science.


When I wasn't at the conference I managed to have several interesting physics discussions. Below are links to some recent work by John Conway and Simon Kochen that came up in two separate conversations I had at Perimeter -- one with Rob Spekkens (he and I were discussing related issues, although he didn't mention these papers specifically) and with Nima Arkani-Hamed, who was visiting from IAS.

The Free Will Theorem

http://arxiv.org/abs/quant-ph/060407

On the basis of three physical axioms, we prove that if the choice of a particular type of spin 1 experiment is not a function of the information accessible to the experimenters, then its outcome is equally not a function of the information accessible to the particles. We show that this result is robust, and deduce that neither hidden variable theories nor mechanisms of the GRW type for wave function collapse can be made relativistic. We also establish the consistency of our axioms and discuss the philosophical implications.

See also the Strong Free Will theorem.

I feel the name of the theorem is a bit misleading, but inevitably so as it is quite difficult to define free will. The main feature of the papers is the great clarity of thought and presentation. In content, they are not so different from the older GHZ result which was formulated in terms of what the authors called local realism. To be honest I don't quite agree with Conway and Kochen's interpretation of their own results -- it's obvious that the results are easily consistent with many worlds quantum mechanics, which does not (at least from my spacetime perspective) allow for free will.

Here are 6 lectures on the theorem and related topics, given by Conway recently at Princeton. See also 'tHooft's response to their result.

Financial crisis meeting at Perimeter

Some snapshots from the conference The Economic Crisis and its Implications for The Science of Economics. I overcame my fear of H1N1 and decided to make the trip anyway. Let's hope all that hand washing pays off :-)

The first photo is the morning panel discussion. From left to right, Eric Weinstein, Nouriel Roubini, Richard Freeman and Nassim Taleb.

The second photo is of Emanuel Derman's talk. In the upper right hand corner you can see Doyne Farmer of Santa Fe.

The third photo is of Mike Brown, former Microsoft CFO and former NASDAQ Chairman telling some amusing stories which involve Bill Gates, Bernie Madoff and simulations of ant traders :-)

The talks today were for a general audience -- my favorite was Andy Lo's. I hope the next two days of the meeting will be a bit more technical.

Financial crisis and economics at Perimeter

I'll be attending an interdisciplinary conference at the Perimeter Institute, May 1-4: The Economic Crisis and its Implications for The Science of Economics. It should be a combustible mix of personalities ;-)

I give the economists credit for entering the lion's den of theoretical physicists. Here is how J. DOYNE FARMER, of the Santa Fe Institute and Prediction Company, described a meeting of this type that occurred 20 years ago.

With some justification, many economists think that the entry of physicists into their world reflects merely audacity, hubris, and arrogance. Physicists are not known for their humility, and some physicists have presented their work in a manner that plays into this stereotype. The cultural barrier between the two groups will be difficult to overcome. This schism was already evident at a conference held at the Santa Fe Institute in 1988 titled “The Economy as an Evolving Complex System.” Roughly half the participants were economists and the other half physicists. Although many of the physicists were largely ignorant of economics, that did not prevent them from openly criticizing the economists. At one point, Nobel laureate Phil Anderson said, “You guys really believe that?” At another point, Larry Summers (now Secretary of the Treasury) accused physicists of having a “Tarzan complex.” This was not just a turf war. Whether due to nature or nurture, this conference clearly showed that there is a deep epistemological divide between physicists and economists that is difficult to cross.

The first day talks are listed below. Perimeter is usually good about putting their talks online, so you can probably enjoy them yourself without schlepping all the way to Canada :-)

9:15 - 10:00 Eric Weinstein Conference Overview

10:00 - 10:45 Nouriel Roubini

11:10 - 11:55 Nassim Taleb

11:55 - 12:40 W. Brian Arthur Neoclassical Economics and its Policy Biases: Is there an alternative?

12:40 - 1:00 Panel Discussion with Speakers

1:00 - 2:30 Lunch Break

2:30 - 3:15 Emanuel Derman Scientists, Sciensters, Anti-Scientists & Economics

3:15 - 4:00 Andrew Lo The Adaptive Markets Hypothesis and Financial Crisis

4:15 - 5:00 Richard Alexander

5:00 - 6:00 Panel Discussion with Speakers and Wrap Up

Perimeter photos

Having a great time here -- the Perimeter Institute is its own little world of theoretical physics!

Thanks to Blackberry founder Mike Lazarides, whose generous donations were largely responsible for creating this place.