Mathematical Theory of Deep Neural Networks (Princeton workshop)

This looks interesting. Deep Learning would benefit from a stronger theoretical understanding of why it works so well. I hope they put the talks online!

Mathematical Theory of Deep Neural Networks

Tuesday March 20th, Princeton Neuroscience Institute.
PNI Psychology Lecture Hall 101

Recent advances in deep networks, combined with open, easily-accessible implementations, have moved empirical results far faster than formal understanding. The lack of rigorous analysis for these techniques limits their use in addressing scientific questions in the physical and biological sciences, and prevents systematic design of the next generation of networks. Recently, long-past-due theoretical results have begun to emerge. These results, and those that will follow in their wake, will begin to shed light on the properties of large, adaptive, distributed learning architectures, and stand to revolutionize how computer science and neuroscience understand these systems.

This intensive one-day technical workshop will focus on state of the art theoretical understanding of deep learning. We aim to bring together researchers from the Princeton Neuroscience Institute (PNI) and of the theoretical machine learning group at the Institute for Advanced Studies (IAS) interested in more rigorously understanding deep networks to foster increased discussion and collaboration across these intrinsically related groups.

The Content of their Character: Ed Blum and Jian Li

See 20 years @15 percent: does Harvard discriminate against Asian-Americans? The excerpt below is from the Harvard lawsuit brief, recalling the parallel between what had been done to limit Jewish enrollment in the early 20th century, and the current situation with Asian-Americans.

... Harvard is engaging in racial balancing. Over an extended period, Harvard’s admission and enrollment figures for each racial category have shown almost no change. Each year, Harvard admits and enrolls essentially the same percentage of African Americans, Hispanics, whites, and Asian Americans even though the application rates and qualifications for each racial group have undergone significant changes over time. This is not the coincidental byproduct of an admissions system that treats each applicant as an individual; indeed, the statistical evidence shows that Harvard modulates its racial admissions preference whenever there is an unanticipated change in the yield rate of a particular racial group in the prior year. Harvard’s remarkably stable admissions and enrollment figures over time are the deliberate result of systemwide intentional racial discrimination designed to achieve a predetermined racial balance of its student body.

... In a letter to the chairman of the committee, President Lowell wrote that “questions of race,” though “delicate and disagreeable,” were not solved by ignoring them. The solution was a new admissions system giving the school wide discretion to limit the admission of Jewish applicants: “To prevent a dangerous increase in the proportion of Jews, I know at present only one way which is at the same time straightforward and effective, and that is a selection by a personal estimate of character on the part of the Admissions authorities ... The only way to make a selection is to limit the numbers, accepting those who appear to be the best.”

... The reduction in Jewish enrollment at Harvard was immediate. The Jewish portion of Harvard’s entering class dropped from over 27 percent in 1925 to 15 percent the following year. For the next 20 years, this percentage (15 percent) remained virtually unchanged.

... The new policy permitted the rejection of scholastically brilliant students considered “undesirable,” and it granted the director of admissions broad latitude to admit those of good background with weaker academic records. The key code word used was “character” — a quality thought to be frequently lacking among Jewish applicants, but present congenitally among affluent Protestants.

DOJ invokes Title VI against Harvard admissions:

WSJ: ... The Justice Department, whose Civil Rights Division is conducting the investigation into similar allegations, said in a letter to Harvard’s lawyers, dated Nov. 17 and reviewed by the Journal, that the school was being investigated under Title VI of the Civil Rights Act of 1964, which bars discrimination on the basis of race, color and national origin for organizations that receive federal funding. The letter also said the school had failed to comply with a Nov. 2 deadline to provide documents related to the university’s admissions policies and practices. ...

I believe I first mentioned Jian Li on this blog back in 2006! It's nice to see that he is still courageous and principled today.



From his closing remarks:

I have a message to every single Asian-American student in the country who is applying to college: your civil rights are being violated and you must speak up in defense of them. If you've suffered discrimination you have the option to file a complaint with the Office for Civil Rights. Let your voice be heard .. not only through formal means but also by simply letting it be known in your schools and your communities, in the press and on social media, that university discrimination is pervasive and that this does not sit well with you. Together we will fight to ensure that universities can no longer treat us as second-class citizens.

Allen Institute meeting on Genetics of Complex Traits

You can probably tell by all the photos below that I love their new building :-)

I was a participant in this event: What Makes Us Human? The Genetics of Complex Traits (Allen Frontiers Group), including in a small second day workshop with just the speakers and the AI leadership. This workshop will, I hope, result in some interesting new initiatives in complex trait genomics!

I'd like to thank the Allen Institute organizers for making this such a pleasant and productive 2 days. I learned some incredible things from the other speakers and I recommend all of their talks -- available here.

My talk:




Action photos:

Working hard on day 2 in the little conference room :-)

The Jiujitsu Philosopher: John Danaher

John Danaher is one of the deepest thinkers in combat sports, MMA, and jiujitsu. He has coached a number of world champions in MMA and jiujitsu/submission grappling (Georges St. Pierre, Garry Tonon, etc.). The recent leg lock technique renaissance is largely due to Danaher and his school.

Danaher was a philosophy PhD student at Columbia before discovering BJJ through Renzo Gracie's academy in NYC. When I was a Yale professor (in the 90s) I made trips to Renzo's for training. I don't recall Danaher (who would have been a student/instructor there at the time), but I do recall Craig Kukuk, Renzo's partner in the school and the first US blackbelt instructor. 

Kukuk had played linebacker at Iowa State University (where I grew up), and we spent time talking about Iowa (a big wrestling hotbed) and the origins of jiujitsu and ultimate fighting in the US. I had trained in Japan and so knew quite a bit about the relationship between traditional Judo and BJJ. At one time I probably knew as much as anyone about the relationship between Judo, BJJ, MMA, and US folk style wrestling.

See Mama said knock you out.

What Makes Us Human? The Genetics of Complex Traits (Allen Frontiers Group)

I'll be attending this meeting in Seattle the next few days.

Recent research has led to new insights on how genes shape brain structure and development, and their impact on individual variation. Although significant inroads have been made in understanding the genetics underlying disease risk, what about the complex traits of extraordinary variation - such as cognition, superior memory, etc.? Can current advances shed light on genetic components underpinning these variations?

Personal genomics, biobank resources, emerging statistical genetics methods and neuroimaging capabilities are opening new frontiers in the field of complex trait analysis. This symposium will highlight experts using diverse approaches to explore a spectrum of individual variation of the human mind.

Paul Allen (MSFT co-founder) is a major supporter of scientific research, including the Allen Institute for Brain Science. Excerpts from his memoir, Idea Man.

We are at a unique moment in bioscience. New ideas, combined with emerging technologies, will create unprecedented and transformational insights into living systems. Accelerating the pace of this change requires a thoughtful and agile exploration of the entire landscape of bioscience, across disciplines and spheres of research. Launched in 2016 with a $100 million commitment toward a larger 10-year plan, The Paul G. Allen Frontiers Group will discover and support scientific ideas that change the world. We are committed to a continuous conversation with the scientific community that allows us to remain at the ever-changing frontiers of science and reimagine what is possible.

My talk is scheduled for 3:55 PM Pacific Weds 1/17. All talks will be streamed on the Allen Institute Facebook page.

Institute for Advanced Study: Genomic Prediction of Complex Traits (seminar)

Genomic Prediction of Complex Traits

After a brief review (suitable for physicists) of computational genomics and complex traits, I describe recent progress in this area. Using methods from Compressed Sensing (L1-penalized regression; Donoho-Tanner phase transition with noise) and the UK BioBank dataset of 500k SNP genotypes, we construct genomic predictors for several complex traits. Our height predictor captures nearly all of the predicted SNP heritability for this trait -- thereby resolving the missing heritability problem. Actual heights of most individuals in validation tests are within a few cm of predicted heights. I also discuss application of these methods to cognitive ability and polygenic disease risk: sparsity estimates (of the number of causal loci), combined with phase transition scaling analysis, allow estimates of the amount of data required to construct good predictors. Finally, I discuss how these advances will affect human health and reproduction (embryo selection for In Vitro Fertilization, genetic editing) in the coming decade.

FEATURING
Steve Hsu

SPEAKER AFFILIATION
Michigan State University

I recently gave a similar talk at 23andMe (slides at link).

Note Added: Many people asked for video of this talk, but alas recording talks is not standard practice at IAS. I did give a similar talk using the same slides just a week later at the Allen Institute in Seattle (Symposium on Genetics of Complex Traits): video here.


Some Comments and Slides:

I tried to make the talk understandable to physicists, and at least according to what I was told (and my impression from the questions asked during and after the talk), largely succeeded. Early on, when presenting the phenotype function y(g), both Nima Arkani-Hamed (my host) and Ed Witten asked some questions about the "units" of the various quantities involved. In the actual computation everything is z-scored: measured in units of SD relative to the sample mean. I didn't realize until later that there was some confusion about how this is done for the "state variable" of the genetic locus g_i. In fact, when the gene array is read the result is 0,1,2 for homozygous common allele, heterozygous, homozygous rare allele, respectively. (I might have that backwards but you get the point.) For each locus there is a minor allele frequency (MAF) and this determines the sample average and SD of the distribution of 0's, 1's, and 2's. It is the z-scored version of this variable that appears in the computation. I didn't realize certain people were following the details so closely in the talk but I should not be surprised ;-) In the future I'll include a slide specifically on this to avoid confusion.

Looking at my slide on missing heritability, Witten immediately noted that estimating SNP heritability (as opposed to total or broad sense heritability) is nontrivial and I had to quickly explain the GCTA technique!

During the talk I discussed the theoretical reason we expect to find a lot of additive variance: nonlinear gadgets are fragile (easy to break through recombination in sexual reproduction), whereas additive genetic variance can be reliably passed on and is easy for natural selection to act on***. (See also Fisher's Fundamental Theorem of Natural Selection. More.) Usually these comments pass over the head of the audience but at IAS I am sure quite a few people understood the point.

One non-physicist reader of this blog braved IAS security and managed to attend the lecture. I am flattered, and I invite him to share his impressions in the comments!

Afterwards there was quite a bit of additional discussion which spilled over into tea time. The important ideas: how Compressed Sensing works, the nature of the phase transition, how we can predict the amount of data required to build a good predictor (capturing most of the SNP heritability) using the universality of the phase transition + estimate of sparsity, etc. were clearly absorbed by the people I talked to.

Slides


*** On the genetic architecture of intelligence and other quantitative traits (p.16):

... The preceding discussion is not intended to convey an overly simplistic view of genetics or systems biology. Complex nonlinear genetic systems certainly exist and are realized in every organism. However, quantitative differences between individuals within a species may be largely due to independent linear effects of specific genetic variants. As noted, linear effects are the most readily evolvable in response to selection, whereas nonlinear gadgets are more likely to be fragile to small changes. (Evolutionary adaptations requiring significant changes to nonlinear gadgets are improbable and therefore require exponentially more time than simple adjustment of frequencies of alleles of linear effect.) One might say that, to first approximation, Biology = linear combinations of nonlinear gadgets, and most of the variation between individuals is in the (linear) way gadgets are combined, rather than in the realization of different gadgets in different individuals.

Linear models work well in practice, allowing, for example, SNP-based prediction of quantitative traits (milk yield, fat and protein content, productive life, etc.) in dairy cattle. ...

Gork revisited, 2018

It's been almost 10 years since I made the post Are you Gork?

Over the last decade, both scientists and non-scientists have become more confident that we will someday create:

A. AGI (= sentient AI, named "Gork" :-)  See Rise of the Machines: Survey of AI Researchers.

B. Quantum Computers. See Quantum Computing at a Tipping Point?

This change in zeitgeist makes the thought experiment proposed below much less outlandish. What, exactly, does Gork perceive? Why couldn't you be Gork? (Note that the AGI in Gork can be an entirely classical algorithm even though he exists in a quantum simulation.)




Slide from this [Caltech IQI] talk. See also illustrations in Big Ed.

Survey questions:

1) Could you be Gork the robot? (Do you split into different branches after observing the outcome of, e.g., a Stern-Gerlach measurement?)

2) If not, why? e.g.,

I have a soul and Gork doesn't!  Copenhagen people, please use exit on your left.

Decoherence solved all that!  Sorry, try again. See previous post.

I don't believe that quantum computers will work as designed, e.g., sufficiently large algorithms or subsystems will lead to real (truly irreversible) collapse. Macroscopic superpositions that are too big (larger than whatever was done in the lab last week!) are impossible.

QM is only an algorithm for computing probabilities -- there is no reality to the quantum state or wavefunction or description of what is happening inside a quantum computer. Tell this to Gork!

Stop bothering me -- I only care about real stuff like the Higgs mass / SUSY-breaking scale / string Landscape / mechanism for high-Tc / LIBOR spread / how to generate alpha. 

[ 2018: Ha Ha -- first 3 real stuff topics turned out to be pretty boring use of the last decade... ]

Just as A. and B. above have become less outlandish assumptions, our ability to create large and complex superposition states with improved technology (largely developed for quantum computing; see Schrodinger's Virus) will make the possibility that we ourselves exist in a superposition state less shocking. Future generations of physicists will wonder why it took their predecessors so long to accept Many Worlds.

Bonus! I will be visiting Caltech next week (Tues and Weds 1/8-9). Any blog readers interested in getting a coffee or beer please feel free to contact me :-)