Spooks drowning in data

Almost every technical endeavor, from finance to high energy physics to biology to internet security to spycraft, is either already or soon to be drowning in Big Data. This is an inevitable consequence of exponential Moore's Laws in bandwidth, processing power, and storage, combined with improved "sensing" capability. The challenge is extracting meaning from all that data.

My impression is that the limiting factor at the moment is the human brainpower necessary to understand the idiosyncrasies of the particular problem, and, simultaneously, develop the appropriate algorithms. There are simply not enough people around who are good at this; it's not just a matter of algorithms, you need insight into the specific situation. Of equal importance is that the (usually non-technical) decision makers who have to act on the data need to have some rough grasp of the strengths and limitations of the methods, so as not to have to treat the results as coming from a black box.

To give you my little example of big data, on my desk (in Oakland, not in Eugene) I have stacks of terabyte drives with copies of essentially every Windows executable (program that runs on a flavor of Windows) that has appeared on the web in the past few years (about 5 percent of this is malware; also stored in our data is what each executable does once it's installed). Gathering this data was only modestly hard; analyzing it in a meaningful way is a lot harder!

NY Review of Books: On a remote edge of Utah's dry and arid high desert, where temperatures often zoom past 100 degrees, hard-hatted construction workers with top-secret clearances are preparing to build what may become America's equivalent of Jorge Luis Borges's "Library of Babel," a place where the collection of information is both infinite and at the same time monstrous, where the entire world's knowledge is stored, but not a single word is understood. At a million square feet, the mammoth $2 billion structure will be one-third larger than the US Capitol and will use the same amount of energy as every house in Salt Lake City combined.

Unlike Borges's "labyrinth of letters," this library expects few visitors. It's being built by the ultra-secret National Security Agency—which is primarily responsible for "signals intelligence," the collection and analysis of various forms of communication—to house trillions of phone calls, e-mail messages, and data trails: Web searches, parking receipts, bookstore visits, and other digital "pocket litter." Lacking adequate space and power at its city-sized Fort Meade, Maryland, headquarters, the NSA is also completing work on another data archive, this one in San Antonio, Texas, which will be nearly the size of the Alamodome.

Just how much information will be stored in these windowless cybertemples? A clue comes from a recent report prepared by the MITRE Corporation, a Pentagon think tank. "As the sensors associated with the various surveillance missions improve," says the report, referring to a variety of technical collection methods, "the data volumes are increasing with a projection that sensor data volume could potentially increase to the level of Yottabytes (1024 Bytes) by 2015."[1] Roughly equal to about a septillion (1,000,000,000,000,000,000,000,000) pages of text, numbers beyond Yottabytes haven't yet been named. Once vacuumed up and stored in these near-infinite "libraries," the data are then analyzed by powerful infoweapons, supercomputers running complex algorithmic programs, to determine who among us may be—or may one day become—a terrorist. In the NSA's world of automated surveillance on steroids, every bit has a history and every keystroke tells a story.

... Where does all this leave us? Aid concludes that the biggest problem facing the agency is not the fact that it's drowning in untranslated, indecipherable, and mostly unusable data, problems that the troubled new modernization plan, Turbulence, is supposed to eventually fix. "These problems may, in fact, be the tip of the iceberg," he writes. Instead, what the agency needs most, Aid says, is more power. But the type of power to which he is referring is the kind that comes from electrical substations, not statutes. "As strange as it may sound," he writes, "one of the most urgent problems facing NSA is a severe shortage of electrical power." With supercomputers measured by the acre and estimated $70 million annual electricity bills for its headquarters, the agency has begun browning out, which is the reason for locating its new data centers in Utah and Texas. And as it pleads for more money to construct newer and bigger power generators, Aid notes, Congress is balking.

The issue is critical because at the NSA, electrical power is political power. In its top-secret world, the coin of the realm is the kilowatt. More electrical power ensures bigger data centers. Bigger data centers, in turn, generate a need for more access to phone calls and e-mail and, conversely, less privacy. The more data that comes in, the more reports flow out. And the more reports that flow out, the more political power for the agency.

Rather than give the NSA more money for more power—electrical and political—some have instead suggested just pulling the plug. "NSA can point to things they have obtained that have been useful," Aid quotes former senior State Department official Herbert Levin, a longtime customer of the agency, "but whether they're worth the billions that are spent, is a genuine question in my mind."

Based on the NSA's history of often being on the wrong end of a surprise and a tendency to mistakenly get the country into, rather than out of, wars, it seems to have a rather disastrous cost-benefit ratio. Were it a corporation, it would likely have gone belly-up years ago. The September 11 attacks are a case in point. For more than a year and a half the NSA was eavesdropping on two of the lead hijackers, knowing they had been sent by bin Laden, while they were in the US preparing for the attacks. The terrorists even chose as their command center a motel in Laurel, Maryland, almost within eyesight of the director's office. Yet the agency never once sought an easy-to-obtain FISA warrant to pinpoint their locations, or even informed the CIA or FBI of their presence.

But pulling the plug, or even allowing the lights to dim, seems unlikely given President Obama's hawkish policies in Afghanistan. However, if the war there turns out to be the train wreck many predict, then Obama may decide to take a much closer look at the spy world's most lavish spender. It is a prospect that has some in the Library of Babel very nervous. "It was a great ride while it lasted," said one.

A Nobel for Barry?

Huh? In a few years he might have actually earned one. This just demonstrates the outright political biases of the Nobel Committee. The Nobel citation reads like this (without actually mentioning GWB): "Bush era bad, Obama good. Hope good."

Note I speak as an Obama supporter!

Will pundits on the left have the guts to point out that the emperor has no clothes? This kind of outcome just reinforces the paranoid fantasies of the far-right: that Obama's Harvard magna is fake, that the World Government has been grooming him for leadership since his student days, that Bill Ayers wrote Dreams From My Father, etc.

Speaking more broadly, it seems to me that our obsession with prizes (an offshoot of winner-take-all culture) is unhealthy, and that prizes these days are less and less correlated with actual achievement.

While my faith in the Nobel process is shaken (although commenters have already pointed out the Kissinger prize and of course there is always Modigliani-Miller ;-), my confidence in Obama is not -- he reacted properly.

Mr. Obama said he was "surprised and deeply humbled" by the committee’s decision, ... he said he would accept it as “a call to action.”

“To be honest,” the president said “I do not feel that I deserve to be in the company of so many of the transformative figures who have been honored by this prize, men and women who’ve inspired me and inspired the entire world through their courageous pursuit of peace.”

Someone just pointed out to me that this opens the door for a string theorist to win the physics prize ;-)

Schrodinger's virus

If the creature above (a tardigrade arthropod) can be placed in a superposition state, will you accept that you probably can be as well? And once you admit this, will you accept that you probably actually DO exist in a superposition state already?

It may be disturbing to learn that we live in a huge quantum multiverse, but was it not also disturbing for Galileo's contemporaries to learn that we live on a giant rotating sphere, hurtling through space at 30 kilometers per second? E pur si muove!

Related posts: Many Worlds: A brief guide for the perplexed, Are you gork?, Label: many worlds

Economist: ONE of the most famous unperformed experiments in science is Schrödinger’s cat. In 1935 Erwin Schrödinger (pictured), who was one of the pioneers of quantum mechanics, imagined putting a cat, a flask of Prussic acid, a radioactive atom, a Geiger counter, an electric relay and a hammer in a sealed box. If the atom decays, the Geiger counter detects the radiation and sends a signal that trips the relay, which releases the hammer, which smashes the flask and poisons the cat.

The point of the experiment is that radioactive decay is a quantum process. The chance of the atom decaying in any given period is known. Whether it has actually decayed (and thus whether the cat is alive or dead) is not—at least until the box is opened. The animal exists, in the argot of the subject, in a “superposition” in which it is both alive and dead at the same time.

Schrödinger’s intention was to illuminate the paradoxes of the quantum world. But superposition (the existence of a thing in two or more quantum states simultaneously) is real and is, for example, the basis of quantum computing. A pair of researchers at the Max Planck Institute for Quantum Optics in Garching, Germany, now propose to do what Schrödinger could not, and put a living organism into a state of quantum superposition.

The organism Ignacio Cirac and Oriol Romero-Isart have in mind is the flu virus. [arxiv] Pedants might object that viruses are not truly alive, but that is a philosophical rather than a naturalistic argument, for they have genes and are capable of reproduction—a capability they lose if they are damaged. The reason for choosing a virus is that it is small. Actual superposition (as opposed to the cat-in-a-box sort) is easiest with small objects, for which there are fewer pathways along which the superposition can break down. Physicists have already put photons, electrons, atoms and even entire molecules into such a state and measured the outcome. In the view of Dr Cirac and Dr Romero-Isart, a virus is just a particularly large molecule, so existing techniques should work on it.

The other thing that helps maintain superposition is low temperature. The less something jiggles about because of heat-induced vibration, the longer it can remain superposed. Dr Cirac and Dr Romero-Isart therefore propose putting the virus inside a microscopic cavity and cooling it down to its state of lowest energy (ground state, in physics parlance) using a piece of apparatus known as a laser trap. This ingenious technique—which won its inventors, one of whom was Steven Chu, now America’s energy secretary, a Nobel prize—works by bombarding an object with laser light at a frequency just below that which it would readily absorb and re-emit if it were stationary. This slows down the movement, and hence the temperature, of its atoms to a fraction of a degree above absolute zero.

Once that is done, another laser pulse will jostle the virus from its ground state into an excited state, just as a single atom is excited by moving one of its electrons from a lower to a higher orbital. By properly applying this pulse, Dr Cirac believes it will be possible to leave the virus in a superposition of the ground and excited states.

For that to work, however, the virus will need to have certain physical properties. It will have to be an insulator and to be transparent to the relevant laser light. And it will have to be able to survive in a vacuum. Such viruses do exist. The influenza virus is one example. Its resilience is legendary. It can survive exposure to a vacuum, and it seems to be an insulator—which is why the researchers have chosen it. And if the experiment works on a virus, they hope to move on to something that is indisputably alive: a tardigrade.

Tardigrades are tiny but resilient arthropods. They can survive in vacuums and at very low temperatures. And, although the difference between ground state and an excited state is not quite the difference between life and death, Schrödinger would no doubt have been amused that his 70-year-old jeu d’esprit has provoked such an earnest following.

More Brainpower and Globalization

The Chronicle of Higher Education is running a series of articles on the competitiveness of US higher education versus Asia; see here and related links (may require subscription). See also earlier posts Meritocracy in China , Brainpower and Globalization.

This describes the steps Tsinghua University took to recruit Turing Award winner Andrew Yao, going as far as effectively creating an entire department of theoretical computer science. (Interestingly, Yao first earned a PhD in physics before switching to computer science.)

Andrew Chi-Chih Yao's trajectory suggests a genius's quick ascent to success. Born in Shanghai, he studied hard, earned two Ph.D.'s from American universities, and at age 35 was a professor of computer science at Stanford University. By 2000, when he received the A.M. Turing Award, one of the most prestigious prizes in his field, younger computer scientists were memorizing a principle bearing his name.

But when Tsinghua University invited him to return to China to lead a new, generously financed institute in 2004, he jumped at the opportunity. "I was very excited," he recalls.

In creating the Institute for Theoretical Computer Science, Mr. Yao says, he had free rein from Tsinghua's administrators to determine everything from research topics to personnel.

Once in Beijing, he threw himself into the project, emerging with an institute that is today one of the university's jewels and is increasingly known outside China.

The way Mr. Yao's institute was created illustrates a central element of China's higher-education strategy. When the government decides it wants to do something, it does it, and fast.

Theoretical computer science—the abstract, intensely mathematical subfield that is Mr. Yao's specialty—is poorly financed in the United States. It wasn't originally a target area for China, either. But China has an approach different from that found in the United States: a desire to build outstanding institutions by attracting the leaders in a field—any field.

Mr. Yao was first approached about joining Tsinghua's faculty in 2003 by Chen Ning Yang, a Nobel Prize-winning physicist, who had himself recently begun lecturing at the university.

Through that and later discussions, Mr. Yao came to understand that the university's officials were more interested in hiring world-class talent than they were in building up specific areas.

"They want to catch up in a global way, and therefore it doesn't matter where they get started," Mr. Yao explains.

That hunger translates into money for leading Western-trained scientists willing to relocate. "China is a very exciting place for science and engineering for people who have vision," he says.

'Yao's Class'

But while the chance to build a world-class institute from scratch required little consideration, how exactly to do it in a country with a developing higher-education system was a difficult question to answer.

Mr. Yao recruited his first class of graduate students, only to discover that many of them lacked basic skills. If he wanted quality applicants, he realized, he would have to train them himself. So he established an undergraduate program within the institute, selecting about 100 students each year and then identifying the most talented ones—a group now known as Yao's Class—for intensive instruction.

The fact that he had the freedom to do so reflects another element of China's higher-education strategy, in which a select number of elite universities are allowed to flourish. ...

Baidu, baby, Baidu

Although Google sits astride the English (and Indo-European)-language Web like a colossus, their market share in Chinese is less than 25 percent and hasn't grown at all in recent years, even under the lead of celebrity hire Kai Fu Lee (who recently left Google to run his own venture fund). Search in China is dominated by Baidu, which appears to be Google's only rival in any language (eat your heart out, Quaero and Agence de l'innovation industrielle :-)

Given that China has the largest number of internet users (over 300 million and still growing very fast) and generates more search queries than the US, this is an interesting strategic development.

For a discussion of Baidu by founder Robin Li, watch this talk. The first 40 minutes or so will be old hat to any startup veteran, but the last 20 minutes is fairly interesting -- Li makes some interesting comments about the future of innovation in China. He notes that the Chinese language search index triples in size each year, whereas in English and other languages it only grows about 50% per year.






Here's a goofy idea for a sci fi story: a conversation or encounter between two big early AIs, one which evolved out of Google and its Indo-European corpus of data + "Mechanical Turk" / user input, and the other which evolved from Baidu and its primarily Sinic corpus and input. Would they be different in a fundamental way?

Supermassive black holes and the entropy of the universe

New Scientist has an article about a recent paper by two Australian researchers (http://www.arxiv.org/abs/0909.3983), which contains detailed estimates of the entropy of various components of the universe (black holes, neutrinos, photons, etc.). This paper is related to some work I did with with Frampton, Kephart and Reeb: What is the entropy of the universe?. (Discussion on Cosmic Variance.)

Our work did not focus on the numerical values of various contributions to the entropy (we made some simple estimates), but rather what the physical meaning is of this entropy -- in particular, that of black holes; see excerpt below.

New Scientist: Mammoth black holes push universe to its doom

30 September 2009 by Rachel Courtland

THE mammoth black holes at the centre of most galaxies may be pushing the universe closer to its final fade-out. And it is all down to the raging disorder within those dark powerhouses.

Disorder is measured by a quantity called entropy, something which has been on the rise ever since the big bang. Chas Egan and Charles Lineweaver of the Australian National University in Canberra used the latest astrophysical data to calculate the total entropy of everything in the universe, from gas to gravitons. It turns out that supermassive black holes are by far the biggest contributors to the universe's entropy. Entropy reflects the number of possible arrangements of matter and energy in an object. The number of different configurations of matter a black hole could contain is staggering because its internal state is completely mysterious.

Egan and Lineweaver found that everything within the observable universe contains about 10^104 units of entropy (joules per Kelvin), a factor of 10 to 1000 times higher than previous estimates that did not include some of the biggest known black holes (www.arxiv.org/abs/0909.3983, submitted to The Astrophysical Journal).

If entropy were ever to reach a maximum level, that would mean the heat death of the universe. In this scenario no energy can flow, because everything is the same temperature and so life and other processes become impossible. "Our results suggest we're a little further along that road than previously thought," Egan says.

But although black holes do boost the universe's total entropy, it is not clear whether they will hasten its heat death. Supermassive black holes don't contribute much to the flows of heat that even out temperature throughout the universe, says physicist Stephen Hsu at the University of Oregon in Eugene.

It's true that these black holes will slowly evaporate by releasing Hawking radiation, particles created near the boundary of the black hole. And this radiation could move the universe towards heat death.

Black holes may evaporate via Hawking radiation, tipping the universe towards its heat death. However, it will take some 10^102 years for a supermassive black hole to evaporate. "The entropy inside those black holes is effectively locked up in there forever," Hsu says. So we may have reached a state approaching heat death long before, as stars burn out and their matter decays.

The large result obtained by the Egan and Lineweaver for the entropy of the universe is primarily due to supermassive black holes. How do we interpret that entropy? Here is an excerpt from our paper (excuse the latex).

Note the entropy used in this paper describes the uncertainty in the precise quantum state of a system. If the system is macroscopic the full quantum state is only accessible to a kind of ``super-observer'' who is unaffected by decoherence \cite{decoherence}. Individual observers within the system who have limited experimental capabilities can only detect particular decoherent outcomes. These outcomes arise, e.g., from an effective density matrix that results from tracing over degrees of freedom which are out of the experimenter's control (i.e., which form the ``environment''). In \cite{BID} the experimental capabilities necessary to distinguish decoherent branches of the wavefunction, or, equivalently, the precise quantum state of Hawking radiation from a black hole, are discussed. It is shown that a super-observer would either need (at minimum) the capability of making very precise measurements of accuracy $\exp(- M^2 )$ (see also the proposal of Maldacena \cite{eternal} for a specific measurement to determine whether black hole evaporation is unitary), or alternatively the capability of engineering very precise non-local operators, which measure a large fraction of the Hawking radiation at once, including correlations (i.e., as opposed to ordinary particle detectors, which only measure Fock state occupation numbers and are not sensitive to phase information).

An observer who lacks the capabilities described in the previous paragraph would be unable to distinguish the states in the $S = M^{3/2}$ subspace in Fig.~\ref{figure1} from those in the larger $S = M^2$ subspace, assuming the unitary evaporation resembles, in gross terms, Hawking evaporation, with the information hidden in correlations among the emitted quanta. In that case, the future uncertainty for ordinary (non-super) observers might be better characterized by the larger $S = M^2$ entropy. Putting it another way, an ordinary (non-super) observer is forced (due to experimental limitations) into a coarse grained description of the radiation; they cannot distinguish between most of the radiation states, and for them the $S = M^2$ entropy is appropriate. For a super-observer, however, due to unitary evolution, the uncertainty in the quantum state does not increase. For them, black holes do not have greater entropy than the precursor states from which they formed.

For the super-observers described above, the large black hole entropies in Table I do not reflect the actual uncertainties in the (current and future) state of the universe and are in that sense misleading. A black hole of mass $M$ whose formation history is typical for our universe (e.g., it originated from gravitational collapse of a star or galactic core) satisfies the bound S [less than] M^{3/2} \cite{MI}. Thus, re-evaluating the numbers in Table I, the total entropy of all black holes in our universe is not bigger than the total matter entropy: the dominant uncertainty in the precise state of the universe, at least as far as arises from known physics, is, in fact, due to CMB photons or neutrinos.

Figure 3 caption: Ordinary matter (star, galactic core, etc.) collapses to form an astrophysical black hole. Under unitary evolution, the number of final Hawking radiation states that are actually accessible from this collapse is $\sim \exp M^{3/2}$, i.e.~precisely the number of ordinary astrophysical precursors (\ref{th1}). It is therefore much smaller than the the number of $\sim \exp M^2$ states a black hole, and its eventual Hawking radiation, could possibly occupy if nothing about its formation process were known.

Final mysterious comment, maximally compressed for the cognoscenti: assuming unitarity, black holes do not push us closer to heat death (equilibrium) in the multiverse, but can contribute (albeit very slowly) to the (coarse grained) heat death experienced by a non-super observer (i.e., an observer subject to decoherence). See here for more on equilibrium in the multiverse.

Copula in cosmology?

A new paper co-authored by one of my collaborators: http://arxiv.org/abs/0909.5187

From Finance to Cosmology: The Copula of Large-Scale Structure

Robert J. Scherrer, Andreas A. Berlind, Qingqing Mao, Cameron K. McBride (Vanderbilt University)

Any multivariate distribution can be uniquely decomposed into marginal (1-point) distributions, and a function called the copula, which contains all of the information on correlations between the distributions. The copula provides an important new methodology for analyzing the density field in large-scale structure. We derive the empirical 2-point copula for the evolved dark matter density field. We find that this empirical copula is well-approximated by a Gaussian copula. We consider the possibility that the full n-point copula is also Gaussian and describe some of the consequences of this hypothesis. Future directions for investigation are discussed.

I don't understand the gravitational dynamics well enough to guess whether we should expect the dark matter distribution today to be described by Gaussian copula. The authors insert a number of caveats, referring to the use of copula to price mortgage securities :-)

See here for some correspondence with Scherrer on the topic of genius and modern science, and here for some popular coverage of the paper we wrote.

Physicist to lead Williams College

I learned yesterday that a contemporary of mine, theoretical physicist Adam Falk, is the new President of Williams College.

Congratulations Adam! I think the future is bright at Williams :-)

See here for more, including some video interviews. Discussion by Williams alumni at Ephblog.

Starship Troopers

This commercial for the video game Halo 3: ODST (Orbital Drop Shock Troops) comes as close as anything I've seen to capturing the flavor of the Heinlein novel Starship Troopers. Forget about the 1997 movie directed by Paul Verhoeven; it's pure camp. Some might classify the original novel as pulp (see excerpt below), but I think it is much more than that.

The ODST project started as a Halo movie project involving Peter Jackson of Lord of the Rings fame. Perhaps this explains the cinematic qualities. (I'm not a gamer; I just noticed the commercial while watching Iowa beat Penn State last night and got interested in what seemed to be an obvious Starship Trooper influence.)

Wikipedia: Starship Troopers is a novel set in an unspecified time in the future, although probably not in the far-flung future. It chronicles the experiences of Juan "Johnnie" Rico, the story's narrator, during his enlistment and training in the Mobile Infantry, and his participation in an interstellar war between the Terran Federation and the Arachnids (referred to as "the Bugs") of Klendathu. It is narrated as a series of flashbacks — one of only a few Heinlein novels to use that narrative device[11] — and contains large sections of character discussion and introspection, as well as exposition, all meant to detail the political theory and philosophical beliefs underlying the society that Juan Rico lives in.

The novel opens with Rico aboard the corvette Rodger Young, platoon transport for "Rasczak's Roughnecks", about to embark on a raid against the planet of the "Skinnies", allies of the Arachnids. We learn that he is a "cap" trooper (called this because they are dropped in capsules from the ship in orbit toward their drop zones) in the Terran Federation's Mobile Infantry (M.I.). The raid itself, one of the few instances of actual combat in the novel, is relatively brief: the Mobile Infantrymen land in the capital city, destroy their targets while trying to avoid unnecessary Skinny casualties, and withdraw, suffering three injured (one fatal) in the process.

The story then flashes back to Rico's graduation from high school and his decision to sign up for Federal Service, rather than attend Harvard University, over the objections of his wealthy father. This is the only chapter that describes Rico's civilian life, and most of it is spent recording the monologues of two people: retired Lt. Colonel Jean V. Dubois, Rico's school instructor in the subject of History and Moral Philosophy, and Fleet Sergeant Ho, a recruiter for the Armed forces of the Terran Federation.

Many readers have felt that Dubois serves as a stand-in for Heinlein throughout the novel. He delivers what is probably the book's most famous soliloquy, on how violence "has settled more issues in history than has any other factor."[12] Fleet Sergeant Ho offers a separate angle on military service to that of Dubois. (Ho has prostheses for several limbs, but does not wear them on duty at the front door of the federal building. This is calculated to remind applicants of the real risks of service, and to weed out those not willing to take such risks in the service of the Federation).

Interspersed throughout the book are other flashbacks to Rico's high school History and Moral Philosophy course, which describe how, in the Terran Federation, the rights of a full Citizen (to vote, and hold public office) must be earned through voluntary Federal service. However, the franchise cannot be exercised until after honorable discharge from the Service, which means that active members of the Service cannot vote. Those residents who opt not to perform Federal Service retain the other rights generally associated with a modern democracy (e.g. free speech, assembly, etc.), but cannot vote or hold public office. This structure arose ad hoc after the collapse of the 20th century Western democracies, brought on by both social failures at home and military defeat by the Chinese Hegemony overseas (i.e. looking forward into the late 20th century from the time the novel was written in the late 1950s).[13]

After enlisting in the Mobile Infantry, Rico is assigned to boot camp at Camp Arthur Currie. Five chapters are spent exploring Rico's training, under the guidance of his chief instructor, Career Ship's Sergeant Charles Zim. Boot camp is deliberately so rigorous that fewer than ten percent of the recruits complete basic training; the rest either resign (with no penalty, save never being able to vote); are expelled (likewise); are given medical discharges (which may however be refused); or assigned to lesser duties (enabling them to vote after their service is finished); or die in training. ...

Here's the version of the book I've had since I was a kid:



Yes, I enjoyed a misspent youth, during which I read all of Heinlein's books and even played the Avalon Hill boardgame version of Starship Troopers.

Give me a few Sergeant Zim's and I'll conquer the galaxy! :-)

Bootcamp for the Mobile Infantry: ... But exercise will keep you warm and they saw to it that we got plenty of that. The first morning we were there they woke us up before daybreak. I had had trouble adjusting to the change in time zones and it seemed to me that I had just got to sleep; I couldn't believe that anyone seriously intended that I should get up in the middle of the night. But they did mean it. A speaker somewhere was blaring out a military march, fit to wake the dead, and a hairy nuisance who had come charging down the company street yelling, "Everybody out! Show a leg! On the bounce!" came marauding back again just as I had pulled the covers over my head, tipped over my cot and dumped me on the cold hard ground. It was an impersonal attention; he didn't even wait to see if I hit. Ten minutes later, dressed in trousers, undershirt, and shoes, I was lined up with the others in ragged ranks for setting-up exercises just as the Sun looked over the eastern horizon.

Facing us was a big broad-shouldered, mean-looking man, dressed just as we were -- except that while I looked and felt like a poor job of embalming, his chin was shaved blue, his trousers were sharply creased, you could have used his shoes for mirrors, and his manner was alert, wide-awake, relaxed, and rested. You got the impression that he never needed to sleep -- just ten-thousand-mile checkups and dust him off occasionally. He bellowed, "C'pnee! Atten . . . shut! I am Career Ship's Sergeant Zim, your company commander. When you speak to me, you will salute and say, `Sir' -- you will salute and `sir' anyone who carries an instructor's baton -- " He was carrying a swagger cane and now made a quick reverse moulinet with it to show what he meant by an instructor's baton; I had noticed men carrying them when we had arrived the night before and had intended to get one myself -- they looked smart. Now I changed my mind. " -- because we don't have enough officers around here for you to practice on. You'll practice on us. ...

Zim turned back to the rest of us, still shivering at attention. He walked up and down, looked us over, and seemed awfully unhappy. At last he stepped out in front of us, shook his head, and said, apparently to himself but he had a voice that carried: "To think that this had to happen to me!" He looked at us. "You apes -- No, not `apes'; you don't rate that much. You pitiful mob of sickly monkeys . . . you sunken-chested, slack-bellied, drooling refugees from apron strings. In my whole life I never saw such a disgraceful huddle of momma's spoiled little darlings in -- you, there! Suck up the gut! Eyes front! I'm talking to you!" I pulled in my belly, even though I was not sure he had addressed me. He went on and on and I began to forget my goose flesh in hearing him storm. He never once repeated himself and he never used either profanity or obscenity. (I learned later that he saved those for very special occasions, which this wasn't.) But he described our shortcomings, physical, mental, moral, and genetic, in great and insulting detail. But somehow I was not insulted; I became greatly interested in studying his command of language. I wished that we had had him on our debate team.

At last he stopped and seemed about to cry. "I can't stand it," he said bitterly. "I've just got to work some of it off -- I had a better set of wooden soldiers when I was six ALL RIGHT! Is there any one of you jungle lice who thinks he can whip me? Is there a man in the crowd? Speak up !" There was a short silence to which I contributed. I didn't have any doubt at all that he could whip me; I was convinced. I heard a voice far down the line, the tall end. "Ah reckon ah can . . . suh." Zim looked happy. "Good! Step out here where I can see you." The recruit did so and he was impressive, at least three inches taller than Sergeant Zim and broader across the shoulders. "What's your name, soldier?" "Breckinridge, suh -- and ah weigh two hundred and ten pounds an' theah ain't any of it `slack-bellied.' " "Any particular way you'd like to fight?" "Suh, you jus' pick youah own method of dyin'. Ah'm not fussy." "Okay, no rules. Start whenever you like." Zim tossed his baton aside. It started -- and it was over. The big recruit was sitting on the ground, holding his left wrist in his right hand. He didn't say anything. Zim bent over him. "Broken?" "Reckon it might be . . . suh." "I'm sorry. You hurried me a little. Do you know where the dispensary is? Never mind -- Jones! Take Breckinridge over to the dispensary."

As they left Zim slapped him on the right shoulder and said quietly, "Let's try it again in a month or so. I'll show you what happened." I think it was meant to be a private remark but they were standing about six feet in front of where I was slowly freezing solid. Zim stepped back and called out, "Okay, we've got one man in this company, at least. I feel better. Do we have another one? Do we have two more? Any two of you scrofulous toads think you can stand up to me?" He looked back and forth along our ranks. "Chicken-livered, spineless -- oh, oh! Yes? Step out." Two men who had been side by side in ranks stepped out together; I suppose they had arranged it in whispers right there, but they also were far down the tall end, so I didn't hear. Zim smiled at them. "Names, for your next of kin, please." "Heinrich." "Heinrich what?" "Heinrich, sir. Bitte." He spoke rapidly to the other recruit and added politely, "He doesn't speak much Standard English yet, sir." "Meyer, mein Herr," the second man supplied. "That's okay, lots of `em don't speak much of it when they get here -I didn't myself. Tell Meyer not to worry, he'll pick it up. But he understands what we are going to do?" "Jawohl," agreed Meyer. "Certainly, sir. He understands Standard, he just can't speak it fluently." "All right. Where did you two pick up those face scars? Heidelberg?" "Nein -- no, sir. Ko:nigsberg." "Same thing." Zim had picked up his baton after fighting Breckinridge; he twirled it and asked, "Perhaps you would each like to borrow one of these?" "It would not be fair to you, sir," Heinrich answered carefully. "Bare hands, if you please." "Suit yourself. Though I might fool you. Ko:nigsberg, eh? Rules?" "How can there be rules, sir, with three?" "An interesting point. Well, let's agree that if eyes are gouged out they must be handed back when it's over. And tell your Korpsbruder that I'm ready now. Start when you like." Zim tossed his baton away; someone caught it. "You joke, sir. We will not gouge eyes." "No eye gouging, agreed. `Fire when ready, Gridley.' " "Please?" "Come on and fight! Or get back into ranks!" Now I am not sure that I saw it happen this way; I may have learned part of it later, in training. But here is what I think happened: The two moved out on each side of our company commander until they had him completely flanked but well out of contact. From this position there is a choice of four basic moves for the man working alone, moves that take advantage of his own mobility and of the superior co-ordination of one man as compared with two -- Sergeant Zim says (correctly) that any group is weaker than a man alone unless they are perfectly trained to work together. For example, Zim could have feinted at one of them, bounced fast to the other with a disabler, such as a broken kneecap then finished off the first at his leisure. Instead he let them attack. Meyer came at him fast, intending to body check and knock him to the ground, I think, while Heinrich would follow through from above, maybe with his boots. That's the way it appeared to start. And here's what I think I saw. Meyer never reached him with that body check. Sergeant Zim whirled to face him, while kicking out and getting Heinrich in the belly -- and then Meyer was sailing through the air, his lunge helped along with a hearty assist from Zim. But all I am sure of is that the fight started and then there were two German boys sleeping peacefully, almost end to end, one face down and one face up, and Zim was standing over them, not even breathing hard.

"Jones," he said. "No, Jones left, didn't he? Mahmud! Let's have the water bucket, then stick them back into their sockets. Who's got my toothpick?" A few moments later the two were conscious, wet, and back in ranks. Zim looked at us and inquired gently, "Anybody else? Or shall we get on with setting-up exercises?" I didn't expect anybody else and I doubt if he did. But from down on the left flank, where the shorties hung out, a boy stepped out of ranks, came front and center. Zim looked down at him. "Just you? Or do you want to pick a partner?" "Just myself, sir." "As you say. Name?" "Shujumi, sir." Zim's eyes widened. "Any relation to Colonel Shujumi?" "I have the honor to be his son, sir." "Ah so! Well! Black Belt?" "No, sir. Not yet." "I'm glad you qualified that. Well, Shujumi, are we going to use contest rules, or shall I send for the ambulance?" "As you wish, sir. But I think, if I may be permitted an opinion, that contest rules would be more prudent." "I don't know just how you mean that, but I agree." Zim tossed his badge of authority aside, then, so help me, they backed off, faced each other, and bowed. After that they circled around each other in a half crouch, making tentative passes with their hands, and looking like a couple of roosters. Suddenly they touched -- and the little chap was down on the ground and Sergeant Zim was flying through the air over his head. But he didn't land with the dull, breath-paralyzing thud that Meyer had; he lit rolling and was on his feet as fast as Shujumi was and facing him. "Banzai!" Zim yelled and grinned. "Arigato," Shujumi answered and grinned back. They touched again almost without a pause and I thought the Sergeant was going to fly again. He didn't; he slithered straight in, there was a confusion of arms and legs and when the motion slowed down you could see that Zim was tucking Shujumi's left foot in his right ear -- a poor fit. Shujumi slapped the ground with a free hand; Zim let him up at once. They again bowed to each other. "Another fall, sir?" "Sorry. We've got work to do. Some other time, eh? For fun . . . and honor. Perhaps I should have told you; your honorable father trained me." "So I had already surmised, sir. Another time it is." Zim slapped him hard on the shoulder. "Back in ranks, soldier. C'pnee!" Then, for twenty minutes, we went through calisthenics that left me as dripping hot as I had been shivering cold. Zim led it himself, doing it all with us and shouting the count.

Cognitive decline of the university population

Razib at GNXP provides some nice figures relevant to my previous post Decline of the humanities.

The figures show the distribution of vocabulary scores (General Social Survey (GSS) WORDSUM) of college graduates and those without college degrees for the periods 1974-1984 and 1998-2008. It looks like the average score for college graduates has dropped by a significant fraction of a standard deviation over 25 years. This effect is entirely predictable given the larger percentage of Americans attending college in recent times.





It should not be surprising that a shrinking percentage of college students can write well or do basic mathematics, let alone appreciate Proust or quantum mechanics.

Additional years of education have not increased verbal abilities in the general population. This observation supports, at least in part, the signaling and sorting model of higher education (the primary value of credentials is that they reflect more or less invariant qualities such as IQ and Conscientiousness), as opposed to the model that higher education builds human capital.

Decline of the humanities

From an essay by William Chace, professor of English and former president of Wesleyan and Emory.

... Here is how the numbers have changed from 1970/71 to 2003/04 (the last academic year with available figures):

English: from 7.6 percent of the majors to 3.9 percent
Foreign languages and literatures: from 2.5 percent to 1.3 percent
Philosophy and religious studies: from 0.9 percent to 0.7 percent
History: from 18.5 percent to 10.7 percent
Business: from 13.7 percent to 21.9 percent

In one generation, then, the numbers of those majoring in the humanities dropped from a total of 30 percent to a total of less than 16 percent; during that same generation, business majors climbed from 14 percent to 22 percent. Despite last year’s debacle on Wall Street, the humanities have not benefited; students are still wagering that business jobs will be there when the economy recovers.

What are the causes for this decline? There are several, but at the root is the failure of departments of English across the country to champion, with passion, the books they teach and to make a strong case to undergraduates that the knowledge of those books and the tradition in which they exist is a human good in and of itself. What departments have done instead is dismember the curriculum, drift away from the notion that historical chronology is important, and substitute for the books themselves a scattered array of secondary considerations (identity studies, abstruse theory, sexuality, film and popular culture). In so doing, they have distanced themselves from the young people interested in good books.

... Alexander W. Astin’s research tells us that in the mid-1960s, more than 80 percent of entering college freshmen reported that nothing was more important than “developing a meaningful philosophy of life.” Astin, director of the Higher Education Research Institute at UCLA, reports that “being very well off financially” was only an afterthought, one that fewer than 45 percent of those freshmen thought to be an essential goal. As the years went on, however, and as tuition shot up, the two traded places; by 1977, financial goals had surged past philosophical ones, and by the year 2001 more than 70 percent of undergraduate students had their eyes trained on financial realities, while only 40 percent were still wrestling with meaningful philosophies.

Regarding the last paragraph, while there has undoubtedly been a general cultural shift, it is also true that a much larger fraction of the population attends college now, with resulting decline of average cognitive ability. Perhaps the elite of the 1960s had the luxury and cognitive ability to concentrate on their philosophy of life, as opposed to earning a living; students today do not.

For more see here:

Education and Verbal Ability over Time: Evidence from Three Multi-Time Sources

Nie, Golde and Butler

Abstract: During the 20th century, there was an unprecedented expansion in the level of educational attainment in America. Using three separate measures, this paper investigates whether there was a concurrent increase in verbal ability and skills. Changes in verbal ability in the general population as well as changes in the verbal ability of graduates of different levels of education are investigated. An additional investigation of how changes in the differences between males' and females' educational attainment are associated with changes in differences between their respective verbal abilities follows. The main finding is that there is little evidence that the large increase in educational attainment has resulted in an increase in any of the measures of verbal abilities and skills.

The origins of behavioral economics: Kahneman interview

Below is a great interview with Daniel Kahneman: Intuition and Rationality (full transcript).

Some years ago I wrote about Choices, Values, and Frames. See here for a nice discussion of behavioral economics.




Obvious to psychologists, difficult for economists. (See here for discussion by Robert Shiller, whose wife is a psychologist.)

Then when we worked on decision making, which was the second thing we did together, we published the theory paper in Econometrica, which is the prestige theory journal in economics. We were not intending to influence economics at all. It was just the best journal for that kind of article. If we had published exactly the same article in Psychological Review it would not have got the Nobel Prize, it [would have been seen as] a trivial detail. If we'd had a hostile referee in Econometrica we would have gone to Psych Review, but the big event that caused the prize is that some economists were interested, particularly an economist who's now my closest friend, Richard Thaler, who became interested in our work. He is the one who both taught me economics and brought these ideas into economics for their behavioral [value.]

So, it's because of the work of economists that I got the prize. I didn't get the prize for work that Amos and I did just because it was very good work. We got the prize (or I got the prize, because he couldn't share in it) because it had an impact on economics; but the impact was because people in economics saw it was relevant. They were a minority of people, and they did brilliant work.

The complexities of life:

It must have been the fall of '41, when there was a curfew for Jews. We were also supposed to be wearing a yellow star, and there was a curfew which I think was 6:00 PM. I was in first or second grade and I'd gone to play with a friend and I was going home and I missed the curfew, I was late. And so, I turned my sweater inside out and walked home, and as I was coming close I remember the street was deserted and there was this German soldier walking towards me. He was wearing the black uniform and I knew that was not good. That was the uniform of the SS. We were walking towards each other and as we were coming close he sort of beckoned me, and of course I went there, and he picked me up and hugged me. I remember being terrified that he would see the Star of David inside my sweater. Then he put me down and took out his wallet and showed me a picture of a boy and gave me some money. That's a formative memory because of what it meant about the complexity of things. I remember being very fascinated at the time by this and by stories of Hitler liking flowers and kissing babies. The complexity of evil was much on my mind as a seven- or eight-year-old.

[While traveling, missing my kids, I find myself smiling goofily at every 3-5 year old I see!]

Psychology research for the Israeli army:

One of the things I did do that I'm still quite proud of is I set up an interviewing system for the Israeli army to interview recruits for combat units. I learned in 2002 [that] that interview is still in place, fifty years later. ... In 1954 Paul Miel published a very important book in which he showed that clinicians are much less good at what they're doing than they think they are, and that their ability to forecast events and psychological behavior in the future is really quite limited. ... I set up an interview that didn't leave a lot of room for clinical intuition on the part of the interviewer. The interviewer went through a script and found out what the person had done as a civilian, basically. It turns out that when you use that information to produce ratings of various strengths, you get ratings that are predictive and are useful on whether somebody is responsible, or appreciates manliness, or is sociable, things like that. [Personality factors are stable and useful! Expert prediction is weak; algorithms beat experts.] In addition, it turned out (and this was important to me years later) that when people had acquired that information, then their clinical judgment was, in fact, good, whereas if they were trying to form a clinical impression [without a script] they couldn't do it.

These observations that I made at that time became quite essential to my research about fifteen years later and they were central to the work that ultimately led to the Nobel Prize. So, it started quite early, by accident.

[Note added: (September 2010) I met Kahneman at Scifoo this year and asked him briefly about his personnel selection experience. He seemed quite proud that the Israeli military is still using his methodology today.]

Advice on research:

My main advice would be to look at the natural traps that await anybody who does research. One of the real traps is to get trapped by a research program that is fundamentally uninteresting, and also, the reluctance to cut your losses so that when you've done something that is not very interesting, wasting time trying to publish it is almost always a big mistake, at least in my judgment. Look for things that are worth doing and discard things that don't work, and know that if you're reasonably good at what you're doing, and otherwise you shouldn't be in that profession, but if you're reasonably good at it, then ideas are a dime a dozen. You'll get lots of ideas. You need some ideas that work and that keep you excited and that you can do something with. And so, not getting stuck on an idea that doesn't work, but looking for other ideas -- that would be my general advice. That certainly has been how I've operated.

The bubble algorithm for human capital allocation

According the this WSJ article, at the peak of our finance bubble about a third of MIT graduates and almost half of all Harvard grads went into finance! Not so long ago a friend of mine at a derivatives desk made offers to 2 percent of the Caltech graduating class (only 4 people, but still).

To what field should society allocate the (N+1)-th big brain to achieve the highest marginal return? For years I've been told that the answer is finance -- markets are efficient after all, and price signals must be correct! ;-)

WSJ: Like nearly 30% of Massachusetts Institute of Technology graduates in recent years, Ted Fernandez set his sights on finance. Though he majored in materials science and engineering, he was wowed by tales of excitement from friends who went to Wall Street.

But when he stopped by an investment bank's booth at a job fair a year ago, it was eerily empty. The booth belonged to Lehman Brothers Holdings Inc., and the date was Sept. 18, three days after the 158-year-old bank filed for bankruptcy. Now Mr. Fernandez, 22 years old, is getting a master's in engineering at M.I.T. and aiming for a career in solar-power technology. [Another bubble? Let's hope not!]

"Undoubtedly, I would have gone into finance if the financial meltdown hadn't occurred," he says. "Now I won't make as much money, but I can go home at night and feel good about what I do. That's worth more than any amount of money."

Over the past 20 years, finance grew faster than almost any other sector of the U.S. economy, offering rich pay and luring a growing share of bright minds to trade securities, make loans, manage portfolios, engineer mergers and turn mortgages into complex derivatives. Now the finance bubble has deflated, forcing hundreds of thousands of employees to search for other work and sending new graduates looking elsewhere for careers.

... Harvard's 2009 graduating class shows the shift in career directions. Those entering finance and consulting tumbled to 20% of graduates this year from nearly twice that in 2008 and 47% the year before, according to a survey by the university's newspaper, the Crimson. ...

Even a modest of shift of talent could have an effect on society. When smart people become entrepreneurs, "they improve technology in the line of business they pursue, and, as a result, productivity and income grow," said a study by economists Kevin M. Murphy, Robert W. Vishny and Andrei Schleifer in 1990. By contrast, they said, allocation of talent to professions such as finance and law -- where returns come from distribution of wealth from others rather than wealth creation -- leads to lower productivity growth, fewer technological opportunities and slower economic growth. [Yes, I know, this argument is far from complete... better people making capital allocation decisions probably does lead to more efficient outcomes... but at what point do we reach saturation?]

"Some professions are socially more useful than others, even if they are not as well compensated," the economists said.

The figure below is from this earlier post:

Ellsberg, Cowen, Meth and My Lai

Our conference is winding down, so I guess I can get back to blogging without feeling guilty that I should be doing something else :-)

One of the pleasures of travel in the pre-internet era was being disconnected from world events. I fondly recall poring over the Herald Tribune in cafes as the preferred method for getting the news. During this trip I've only scanned the news and blogs in a cursory fashion, but I did listen to a fair number of podcasts while on planes, a ferry, and falling asleep near lapping waves. Below are some recommendations.


Daniel Ellsberg (Pentagon Papers) interview. In the interview Ellsberg compares Afghanistan to Vietnam; I don't know enough to judge whether the analogy is reasonable. I recommend this biography, which also covers his early history as an academic, working on game theory and decision theory. Ellsberg wrote this influential 1957 paper discussing the difference between Knightian (unmeasurable) uncertainty and (measurable) risk while a Junior Fellow at Harvard. These comments by Nobelist Thomas Schelling, who was something of a mentor to Ellsberg, are fascinating.

Interview with Tyler Cowen on Econtalk, about his recent book Create Your Own Economy. Buddhists vs autists?

Interview with Nick Reding, author of Methland: the Death and Life of an American Small Town, about methamphetamines and their impact on a small Iowa town.

The My Lai tapes (BBC documentary). Part 2. Sad but true.

The view from Crete

Click for larger version of photo. The first two photos are facing east in the morning. I've learned that digital cameras (even the crappy one in my iPhone) can process direct shots of the sun (nice smoothing algorithms!). The bottom photo is of the Orthodox Academy of Crete, where our conference is being held.

Black holes and decoherence talk

Slides for my talk in Crete. Previous discussion and link to paper.

Greetings from Athens

Click photos for larger versions

My advice: get to the Parthenon early, before the hordes of tourists arrive. I got there just after the opening and only had to share the place with a few dozen others. By the time I left the entrance area looked like a staging zone for D-Day.

The New Acropolis Museum (discussed in an earlier post) is beautiful, but they have a strict no-photographs policy.

The pictures below are from the National Archeological Museum. The weird gizmo is the famous Antikythera mechanism, which I almost missed -- it's in the Bronze Collection, which is overshadowed by all the statuary. The shiny modern thing is the Price model, built to illustrate the device's operation as an analog computer of astronomical motions. Yes, an analog computer built in 150 BC or so that calculates the positions of the sun, moon and planets -- it's unbelievable how advanced the ancient Greeks were!

Some favorite posts

NEW!  Podcast show Manifold.

I started writing this blog in 2004 (it has had millions of visitors!), and by now the content is a bit unwieldy to navigate, even with labels and search. I thought I'd make a list of some of my favorite posts and topics. Please suggest other posts to add to the list!

Pessimism of the Intellect, Optimism of the Will.

Feynman and me, Memories of Feynman. Labels: Feynman, path integrals.

Richard Feynman and the 19 year old me at my Caltech graduation:

Mama said knock you out (learning how to fight).

Many Worlds and quantum mechanics, a brief guide. Label: Many Worlds. Note, the usual quantum probabilities do not emerge naturally in this interpretation. See my papers On the Origin of Probability in Quantum Mechanics and The measure problem in no-collapse (many worlds) quantum mechanics.

Cognitive limitations: statistics , higher ed. Label: bounded rationality, human capital. Brainpower and globalization.

Expert predictions in soft subjects are unreliable. Intellectual honesty. Frauds! Label: expert predictions.

We can (crudely) measure cognitive ability using simple tests. (It is amazing to me that this is a controversial statement.) Randomly sampled eminent scientists have (very) high IQs, and given the observed stability of adult IQ the causality is clear: psychometrics works. The cult of genius? Income, Wealth, and IQ, One hundred thousand brains. Bezos on the Big Brains. Label: psychometrics.

Historically isolated groups of humans cluster genetically according to geographical ancestry. Explained in pictures , words , more words.

I am skeptical of all but the weakest claims of market efficiency. My talk on the 2008 credit crisis. Venn diagram for economics.

Careers, advice to geeks: A tale of two geeks , success vs ability. Labels: careers , startups , entrepreneurs.

Net worth , life satisfaction , happiness , the gilded age.

What is the likely development path for China in the next decades? Sustainability of China growth , China development: how big is the middle class? , Back to the future , Shanghai from an Indian perspective.

That curious institution, Caltech. How did a 16 year old kid from Iowa end up there? (See memories of Feynman above.)

There are geniuses in the world. The cult of genius.

My lovely kids. Photos. Autobiographical.


Update:

Credentialism and elite careers , Defining merit , elitism , brainpower

Recent videos (talks on genomics): https://www.youtube.com/results?search_query=hsu+genomics

Talks (some with slides + video):

Cold Spring Harbor Laboratory
Berkeley Innovative Genomics Institute and OpenAI
Janelia Research Campus (HHMI)
Allen Institute (Seattle) meeting on Genetics of Complex Traits

Review article: On the genetic architecture of cognitive ability and other quantitative traits (2014)

I work on algorithms for phenotype prediction from genotype, using new methods from high dimensional statistics. My estimate is that prediction of complex traits such as height, cognitive ability, or highly polygenic disease conditions will require data sets of order one million individuals (i.e., to build a model which accounts for most of the genetic variance). Once these models are available, human reproduction (and evolution!) will be revolutionized.

These papers are somewhat technical:
https://arxiv.org/abs/1310.2264
http://arxiv.org/abs/1408.6583

This one is a bit less technical and gives a broader overview:
http://arxiv.org/abs/1408.3421

Cow genomics (an existence proof):
http://infoproc.blogspot.com/2012/08/genomic-prediction-no-bull.html
http://infoproc.blogspot.com/2014/08/its-all-in-gene-cows.html

These are for popular audiences (Nautilus Magazine):
http://nautil.us/issue/18/genius/super_intelligent-humans-are-coming
http://nautil.us/issue/28/2050/dont-worry-smart-machines-will-take-us-with-them

2018: As anticipated, we now have good height predictors thanks to the 500k genome release of UK Biobank data: Scientists of Stature

Genomic predictors for common disease risk, constructed via machine learning on hundreds of thousands of genotypes. The predictors use anywhere from a few tens (e.g., 20 or 50) to thousands of SNPs to compute the risk PGS (Poly-Genic Score) for conditions such as diabetes, breast cancer, heart attack, and more: Genomic Prediction of Complex Disease Risk.

The Economist on polygenic risk scores (2019).

Detailed analysis of genetic architectures of disease risk predictors. Implications for pleiotropy.

Sibling validation of genomic predictors.

Recent papers from my group:

https://www.genetics.org/content/210/2/477
https://www.nature.com/articles/s41598-019-51258-x
https://www.nature.com/articles/s41598-020-68881-8
https://www.nature.com/articles/s41598-020-69927-7


2021 review article, prepared for the book Genomic Prediction of Complex Traits, Springer Nature series Methods in Molecular Biology: 

From Genotype to Phenotype: polygenic prediction of complex human traits

Krugman: the Economists have no clothes

Only Nixon could go to China, and only Paul Krugman could write this mea culpa for the discipline of economics.

There was a bubble in housing -- everybody knows that now. There was a bubble in finance itself -- the financial share of national income reached an all-time high just before the crisis; not too many people know this. There was even an academic bubble in the field of economics: the perceived quality of results in the field, reflected in the salaries and prestige of economics professors (e.g., relative to other social scientists), was as inflated as the price of any McMansion -- surely every university dean must understand this now? (Bayesian / Machine Learning comment: do these guys ever update? Or is it "All priors, all the time"?)

I suggest reading the whole thing. I've only excerpted the last three paragraphs below. (See also related essay by Richard Posner and this interview with Bill Janeway.)

How Did Economists Get It So Wrong?: ... So here’s what I think economists have to do. First, they have to face up to the inconvenient reality that financial markets fall far short of perfection, that they are subject to extraordinary delusions and the madness of crowds. Second, they have to admit — and this will be very hard for the people who giggled and whispered over Keynes — that Keynesian economics remains the best framework we have for making sense of recessions and depressions. Third, they’ll have to do their best to incorporate the realities of finance into macroeconomics.

Many economists will find these changes deeply disturbing. It will be a long time, if ever, before the new, more realistic approaches to finance and macroeconomics offer the same kind of clarity, completeness and sheer beauty that characterizes the full neoclassical approach. To some economists that will be a reason to cling to neoclassicism, despite its utter failure to make sense of the greatest economic crisis in three generations. This seems, however, like a good time to recall the words of H. L. Mencken: “There is always an easy solution to every human problem — neat, plausible and wrong.”

When it comes to the all-too-human problem of recessions and depressions, economists need to abandon the neat but wrong solution of assuming that everyone is rational and markets work perfectly. The vision that emerges as the profession rethinks its foundations may not be all that clear; it certainly won’t be neat; but we can hope that it will have the virtue of being at least partly right.

Related posts on Krugman's article from two thoughtful economists, Brad DeLong: Which economists got it so wrong? , Where does macro go from here? and Arnold Kling: Krugman vs Blanchard.

Athens and Crete

I'm off to Athens and Crete next week. The main purpose of the trip is to attend a meeting on quantum gravity, but I'm also looking forward to visiting this new museum:



See slideshow.

Now that they have an appropriate place for them, will the Greeks get the Elgin Marbles back from the British?

NYTimes: ... For more than 30 years, Greece has been working, through diplomacy and public relations offensives, to regain the Elgin Marbles, sections of a decorative frieze that adorned the Parthenon until Lord Elgin ordered them removed in the early 19th century, during his tenure as British ambassador to the Ottoman Empire. Years later, bankrupt, he sold them to the British Museum, where they have been a major attraction since 1816.

Today, almost two centuries on, many Greeks hope the opening of the museum will focus international attention on their country’s claim to the so-called Elgin Marbles, and put an end to Britain’s longtime argument that it is in a better position to look after those 2,500-year-old panels. Last week the Greek government turned down an offer from the British Museum for a three-month loan of the collection, because it came with the condition that the Greeks formally acknowledge British ownership.

“This is a nonstarter for any discussion,” Mr. Samaras said. “No Greek can sign up to that.”

The new museum, 226,000 square feet of glass and concrete designed by the New York architect Bernard Tschumi, replaces the old Acropolis Museum, a small 1874 building tucked into the rock of the Acropolis next to the Parthenon. The design, introduced in 2001, was meant to be completed in time for the 2004 Olympics, but dozens of legal battles — many having to do with some 25 buildings that were demolished to make room for it — delayed the process for years.

Eventually I suspect the Chinese, Egyptians and many others will recover their stolen national treasures from museums in Europe and the US.

Flowers for Algernon

I read this when I was a kid, and it had a strong effect on me, at least in part because I had a next-door neighbor who was my age and mentally retarded. Now that I'm a parent it strikes me that we all follow the path of Charlie Gordon -- starting from modest intellectual ability, developing to a peak, followed by inevitable decline. It happens more gradually for us than for him, but we are equally aware of the process and where it leads.

Wikipedia: ... The titular Algernon is a laboratory mouse who has undergone surgery to increase his intelligence by artificial means. The story is told as a series of progress reports written by Charlie, the first human test subject for the surgery.

... Another key moment came in 1957, while [author] Keyes was teaching English to students with special needs; one student asked him if it would be possible to be put into a regular class if he worked hard and became smart.

... The procedure is a success and, three months later, Charlie's IQ has reached 185. However, as his intelligence, education and understanding of the world around him increases, his relationships with people deteriorate.

... Charlie discovers a flaw in the theories that led Nemur and Strauss to develop their intelligence-enhancing procedure. Shortly thereafter, Algernon starts behaving erratically, loses his new intelligence, and dies. As Charlie does further research, he determines that he too will inevitably revert to his old condition.

From the Wikipedia entry I learned the novel is based on a short story that appeared years earlier and won the science fiction Hugo award. The full text is here (at least for the moment). I think the novel is stronger than the short story.

Here's an excerpt (Google Books).

June 29 -- ... I phoned Landsdoff at the New Institute for Advanced Study, about the possibility of using the pair-production nuclear photoeffect for exploratory work in biophysics. At first he thought I was a crackpot, but when I pointed out to him the flaws in his article in the New Institute Journal he kept me on the phone nearly an hour. He wants me to come to the Institute to discuss my ideas with his group. ... That's the problem, of course. I don't know how much time I have. A month? A year? The rest of my life? That depends on what I find out about the psychophysical side-effects of the experiment. ...