Copula in cosmology?

A new paper co-authored by one of my collaborators: http://arxiv.org/abs/0909.5187

From Finance to Cosmology: The Copula of Large-Scale Structure

Robert J. Scherrer, Andreas A. Berlind, Qingqing Mao, Cameron K. McBride (Vanderbilt University)

Any multivariate distribution can be uniquely decomposed into marginal (1-point) distributions, and a function called the copula, which contains all of the information on correlations between the distributions. The copula provides an important new methodology for analyzing the density field in large-scale structure. We derive the empirical 2-point copula for the evolved dark matter density field. We find that this empirical copula is well-approximated by a Gaussian copula. We consider the possibility that the full n-point copula is also Gaussian and describe some of the consequences of this hypothesis. Future directions for investigation are discussed.

I don't understand the gravitational dynamics well enough to guess whether we should expect the dark matter distribution today to be described by Gaussian copula. The authors insert a number of caveats, referring to the use of copula to price mortgage securities :-)

See here for some correspondence with Scherrer on the topic of genius and modern science, and here for some popular coverage of the paper we wrote.

Physicist to lead Williams College

I learned yesterday that a contemporary of mine, theoretical physicist Adam Falk, is the new President of Williams College.

Congratulations Adam! I think the future is bright at Williams :-)

See here for more, including some video interviews. Discussion by Williams alumni at Ephblog.

Starship Troopers

This commercial for the video game Halo 3: ODST (Orbital Drop Shock Troops) comes as close as anything I've seen to capturing the flavor of the Heinlein novel Starship Troopers. Forget about the 1997 movie directed by Paul Verhoeven; it's pure camp. Some might classify the original novel as pulp (see excerpt below), but I think it is much more than that.

The ODST project started as a Halo movie project involving Peter Jackson of Lord of the Rings fame. Perhaps this explains the cinematic qualities. (I'm not a gamer; I just noticed the commercial while watching Iowa beat Penn State last night and got interested in what seemed to be an obvious Starship Trooper influence.)

Wikipedia: Starship Troopers is a novel set in an unspecified time in the future, although probably not in the far-flung future. It chronicles the experiences of Juan "Johnnie" Rico, the story's narrator, during his enlistment and training in the Mobile Infantry, and his participation in an interstellar war between the Terran Federation and the Arachnids (referred to as "the Bugs") of Klendathu. It is narrated as a series of flashbacks — one of only a few Heinlein novels to use that narrative device[11] — and contains large sections of character discussion and introspection, as well as exposition, all meant to detail the political theory and philosophical beliefs underlying the society that Juan Rico lives in.

The novel opens with Rico aboard the corvette Rodger Young, platoon transport for "Rasczak's Roughnecks", about to embark on a raid against the planet of the "Skinnies", allies of the Arachnids. We learn that he is a "cap" trooper (called this because they are dropped in capsules from the ship in orbit toward their drop zones) in the Terran Federation's Mobile Infantry (M.I.). The raid itself, one of the few instances of actual combat in the novel, is relatively brief: the Mobile Infantrymen land in the capital city, destroy their targets while trying to avoid unnecessary Skinny casualties, and withdraw, suffering three injured (one fatal) in the process.

The story then flashes back to Rico's graduation from high school and his decision to sign up for Federal Service, rather than attend Harvard University, over the objections of his wealthy father. This is the only chapter that describes Rico's civilian life, and most of it is spent recording the monologues of two people: retired Lt. Colonel Jean V. Dubois, Rico's school instructor in the subject of History and Moral Philosophy, and Fleet Sergeant Ho, a recruiter for the Armed forces of the Terran Federation.

Many readers have felt that Dubois serves as a stand-in for Heinlein throughout the novel. He delivers what is probably the book's most famous soliloquy, on how violence "has settled more issues in history than has any other factor."[12] Fleet Sergeant Ho offers a separate angle on military service to that of Dubois. (Ho has prostheses for several limbs, but does not wear them on duty at the front door of the federal building. This is calculated to remind applicants of the real risks of service, and to weed out those not willing to take such risks in the service of the Federation).

Interspersed throughout the book are other flashbacks to Rico's high school History and Moral Philosophy course, which describe how, in the Terran Federation, the rights of a full Citizen (to vote, and hold public office) must be earned through voluntary Federal service. However, the franchise cannot be exercised until after honorable discharge from the Service, which means that active members of the Service cannot vote. Those residents who opt not to perform Federal Service retain the other rights generally associated with a modern democracy (e.g. free speech, assembly, etc.), but cannot vote or hold public office. This structure arose ad hoc after the collapse of the 20th century Western democracies, brought on by both social failures at home and military defeat by the Chinese Hegemony overseas (i.e. looking forward into the late 20th century from the time the novel was written in the late 1950s).[13]

After enlisting in the Mobile Infantry, Rico is assigned to boot camp at Camp Arthur Currie. Five chapters are spent exploring Rico's training, under the guidance of his chief instructor, Career Ship's Sergeant Charles Zim. Boot camp is deliberately so rigorous that fewer than ten percent of the recruits complete basic training; the rest either resign (with no penalty, save never being able to vote); are expelled (likewise); are given medical discharges (which may however be refused); or assigned to lesser duties (enabling them to vote after their service is finished); or die in training. ...

Here's the version of the book I've had since I was a kid:



Yes, I enjoyed a misspent youth, during which I read all of Heinlein's books and even played the Avalon Hill boardgame version of Starship Troopers.

Give me a few Sergeant Zim's and I'll conquer the galaxy! :-)

Bootcamp for the Mobile Infantry: ... But exercise will keep you warm and they saw to it that we got plenty of that. The first morning we were there they woke us up before daybreak. I had had trouble adjusting to the change in time zones and it seemed to me that I had just got to sleep; I couldn't believe that anyone seriously intended that I should get up in the middle of the night. But they did mean it. A speaker somewhere was blaring out a military march, fit to wake the dead, and a hairy nuisance who had come charging down the company street yelling, "Everybody out! Show a leg! On the bounce!" came marauding back again just as I had pulled the covers over my head, tipped over my cot and dumped me on the cold hard ground. It was an impersonal attention; he didn't even wait to see if I hit. Ten minutes later, dressed in trousers, undershirt, and shoes, I was lined up with the others in ragged ranks for setting-up exercises just as the Sun looked over the eastern horizon.

Facing us was a big broad-shouldered, mean-looking man, dressed just as we were -- except that while I looked and felt like a poor job of embalming, his chin was shaved blue, his trousers were sharply creased, you could have used his shoes for mirrors, and his manner was alert, wide-awake, relaxed, and rested. You got the impression that he never needed to sleep -- just ten-thousand-mile checkups and dust him off occasionally. He bellowed, "C'pnee! Atten . . . shut! I am Career Ship's Sergeant Zim, your company commander. When you speak to me, you will salute and say, `Sir' -- you will salute and `sir' anyone who carries an instructor's baton -- " He was carrying a swagger cane and now made a quick reverse moulinet with it to show what he meant by an instructor's baton; I had noticed men carrying them when we had arrived the night before and had intended to get one myself -- they looked smart. Now I changed my mind. " -- because we don't have enough officers around here for you to practice on. You'll practice on us. ...

Zim turned back to the rest of us, still shivering at attention. He walked up and down, looked us over, and seemed awfully unhappy. At last he stepped out in front of us, shook his head, and said, apparently to himself but he had a voice that carried: "To think that this had to happen to me!" He looked at us. "You apes -- No, not `apes'; you don't rate that much. You pitiful mob of sickly monkeys . . . you sunken-chested, slack-bellied, drooling refugees from apron strings. In my whole life I never saw such a disgraceful huddle of momma's spoiled little darlings in -- you, there! Suck up the gut! Eyes front! I'm talking to you!" I pulled in my belly, even though I was not sure he had addressed me. He went on and on and I began to forget my goose flesh in hearing him storm. He never once repeated himself and he never used either profanity or obscenity. (I learned later that he saved those for very special occasions, which this wasn't.) But he described our shortcomings, physical, mental, moral, and genetic, in great and insulting detail. But somehow I was not insulted; I became greatly interested in studying his command of language. I wished that we had had him on our debate team.

At last he stopped and seemed about to cry. "I can't stand it," he said bitterly. "I've just got to work some of it off -- I had a better set of wooden soldiers when I was six ALL RIGHT! Is there any one of you jungle lice who thinks he can whip me? Is there a man in the crowd? Speak up !" There was a short silence to which I contributed. I didn't have any doubt at all that he could whip me; I was convinced. I heard a voice far down the line, the tall end. "Ah reckon ah can . . . suh." Zim looked happy. "Good! Step out here where I can see you." The recruit did so and he was impressive, at least three inches taller than Sergeant Zim and broader across the shoulders. "What's your name, soldier?" "Breckinridge, suh -- and ah weigh two hundred and ten pounds an' theah ain't any of it `slack-bellied.' " "Any particular way you'd like to fight?" "Suh, you jus' pick youah own method of dyin'. Ah'm not fussy." "Okay, no rules. Start whenever you like." Zim tossed his baton aside. It started -- and it was over. The big recruit was sitting on the ground, holding his left wrist in his right hand. He didn't say anything. Zim bent over him. "Broken?" "Reckon it might be . . . suh." "I'm sorry. You hurried me a little. Do you know where the dispensary is? Never mind -- Jones! Take Breckinridge over to the dispensary."

As they left Zim slapped him on the right shoulder and said quietly, "Let's try it again in a month or so. I'll show you what happened." I think it was meant to be a private remark but they were standing about six feet in front of where I was slowly freezing solid. Zim stepped back and called out, "Okay, we've got one man in this company, at least. I feel better. Do we have another one? Do we have two more? Any two of you scrofulous toads think you can stand up to me?" He looked back and forth along our ranks. "Chicken-livered, spineless -- oh, oh! Yes? Step out." Two men who had been side by side in ranks stepped out together; I suppose they had arranged it in whispers right there, but they also were far down the tall end, so I didn't hear. Zim smiled at them. "Names, for your next of kin, please." "Heinrich." "Heinrich what?" "Heinrich, sir. Bitte." He spoke rapidly to the other recruit and added politely, "He doesn't speak much Standard English yet, sir." "Meyer, mein Herr," the second man supplied. "That's okay, lots of `em don't speak much of it when they get here -I didn't myself. Tell Meyer not to worry, he'll pick it up. But he understands what we are going to do?" "Jawohl," agreed Meyer. "Certainly, sir. He understands Standard, he just can't speak it fluently." "All right. Where did you two pick up those face scars? Heidelberg?" "Nein -- no, sir. Ko:nigsberg." "Same thing." Zim had picked up his baton after fighting Breckinridge; he twirled it and asked, "Perhaps you would each like to borrow one of these?" "It would not be fair to you, sir," Heinrich answered carefully. "Bare hands, if you please." "Suit yourself. Though I might fool you. Ko:nigsberg, eh? Rules?" "How can there be rules, sir, with three?" "An interesting point. Well, let's agree that if eyes are gouged out they must be handed back when it's over. And tell your Korpsbruder that I'm ready now. Start when you like." Zim tossed his baton away; someone caught it. "You joke, sir. We will not gouge eyes." "No eye gouging, agreed. `Fire when ready, Gridley.' " "Please?" "Come on and fight! Or get back into ranks!" Now I am not sure that I saw it happen this way; I may have learned part of it later, in training. But here is what I think happened: The two moved out on each side of our company commander until they had him completely flanked but well out of contact. From this position there is a choice of four basic moves for the man working alone, moves that take advantage of his own mobility and of the superior co-ordination of one man as compared with two -- Sergeant Zim says (correctly) that any group is weaker than a man alone unless they are perfectly trained to work together. For example, Zim could have feinted at one of them, bounced fast to the other with a disabler, such as a broken kneecap then finished off the first at his leisure. Instead he let them attack. Meyer came at him fast, intending to body check and knock him to the ground, I think, while Heinrich would follow through from above, maybe with his boots. That's the way it appeared to start. And here's what I think I saw. Meyer never reached him with that body check. Sergeant Zim whirled to face him, while kicking out and getting Heinrich in the belly -- and then Meyer was sailing through the air, his lunge helped along with a hearty assist from Zim. But all I am sure of is that the fight started and then there were two German boys sleeping peacefully, almost end to end, one face down and one face up, and Zim was standing over them, not even breathing hard.

"Jones," he said. "No, Jones left, didn't he? Mahmud! Let's have the water bucket, then stick them back into their sockets. Who's got my toothpick?" A few moments later the two were conscious, wet, and back in ranks. Zim looked at us and inquired gently, "Anybody else? Or shall we get on with setting-up exercises?" I didn't expect anybody else and I doubt if he did. But from down on the left flank, where the shorties hung out, a boy stepped out of ranks, came front and center. Zim looked down at him. "Just you? Or do you want to pick a partner?" "Just myself, sir." "As you say. Name?" "Shujumi, sir." Zim's eyes widened. "Any relation to Colonel Shujumi?" "I have the honor to be his son, sir." "Ah so! Well! Black Belt?" "No, sir. Not yet." "I'm glad you qualified that. Well, Shujumi, are we going to use contest rules, or shall I send for the ambulance?" "As you wish, sir. But I think, if I may be permitted an opinion, that contest rules would be more prudent." "I don't know just how you mean that, but I agree." Zim tossed his badge of authority aside, then, so help me, they backed off, faced each other, and bowed. After that they circled around each other in a half crouch, making tentative passes with their hands, and looking like a couple of roosters. Suddenly they touched -- and the little chap was down on the ground and Sergeant Zim was flying through the air over his head. But he didn't land with the dull, breath-paralyzing thud that Meyer had; he lit rolling and was on his feet as fast as Shujumi was and facing him. "Banzai!" Zim yelled and grinned. "Arigato," Shujumi answered and grinned back. They touched again almost without a pause and I thought the Sergeant was going to fly again. He didn't; he slithered straight in, there was a confusion of arms and legs and when the motion slowed down you could see that Zim was tucking Shujumi's left foot in his right ear -- a poor fit. Shujumi slapped the ground with a free hand; Zim let him up at once. They again bowed to each other. "Another fall, sir?" "Sorry. We've got work to do. Some other time, eh? For fun . . . and honor. Perhaps I should have told you; your honorable father trained me." "So I had already surmised, sir. Another time it is." Zim slapped him hard on the shoulder. "Back in ranks, soldier. C'pnee!" Then, for twenty minutes, we went through calisthenics that left me as dripping hot as I had been shivering cold. Zim led it himself, doing it all with us and shouting the count.

Cognitive decline of the university population

Razib at GNXP provides some nice figures relevant to my previous post Decline of the humanities.

The figures show the distribution of vocabulary scores (General Social Survey (GSS) WORDSUM) of college graduates and those without college degrees for the periods 1974-1984 and 1998-2008. It looks like the average score for college graduates has dropped by a significant fraction of a standard deviation over 25 years. This effect is entirely predictable given the larger percentage of Americans attending college in recent times.





It should not be surprising that a shrinking percentage of college students can write well or do basic mathematics, let alone appreciate Proust or quantum mechanics.

Additional years of education have not increased verbal abilities in the general population. This observation supports, at least in part, the signaling and sorting model of higher education (the primary value of credentials is that they reflect more or less invariant qualities such as IQ and Conscientiousness), as opposed to the model that higher education builds human capital.

Decline of the humanities

From an essay by William Chace, professor of English and former president of Wesleyan and Emory.

... Here is how the numbers have changed from 1970/71 to 2003/04 (the last academic year with available figures):

English: from 7.6 percent of the majors to 3.9 percent
Foreign languages and literatures: from 2.5 percent to 1.3 percent
Philosophy and religious studies: from 0.9 percent to 0.7 percent
History: from 18.5 percent to 10.7 percent
Business: from 13.7 percent to 21.9 percent

In one generation, then, the numbers of those majoring in the humanities dropped from a total of 30 percent to a total of less than 16 percent; during that same generation, business majors climbed from 14 percent to 22 percent. Despite last year’s debacle on Wall Street, the humanities have not benefited; students are still wagering that business jobs will be there when the economy recovers.

What are the causes for this decline? There are several, but at the root is the failure of departments of English across the country to champion, with passion, the books they teach and to make a strong case to undergraduates that the knowledge of those books and the tradition in which they exist is a human good in and of itself. What departments have done instead is dismember the curriculum, drift away from the notion that historical chronology is important, and substitute for the books themselves a scattered array of secondary considerations (identity studies, abstruse theory, sexuality, film and popular culture). In so doing, they have distanced themselves from the young people interested in good books.

... Alexander W. Astin’s research tells us that in the mid-1960s, more than 80 percent of entering college freshmen reported that nothing was more important than “developing a meaningful philosophy of life.” Astin, director of the Higher Education Research Institute at UCLA, reports that “being very well off financially” was only an afterthought, one that fewer than 45 percent of those freshmen thought to be an essential goal. As the years went on, however, and as tuition shot up, the two traded places; by 1977, financial goals had surged past philosophical ones, and by the year 2001 more than 70 percent of undergraduate students had their eyes trained on financial realities, while only 40 percent were still wrestling with meaningful philosophies.

Regarding the last paragraph, while there has undoubtedly been a general cultural shift, it is also true that a much larger fraction of the population attends college now, with resulting decline of average cognitive ability. Perhaps the elite of the 1960s had the luxury and cognitive ability to concentrate on their philosophy of life, as opposed to earning a living; students today do not.

For more see here:

Education and Verbal Ability over Time: Evidence from Three Multi-Time Sources

Nie, Golde and Butler

Abstract: During the 20th century, there was an unprecedented expansion in the level of educational attainment in America. Using three separate measures, this paper investigates whether there was a concurrent increase in verbal ability and skills. Changes in verbal ability in the general population as well as changes in the verbal ability of graduates of different levels of education are investigated. An additional investigation of how changes in the differences between males' and females' educational attainment are associated with changes in differences between their respective verbal abilities follows. The main finding is that there is little evidence that the large increase in educational attainment has resulted in an increase in any of the measures of verbal abilities and skills.

The origins of behavioral economics: Kahneman interview

Below is a great interview with Daniel Kahneman: Intuition and Rationality (full transcript).

Some years ago I wrote about Choices, Values, and Frames. See here for a nice discussion of behavioral economics.




Obvious to psychologists, difficult for economists. (See here for discussion by Robert Shiller, whose wife is a psychologist.)

Then when we worked on decision making, which was the second thing we did together, we published the theory paper in Econometrica, which is the prestige theory journal in economics. We were not intending to influence economics at all. It was just the best journal for that kind of article. If we had published exactly the same article in Psychological Review it would not have got the Nobel Prize, it [would have been seen as] a trivial detail. If we'd had a hostile referee in Econometrica we would have gone to Psych Review, but the big event that caused the prize is that some economists were interested, particularly an economist who's now my closest friend, Richard Thaler, who became interested in our work. He is the one who both taught me economics and brought these ideas into economics for their behavioral [value.]

So, it's because of the work of economists that I got the prize. I didn't get the prize for work that Amos and I did just because it was very good work. We got the prize (or I got the prize, because he couldn't share in it) because it had an impact on economics; but the impact was because people in economics saw it was relevant. They were a minority of people, and they did brilliant work.

The complexities of life:

It must have been the fall of '41, when there was a curfew for Jews. We were also supposed to be wearing a yellow star, and there was a curfew which I think was 6:00 PM. I was in first or second grade and I'd gone to play with a friend and I was going home and I missed the curfew, I was late. And so, I turned my sweater inside out and walked home, and as I was coming close I remember the street was deserted and there was this German soldier walking towards me. He was wearing the black uniform and I knew that was not good. That was the uniform of the SS. We were walking towards each other and as we were coming close he sort of beckoned me, and of course I went there, and he picked me up and hugged me. I remember being terrified that he would see the Star of David inside my sweater. Then he put me down and took out his wallet and showed me a picture of a boy and gave me some money. That's a formative memory because of what it meant about the complexity of things. I remember being very fascinated at the time by this and by stories of Hitler liking flowers and kissing babies. The complexity of evil was much on my mind as a seven- or eight-year-old.

[While traveling, missing my kids, I find myself smiling goofily at every 3-5 year old I see!]

Psychology research for the Israeli army:

One of the things I did do that I'm still quite proud of is I set up an interviewing system for the Israeli army to interview recruits for combat units. I learned in 2002 [that] that interview is still in place, fifty years later. ... In 1954 Paul Miel published a very important book in which he showed that clinicians are much less good at what they're doing than they think they are, and that their ability to forecast events and psychological behavior in the future is really quite limited. ... I set up an interview that didn't leave a lot of room for clinical intuition on the part of the interviewer. The interviewer went through a script and found out what the person had done as a civilian, basically. It turns out that when you use that information to produce ratings of various strengths, you get ratings that are predictive and are useful on whether somebody is responsible, or appreciates manliness, or is sociable, things like that. [Personality factors are stable and useful! Expert prediction is weak; algorithms beat experts.] In addition, it turned out (and this was important to me years later) that when people had acquired that information, then their clinical judgment was, in fact, good, whereas if they were trying to form a clinical impression [without a script] they couldn't do it.

These observations that I made at that time became quite essential to my research about fifteen years later and they were central to the work that ultimately led to the Nobel Prize. So, it started quite early, by accident.

[Note added: (September 2010) I met Kahneman at Scifoo this year and asked him briefly about his personnel selection experience. He seemed quite proud that the Israeli military is still using his methodology today.]

Advice on research:

My main advice would be to look at the natural traps that await anybody who does research. One of the real traps is to get trapped by a research program that is fundamentally uninteresting, and also, the reluctance to cut your losses so that when you've done something that is not very interesting, wasting time trying to publish it is almost always a big mistake, at least in my judgment. Look for things that are worth doing and discard things that don't work, and know that if you're reasonably good at what you're doing, and otherwise you shouldn't be in that profession, but if you're reasonably good at it, then ideas are a dime a dozen. You'll get lots of ideas. You need some ideas that work and that keep you excited and that you can do something with. And so, not getting stuck on an idea that doesn't work, but looking for other ideas -- that would be my general advice. That certainly has been how I've operated.

The bubble algorithm for human capital allocation

According the this WSJ article, at the peak of our finance bubble about a third of MIT graduates and almost half of all Harvard grads went into finance! Not so long ago a friend of mine at a derivatives desk made offers to 2 percent of the Caltech graduating class (only 4 people, but still).

To what field should society allocate the (N+1)-th big brain to achieve the highest marginal return? For years I've been told that the answer is finance -- markets are efficient after all, and price signals must be correct! ;-)

WSJ: Like nearly 30% of Massachusetts Institute of Technology graduates in recent years, Ted Fernandez set his sights on finance. Though he majored in materials science and engineering, he was wowed by tales of excitement from friends who went to Wall Street.

But when he stopped by an investment bank's booth at a job fair a year ago, it was eerily empty. The booth belonged to Lehman Brothers Holdings Inc., and the date was Sept. 18, three days after the 158-year-old bank filed for bankruptcy. Now Mr. Fernandez, 22 years old, is getting a master's in engineering at M.I.T. and aiming for a career in solar-power technology. [Another bubble? Let's hope not!]

"Undoubtedly, I would have gone into finance if the financial meltdown hadn't occurred," he says. "Now I won't make as much money, but I can go home at night and feel good about what I do. That's worth more than any amount of money."

Over the past 20 years, finance grew faster than almost any other sector of the U.S. economy, offering rich pay and luring a growing share of bright minds to trade securities, make loans, manage portfolios, engineer mergers and turn mortgages into complex derivatives. Now the finance bubble has deflated, forcing hundreds of thousands of employees to search for other work and sending new graduates looking elsewhere for careers.

... Harvard's 2009 graduating class shows the shift in career directions. Those entering finance and consulting tumbled to 20% of graduates this year from nearly twice that in 2008 and 47% the year before, according to a survey by the university's newspaper, the Crimson. ...

Even a modest of shift of talent could have an effect on society. When smart people become entrepreneurs, "they improve technology in the line of business they pursue, and, as a result, productivity and income grow," said a study by economists Kevin M. Murphy, Robert W. Vishny and Andrei Schleifer in 1990. By contrast, they said, allocation of talent to professions such as finance and law -- where returns come from distribution of wealth from others rather than wealth creation -- leads to lower productivity growth, fewer technological opportunities and slower economic growth. [Yes, I know, this argument is far from complete... better people making capital allocation decisions probably does lead to more efficient outcomes... but at what point do we reach saturation?]

"Some professions are socially more useful than others, even if they are not as well compensated," the economists said.

The figure below is from this earlier post:

Ellsberg, Cowen, Meth and My Lai

Our conference is winding down, so I guess I can get back to blogging without feeling guilty that I should be doing something else :-)

One of the pleasures of travel in the pre-internet era was being disconnected from world events. I fondly recall poring over the Herald Tribune in cafes as the preferred method for getting the news. During this trip I've only scanned the news and blogs in a cursory fashion, but I did listen to a fair number of podcasts while on planes, a ferry, and falling asleep near lapping waves. Below are some recommendations.


Daniel Ellsberg (Pentagon Papers) interview. In the interview Ellsberg compares Afghanistan to Vietnam; I don't know enough to judge whether the analogy is reasonable. I recommend this biography, which also covers his early history as an academic, working on game theory and decision theory. Ellsberg wrote this influential 1957 paper discussing the difference between Knightian (unmeasurable) uncertainty and (measurable) risk while a Junior Fellow at Harvard. These comments by Nobelist Thomas Schelling, who was something of a mentor to Ellsberg, are fascinating.

Interview with Tyler Cowen on Econtalk, about his recent book Create Your Own Economy. Buddhists vs autists?

Interview with Nick Reding, author of Methland: the Death and Life of an American Small Town, about methamphetamines and their impact on a small Iowa town.

The My Lai tapes (BBC documentary). Part 2. Sad but true.

The view from Crete

Click for larger version of photo. The first two photos are facing east in the morning. I've learned that digital cameras (even the crappy one in my iPhone) can process direct shots of the sun (nice smoothing algorithms!). The bottom photo is of the Orthodox Academy of Crete, where our conference is being held.

Black holes and decoherence talk

Slides for my talk in Crete. Previous discussion and link to paper.

Greetings from Athens

Click photos for larger versions

My advice: get to the Parthenon early, before the hordes of tourists arrive. I got there just after the opening and only had to share the place with a few dozen others. By the time I left the entrance area looked like a staging zone for D-Day.

The New Acropolis Museum (discussed in an earlier post) is beautiful, but they have a strict no-photographs policy.

The pictures below are from the National Archeological Museum. The weird gizmo is the famous Antikythera mechanism, which I almost missed -- it's in the Bronze Collection, which is overshadowed by all the statuary. The shiny modern thing is the Price model, built to illustrate the device's operation as an analog computer of astronomical motions. Yes, an analog computer built in 150 BC or so that calculates the positions of the sun, moon and planets -- it's unbelievable how advanced the ancient Greeks were!

Some favorite posts

NEW!  Podcast show Manifold.

I started writing this blog in 2004 (it has had millions of visitors!), and by now the content is a bit unwieldy to navigate, even with labels and search. I thought I'd make a list of some of my favorite posts and topics. Please suggest other posts to add to the list!

Pessimism of the Intellect, Optimism of the Will.

Feynman and me, Memories of Feynman. Labels: Feynman, path integrals.

Richard Feynman and the 19 year old me at my Caltech graduation:

Mama said knock you out (learning how to fight).

Many Worlds and quantum mechanics, a brief guide. Label: Many Worlds. Note, the usual quantum probabilities do not emerge naturally in this interpretation. See my papers On the Origin of Probability in Quantum Mechanics and The measure problem in no-collapse (many worlds) quantum mechanics.

Cognitive limitations: statistics , higher ed. Label: bounded rationality, human capital. Brainpower and globalization.

Expert predictions in soft subjects are unreliable. Intellectual honesty. Frauds! Label: expert predictions.

We can (crudely) measure cognitive ability using simple tests. (It is amazing to me that this is a controversial statement.) Randomly sampled eminent scientists have (very) high IQs, and given the observed stability of adult IQ the causality is clear: psychometrics works. The cult of genius? Income, Wealth, and IQ, One hundred thousand brains. Bezos on the Big Brains. Label: psychometrics.

Historically isolated groups of humans cluster genetically according to geographical ancestry. Explained in pictures , words , more words.

I am skeptical of all but the weakest claims of market efficiency. My talk on the 2008 credit crisis. Venn diagram for economics.

Careers, advice to geeks: A tale of two geeks , success vs ability. Labels: careers , startups , entrepreneurs.

Net worth , life satisfaction , happiness , the gilded age.

What is the likely development path for China in the next decades? Sustainability of China growth , China development: how big is the middle class? , Back to the future , Shanghai from an Indian perspective.

That curious institution, Caltech. How did a 16 year old kid from Iowa end up there? (See memories of Feynman above.)

There are geniuses in the world. The cult of genius.

My lovely kids. Photos. Autobiographical.


Update:

Credentialism and elite careers , Defining merit , elitism , brainpower

Recent videos (talks on genomics): https://www.youtube.com/results?search_query=hsu+genomics

Talks (some with slides + video):

Cold Spring Harbor Laboratory
Berkeley Innovative Genomics Institute and OpenAI
Janelia Research Campus (HHMI)
Allen Institute (Seattle) meeting on Genetics of Complex Traits

Review article: On the genetic architecture of cognitive ability and other quantitative traits (2014)

I work on algorithms for phenotype prediction from genotype, using new methods from high dimensional statistics. My estimate is that prediction of complex traits such as height, cognitive ability, or highly polygenic disease conditions will require data sets of order one million individuals (i.e., to build a model which accounts for most of the genetic variance). Once these models are available, human reproduction (and evolution!) will be revolutionized.

These papers are somewhat technical:
https://arxiv.org/abs/1310.2264
http://arxiv.org/abs/1408.6583

This one is a bit less technical and gives a broader overview:
http://arxiv.org/abs/1408.3421

Cow genomics (an existence proof):
http://infoproc.blogspot.com/2012/08/genomic-prediction-no-bull.html
http://infoproc.blogspot.com/2014/08/its-all-in-gene-cows.html

These are for popular audiences (Nautilus Magazine):
http://nautil.us/issue/18/genius/super_intelligent-humans-are-coming
http://nautil.us/issue/28/2050/dont-worry-smart-machines-will-take-us-with-them

2018: As anticipated, we now have good height predictors thanks to the 500k genome release of UK Biobank data: Scientists of Stature

Genomic predictors for common disease risk, constructed via machine learning on hundreds of thousands of genotypes. The predictors use anywhere from a few tens (e.g., 20 or 50) to thousands of SNPs to compute the risk PGS (Poly-Genic Score) for conditions such as diabetes, breast cancer, heart attack, and more: Genomic Prediction of Complex Disease Risk.

The Economist on polygenic risk scores (2019).

Detailed analysis of genetic architectures of disease risk predictors. Implications for pleiotropy.

Sibling validation of genomic predictors.

Recent papers from my group:

https://www.genetics.org/content/210/2/477
https://www.nature.com/articles/s41598-019-51258-x
https://www.nature.com/articles/s41598-020-68881-8
https://www.nature.com/articles/s41598-020-69927-7


2021 review article, prepared for the book Genomic Prediction of Complex Traits, Springer Nature series Methods in Molecular Biology: 

From Genotype to Phenotype: polygenic prediction of complex human traits

Krugman: the Economists have no clothes

Only Nixon could go to China, and only Paul Krugman could write this mea culpa for the discipline of economics.

There was a bubble in housing -- everybody knows that now. There was a bubble in finance itself -- the financial share of national income reached an all-time high just before the crisis; not too many people know this. There was even an academic bubble in the field of economics: the perceived quality of results in the field, reflected in the salaries and prestige of economics professors (e.g., relative to other social scientists), was as inflated as the price of any McMansion -- surely every university dean must understand this now? (Bayesian / Machine Learning comment: do these guys ever update? Or is it "All priors, all the time"?)

I suggest reading the whole thing. I've only excerpted the last three paragraphs below. (See also related essay by Richard Posner and this interview with Bill Janeway.)

How Did Economists Get It So Wrong?: ... So here’s what I think economists have to do. First, they have to face up to the inconvenient reality that financial markets fall far short of perfection, that they are subject to extraordinary delusions and the madness of crowds. Second, they have to admit — and this will be very hard for the people who giggled and whispered over Keynes — that Keynesian economics remains the best framework we have for making sense of recessions and depressions. Third, they’ll have to do their best to incorporate the realities of finance into macroeconomics.

Many economists will find these changes deeply disturbing. It will be a long time, if ever, before the new, more realistic approaches to finance and macroeconomics offer the same kind of clarity, completeness and sheer beauty that characterizes the full neoclassical approach. To some economists that will be a reason to cling to neoclassicism, despite its utter failure to make sense of the greatest economic crisis in three generations. This seems, however, like a good time to recall the words of H. L. Mencken: “There is always an easy solution to every human problem — neat, plausible and wrong.”

When it comes to the all-too-human problem of recessions and depressions, economists need to abandon the neat but wrong solution of assuming that everyone is rational and markets work perfectly. The vision that emerges as the profession rethinks its foundations may not be all that clear; it certainly won’t be neat; but we can hope that it will have the virtue of being at least partly right.

Related posts on Krugman's article from two thoughtful economists, Brad DeLong: Which economists got it so wrong? , Where does macro go from here? and Arnold Kling: Krugman vs Blanchard.

Athens and Crete

I'm off to Athens and Crete next week. The main purpose of the trip is to attend a meeting on quantum gravity, but I'm also looking forward to visiting this new museum:



See slideshow.

Now that they have an appropriate place for them, will the Greeks get the Elgin Marbles back from the British?

NYTimes: ... For more than 30 years, Greece has been working, through diplomacy and public relations offensives, to regain the Elgin Marbles, sections of a decorative frieze that adorned the Parthenon until Lord Elgin ordered them removed in the early 19th century, during his tenure as British ambassador to the Ottoman Empire. Years later, bankrupt, he sold them to the British Museum, where they have been a major attraction since 1816.

Today, almost two centuries on, many Greeks hope the opening of the museum will focus international attention on their country’s claim to the so-called Elgin Marbles, and put an end to Britain’s longtime argument that it is in a better position to look after those 2,500-year-old panels. Last week the Greek government turned down an offer from the British Museum for a three-month loan of the collection, because it came with the condition that the Greeks formally acknowledge British ownership.

“This is a nonstarter for any discussion,” Mr. Samaras said. “No Greek can sign up to that.”

The new museum, 226,000 square feet of glass and concrete designed by the New York architect Bernard Tschumi, replaces the old Acropolis Museum, a small 1874 building tucked into the rock of the Acropolis next to the Parthenon. The design, introduced in 2001, was meant to be completed in time for the 2004 Olympics, but dozens of legal battles — many having to do with some 25 buildings that were demolished to make room for it — delayed the process for years.

Eventually I suspect the Chinese, Egyptians and many others will recover their stolen national treasures from museums in Europe and the US.

Flowers for Algernon

I read this when I was a kid, and it had a strong effect on me, at least in part because I had a next-door neighbor who was my age and mentally retarded. Now that I'm a parent it strikes me that we all follow the path of Charlie Gordon -- starting from modest intellectual ability, developing to a peak, followed by inevitable decline. It happens more gradually for us than for him, but we are equally aware of the process and where it leads.

Wikipedia: ... The titular Algernon is a laboratory mouse who has undergone surgery to increase his intelligence by artificial means. The story is told as a series of progress reports written by Charlie, the first human test subject for the surgery.

... Another key moment came in 1957, while [author] Keyes was teaching English to students with special needs; one student asked him if it would be possible to be put into a regular class if he worked hard and became smart.

... The procedure is a success and, three months later, Charlie's IQ has reached 185. However, as his intelligence, education and understanding of the world around him increases, his relationships with people deteriorate.

... Charlie discovers a flaw in the theories that led Nemur and Strauss to develop their intelligence-enhancing procedure. Shortly thereafter, Algernon starts behaving erratically, loses his new intelligence, and dies. As Charlie does further research, he determines that he too will inevitably revert to his old condition.

From the Wikipedia entry I learned the novel is based on a short story that appeared years earlier and won the science fiction Hugo award. The full text is here (at least for the moment). I think the novel is stronger than the short story.

Here's an excerpt (Google Books).

June 29 -- ... I phoned Landsdoff at the New Institute for Advanced Study, about the possibility of using the pair-production nuclear photoeffect for exploratory work in biophysics. At first he thought I was a crackpot, but when I pointed out to him the flaws in his article in the New Institute Journal he kept me on the phone nearly an hour. He wants me to come to the Institute to discuss my ideas with his group. ... That's the problem, of course. I don't know how much time I have. A month? A year? The rest of my life? That depends on what I find out about the psychophysical side-effects of the experiment. ...

The cost of liquidity?

Speaking of high frequency trading, here's Zero Hedge on the Renaissance vs. Volfbeyn and Belpolsky matter. Related posts here and here.

130. While employed by Renaissance, Dr. Volfbeyn's superiors repeatedly asked him to assist Renaissance in conducting securities transaction that Dr. Volfbeyn believed to be illegal.

The illegality of these activites touched upon three main topical areas:

ITG-POSIT (The Dark Pool angle)

131. In particular, Dr. Volfbeyn was instructed to devise a strategy to defraud investors trading through the Portfolio System for Institutional Trading ("POSIT"). POSIT is an electronic trading system operating by Investment Technology Group ("ITG" ). POSIT collects buy and sell orders from large traders and attempts to match them.

132. On information and belief, POSIT is completely confidential. It does not reveal information about orders to anyone. For its customers, this confidentiality is an essential aspect of the system.

133. Renaissance asked Dr. Volfbeyn to create a computer algorithm to reveal information that POSIT intended to keep confidential [REDACTED]

134. Renaissance intended to, and did, use this trading strategy [the POSIT strategy] to profit [REDACTED]

135. Dr. Volfbeyn believed that [REDACTED] [the POSIT strategy] violated securities laws. He expressed his opinion to his superiors at Renaissance and refused to build the computer algorithm as they requested.

Limit Order Strategy [Stealing Liquidity]

139. Renaissance asked Dr. Volfbeyn to develop a computer algorithm [REDACTED] [the "limit order strategy"]

140. A limit order is an instruction to trade at the best price available, but only if the price is no worse that a "limit price" specified by the trader. Standing limit orders are placed in a file, called a limit-order book. Limit-order books on the New York Stock Exchange and NASDAQ are available to be viewed by anyone.

141. By [REDACTED], Renaissance intended to profit illegally.

142. Dr. Volfbeyn refused to participate in such activities. He explained that his refusal was based on his belief that the proposed transactions violated securities laws [2nd time RenTec allegedly used an illegel strategy]

143. Senior Renaissance personnel, including Executive Vice President Peter Brown and Vice President Mark Silber, attempted to persuade Dr. Volfbeyn to engage in the [REDACTED] limit order strategy, despite his objections. Mark Silber is the compliance officer for Renaissance, responsible for implementing systems to ensure that Renaissance does not violate the securities laws, and for protecting employees who complain about potentially illegal conduct.

144. On information and belief, Renaissance did not implement the [REDACTED] limit order strategy prior to Dr. Volfbeyn's termination. [What about after?]

Swap Transactions [The Naked Short Scam]

145. At all times relevant to this action, Rule 3350 of the NASD, prohibited NASD members, with certain exceptions from effecting short sales in any Nasdaq security at or below the current national best (inside) bid when the current national best (inside) bid is below the preceding national best (inside) bid in the security.

146. At all times relevant to this action, Rule 10a-1 under the Securities Exchange Act of 1934 provided that, subject to certain exceptions, an exchange-listed security could only be sold short at a price above the immediately preceding reported price or at the last sale price if it is higher than the last different reported price.

147. During the period when Dr. Volfbeyn and Dr. Belopolsky were employed at Renaissance, plaintiff engaged in a massive scam [REDACTED] [the "swap transaction strategy"]

148. [REDACTED]

149. [REDACTED]

150. Renaissance conducted [REDACTED] in violation of Rule 3350 and Rule 10a-1. Renaissance also intentionally [REDACTED] in violation of SEC and NASD rules. [REDACTED] Renaissance profited from the strategy [REDACTED].

151. Researchers at Renaissance expressed their concern to Executive Vice President Peter Brown and other officials of Renaissance about the legality of these swap transactions, including concerns that the transactions violated the tax laws and securities laws. Renaissance failed to halt the transactions. On information and belief, the swap transactions are continuing and generate substantial profits for Renaissance.

Whatever happened to Whit Stillman?

Whit Stillman wrote and directed the trilogy Metropolitan, Barcelona and Last Days of Disco, three of my favorite movies. It's been 10 years since his last film and I've often wondered what became of him. He recently resurfaced on the Leonard Lopate show and gave the interview below.



Older Stillman interviews and reviews of his films here. Related video (for a while you could watch all of Last Days of Disco on hulu.com).

My favorite of the trilogy is Metropolitan, which was shot for almost nothing around NYC. In the clip below we learn about the UHB: Urban Haute Bourgeoisie. Metropolitan reminds me of early Woody Allen, but set in the world of 1980s college age upper east side WASPs.

Me and Twitter

No, I'm not on Twitter. I don't need to be immersed in distracting stimulus -- the web is bad enough. My thoughts are already shallow; don't force my brain to swim in even shallower water :-)

I met Twitter founder Evan Williams a few years ago, before Twitter was anywhere near a big thing. He told me about Blogger, which he sold to Google, and then the inevitable "So what are you working on now?" question came up.

He described Twitter to me, and two thoughts entered my mind. The first shows I am old, or out of touch, or have no feel for Web 2.0 consumer startups: "Who would use that?" I said to myself.

The second thought, which I actually verbalized, turns out to be a good question (still unanswered) and shows I may have VC potential: "How are you going to monetize that?" :-)

Here is a great talk by Williams about his podcasting startup Odeo, which somehow spun out Twitter. Note it's from 2005, but his insights on entrepreneurship are great; he's a low-key midwestern guy like me (from Nebraska).


Nevertheless, this Twitter feed cracked me up today:
http://twitter.com/shitmydadsays

"You need to flush the toilet more than once...No, YOU, YOU specifically need to. You know what, use a different toilet. This is my toilet."
1:07 PM Aug 23rd from web

"Don't touch the bacon, it's not done yet. You let me handle the bacon, and i'll let you handle..what ever it is you do. I guess nothing."
11:15 AM Aug 22nd from web

"Your mother made a batch of meatballs last night. Some are for you, some are for me, but more are for me. Remember that. More. Me."
8:57 AM Aug 21st from web

"Your brother brought his baby over this morning. He told me it could stand. It couldn't stand for shit. Just sat there. Big let down."
9:35 AM Aug 20th from web

"Love this Mrs. Dash. The bitch can make spices... Jesus, Joni (my mom) it's a joke. I was making a joke! Mrs. Dash isn't even real dammit!"
9:28 AM Aug 19th from web

"The dog is not bored, it's a fucking dog. It's not like he's waiting for me to give him a fucking rubix cube. He's a god damned dog."
10:43 AM Aug 18th from web

"I didn't live to be 73 years old so I could eat kale. Don't fix me your breakfast and pretend you're fixing mine."
11:24 AM Aug 3rd from web

High speed trading: an $8 billion per year tax?

Because there is money to be made at this game (estimated in the article to be in excess of $8 billion per year), it is attracting both financial and human capital. The human capital might be doing more productive things elsewhere. Let's suppose that the social utility provided by the market is efficient resource allocation via price signals. I see no benefit to society from making these price signals "efficient" over time scales as small as a tenth of a second (i.e., much smaller than the time scales over which physical resources can be allocated, or humans can make executive decisions). Imposing a 1-100 second random delay on any order placed at an exchange would not (as far as I can see) have any negative impact on the actual economy, but would eliminate an expensive arms race that transfers money from small or long-term investors to brainy hedge funds.

These activities are particularly appealing to banks and hedge funds in the current environment because a trader can book a real profit or loss at the end of each day (or even every few seconds!) -- very different from the illiquid positions that led to the credit crisis. In the long run, even complex derivatives like CDO and CDS contracts have the potential of providing some social good -- the ability to diversify risks, etc. I see no comparable redeeming value in high speed trading.

A couple years ago I listened to a talk by a former high energy physicist about how his firm was using FPGAs to execute their trading algorithms in hardware. Sound like a good use of brainpower and resources to you?

Regarding the Aleynikov case reported on below, 32 MB of proprietary source code is not a small amount of code. It could be the core of Goldman's algorithms. I would think a court-appointed expert could easily determine the value of the code Aleynikov downloaded. If he merely grabbed it by mistake while downloading some open source directories (his claim), it would be unlikely to contain the key algorithms.

Related posts here (Aleynikov vs. Goldman) and here (Volfbeyn and Belopolsky vs. Renaissance).

NYTimes: ... the charges, along with civil cases in Chicago and New York involving other Wall Street firms, offer a glimpse into the turbulent world of ultrafast computerized stock trading.

Little understood outside the securities industry, the business has suddenly become one of the most competitive and controversial on Wall Street. At its heart are computer programs that take years to develop and are treated as closely guarded secrets.

Mr. Aleynikov, who is free on $750,000 bond, is suspected of having taken pieces of Goldman software that enables the buying and selling of shares in milliseconds. Banks and hedge funds use such programs to profit from tiny price discrepancies among markets and in some instances leap in front of bigger orders.

Defenders of the programs say they make trading more efficient. Critics say they are little more than a tax on long-term investors and can even worsen market swings.

But no one disputes that high-frequency trading is highly profitable. The Tabb Group, a financial markets research firm, estimates that the programs will make $8 billion this year for Wall Street firms. Bernard S. Donefer, a distinguished lecturer at Baruch College and the former head of markets systems at Fidelity Investments, says profits are even higher.

“It is certainly growing,” said Larry Tabb, founder of the Tabb Group. “There’s more talent around, and the technology is getting cheaper.”

The profits have led to a gold rush, with hedge funds and investment banks dangling million-dollar salaries at software engineers. In one lawsuit, the Citadel Investment Group, a $12 billion hedge fund, revealed that it had paid tens of millions to two top programmers in the last seven years.

“A geek who writes code — those guys are now the valuable guys,” Mr. Donefer said.

The spate of lawsuits reflects the highly competitive nature of ultrafast trading, which is evolving quickly, largely because of broader changes in stock trading, securities industry experts say.

Until the late 1990s, big investors bought and sold large blocks of shares through securities firms like Morgan Stanley. But in the last decade, the profits from making big trades have vanished, so investment banks have become reluctant to take such risks.

Today, big investors divide large orders into smaller trades and parcel them to many exchanges, where traders compete to make a penny or two a share on each order. Ultrafast trading is an outgrowth of that strategy.

As Mr. Aleynikov and other programmers have discovered, investment banks do not take kindly to their leaving, especially if the banks believe that the programmers are taking code — the engine that drives trading — on their way out.

Mr. Aleynikov immigrated to the United States from Russia in 1991. In 1998, he joined IDT, a telecommunications company, where he wrote software to route calls and data more efficiently. In 2007, Goldman hired him as a vice president, paying him $400,000 a year, according to the federal complaint against him.

He lived in the central New Jersey suburbs with his wife and three young daughters. This year, the family moved to a $1.14 million mansion in North Caldwell, best known as Tony Soprano’s hometown. ...

This spring, Mr. Aleynikov quit Goldman to join Teza Technologies, a new trading firm, tripling his salary to about $1.2 million, according to the complaint. He left Goldman on June 5. In the days before he left, he transferred code to a server in Germany that offers free data hosting.

At Mr. Aleynikov’s bail hearing, Joseph Facciponti, the assistant United States attorney prosecuting the case, said that Goldman discovered the transfer in late June. On July 1, the company told the government about the suspected theft. Two days later, agents arrested Mr. Aleynikov at Newark.

After his arrest, Mr. Aleynikov was taken for interrogation to F.B.I. offices in Manhattan. Mr. Aleynikov waived his rights against self-incrimination, and agreed to allow agents to search his house.

He said that he had inadvertently downloaded a portion of Goldman’s proprietary code while trying to take files of open source software — programs that are not proprietary and can be used freely by anyone. He said he had not used the Goldman code at his new job or distributed it to anyone else, and the criminal complaint offers no evidence that he has.

Why he downloaded the open source software from Goldman, rather than getting it elsewhere, and how he could at the same time have inadvertently downloaded some of the firm’s most confidential software, is not yet clear.

At Mr. Aleynikov’s bail hearing, Mr. Facciponti said that simply by sending the code to the German server, he had badly damaged Goldman.

“The bank itself stands to lose its entire investment in creating this software to begin with, which is millions upon millions of dollars,” Mr. Facciponti said.

Sabrina Shroff, a public defender who represents Mr. Aleynikov, responded that he had transferred less than 32 megabytes of Goldman proprietary code, a small fraction of the overall program, which is at least 1,224 megabytes. Kevin N. Fox, the magistrate judge, ordered Mr. Aleynikov released on bond.

Clarification in response to comments: I guess I should make my claims more precise. It seems to me that imposing a random delay of average length T would

1) remove the possibility of gaming the system on timescales much less than T (thereby sending lots of smart people back to productive activities)

2) not affect market liquidity on timescales much larger than T. I claim that for T of order a minute (or even longer) there is no social value from liquidity on much smaller time scales.

I am not highly confident of my statements because market making is a dynamical system, with interacting agents, etc. Doyne Farmer and Sante Fe researchers did some modeling for NASDAQ in anticipation of their 2001 decimalization (see here); the details are complicated. Yes, the price of decimalization and liquidity on very short time scales is the current arms race, but I have yet to see an argument for why those things are good for society.

For a defense of high frequency trading, see discussion at Scott Locklin's blog here and here.

The Heisenberg uncertainty

Why was the German atomic bomb effort during WWII unsuccessful? Was it because Heisenberg deliberately undermined the project (perhaps because he did not want Hitler to have atomic weapons), or because of simple incompetence?

I thought the question had been answered definitively by the release of the Farm Hall transcripts (Operation Epsilon). In the spring and summer of 1945 ten of the leading German nuclear physicists, including Heisenberg, were sequestered at a house near Cambridge, England. The house was bugged, and there is no reason to doubt that the conversations recorded were genuine:

Diebner: "I wonder whether there are microphones installed here?"

Heisenberg: "Microphones installed? (laughing) Oh no, they're not as cute as all that. I don't think they know the real Gestapo methods; they're a bit old fashioned in that respect."

I read the transcripts a few years ago, and my impression was that Heisenberg had grossly overestimated the amount of U235 necessary for a bomb. Here is what appears in the Wikipedia entry on Farm Hall:

Most historians have no reason to believe that he was not being genuine, and the attitude of Heisenberg and the other scientists over all the months and especially their reaction to the shattering news of the bomb explosion was so genuine that it is almost inconceivable it was staged.

All of the scientists expressed shock when informed of the atomic bombing of Hiroshima on August 6, 1945. The transcripts seem to indicate that the physicists, in particular Heisenberg, had either overestimated the amount of enriched uranium that an atomic bomb would require or consciously overstated it, and that the German project was at best in a very early, theoretical stage of thinking about how atomic bombs would work.

Recently I came across an exchange in the NY Review of Books, between physicist Jeremey Bernstein, who wrote Hitler's Uranium Club: The Secret Recordings at Farm Hall, and Thomas Powers, author of Heisenberg's War. Powers believes that Heisenberg had a much better understanding of the requirements for a bomb than are suggested by his Farm Hall comments immediately after learning of Hiroshima.

[Bernstein:] ... In June of 1942 Heisenberg was asked how much uranium a nuclear bomb would use that could destroy London. If one kilogram of uranium could be completely fissioned it would produce an explosion equal to about twenty thousand tons of TNT; about the size of the Hiroshima bomb. This would correspond to a sphere of uranium with a radius of a little over two centimeters—a ping- pong ball. But in a real bomb only about 2 percent of the uranium is fissioned so that about a hundred kilograms of uranium are needed, which corresponds to a radius of about eight and a half centimeters. At Farm Hall Heisenberg did not seem to have a clear idea of the efficiency question, so it is totally unclear to me what he had in mind when, in 1942, he apparently indicated that it was the shape of a pineapple. ...

[Letter from Charles Frank, British physicist and intelligence advisor to Operation Epsilon]: As I remember it, Heisenberg gave just the calculation you quote, except that my memory says he reached the answer 5 tons.... He gave this calculation at the beginning of an elegant colloquium, delivered the day after they heard about Hiroshima, in which he used a rather polished version of diffusion-and-multiplication theory (which he was no doubt familiar with from their pile work) to arrive at an answer for critical mass of the order of 1 or a few kilograms. He gave the crude and faulty calculation at the beginning, as the way they had worked it out before, and I think he said it was the way he had worked it out, and I think he said it was the estimate he gave at the 1940 conference about what was to be done with nuclear fission in relation to the war. I think he said he had done the revised calculation overnight, and I think that the whole style of the lecture implied that he was presenting a result and argument new to him and to his audience.

[Powers:] Bernstein interprets Heisenberg's remarks, available to us since the 1992 release of the Farm Hall tapes recorded by British intelligence during the war, to mean that Heisenberg failed to calculate the critical mass of an atomic bomb earlier in the war. Because fissionable material is extremely difficult and expensive to manufacture, the amount of it necessary for a bomb—the "critical mass"—is by far the most important technical question to be asked before embarking on a bomb program. A figure on the order of tons would have made building a bomb impossible for any nation at the time, the United States included. From his interpretation of Heisenberg's remarks on this point Bernstein concludes there was no German bomb, and no German bomb program, as the result of "simple incompetence"—the phrase he used in an article in the May 1999 issue of Commentary. In his letter he writes, "There is no evidence that he [Heisenberg] or anyone else in the German project did this calculation correctly during the war," but in fact there is such evidence—in the Farm Hall discussion recorded on the night after Hiroshima, Otto Hahn, one of the ten German scientists, protested Heisenberg's claim that huge amounts of fissionable uranium were required for a bomb. "But tell me why you used to tell me that one needed 50 kilograms of '235' in order to do anything. Now you say one needs two tons." ...

The German physicist Manfred von Ardenne confirms in his memoirs, as he did to me personally in an interview in 1989, that Hahn told him in 1940 that critical mass would be on the order of kilograms, not tons, citing Heisenberg as his source. The fact that Heisenberg had calculated a roughly correct value for critical mass is also demonstrated by his answer to a question during the June 1942 conference in Berlin with Albert Speer. In a letter to Samuel Goudsmit of October 3, 1948, Heisenberg wrote: "General Field Marshall Milch asked me approximately how large a bomb would be, of which the action was sufficient to destroy a large city. I answered at that time, that the bomb, that is the essentially active part, would have been about the size of a pineapple." (Goudsmit papers, American Institute of Physics) The "essentially active part" of a bomb is called the core. Erich Bagge, who was also present at the meeting with Speer, told interviewers, including me, that Heisenberg had shaped his hands in the air to suggest an object about the size of a "football." ...

So there seems to be some residual uncertainty over what Heisenberg knew and when. I guess I'll have to look at the transcripts again, but I distinctly recall Heisenberg being confused about the required critical mass for a bomb. He may very well have worked out the correct figure by his colloquium the following day, but of course that was with the additional knowledge that the Americans had actually produced a working bomb! As all theoreticians know, it's much easier to produce the right calculation once you know the final answer ;-)

An amusing story related by Powers:

... In a letter written to me a few years back the Italian physicist Ugo Fano described a party at the Ann Arbor, Michigan, home of Samuel Goudsmit, scientific director of the Alsos mission, in late August 1939. There two Nobel Prize–winning physicists, Werner Heisenberg, soon to return to Germany, and Enrico Fermi, a refugee from Mussolini's Italy, were both honored guests. "At that party," Fano writes, "[Edoardo] Amaldi drew me aside to point out its humor: 'See Fermi, see Heisenberg, sitting in that corner. Everyone in this room expects a big war and the two of them to lead fission work on opposite sides, but nobody says!'" (Letter of September 18, 1993) Within a month Heisenberg had in effect been drafted to do theoretical work on bomb physics and he soon wrote two papers which were the basis of further German research during the war.

Bolt: 19.19 200m!!!

Shocking! I never expected to see 19.19 in my lifetime, but then I expected Michael Johnson's 19.32 WR to last for decades, like Bob Beamon's long jump WR. (Bolt is such a mutant that I won't be surprised if no one else comes close to MJ's 19.32 for a long time.)

During the Beijing Olympics some readers objected to my stating flat out that Bolt is a more exceptional and impressive athlete than Michael Phelps. Does anyone (who knows something about both track and swimming) still disagree?

God save the Queen, and the economists

Richard Posner writes in the Atlantic. Read the whole thing!

I doubt academic economists are to blame for the crisis in any direct sense -- the real perps are on Wall Street and Main Street -- but they do deserve blame for the "ideological capture" of regulators and policy makers, many of whom, through years of indoctrination in undergraduate and professional school courses have come to believe in a semi-strong version of efficient markets (see below). The academics also deserve blame for intellectual dishonesty if they fail to admit that we know almost nothing about macroeconomics and that the profession itself cannot agree on what caused the crisis and how best to recover.

Will Economists Escape a Whipping?

It is remarkable how economists have been able to deflect blame for the economic crisis that erupted last September with the sudden collapse of the international banking industry and that continues to afflict the world's economies. Last November, Queen Elizabeth visited the London School of Economics and asked the faculty why "nobody [had] noticed [before September 2008] that the credit crunch was on its way?" ...

Yesterday, another letter to the Queen, this one dated August 10 and signed by ten English and Australian economists, was released ..., also responding to the Queen's question ... The August 10 letter states that "their overall analysis is inadequate because it fails to acknowledge any deficiency in the training or culture of economists themselves." The nub of their criticism is that "in recent years economics has turned virtually into a branch of applied mathematics, and has...become detached from real-world institutions and events." They criticize economics education as excessively narrow, "to the detriment of any synthetic vision," and fault Besley and Hennessy for saying nothing about "the typical omission of psychology, philosophy or economic history from the current education of economists" and for mentioning neither "the highly questionable belief in universal 'rationality' nor the 'efficient markets hypothesis.'"

A more focused criticism would, in my opinion, be more effective. The Queen was asking about the failure to foresee the financial collapse of last September, rather than about the health of modern economics in the large. That failure was I think due in significant part to a concept of rationality that exaggerates the amount of information that people have about the future, even experts, and to a disregard of economic factors that don't lend themselves to expression in mathematical models, or are intractable to formal analysis. The efficient markets theory, when understood not as teaching merely that markets are hard to beat even for experts and therefore passive management of a diversified portfolio of assets is likely to outperform a strategy of picking underpriced stocks or other securities to buy and overpriced ones to sell, but as demonstrating that asset prices are always an adequate gauge of value--that there are not asset "bubbles"--blinded most economists to the housing bubble of the early 2000s and the stock market bubble that expanded with it. In modeling the business cycle, economists not only ignored, because difficult to accommodate in their mathematical models, vital institutional detail (such as the rise of the "shadow banking industry," which is what mainly collapsed last September)--often indeed ignoring money itself, on the ground that it doesn't really affect the "real" (that is, the nonfinancial) economy. They also ignored key concepts in Keynes's analysis of the business cycle, such as hoarding and uncertainty and business confidence ("animal spirits") and worker resistance to nominal (as distinct from real) wage reductions in depressions. Lessons of economic history were ignored, too, leading to a belief that there would never be another depression, let alone a collapse of the banking industry. Even when the collapse occurred, in September, many macroeconomists denied that it would lead to anything worse than a mild recession; the measures that the government has taken to recover from what has turned into a depression owe little to post-Keynesian economic thinking; and the economists cannot agree on what further, if anything, should be done, and which of the government's recovery measures has worked or will work.

Besley and Hennessy's letter, when first published, was described in some quarters as a letter of "apology" by English economists. It was not that; nor is the August 10 letter--the latter is a denunciation of mainstream economics.

The notion of a profession's apologizing for its failure in a letter to the monarch is charming, however. It would be an apology to the nation, personified in its monarch. The English monarch does not exercise political power, but does personify the nation, and it is easier to write a letter to a person than to a nation.

The English economics profession failed the United Kingdom; the American economics profession failed the United States. Not that the profession should be equated to its macroeconomic and financial divisions. The study of business cycles is only a small part of modern economics. Other areas of economics bear significantly on the study of business cycles, such as labor economics, without being implicated in the failures of response to the current crisis. But the control of the business cycle had until the present crisis been regarded as a principal triumph of modern economics and justification for regarding economics as the queen of the social sciences. We have no monarch; the President is not a personification of the nation but rather the head of the national government; there is no one to write the letter of apology to. No matter. The urgent need is for the part of the profession that concerns itself with business cycles to acknowledge its inadequacies and reorient its training and research.