Money never sleeps

Woo hoo! Michael Douglas is back as Gordon Gekko, and just in time. Too bad Oliver Stone isn't back to direct as well. I predict that this time Gekko will run a huge private equity fund with 10x the capital of the previous one. Jail time will not have been an impediment to raising his latest fund.

From the original:

Gekko: Greed, for lack of a better word, is good. Greed is right. Greed works. Greed clarifies and cuts through and captures the essence of evolutionary spirit. Greed in all of its forms, greed for life, for money, for love, knowledge has marked the upward surge of mankind. And greed, you mark my words, will not only save Teldar Paper, but that other malfunctioning corporation called the USA.

Gekko: The richest one percent of this country owns half our country's wealth, five trillion dollars. One third of that comes from hard work, two thirds comes from inheritance, interest on interest accumulating to widows and idiot sons and what I do, stock and real estate speculation. It's bullshit. You got ninety percent of the American public out there with little or no net worth. I create nothing. I own.

Carl Fox: Stop going for the easy buck and start producing something with your life. Create, instead of living off the buying and selling of others.

About the sequel:

NYTimes: Greed is still good.

Or so those at 20th Century Fox hope. Even as their boss, Rupert Murdoch, pursued an uninvited takeover bid for Dow Jones this week, Fox movie executives quietly sealed a deal to revive Gordon Gekko, the suspender-loving financial prowler who made grabbing seem good in Oliver Stone’s 1987 film, “Wall Street.”

When last seen, the corrupt Gekko, an Oscar-winning role for Michael Douglas, was on the brink of surrendering his white cuffs for handcuffs, having been sold out by his protégé Bud Fox, played by Charlie Sheen.

“He went to jail,” acknowledged Edward R. Pressman, who produced the original movie and reached an agreement with Fox this week to develop a sequel in which Mr. Douglas will resume his machinations on a global scale in the hedge-fund era. Mr. Pressman declined to say more about the plot. But the title, he said, will be “Money Never Sleeps,” after one of Gekko’s guiding principles in the first film, written by Stanley Weiser and Mr. Stone.

“Wall Street” was only a modest hit when Fox released it. But it won a passionate following in the financial world, where many found something to love in the predatory Gekko. Speaking by telephone from Bermuda, Mr. Douglas said he wouldn’t mind if he never had “one more drunken Wall Street broker come up to me and say, ‘You’re the man!’ ”

Mr. Stone will not direct the sequel, although the producer said that Messrs. Pressman and Douglas and their new writer, Stephen Schiff (“True Crime”), pressed him to do so for months. Mr. Schiff, who expects to deliver a script later this year, said the Bud Fox character was likely to be missing as well. But a restyled Gekko, he predicted, might start setting trends all over again.

“If you weren’t wearing suspenders before ‘Wall Street,’ you were certainly wearing them after,” he said. As for moral development, don’t expect too much from a villain who taught us that lunch is for wimps, and who bragged: “I create nothing. I own.”

“I don’t think he’s much different,” Mr. Douglas said. “He’s just had more time to think about what to do.”

Energetics

Yesterday I heard a great interview (Bloomberg podcast -- unfortunately no longer available online; but see here) with venture capitalist Vinod Khosla (formerly of Kleiner-Perkins, co-founder of Sun), who has been investing in energy technologies of late. One interesting comment he made was that no technology or environmental solution will scale (certainly not enough to affect global warming) unless there is a profit to be made along the way.

Some numbers from this DOE report.

Sources of Electricity

• Currently - around 13 TW of Annualized Average Power Consumption on the Earth.

• Fossil Fuel → Annual Production of 25 billion Metric tons of CO2 ⇒ Requires Massive & Efficient Underground Storage.

• 10 TW ≡ 10^4 One GW Nuclear Power Plants – Will exhaust Terrestrial Uranium Resources in 10 years if constructed.

• Maximum Practical Hydroelectric Resources 0.5 TW.

• Cumulative Energy of Tides and Ocean Currents < 2 TW.

• Total Geothermal Energy on Earth 12 TW.

• Maximum Amount of Extractable Wind Power 2 - 4 TW.

• 120,000 TW of Solar Radiation Strikes the Earth Surface in the form of sunlight.

Robot Genius!

Sorry for the lack of posts -- I was traveling all of last week.

Some recent press coverage of Robot Genius:

InfoWorld
Network World
Dark Reading
SC Magazine
Network World 2
PC World
Net Security

InfoWorld April 25, 2007: With a name like Robot Genius you wouldn't expect the company's leaders to be modest, but the more you hear the firm's Chairman Stephen Hsu talk about his startup's new approach to anti-malware, the more you believe the name might fit.

On April 30, the company founded by Hsu and James Hormuzdiar-- a partnership best known for building SSL VPN provider SafeWeb and selling it to Symantec for $26 million in 2003 -- will formally introduce a trio of behavior-based security products.

By taking a radically different approach to scanning the Web for malware code and using massive computing power to filter out every URL on the Internet responsible for serving up infections, Robot Genius' technologies will change the way people view anti-virus tools, Hsu said.

"We're entering an era where the scalability and bandwidth of machines allows for constant monitoring for malware by looking at the entire Web and studying every piece of downloadable software that's available," said Hsu. "Meanwhile, the current generation of virus-scanning applications, even behavior-based tools, has reached its limits in terms of finding the most sophisticated attacks, and lacks the ability to adequately fix problems once they're found."

...

Stata center, MIT

One of the things I often complain about is that doing fundamental physics is like trying to push a big rock -- it's very hard to tell whether you're having any effect or making any progress!

Here I am trying my luck with the Frank Gehry-designed Stata center at MIT, home of their Computer Science and Artificial Intelligence Laboratory (CSAIL).

A funny coincidence -- in the elevator I noticed a poster for a talk last Friday on blogging and tagging given by an old friend of mine from Berkeley!

Vol and tail risk

Some nice discussion here by financier turned internet entrepreneur Roger Ehrenberg. His blog is on my Google Reader list. See also here.

Volatility Management in a Complacent World

Volatility has a corrosive effect on returns. Two cash flow streams that generate similar average returns, where one is more volatile than the other, can result in sharply divergent IRRs with the less volatile stream yielding the superior result. Therefore, it is clear that volatility reduction has a real and calculable value, but requires a probabilistic view of the world that is often difficult to quantify and harder to pay for. Further, sometimes the pressure for short-term returns can skew rational long-term thinking, causing an increase in the willingness to accept volatility that, in turn, further depresses the value of accepting such risks. And with this the cycle of complacency begins, is reinforced, and feeds on itself until the inevitable happens: event shock. And people will throw up their hands and and say "Look at this tail risk; this is once-in-a-(decade/century/millennium) event." And those with a keen appreciation for such things will say - I told you so. "Fat tails" and "three sigma events" are now and have always been a relatively ordinary phenomenon. Tail risk is risk that can and should be priced and, depending upon your objectives and stakeholders, actively mitigated. But this requires discipline and cost, two things commonly lacking in those compensated for short-term objectives. And herein lies the issue.

The Economist has an interesting piece titled Sting in the Tail, positing whether low volatility is making the world too complacent about risk. There are some derivatives details in the article that I will address in a separate post that are either unclear or incorrect and, in my opinion, confuse the issue and muddy the picture unnecessarily. However, the questions raised by the article are spot on and important for investors and policy-makers to consider...

1. Have developments in the global financial markets - the rise of derivatives and risk dispersion, stronger, more competent central banks, and a more diversified base of economic power spread across countries, companies and currencies - conspired to reduce the "structural" level of market volatility?

I think the answer is yes. In general, diversification reduces the variability of outcomes, and with the increased diversification of both risks and economic power it must logically follow that some degree of variability has to have been stripped from the global financial markets. This doesn't mean that markets still can't get pushed out of whack or that exogenous, non-economic shocks (i.e., war, terrorism, etc.) don't have profound economic impacts that give rise to volatility. It only means that if one were to take a longitudinal view of global economic performance, that one would expect a less variable set of returns than previously generated in a world with greater concentration of risks and economic power. My only qualifier to this is: in a more complex, diversified world, does this possibly give rise to a third dimension of exposures related to the non-linear increase in the complexity of relationships managing the world's global economic resources? It was easy to navigate when you had a few economic superpowers and a few powerful financial markets. But what about now? The picture is painfully more complicated. And this might, in and of itself, effect the volatility landscape.

2. Are the perceptions of risk and, therefore, volatility, cyclical?

The answer is unquestionably yes. People and markets have short memories, and that is a fact. Which means that if liquidity is plentiful, spreads are tight and investors are looking everywhere for returns, spreads will continue to tighten as previously risky assets are no longer viewed as risky (i.e., high-yield debt, emerging markets debt and equities, etc.). This also means that those buying insurance are few and those selling insurance are plentiful, further depressing volatility and only amplifying the effects of a complacent market psychology. Then seemingly out of nowhere - bang! Emerging markets debt spreads move from 200 bps to 1000 bps, equity markets drop 10-20%, and risk premia magically expand to meet the heightened anxiety and uncertainty. And this will persist for a while. Until people forget about the shock and it's once again business as usual. This is how it always has been and this is how it always will be. Will the current liquidity-fueled securities-buying boozer persist indefinitely? Of course not. It is just a matter of time.

3. Are the costs of insuring against tail risk relatively expensive?

Yes. And they always will be. Selling out-of-the-money put options is a risky business, and the implied volatility that is required to buy these instruments is generally very high. In fact, broker/dealers are not the ones best positioned to sell such instruments. While the implied volatility charged the buyer is relatively high the cost of the option is absolutely cheap, due to its out-of-the-moneyness and the low probability of its being exercised. Therefore, the broker/dealer selling the option is not making a whole lot of money even when charging high implied volatility, and the dynamic hedging (usually "delta hedging" - the probability-weighted amount of the hedge underlying that needs to be held based upon the spot price of the underlying and the remaining time to maturity) that needs to take place is subject to "gap risk." This basically means that when the dealer most needs to sell to adjust its hedge it will be forced to sell at progressively less attractive prices in a market meltdown. This is how dealers can lose hundreds of millions of dollars very, very quickly. However, insurance companies and others with durable, ultra-long dated asset portfolios can sell these options as a vehicle for enhancing returns (the flip side of covered call writing) without the need for dynamic hedging. Bottom line, implied volatility will always be high for these types of instruments, and necessarily so.

4. Can one generate seemingly superior short-term returns by avoiding the costs of insuring long-term risk?

Absolutely. Whether the investor is buying risky assets at progressively tighter spreads or selling optionality as a vehicle for collecting premium it hopes never to have to pay back (and then some), these activities generate returns that are relatively attractive when compared to common benchmarks. However, Mr. Market ensures that investors don't get something for nothing, so when that "unexpected" shock occurs - spreads blow out, markets drop and margin calls come knocking - the benefits associated with making short-term numbers look good are generally far outweighed by the costs associated with the unwinding of these risky asset/short-option positions. Messrs. Meriwether and Niederhoffer know all about this. So let's just say that investors should look pretty carefully at fund documents before investing, because often these documents give managers tremendous amounts of latitude, and one will need to dig pretty deep to properly analyze the quality of a fund's earnings. Is it due to good securities selection or an increase in portfolio risk? This is the question the needs to be answered.

5. Has complacency driven this historical tightening of risk premia across markets, to the point where it is poised to explode in the face of said tail events?

Oh, yes. I think liquidity is a great thing, except when it causes investors to make irrational decisions. Chasing returns. Getting away from a fund's mission. Assuming risks that are properly absorbed by those better able (and more appropriately positioned) to take them. And when the time comes, no amount of liqudity is going to buttress what looks like an inexorably declining market. Changes in market psychology can be abrupt and harsh. People and governments will move to the sidelines until the detritus is cleared, and this will take exactly how long? Who knows.

So where we are today is at a time when the costs of insurance are both relatively and absolutely low yet the urge is for investors to sell it, not buy it. Because short-term performance considerations (which directly drive most fund managers' compensation, as well as the ability to gather additional assets to manage) can often drive sub-optimal portfolio decisions. And this is certainly not good for fund investors. And it is at times like these when the smart, savvy, long-term oriented managers with an appreciation for history take a contrarian position. And I might wager that this is precisely what is happening. We'll see the wheat separated from the chaff in short order. Just wait and see.

Tom Wolfe on hedge funds

There's a new glossy business/finance magazine on the racks, called Portfolio. In their first issue they've got a profile of Citadel's Ken Griffin and a long piece by Tom Wolfe on hedge funds. As I've said many times here, it's the new Gilded Age.

I can't think of anyone better to write this story except possibly Michael Lewis, author of Liar's Poker.

Twenty years after Bonfire of the Vanities, the author checks in on the new masters of the universe and finds them even coarser and ruder than their predecessors could have ever imagined being.

...First, they have more money, infi-nitely more, than any of the various waves of new money that preceded them, with the possible exception of robber barons on the order of John D. Rockefeller, who, incidentally, was regarded as a rude Pocantico hillbilly Baptist by society in New York a hundred years ago. Hedge funds have what investment managers call “the greatest business plan of all time,” known as “2 and 20.” Each year the typical fund takes 20 percent of the return plus 2 percent of the total investments. Some of the hottest managers, such as Icahn and Stevie Cohen, reportedly take 50 percent of the profits.

In 2005—the figures for 2006 are not yet available—Cohen’s SAC Capital Advisors, of Stamford (he himself lives in Greenwich), made an 18 percent profit on its investors’ $8.5 billion, meaning that Cohen’s income for that one year was in the neighborhood of a billion dollars. This and the figures that follow are the calculations of Trader’s Monthly and Alpha magazine, a niche publication created for the hedge fund industry. No magazine was ever named with greater psychological accuracy, as we are about to see.

Neck and neck with Cohen were James Simons of East Setauket, New York (between $900 million and $1 billion), and Paul Tudor Jones II of Greenwich (between $800 million and $900 million). Further back in the field, nose to nose at $500 million to $600 million each, were Eddie Lampert of Greenwich and Stephen A. Feinberg and Bruce Kovner of New York City. But all six were far, far behind the old man, T. Boone Pickens, who you’ll recall made $1.5 billion in 2005.

Second, hedge fund managers are possessed by a previously unheard-of status fixation. The bellowing door-banger had that status fixation and then some. In Greenwich such characters are not shoehorned into the same buildings as ordinary rich people, which is to say, those with older and far less money. Given Greenwich’s zoning, these people are not likely to express that status fixation neighbor to neighbor. It comes out in other ways.

...The tales are endless: the hedge fund founder desperate to get his son into one of Greenwich’s socially swell private schools who clips a six-figure check to the first page of the application, witlessly forcing the school to reject both his son and his check or lose all credibility—

The lone-wolf entrepreneur who keeps an old-money matron and charity fundraiser waiting outside his office in Greenwich for an hour, remains reared back in his chair with his feet propped up on top of his desk as she comes in, listens to her pitch with his feet on top of his desk, utters a sum total of two words, “Not interested,” with his feet up on top of his desk, and offers no farewell, not even a Godspeed tap-tap of the shoes on his feet up on top of the desk—

The many of these people who spend entire meetings with eyes cast down at their BlackBerrys, thumbing out text messages to God-knows-what-people elsewhere—

The hedge fund manager who, during a 40-minute meeting, takes four telephone calls from his wife on the subject of a dinner party they’re planning, down to the level of who should sit next to whom, whether to serve the champagne in the new flutes or the art deco bowl-and-stem glasses, whether or not endive works as an hors d’oeuvre or is it a little too bitter?—

The hedge fund managers who hold meetings with their shirttails hanging outside their jeans, like college boys—

The former manager of Tremont Capital Group who came to meetings with the fund’s investors barefoot—

The twinkie wives of these people who arrive at real estate offices seeking to-die-for houses and apartments wearing jeans and stiletto-heel boots, with gotta-be-blond hair streaming down to their shoulder blades, holding a baby on a cocked hip with one hand and a cell phone to the ear with the other while a limousine waits outside, motor running—

The twinkies who have their eggs fertilized by their husbands’ sperm in a laboratory, creating embryos for implantation in the wombs of surrogate mothers who are paid to manufacture children for delivery in nine months, since why on earth should any wife whose husband is worth a billion or even $500 million have to endure the distended belly, bilious mornings, back cramps, not to mention a cramped social life, to end up with her perfect personal-trainer-sculpted boy-with-breasts body she has spent thousands of sweaty hours attaining, ruined … tempting her husband to survey all the little man-eaters out there, including those former wives who used to meet regularly at the Boxing Cat Grill until it burned down, whereas the current wives leave their husbands catatonic before the plasma TV and meet three or four times a week at one local bar or another and drive home in their Hummers and bobtail Mercedes S.U.V.’s, bombed out of their minds, while waiting for the baby to come from the factory—

Whenever such rich gossip is re-peated, somebody invariably says, “Who are these people?”

Nonlinearity and noisy outcomes

The Times magazine has a great little summary of some recent social science research, which studied the effects of social influence on judgements of quality. The researchers placed a number of songs online and asked volunteers to rate them. One group rated them without seeing others' opinions. In a number of "worlds" the raters were allowed to see the opinions of others in their world. Unsurprisingly, the interactive worlds exhibited large fluctuations, in which songs judged as mediocre by isolated listeners rose on the basis of small initial fluctuations in their ratings (e.g., in a particular world, the first 10 raters may have all liked an otherwise mediocre song, and subsequent listeners were influenced by this, leading to a positive feedback loop).

It isn't hard to think of a number of other contexts where this effect plays out. Think of the careers of two otherwise identical competitors (e.g., in science, business, academia). The one who enjoys an intial positive fluctuation may be carried along far beyond their competitor, for no reason of superior merit. The effect also appears in competing technologies or brands or fashion trends.

If outcomes are so noisy, then successful prediction is more a matter of luck than skill. The successful predictor is not necessarily a better judge of intrinsic quality, since quality is swamped by random fluctuations that are amplified nonlinearly. This picture undermines the rationale for the high compensation awarded to certain CEOs, studio and recording executives, even portfolio managers. In recent years I've often heard the argument that these people deserve their compensation because they generate tremendous value for society by making correct decisions about resource allocation (especially if they sit at the cash nexus of finance). However, the argument depends heavily on the assumption that the people in question are really adding value, rather than just throwing darts. If the system is sufficiently noisy it may be almost impossible to tell one way or the other. We may be rewarding the lucky, rather than the good, and a society with fewer incentives for these people may be equally or nearly equally efficient.

See related discussion of studio executives here, and another related discussion here.

As anyone who follows the business of culture is aware, the profits of cultural industries depend disproportionately on the occasional outsize success — a blockbuster movie, a best-selling book or a superstar artist — to offset the many investments that fail dismally. What may be less clear to casual observers is why professional editors, studio executives and talent managers, many of whom have a lifetime of experience in their businesses, are so bad at predicting which of their many potential projects will make it big. How could it be that industry executives rejected, passed over or even disparaged smash hits like “Star Wars,” “Harry Potter” and the Beatles, even as many of their most confident bets turned out to be flops? It may be true, in other words, that “nobody knows anything,” as the screenwriter William Goldman once said about Hollywood. But why? Of course, the experts may simply not be as smart as they would like us to believe. Recent research, however, suggests that reliable hit prediction is impossible no matter how much you know — a result that has implications not only for our understanding of best-seller lists but for business and politics as well.

...But people almost never make decisions independently — in part because the world abounds with so many choices that we have little hope of ever finding what we want on our own; in part because we are never really sure what we want anyway; and in part because what we often want is not so much to experience the “best” of everything as it is to experience the same things as other people and thereby also experience the benefits of sharing.

There’s nothing wrong with these tendencies. Ultimately, we’re all social beings, and without one another to rely on, life would be not only intolerable but meaningless. Yet our mutual dependence has unexpected consequences, one of which is that if people do not make decisions independently — if even in part they like things because other people like them — then predicting hits is not only difficult but actually impossible, no matter how much you know about individual tastes.

The reason is that when people tend to like what other people like, differences in popularity are subject to what is called “cumulative advantage,” or the “rich get richer” effect. This means that if one object happens to be slightly more popular than another at just the right point, it will tend to become more popular still. As a result, even tiny, random fluctuations can blow up, generating potentially enormous long-run differences among even indistinguishable competitors — a phenomenon that is similar in some ways to the famous “butterfly effect” from chaos theory. Thus, if history were to be somehow rerun many times, seemingly identical universes with the same set of competitors and the same overall market tastes would quickly generate different winners: Madonna would have been popular in this world, but in some other version of history, she would be a nobody, and someone we have never heard of would be in her place.

...Fortunately, the explosive growth of the Internet has made it possible to study human activity in a controlled manner for thousands or even millions of people at the same time. Recently, my collaborators, Matthew Salganik and Peter Dodds, and I conducted just such a Web-based experiment. In our study, published last year in Science, more than 14,000 participants registered at our Web site, Music Lab (www.musiclab.columbia.edu), and were asked to listen to, rate and, if they chose, download songs by bands they had never heard of. Some of the participants saw only the names of the songs and bands, while others also saw how many times the songs had been downloaded by previous participants. This second group — in what we called the “social influence” condition — was further split into eight parallel “worlds” such that participants could see the prior downloads of people only in their own world. We didn’t manipulate any of these rankings — all the artists in all the worlds started out identically, with zero downloads — but because the different worlds were kept separate, they subsequently evolved independently of one another.

This setup let us test the possibility of prediction in two very direct ways. First, if people know what they like regardless of what they think other people like, the most successful songs should draw about the same amount of the total market share in both the independent and social-influence conditions — that is, hits shouldn’t be any bigger just because the people downloading them know what other people downloaded. And second, the very same songs — the “best” ones — should become hits in all social-influence worlds.

What we found, however, was exactly the opposite. In all the social-influence worlds, the most popular songs were much more popular (and the least popular songs were less popular) than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds, just as cumulative-advantage theory would predict. Introducing social influence into human decision making, in other words, didn’t just make the hits bigger; it also made them more unpredictable. ...

Inequality and NYC real estate

Felix Salmon argues that Manhattan property values have nowhere to go but up! (Last I checked, desirable apartments were over $1k per square foot, or > $1M for a 1000 square foot pad.)

His argument is simple: thanks to growing inequality (nonlinear returns to big winners in fields like finance, entertainment, technology), and the unique desirability of living in Manhattan (he details these ad nauseum in his post -- as a professional blogger, I suppose he has a lot of free time :-), there will always be high demand for the limited number of units in the Big Apple. I think he's right, at least based on current trends.

In the first quarter of this year, the New York City housing market boomed even as the rest of the country saw some nasty falls in house prices. And I suspect that the same trend might continue for quite a while. Partly, that's because precious few Manhattan homeowners have subprime mortgages. But on a much larger scale, it's because New York is one of a handful of global cities which are the winners in the location stakes. The set of things you buy when you buy an apartment here can't be measured in square feet.

At 11:18am this morning, I got an email which told me that the Committee on Global Thought at Columbia University was having a discussion about the economics of climate change. The discussants? Jeff Sachs, Joe Stiglitz, and Nick Stern. Said discussion was happening at 4pm, and was free and open to the public. Of course, I went. I was even fortunate enough to be able to put to Stern directly my single biggest question/problem on the subject of climate change. He gave a great answer – and then Sachs answered the question too, and then Stiglitz gave his answer, and then Stern came back and added to his answer. (I'll blog it in a minute.) It was a wonderful moment, and I thank New York City for it.

...The climate change event took place one week to the day after I went out for lunch with Nassim Nicholas Taleb, and had a fascinating and wide-ranging conversation with him.

...And because New York is a global town, demand for property here is global as well. Every time the dollar falls, New York property becomes that much more appealing to millions of Europeans and Asians who have visited and dreamed of living here: it's not even expensive, by London or Hong Kong standards.

I wouldn't be at all surprised were someone to tell me that Sachs, Stiglitz and Stern were all having dinner tonight with Bill Clinton, maybe at the house of Mike Bloomberg or George Soros. It's the kind of thing which happens in New York – and in precious few other places. Davos, maybe, once a year. As such people move to New York, other such people follow them here, in a self-perpetuating virtuous cycle.

Taleb says, in his latest book, that there's no particular reason why New York rose and Baltimore fell. But now it has happened, it can't be stopped. Baltimore will never again be a leading global city. And – I feel comfortable in saying – New York will never again (not in the next few decades, anyway) be a crime-addled drug den like it was in the 1980s. The road from there to here was not foreseeable. But the road ahead is clear: New York City is pulling away from the pack, and the bigger a lead it takes, the faster it goes.

Of course, Felix's perspective might change after he has kids :-)

Skills shortage in India

Thanks to a correspondent who sent me the link. I get the New Yorker in the mail, but strangely the last issue or two seem not to have arrived!

I think the main advantage India has over China is linguistic: thanks to their colonial history and the similarity between Hindi and English (both Indo-European languages with lots of cognates), it is much easier for Indians to operate comfortably in the lingua franca of business and science. While the elite of Indian science and engineering are fantastic, the article below points out that depth is still somewhat lacking. I suspect this is less true for China -- university-trained engineers there are relatively close to world standards.

Previous related posts.

New Yorker: James Surowiecki April 16, 2007

The economic transformation of India is one of the great business storiea of our time. As stifling government regulations have been lifted entrepreneurship has flourished, and the country has becom a high-powered center for information technology and pharmaceuticals. Indian companies like Infosys and Wipro are powerful global players, while Western firms like G.E and I.B.M. now have major research facilities in India employing thousands. India’s seemingly endless flow of young, motivated engineers, scientists, and managers offering developed-world skills at developing-world wages is held to be putting American jobs at risk, and the country is frequently heralded as “the next economic superpower.

But India has run into a surprising hitch on its way to superpower status: its inexhaustible supply of workers is becoming exhausted. Although India has one of the youngest workforces on the planet, the head of Infosys said recently that there was an “acute shortage of skilled manpower,” and a study by Hewitt Associates projects that this year salaries for skilled workers will rise fourteen and a half per cent, a sure sign that demand for skilled labor is outstripping supply.

How is this possible in a country that every year produces two and a half million college graduates and four hundred thousand engineers? Start with the fact that just ten per cent of Indians get any kind of post-secondary education, compared with some fifty per cent who do in the U.S. Moreover, of that ten per cent, the vast majority go to one of India’s seventeen thousand colleges, many of which are closer to community colleges than to four-year institutions. India does have more than three hundred universities, but a recent survey by the London Times Higher Education Supplement put only two of them among the top hundred in the world. Many Indian graduates therefore enter the workforce with a low level of skills. A current study led by Vivek Wadhwa, of Duke University, has found that if you define “engineer” by U.S. standards, India produces just a hundred and seventy thousand engineers a year, not four hundred thousand. Infosys says that, of 1.3 million applicants for jobs last year, it found only two per cent acceptable.

There was a time when many economists believed that post-secondary education didn’t have much impact on economic growth. The really important educational gains, they thought, came from giving rudimentary skills to large numbers of people (which India still needs to do—at least thirty per cent of the population is illiterate). They believed that, in economic terms, society got a very low rate of return on its investment in higher education. But lately that assumption has been overturned, and the social rate of return on investment in university education in India has been calculated at an impressive nine or ten per cent. In other words, every dollar India puts into higher education creates value for the economy as a whole. Yet India spends roughly three and a half per cent of its G.D.P. on education, significantly below the percentage spent by the U.S., even though India’s population is much younger, and spending on education should be proportionately higher. ...

Information, information processing and black holes

I wrote up the talk I gave last month in Barbados.

The excerpt below the abstract is from the final section of the paper (apologies for the latex remnants). In that section I discuss a rather strong implication of quantum mechanics. Simple entropic or information theoretic arguments, together with standard big bang cosmology, imply that essentially all the detailed aspects of the world around us (the arrangement of galaxies in clusters, electrons in stars, leaves on trees, or books on bookshelves) are random consequences of quantum outcomes. There is simply not enough information in the initial conditions to specify all of these things. Unless their variability is illusory, it must result from quantum randomness. Very little about the universe today is predictable, even with perfect knowledge of the initial conditions and subsequent dynamical evolution.


hep-th > arXiv:0704.1154

Information, information processing and gravity

Abstract: I discuss fundamental limits placed on information and information processing by gravity. Such limits arise because both information and its processing require energy, while gravitational collapse (formation of a horizon or black hole) restricts the amount of energy allowed in a finite region. Specifically, I use a criterion for gravitational collapse called the hoop conjecture. Once the hoop conjecture is assumed a number of results can be obtained directly: the existence of a fundamental uncertainty in spatial distance of order the Planck length, bounds on information (entropy) in a finite region, and a bound on the rate of information processing in a finite region. In the final section I discuss some cosmological issues related to the total amount of information in the universe, and note that almost all detailed aspects of the late universe are determined by the randomness of quantum outcomes. This paper is based on a talk presented at a 2007 Bellairs Research Institute (McGill University) workshop on black holes and quantum information.

How much information in the universe?

In this final section we ask how much information is necessary to specify the current state of the universe, and where did it come from?

There is convincing observational evidence for the big bang model of cosmology, and specifically for the fact that the universe is and has been expanding. In a radiation-dominated universe, the FRW scale factor grows as $R(t) \sim t^{1/2}$, where $t$ is the comoving cosmological time. From this, it is clear that our universe evolved from a much smaller volume at early times. Indeed, in inflationary cosmology (Fig.~\ref{inflation}) the visible universe results from an initial patch which is exponentially smaller than our current horizon volume. The corresponding ratio of entropies is similarly gigantic, meaning that there is much more information in the universe today than in the small primordial patch from which it originated. Therefore, the set of possible early universe initial conditions is much, much smaller than the set of possible late time universes. A mapping between all the detailed rearrangements or modifications of the universe today and the set of possible initial data is many to one, not one to one \cite{rnd}.

Thus, the richness and variability of the universe we inhabit cannot be attributed to the range of initial conditions. The fact that I am typing this on a sunny day, or that our planet has a single moon, or that the books on my office shelves have their current arrangement, was not determined by big bang initial data.

How, then, do the richness and variability of our world arise? The answer is quantum randomness -- the randomness inherent in measurements of quantum outcomes.

Imagine an ensemble $\Psi$ of $n$ qubits, each prepared in an identical state $\psi$. Now imagine that each qubit is measured, with a resulting spin up ($+$) or spin down ($-$) result. There are $2^n$ possible records, or histories, of this measurement. This is an exponentially large set of outcomes; among them are all possible $n$-bit strings, including every $n$-bit work of literature it is possible to write! Although the initial state $\Psi$ contained very little information (essentially, only a single qubit of information, since each spin is in an identical state), $n$ bits of classical information are required to specify {\it which} of the $2^n$ outcomes is observed in a particular universe. For $n \rightarrow \infty$ the set of possible records is arbitrarily rich and varied despite the simplicity of initial state $\Psi$.

In the same way, given an initial quantum state $\Psi$ describing the primordial patch of the big bang from which our horizon volume evolved, one must still know the outcomes of a large number of quantum measurements in order to specify the particulars of the universe today. From a many worlds perspective, one must specify all the decoherent outcomes to indicate a particular branch of the wavefunction -- a staggering amount of information. Equivalently, from the traditional Copenhagen perspective, each quantum measurement injects a bit (or more) of truly random information into our universe, and this randomness accounts for its variability.

The most familiar cosmological quantum randomness comes from fluctuations of the inflaton field, which determine the spectrum of primordial energy density fluctuations. It is these density fluctuations that determine the locations of galaxies, stars and planets today. However, from entropic or information theoretic considerations we readily deduce that essentially {\it every} detailed aspect of our universe (beyond the fundamental Lagrangian and some general features of our spacetime and its contents) is a consequence of quantum fluctuations!

Back to the future

Nice survey in the Economist on China development.

...The favourite reading at the moment among a younger, more cosmopolitan generation of Chinese diplomats is “Power Shift”, a collection of essays by mainly American-based academics. Its premise is that the tectonic plates that have defined Asia for the past half-century are moving, and that China is the chief agent of change as it resumes its historical role as Asia's central actor. Gone, largely, are China's fears of encirclement. “Impossible!” a senior Chinese diplomat laughs. “China is now far too powerful to be contained.” One of Deng Xiaoping's tenets—that the country should, as a Chinese saying has it, disguise its ambition and hide its claws—seems to have been buried.

But what kind of power is China becoming? Some Western hawks find it unsettling that this is even being debated within China, but it is better to talk about it than not.

Only once a decade or so does a piece of television programming break through the variety shows and the propaganda to capture China's attention. A hugely popular 12-part series on China Central Television has just done so, showing how nine countries rose to prominence, beginning with Portugal in the 15th century and ending with the United States in the 20th. The conclusion, as befits state television, delivers an explicit political message, but one that may surprise outsiders. In finding plenty of lessons to learn from, the series attaches greater importance to social stability and peaceful foreign relations than to jingoism and brute military strength.

Indeed, a propos of the television series, the same senior Chinese diplomat mentioned earlier argued energetically that pacifist Japan's post-war rise was a model of good-neighbourliness that China itself could usefully emulate. That is intriguing. Much of the present bad blood between China and Japan has to do with China's constant harping on Japan's brutal deeds in the first half of the 20th century while glossing over its positive regional influence in the second half.

...Yet suspicions remain. Mr Hu may have embraced the notion of China's “peaceful rise”, first advanced by Chinese academics in 2003, yet even the phrase itself is unsettling. As Lee Kuan Yew, Singapore's former prime minister and now its “minister mentor”, puts it: “‘Peaceful rise' is a contradiction in terms. I told China's leaders that. I said: ‘Why not call it a renaissance, a return to a golden age when poetry, painting, clothes, music and drama flourished?'”

China's economic rise is certainly impressive. The economy's growth—an average of 10% a year since 1990—is not really more remarkable than the earlier rise of other Asian economies, led by Japan, but there is a difference: the huge size of China's population, at 1.3 billion. In 2005 China overtook Japan in the volume of trade it conducts. Depending on how you measure size and guess at future growth rates, it may overtake both Germany and Japan within 15 years to become the world's second-biggest economy. Measured at purchasing-power parity, China's share of the world economy is already much closer to the rich countries' (see chart 1). But bear in mind that the average Chinese income remains low. If China is on its way to becoming a superpower, it will be the world's poorest one yet.

Opinion polls suggest that the vast majority of Chinese see their rise as nothing that should trouble others. For many of them it merely marks a return to historical norms. Angus Maddison, an economic historian at the University of Groningen, has estimated that between 1600 and the early 19th century China accounted for between a quarter and a third of global output (see chart 2). At that time China's agriculture was more advanced than the West's, its cities bigger and more literate and its ruling classes more meritocratic. The country had also proved itself capable of long-distance exploration by sea. Another historian, Niall Ferguson, reckons that what went so spectacularly wrong for China then is more remarkable and worthy of investigation than why things should now be going right.

It's all in da gene...

"Horses ain't like people, man, they can't make themselves better than they're born. See, with a horse, it's all in the gene. It's the fucking gene that does the running. The horse has got absolutely nothing to do with it." --- Paulie (Eric Roberts) in The Pope of Greenwich Village.

The size of dogs -- from Chihuahua to Great Dane -- is controlled by a single gene!

NYTimes: Scientists have just discovered which gene fragment controls the size of dogs, which have the greatest size range of any mammal — no other species produces adults with 100-fold differences, like that between a two-pound chihuahua and a 200-pound Newfoundland.

In a study to be published tomorrow in the journal Science, researchers analyzed 3,241 purebred dogs from 143 breeds. Genetically, the yapper arguing with your ankle is almost identical to the drooling behemoth bred to hunt bears, except for a tiny bit of DNA that suppresses the “insulin-like growth factor 1” gene.

Dog breeders have unwittingly been selecting for it since the last Ice Age. Dogs emerged from the wolf about 15,000 years ago, and as far back as 10,000 years ago, domesticated dogs as big as mastiffs and as small as Jack Russell terriers were trotting the earth.

Bonus! From Gene Expression, the following figure is taken from this PLoS Genetics paper (excerpt from abstract below). See related earlier post here.



Neighbor-Joining Trees Depicting the Genetic Relationships of 1,040 Individuals from 51 World Populations Collected by the CEPH-HGDP

(A) Individuals are color coded according to which of five major geographic regions of the globe they are collected from.

(B) Individuals are color coded according to which of the 51 populations they are associated with (1: Biaka Pygmy, 2: San, 3: Mbuti Pygmy, 4: Druze; 5: Bedouin, 6: Mozabite, 7: Palestinian, 8: Kalash, 9: Pima, 10: Columbian, 11: Karitiana, 12: Surui, 13: New Guinea, 14: Yakut).

Generalized Analysis of Molecular Variance: ... As in the original AMOVA strategy, the proposed approach, termed generalized AMOVA (GAMOVA), requires a genetic similarity matrix constructed from the allelic profiles of individuals under study and/or allele frequency summaries of the populations from which the individuals have been sampled. The proposed strategy can be used to either estimate the fraction of genetic variation explained by grouping factors such as country of origin, race, or ethnicity, or to quantify the strength of the relationship of the observed genetic background variation to quantitative measures collected on the subjects, such as blood pressure levels or anthropometric measures. Since the formulation of our test statistic is rooted in multivariate linear models, sets of variables can be related to genetic background in multiple regression-like contexts. GAMOVA can also be used to complement graphical representations of genetic diversity such as tree diagrams (dendrograms) or heatmaps. We examine features, advantages, and power of the proposed procedure and showcase its flexibility by using it to analyze a wide variety of published data sets, including data from the Human Genome Diversity Project, classical anthropometry data collected by Howells, and the International HapMap Project.

Buffet and LTCM

Interesting tidbit for students of the LTCM meltdown, as relayed by Wharton professor Jeremy Siegel.

Apologies to people who believe in efficient markets and similar orthodoxy ;-)

Long Term Capital Management

One student asked about Berkshire’s role in the Long-Term Capital Management (LTCM) crisis that rocked the financial markets in 1998. It was obvious that Warren wished that he had been able to buy LTCM’s positions when the Fed forced a resolution of the crisis that was crippling the government bond market.

The LTCM crisis was a ready-made example of Warren’s philosophy of buying firms when the economics was right, yet fear ruled the markets. He noted that “off-the-run” (non-benchmark) government bonds were selling to yield 30 basis points more than the “on-the-run” (benchmark) bonds that were maturing just six months later. He rightly claimed that this made no sense economically.

LTCM had taken a huge leveraged position in these bonds when the spreads were much smaller, but didn’t have the collateral to hold on to it when the spread widened. Buffett quoted John Maynard Keynes, who wrote in 1931 that “The market can stay irrational longer than you can stay solvent.” As the spread widened, Keynes’ dictum became devastatingly relevant for LTCM. But Berkshire, with its huge cash hoard, could withstand the pressure of even more market irrationality before the spread eventually returned to normal.

Unfortunately, Warren was never able to consummate the deal. He had been invited by Bill Gates to vacation in Alaska when the crisis broke and it was hard to negotiate such a deal on a cell phone. Eventually the banks holding the collateral made a deal with LTCM and Berkshire didn’t have a chance to top their offer.

Warren is philosophical about the loss of this opportunity, noting it was the most expensive vacation he ever took. “Bill Gates cost me about $3 billion,” he shrugged. But that incident certainly has not soured their friendship and didn’t prevent him from giving the Bill and Melinda Gates Foundation the lion’s share of his wealth.

Letter to a former student

Here's some advice I wrote to a former student who is headed into software development. Can anyone add to or improve my comments?

1) software development in general

The Mythical Man Month (overrated, in my view, but everyone in the industry has read it)

Joel Spolsky on Software (a successful entrepreneur who has a big following in the developer community; he has an extensive web site)

Paul Graham (a CS PhD who writes about software and startups; very opinionated; check out his web site)


2) algorithms

The standard text is Rivest et al., but for an informal introduction to some of the best stuff in CS try The Turing Omnibus. For security and crypto, try Applied Cryptography by Schneier. Knuth's books on the art of computer programming are probably mainly of academic interest, but you might enjoy a look.


3) general

Anyone in the working world should read How To Win Friends and Influence People by Dale Carnegie -- I swear by it. There is an outline of the whole book on my blog somewhere :-)


4) miscellaneous comments

Unix shell tools are amazingly powerful and will serve you time and again -- grep, piping, awk, sed, emacs, etc. Many Windows programmers are unfamiliar with these and are blown away by their power to do quick and dirty stuff.

The more you know the more effective you become -- experienced programmers think a lot before they code, and they tend to reuse existing code or libraries.

Don't forget to think like a physicist -- get to the heart of the problem before starting in any direction; make quick and dirty models and test them out; test your assumptions throughout the process by checking against the accumulating evidence. After you have a feel for the problem spend some time generalizing about or abstracting from what you've done.

Who pays?

I'm trying to understand how the subprime mortgage mess is going to unwind. Below are links to two recent articles I found useful.

1) WSJ on leading subprime lender New Century and how its bankruptcy was driven by decisions on Wall St. by firms like Citi, JP Morgan and Merrill that both lend to mortgage issuers and repackage their loans for resale as CMOs.

2) The Economist's remarkably sanguine summary of the situation. (Some useful excerpts below -- the first hard analysis numbers I've yet seen.)

America's residential mortgage market is huge. It consists of some $10 trillion worth of loans, of which around 75% are repackaged into securities, mainly by the government-sponsored mortgage giants, Fannie Mae and Freddie Mac. Most of this market involves little risk. Two-thirds of mortgage borrowers enjoy good credit and a fixed interest rate and can depend on the value of their houses remaining far higher than their borrowings. But a growing minority of loans look very different, with weak borrowers, adjustable rates and little, or no, cushion of home equity.

For a decade, the fastest growth in America's mortgage markets has been at the bottom. Subprime borrowers—long shut out of home ownership—now account for one in five new mortgages and 10% of all mortgage debt, thanks to the expansion of mortgage-backed securities (and derivatives based on them). Low short-term interest rates earlier this decade led to a bonanza in adjustable-rate mortgages (ARMs). Ever more exotic products were dreamt up, including “teaser” loans with an introductory period of interest rates as low as 1%.

When the housing market began to slow, lenders pepped up the pace of sales by dramatically loosening credit standards, lending more against each property and cutting the need for documentation. Wall Street cheered them on. Investors were hungry for high-yielding assets and banks and brokers could earn fat fees by pooling and slicing the risks in these loans.

Standards fell furthest at the bottom of the credit ladder: subprime mortgages and those one rung higher, known as Alt-As. A recent report by analysts at Credit Suisse estimates that 80% of subprime loans made in 2006 included low “teaser” rates; almost eight out of ten Alt-A loans were “liar loans”, based on little or no documentation; loan-to-value ratios were often over 90% with a second piggy-bank loan routinely thrown in. America's weakest borrowers, in short, were often able to buy a house without handing over a penny.

Lenders got the demand for loans that they wanted—and more fool them. Amid the continuing boom, some 40% of all originations last year were subprime or Alt-A. But as these mortgages were reset to higher rates and borrowers who had lied about their income failed to pay up, the trap was sprung. A new study by Christopher Cagan, an economist at First American CoreLogic, based on his firm's database of most American mortgages, calculates that 60% of all adjustable-rate loans made since 2004 will be reset to payments that will be 25% higher or more. A fifth will see monthly payments soar by 50% or more.

...Mr Cagan marries the statistics and concludes that—going by today's prices—some 1.1m mortgages (or 13% of all adjustable-rate mortgages originated between 2004 and 2006), worth $326 billion, are heading for repossession in the next few years. The suffering will be concentrated: only 7% of mainstream adjustable mortgages will be affected, whereas one in three of the recent “teaser” loans will end in default. The harshest year will be 2008, when many mortgages will be reset and few borrowers will have much equity.

Mr Cagan's study considers only the effect of higher payments (ignoring defaults from job loss, divorce, and so on). But it is a guide to how much default rates may worsen even if the economy stays strong and house prices stabilise. According to RealtyTrac, some 1.3m homes were in default on their mortgages in 2006, up 42% from the year before. This study suggests that figure could rise much further. And if house prices fall, the picture darkens. Mr Cagan's work suggests that every percentage point drop in house prices would bring 70,000 extra repossessions.

The direct damage to Wall Street is likely to be modest. A repossessed property will eventually be sold, albeit at a discount. As a result, Mr Cagan's estimate of $326 billion of repossessed mortgages translates into roughly $112 billion of losses, spread over several years. Even a loss several times larger than that would barely ruffle America's vast financial markets: about $600 billion was wiped out on the stockmarkets as share prices fell on February 27th.

In theory, the chopping up and selling on of risk should spread the pain. The losses ought to be manageable even for banks such as HSBC and Wells Fargo, the two biggest subprime mortgage lenders, and Bear Stearns, Wall Street's largest underwriter of mortgage-backed securities. Subprime mortgages make up only a small part of their business. Indeed, banks so far smell an opportunity to buy the assets of imploding subprime lenders on the cheap.

Some open questions (experts please help!):

1) Effect on housing bubble: how much of the recent bubble was driven specifically by increased availability of credit (as opposed to the usual irrational exuberance or speculation)?

2) How much of bad mortgage debt is insured by CDO derivatives? Who is on the hook? Selling this kind of insurance was reportedly a popular income strategy for hedge funds.

3) Is $100B of mortgage-related losses over several years a big number or a small one? Will anyone blow up (CMO insurers)? Who is holding the riskiest CMO tranches?

4) If $100B over several years is chump change, what are the chances of contagion still leading to a housing bust, credit crunch and recession?

Earnings of big Wall St. banks will be negatively impacted, but some smart guys are surely buying up these loans on the cheap, as markets overreact in the negative direction.

Proton pumps: modular, swappable genetic units

Wow! It's the modularity that is amazing. Swap a bit of DNA and suddenly you have bacteria that can harvest energy from light! (Via GNXP.)

Technology Review: Some bacteria, such as cyanobacteria, use photosynthesis to make sugars, just as plants do. But others have a newly discovered ability to harvest light through a different mechanism: using light-activated proteins known as proteorhodopsins, which are similar to proteins found in our retinas. When the protein is bound to a light-sensitive molecule called retinal and hit with light, it pumps positively charged protons across the cell membrane. That creates an electrical gradient that acts as a source of energy, much like the voltage, or electromotive force, supplied by batteries.

First discovered in marine organisms in 2000, scientists recently found that the genes for the proteorhodopsin system--essentially a genetic module that includes the genes that code for both the protein and the enzymes required to produce retinal--are frequently swapped among different microorganisms in the ocean. (While we usually think of genes being passed from parent to offspring, microorganisms can exchange bits of DNA laterally.)

Intrigued by the prospect that a single piece of DNA is really all an organism needs to harvest energy from light, the researchers inserted it into E. coli. They found that the microorganisms synthesized all the necessary components and assembled them in the cell membrane, using the system to generate energy. "All it takes to derive energy from sunlight is that bit of DNA," says Ed Delong, professor of biological engineering at MIT and author of the study. The results were published last week in the Proceedings of the National Academy of Sciences.

The findings have implications for both marine ecology and for synthetic biology, an emerging field that aims to design and build new life forms that can perform useful functions. Giant genomic studies of the ocean have found that the rhodopsin system is surprisingly widespread. The fact that a single gene transfer can result in an entirely new functionality helps explain how this genetic module traveled so widely. In fact for microbes, this kind of module swapping may be the rule rather than the exception."A new paradigm is emerging in microbiology: [microorganisms] are much more fluid than we thought," says Ford Doolittle, Canada Research Chair in comparative genomics at DalhousieUniversity, in Nova Scotia.

Income inequality: Manhattan toddlers

From the Times, this story tells a lot about what's happening in Manhattan. My friends there say it's very kid-friendly these days, with crime way down from a few decades ago.

Given how the hedonic treadmill works, I can't imagine living in Manhattan if I were a Columbia or NYU professor. Who wants to be the poorest family in the neighborhood? ;-) One of the families in the article, the father a management consultant, says they won't be able to afford the upper West Side once their kids each need a bedroom of their own.

The analysis shows that Manhattan’s 35,000 or so white non-Hispanic toddlers are being raised by parents whose median income was $284,208 a year in 2005, which means they are growing up in wealthier households than similar youngsters in any other large county in the country.

Among white families with toddlers, San Francisco ranked second, with a median income of $150,763, followed by Somerset, N.J. ($136,807); San Jose, Calif. ($134,668); Fairfield, Conn. ($132,427); and Westchester ($122,240).

Median household income of families with children ages 0 to 4. (Left is all ethnic groups, right is non-Hispanic whites only.)



For more on income inequality, including the interesting observation that it is primarily driven by financiers and tech entrepreneurs (third link), see here.

The Mechanical Turk and Searle's Chinese Room

The Times has an article about Jeff Bezos' Mechanical Turk project, which lets machines outsource certain tasks to humans. (The orginal mechanical Turk was an 18th century hoax in which a hidden human operated a chess-playing automaton.) As Bezos describes,

“Normally, a human makes a request of a computer, and the computer does the computation of the task,” he said. “But artificial artificial intelligences like Mechanical Turk invert all that. The computer has a task that is easy for a human but extraordinarily hard for the computer. So instead of calling a computer service to perform the function, it calls a human.”

...The company opened Mechanical Turk as a public site in November 2005. Today, there are more than 100,000 “Turk Workers” in more than 100 countries who earn micropayments in exchange for completing a wide range of quick tasks called HITs, for human intelligence tasks, for various companies.

The Times writer Jason Pontin (who is also editor and publisher of MIT's Technology Review), gives Turk working a try, and finds it disorienting:

What is it like to be an individual component of these digital, collective minds?

To find out, I experimented. After registering at www.mturk.com, I was confronted with a table of HITs that I could perform, together with the price that I would be paid. I first accepted a job from ContentSpooling.net that asked me to write three titles for an article about annuities and their use in retirement planning. Then I viewed a series of images apparently captured from a vehicle moving through the gray suburbs of North London, and, at the request of Geospatial Vision, a division of the British technology company Oxford Metrics Group, identified objects like road signs and markings.

For all this, my Amazon account was credited the lordly sum of 12 cents. The entire experience lasted no more than 15 minutes, and from my point of view, as an occluded part of the hive-mind, it made no sense at all.

This is reminiscent of philospher John Searle's thought experiment called the Chinese Room, in which he posits a large team of humans implementing an algorithm that translates Chinese to English. Since each human performs only a small task (e.g., sorting acording to a rule set), none have any understanding of the overall process. Searle asks where, exactly, does the understanding of Chinese and English reside in this device? Searle considered his thought experiment as evidence against strong AI, whereas I just consider Searle to be confused. It's obvious that a Turk worker might be a small cog in some larger process that "understands" the world and processes information in a useful way. This depends not at all on what the little cog understands or does not understand.

Academic impact

Berkeley PhD student Jo Guldi describes how Google Book Search is revolutionizing her historical research.

Google Book Search is a relatively recent phenomenon... six months ago, right? About six months ago I was pottering around there, finding a few illustrated nineteenth-century texts, a lot of contemporary books for sale, and not much of too much interest. Six months turns out to be a long time in book land. In that period of time, Book Search has accomplished enough to transform the academic profession.

I was idly trying a search on "roads" to see what sort of a literature would turn up for the period of my dissertation research, 1740-1850. I didn't expect much. I've spent the last two years wandering through the Yale, Harvard, and California libraries, the British Library, Britain's National Archives, and the immense reserves of North American Inter Library Loan reading every book on London, pavement, or travel I could get my hands on.

Surprise. In a single idle search I just added twenty extra full-text books to my list.

Which are, by the way, full-text searchable --

-- and subject to word-count analysis --

-- and replete with full illustrations --

-- and instantly digestable into visuals for powerpoint presentations.

Hallelujah, GoogleBooks. And holy mackeral! Good work.

By now, the first half of the nineteenth century exists in a very complete form on Google Books. In the last six months, while academic history has meandered in its habituated paths of grinding research, the possibilities of scholarship have been utterly transformed.

...

What this signals, by the way, is the opportunity for a new age of scholarship. Cultural and image analysis used to be painfully time-consuming, heavy lifting, involving rare kinds of access, full fellowships, immense travel, and long waits for delicate books. Comparison between different cultural sources was even harder, placing absurd demands on the cultural historian's personal memory and note-taking skills. Cultural historians, despite their many skills, stood second in depth of research on any particular topic to political historians, for whom one visit to a Parliamentary archive and one visit to a personal residence outfitted them with every last detail of historical change. Now all that is changing. Comparing a hundred images is no longer a problem for a year's labor in an out-of-the-way museum reading room. Comparing a hundred personal accounts from working men is no longer a task to eat up a social historian's entire year.

Eric Schmidt interview

Interesting interview (podcast) with Google CEO Eric Schmidt.

On the infrastructure barrier of entry to compete with Google:

Q: Ray Ozzie recently made some comments about Microsoft. Their data center construction being a big competitive advantage. I wanted to see how you think that what Google is doing is different from what their competitors are doing with their data centers.

Eric: Google, by any measure, is much more capital-intensive than our competitors, and we’re much further along in the build-out of data centers. So without commenting on our competitor’s quote and positioning, let me say that in an internet market where you deliver your services by computers with spinning disks, we have a competitive advantage... because we have the cheapest and most scalable such architecture. We hope that in the course of innovation we will be able to build products which are almost impossible for our competitors to replicate, because we simply learn how to implement them at scale. The internet is a scaled business, and running large internet scale businesses is very, very difficult. All the challenges – staying up, dealing with spammers, dealing with other access problems – and we believe we do it best in the world.

Q: What are some of those big areas of capital expenditures today... kinda how do you see those going forward?

Eric: Well, virtually all the capital goes into the expenses that are associated with running data centers. And that literally are the computers, and the disks, and the appropriate networking harboring, and literally the buildings now that house them. It used to be, right after the bubble burst, that you could go and purchase these inexpensive data centers that had been written off by people who overbuilt. But those data centers are long gone, and that’s one of our primary focus.

On entrepreneurship:

Q: Talking before, you’ve spoken glowingly about Larry and Sergey, and also Chad and Steve from YouTube. What are the characteristics of very successful entrepreneurs?

Eric: I think the most important characteristic of an entrepeneur is that they’re going to do it whether you give them permission or not. They are motivated by something inside of them. It’s not something that can be taught... they feel it, they want it, they’re driven to it. And when you find such a person, they’re usually a pretty good person to hang out with. They’re gonna do some interesting things. In Larry and Sergey’s case, and more recently with YouTube with Steve and Chad, they were going to be successful, and you can tell when you talk to them. Everything else is a tactic compared to that passion and vision that an entrepreneur has. It’s relatively rare, and it’s important to identify and respect it. (...)

Q: Eric, you’ve been a leader in the Valley for a long time now. What strategy is different today than in the past about the areas of innovation and entrepeneurship?

Eric: Yeah, it’s funny that every generation thinks that they’ve invented cold fusion... but every previous generation did too. And one of the rules about innovation is that it’s a constant process. The innovators today are not different – they’re just innovators now, as opposed to innovators from 10 years ago, or probably 100 years ago. In watching old BBC movies about England in the late 1900s/ early 1900s, you have to imagine what it meant to be an innovator, when roads and gaslights and electricity and so forth were being invented. It’s the same person, it’s the same type, it’s the same economic sense of leverage, it’s the same passion that drives innovators. The Valley is fortunate that these cycles are almost perpetual here, and innovators are drawn to both the culture, and I think the good weather...

Partial transcript found here.

Barbados pix

Some photos from Barbados, regrettably taken with my wimpy cellphone camera. The first two are from the beach near Bellairs Research Institute (where we had the workshop) and the third is Jonathan Oppenheim of Cambridge lecturing on black hole information.

I also added a fourth photo of the whole group, taken with Charles Bennett's non-wimpy camera. Charles is a talented photographer as well as information theorist :-)

Barbados, here I come!

I'm off to a workshop on black holes and quantum information. Although I'll have network access you might not hear much from me for the next week or so :-)

Bellairs workshop schedule.

Update: As predicted, I haven't had much time to post about this meeting. It's been a wonderful chance to talk to quantum information theorists while enjoying the Caribbean. I don't know whether the high point has been the excellent talks (which have often gone 2 hours, due to the informal atmosphere), getting to meet well-known theorists like Don Page (a relativist whose papers I first read while in grad school) or Charlie Bennett (IBM Fellow and co-inventor of quantum teleportation), or swimming with turtles yesterday. I'll post some photos when I get back this weekend.

Machine intelligence

We just completed some head to head testing of Robot Genius' web crawl data. We've built a fully automated process that downloads every Windows executable on the web (terabytes of data, now in the can), installs it on an analysis machine, and determines whether it is malware.

We compared our results for 8000 executable urls (something our farm can do in much less than a day) against the databases of two public security companies. The first is a leader in the web protection space, and the second is one of the 3 largest antivirus and desktop security vendors. The results of the second head to head comparison are below (in the first comparison the public company did even worse).

Conclusion: the Robot Genius defeats teams of hundreds of security engineers located in multiple countries.

Summary statistics:

Caught be RG: 100
Caught by leading AV company: 96

False negatives by RG: 4 (6 if you include Alexa toolbar, which we do not)
False negatives by leading AV company: 58

False positives by RG: 2, due to a dumb mistake that we have fixed
False positives in leading AV company data: 52 (50 if you consider Alexa toolbar bad, which we do not)

Hedge fund intelligence

It's a hedge fund universe: over a trillion dollars under management now by the largest US funds. Thanks to an intrepid correspondent for the data :-)

Hedge Fund Intelligence: As money continued to flow into alternatives last year, combined assets at the largest U.S. hedge funds finally crossed the trillion-dollar mark. The top three firms each exceeded $30 billion in assets for the first time, with JPMorgan Asset Management unseating Goldman Sachs Asset Management as the largest hedge fund platform, with $34 billion. Among the top tier, Renaissance Technologies joins the top 10 for the first time, and Cerberus Capital Management rejoins the list after a three-year hiatus.

The Absolute Return Billion Dollar Club, our biannual survey of U.S. hedge funds, shows that 241 firms, each managing more than $1 billion, held a combined total of nearly $1.2 trillion as of January 1. That is about $215 billion more than the top 218 firms were managing this past summer and $347 billion more than the top 207 firms were running at the beginning of last year. And the huge increases came during a year when hedge fund closures, including that of $9.1 billion Amaranth Advisors, erased $35 billion from the market. (See "Poof! $35 billion gone from hedge funds," p14.)

I originally posted more information -- including a long list of >$4B funds, but was asked by Hedge Fund Intelligence to remove most of it.

AI and Google

Here is a little video snippet of Larry Page talking about AI at the recent AAAS meeting. He points out that our genetic information is about 600MB compressed, so smaller than any modern operating system. The connection between compression and AI has been made by many people -- what is intelligence, after all, if not an algorithm capable of taking in data (observations about the world) and turning it into predictions about what will happen next? Prediction (or, equivalently, modeling) is nothing more than compression! Newton's laws plus some initial data compress all the information about the trajectory of a spaceship -- trading bits of stored information (the spacetime coordinates of the trajectory) for CPU flops (necessary to uncompress the trajectory from the initial position and velocity -- i.e., evolve it forward in time).

Page guesses that AI will result more from "lots of computation" than from "whiteboard stuff" (i.e., we won't really "understand" how it happens from a theoretical or analytical perspective) and that "we aren't as far off as many people think"!

A lot of people like to speculate that Google is working like mad on AI, and indeed certain related problems like machine translation, pattern recognition and, of course, search, are things they devote a lot of resources to. However, the vast majority of their 10,000 employees (yes, the number really has been doubling every year for a while) are working on just keeping the existing services up and running. There isn't yet a blue-sky, Bell Labs-like research arm at Google.

See here for previous related posts. I have a bet with a former PhD student about machines passing a strong version of the Turing test in the next 50 years.