On data mining

Last week we had Jiawei Han of UIUC here to give a talk: Exploring the Power of Links in Information Network Mining. He's the author of a well-known book on data mining.

During our conversation we discussed a number of projects his group has worked on in the past, all of which involve teasing out the structure in large bodies of data. Being a lazy theorist, my attitude in the past about data mining has been as follows: sit and think about the problem, come up with list of potential signals, analyze data to see which signals actually work. The point being that the good signals would turn out to be a subset (or possibly combination) of the ones you could think of a priori -- i.e., for which there is a plausible, human-comprehensible, reason.

In many of the examples we discussed I was able to guess the main signals that turned out to be useful. However, Han impressed on me that, these days, with gigantic corpora of data available, one often encounters very subtle signals that are identified only by algorithm -- that human intuition completely fails to identify. (Gee, why that weird linear combination of those inputs, with alternating signs, even?! :-)

Our conversation made me want to get my hands dirty on some big data mining project. Of course, it's much easier for him -- his group has something like ten graduate students at a time! Interestingly, he identified this ability to tap into large chunks of manpower as an advantage of being in academia as opposed to, e.g., at Microsoft Research. Of course, if you are doing very commercially applicable research you can access even greater resources at a company lab/startup, but for blue sky academic work it wouldn't be the case.

Conference fun

My summer schedule is filling up already!

Paris, May 26-28: Black hole information at Institut Henri Poincaré.

Foo camp, Sebastapol, July 11-13: woo hoo! O'Reilly Media's annual un-conference. My report from last year, including video.

Sci Foo, Googleplex, Aug 8-10: co-organized by Google, O'Reilly Media and Nature. Flickr photos from last year (2007).

Trento, Italy Sept. 1-5: statistical thermalization, at the European Center for Nuclear Theory (ECT).

Gladwell amongst the patent trolls

Malcolm Gladwell writes about Nathan Myhrvold's company Intellectual Ventures in the recent New Yorker. (Myhrvold is the former cosmologist who left physics and eventually became consigliere to Bill Gates, founding Microsoft Research and charting Microsoft's blue sky research direction. He famously missed the importance of the Internet until the mid 90's.) If you read this blog often, you know my opinion about Gladwell: he has a good nose for interesting topics, but not enough brainpower or common sense for reliable analysis. The same is true here: he produces an interesting profile of Myhrvold (although see here for a much better one from 1997 by Ken Auletta) and friends, but seems to entirely miss a number of important points. Intellectual Ventures is not about real inventions, but about patenting around ideas so that they have a future claim on the ones that turn out the be useful. In other words, they are patent trolls. Gladwell does not seem to realize the difference between rampant speculation and true invention: the hours of painstaking work in the lab required to convert an idea into reality.

Here's an excerpt about how the "invention" process works -- get some smart guys in a room and let them talk (every theory group lounge is a fount of commercializable ideas ;-). Yes! if your inventors are smart enough, they can produce 36 new inventions at dinner! Is this a statement about real innovation, or about what a patent attorney might manage to get the understaffed, overburdened USPTO to approve? It makes a mockery of what real inventors and innovators do. Why start a company and hire engineers to build a prototype? Just get a few lawyers and patent everything in sight...

How useful is it to have a group of really smart people brainstorm for a day? When Myhrvold started out, his expectations were modest. Although he wanted insights like Alexander Graham Bell’s, Bell was clearly one in a million, a genius who went on to have ideas in an extraordinary number of areas—sound recording, flight, lasers, tetrahedral construction, and hydrofoil boats, to name a few. ...

But then, in August of 2003, I.V. held its first invention session, and it was a revelation. “Afterward, Nathan kept saying, ‘There are so many inventions,’ ” Wood recalled. “He thought if we came up with a half-dozen good ideas it would be great, and we came up with somewhere between fifty and a hundred. I said to him, ‘But you had eight people in that room who are seasoned inventors. Weren’t you expecting a multiplier effect?’ And he said, ‘Yeah, but it was more than multiplicity.’ Not even Nathan had any idea of what it was going to be like.”

The original expectation was that I.V. would file a hundred patents a year. Currently, it’s filing five hundred a year. It has a backlog of three thousand ideas. Wood said that he once attended a two-day invention session presided over by Jung, and after the first day the group went out to dinner. “So Edward took his people out, plus me,” Wood said. “And the eight of us sat down at a table and the attorney said, ‘Do you mind if I record the evening?’ And we all said no, of course not. We sat there. It was a long dinner. I thought we were lightly chewing the rag. But the next day the attorney comes up with eight single-spaced pages flagging thirty-six different inventions from dinner. Dinner.”

For the cognoscenti out there, yes, the Wood mentioned in the article is none other than Star Warrior Lowell Wood, former head of the zany (useless?) O Group (NYTimes 1984) at Livermore. Wood is perfect for Myhrvold's purposes -- for decades his group bamboozled the US defense establishment with wild ideas that cost taxpayers billions of dollars. Follow the link to the Times article and tell me how many of the ideas mentioned turned into something useful, now almost a quarter century later.

Rather than leave you with a completely negative impression of the article, I include the following excerpt, which has Wood noticing something about cancer cells in the bloodstream that seems to have eluded biologists and medical researchers for some time. It is true that there are great ideas out there just waiting to be discovered, but lots of people can have the same idea. The hard part is making the idea into a practical, commercially viable reality.

...Last March, Myhrvold decided to do an invention session with Eric Leuthardt and several other physicians in St. Louis. Rod Hyde came, along with a scientist from M.I.T. named Ed Boyden. Wood was there as well.

“Lowell came in looking like the Cheshire Cat,” Myhrvold recalled. “He said, ‘I have a question for everyone. You have a tumor, and the tumor becomes metastatic, and it sheds metastatic cancer cells. How long do those circulate in the bloodstream before they land?’ And we all said, ‘We don’t know. Ten times?’ ‘No,’ he said. ‘As many as a million times.’ Isn’t that amazing? If you had no time, you’d be screwed. But it turns out that these cells are in your blood for as long as a year before they land somewhere. What that says is that you’ve got a chance to intercept them.”

How did Wood come to this conclusion? He had run across a stray fact in a recent issue of The New England Journal of Medicine. “It was an article that talked about, at one point, the number of cancer cells per millilitre of blood,” he said. “And I looked at that figure and said, ‘Something’s wrong here. That can’t possibly be true.’ The number was incredibly high. Too high. It has to be one cell in a hundred litres, not what they were saying—one cell in a millilitre. Yet they spoke of it so confidently. I clicked through to the references. It was a commonplace. There really were that many cancer cells.”

Wood did some arithmetic. He knew that human beings have only about five litres of blood. He knew that the heart pumps close to a hundred millilitres of blood per beat, which means that all of our blood circulates through our bloodstream in a matter of minutes. The New England Journal article was about metastatic breast cancer, and it seemed to Wood that when women die of metastatic breast cancer they don’t die with thousands of tumors. The vast majority of circulating cancer cells don’t do anything.

“It turns out that some small per cent of tumor cells are actually the deadly ones,” he went on. “Tumor stem cells are what really initiate metastases. And isn’t it astonishing that they have to turn over at least ten thousand times before they can find a happy home? You naïvely think it’s once or twice or three times. Maybe five times at most. It isn’t. In other words, metastatic cancer—the brand of cancer that kills us—is an amazingly hard thing to initiate. Which strongly suggests that if you tip things just a little bit you essentially turn off the process.”

That was the idea that Wood presented to the room in St. Louis. From there, the discussion raced ahead. Myhrvold and his inventors had already done a lot of thinking about using tiny optical filters capable of identifying and zapping microscopic particles. They also knew that finding cancer cells in blood is not hard. They’re often the wrong size or the wrong shape. So what if you slid a tiny filter into a blood vessel of a cancer patient? “You don’t have to intercept very much of the blood for it to work,” Wood went on. “Maybe one ten-thousandth of it. The filter could be put in a little tiny vein in the back of the hand, because that’s all you need. Or maybe I intercept all of the blood, but then it doesn’t have to be a particularly efficient filter.”

Wood was a physicist, not a doctor, but that wasn’t necessarily a liability, at this stage. “People in biology and medicine don’t do arithmetic,” he said. He wasn’t being critical of biologists and physicians: this was, after all, a man who read medical journals for fun. He meant that the traditions of medicine encouraged qualitative observation and interpretation. But what physicists do—out of sheer force of habit and training—is measure things and compare measurements, and do the math to put measurements in context. At that moment, while reading The New England Journal, Wood had the advantages of someone looking at a familiar fact with a fresh perspective.

That was also why Myhrvold had wanted to take his crew to St. Louis to meet with the surgeons. He likes to say that the only time a physicist and a brain surgeon meet is when the physicist is about to be cut open—and to his mind that made no sense. Surgeons had all kinds of problems that they didn’t realize had solutions, and physicists had all kinds of solutions to things that they didn’t realize were problems. At one point, Myhrvold asked the surgeons what, in a perfect world, would make their lives easier, and they said that they wanted an X-ray that went only skin deep. They wanted to know, before they made their first incision, what was just below the surface. When the Intellectual Ventures crew heard that, their response was amazement. “That’s your dream? A subcutaneous X-ray? We can do that.”

Let me close with my usual observation (specifically aimed at venture capitalists, research lab directors and university administrators) concerning an asymmetry in cognitive depth: yes, physicists can casually read the New England Journal of Medicine and come up with interesting insights, but, no, biologists and medical doctors cannot read Physical Review.

How the other half works

Valleywag lists its top ten tech workspaces here, including photos. Fancy cubicles, workout facilities and kitchens I expected, but stripper/fireman poles? :-) How does your office compare?

Here's my favorite shot, from Pierpoint Communications in downtown Austin.



Actually, Valleywag's top 10 are unimpressive compared to the some of the hedge fund offices I've seen in Manhattan :-)

Here's where I work, home of the largest Feynman diagram in the world:

Brainpower ain't free

This NYTimes article describes research on the fitness costs and benefits of increased intelligence (learning ability). The specific results are for fruit flies, C. Elegans (worms) and E. Coli (bacteria), but the theoretical basis is well understood already. Evolutionary equilibrium occurs at a local fitness maximum, which means that further increases in brainpower come with negative fitness costs in some other area (e.g., disease resistance, physical capability). If brainpower could continue to increase without negative side effects, it would have. The fact that it hasn't suggests that genes with beneficial effects on intelligence may also come with negative consequences.

Note that equilibrium is only an approximate condition -- there may be directions in gene space in which overall fitness can still increase (even substantially), but it takes time for the random mutational process of evolution to find them. In most directions one would expect to find either only a very small positive (or zero) fitness gradient or a negative gradient, assuming a population that has been genotypically stable for a long time. Recent studies suggest that humans may have experienced rapid evolution in the last 10-50 thousand years due to the advent of agriculture, population growth, etc.

At the end of the article, one of the biologists seems ready to rediscover the Cochran-Harpending hypothesis :-) See also here.

NYTimes: ... It takes just 15 generations under these conditions for the flies to become genetically programmed to learn better. At the beginning of the experiment, the flies take many hours to learn the difference between the normal and quinine-spiked jellies. The fast-learning strain of flies needs less than an hour.

But the flies pay a price for fast learning. Dr. Kawecki and his colleagues pitted smart fly larvae against a different strain of flies, mixing the insects and giving them a meager supply of yeast to see who would survive. The scientists then ran the same experiment, but with the ordinary relatives of the smart flies competing against the new strain. About half the smart flies survived; 80 percent of the ordinary flies did.

Reversing the experiment showed that being smart does not ensure survival. “We took some population of flies and kept them over 30 generations on really poor food so they adapted so they could develop better on it,” Dr. Kawecki said. “And then we asked what happened to the learning ability. It went down.”

The ability to learn does not just harm the flies in their youth, though. In a paper to be published in the journal Evolution, Dr. Kawecki and his colleagues report that their fast-learning flies live on average 15 percent shorter lives than flies that had not experienced selection on the quinine-spiked jelly. Flies that have undergone selection for long life were up to 40 percent worse at learning than ordinary flies.

... “Humans have gone to the extreme,” said Dr. Dukas, both in the ability of our species to learn and in the cost for that ability.

Humans’ oversize brains require 20 percent of all the calories burned at rest. A newborn’s brain is so big that it can create serious risks for mother and child at birth. Yet newborns know so little that they are entirely helpless. It takes many years for humans to learn enough to live on their own.

Dr. Kawecki says it is worth investigating whether humans also pay hidden costs for extreme learning. “We could speculate that some diseases are a byproduct of intelligence,” he said.

The benefits of learning must have been enormous for evolution to have overcome those costs, Dr. Kawecki argues. For many animals, learning mainly offers a benefit in finding food or a mate. But humans also live in complex societies where learning has benefits, as well.

“If you’re using your intelligence to outsmart your group, then there’s an arms race,” Dr. Kawecki said. “So there’s no absolute optimal level. You just have to be smarter than the others.”

Inflation, deconstructed

This NYTimes illustration of the various components of the CPI (inflation index) is one of the most impressive web graphics I've seen in a while. I suggest you click through and look at the original -- it allows you to zoom in and see the contribution from individual components (gasoline, computers, college tuition, eyeglasses, etc.) to the overall index. Blue regions represent deflation (reduction in prices); reddish regions are strong inflation (the big red blob is gasoline).

One interesting point is that the CPI uses "owner's equivalent rent" to calculate the housing part of the index. This missed the run up in house prices (rents were pretty flat over the last few years, meaning price to rent ratios were very high, a strong signal of a bubble). Had the cost of ownership, as opposed to renting, been factored in, inflation would have been significantly higher in recent years. Of course, most of that will go away now that the housing bubble has popped :-)

See previous discussion here.

Grand unification and quantum gravitational effects

New paper! http://arxiv.org/abs/0805.0145

Ever notice that the energy scale for grand unification (> 10^16 GeV, as dictated by proton decay) is uncomfortably close to the scale where (currently uncalculable) quantum gravity effects are strong (Planck scale = 10^19 GeV)? Does this make you nervous about whether one can trust low-energy predictions of grand unified models? Or, conversely, whether evolution of coupling constants from low-energies can really offer evidence for or against unification?

In this new paper, we add an additional wrinkle: the scale of quantum gravity might be lower than previously expected, due to renormalization effects. In models with large numbers of particle species (e.g., N=1000, common in many SO(10), E8xE8 and even SU(5) models), these effects can reduce the Planck scale from the conventional value (10^19 GeV, obtained by dimensional analysis) by almost an order of magnitude, thereby increasing the size of quantum gravity uncertainties.

Grand unification and enhanced quantum gravitational effects

Xavier Calmet, Stephen D.H. Hsu, David Reeb

In grand unified theories with large numbers of fields, renormalization effects significantly modify the scale at which quantum gravity becomes strong. This in turn can modify the boundary conditions for coupling constant unification, if higher dimensional operators induced by gravity are taken into consideration. We show that the generic size of these effects from gravity can be larger than the two-loop corrections typically considered in renormalization group analyses of unification. In some cases, gravitational effects of modest size can render unification impossible.

Figure caption: For $\eta$ fixed by the particle content of the theory, solid lines are contours of constant $c$ such that, under the presence of the gravitationally induced and enhanced operator (\ref{dim5}), SUSY-SU(5) perfectly unifies at two loops for given values of the initial strong coupling constant $\alpha_3(M_Z)$ and SUSY breaking scale $M_{{\rm SUSY}}$. Over the whole range, unification happens for some value of the coefficient $c$, with unification scale and unified coupling between $M_X=9.3\times 10^{14}\,{\rm GeV}$, $\alpha_G=0.033$ (lower right corner) and $M_X=5.5\times 10^{16}\,{\rm GeV}$, $\alpha_G=0.045$ (upper left).

Don't become a scientist! Philip Greenspun edition

I recently came across this essay by Philip Greenspun, which examines, in brutal detail, the negative aspects of a career in science. It goes way beyond my previous writings on the subject :-) Greenspun's essay was occasioned by the Larry Summers affair, and his main point regarding women in science is that science is such a crummy career choice that only testosterone-poisoned (overly competitive and status-driven) men would be stupid enough to pursue it. I'm not sure I agree completely with that perspective, but I like his essay quite a bit.

Some other themes he touches on: (a) sample bias; people are typically only familiar with the lives and careers of exceptionally successful scientists: In short, some young people think that science is a good career for the same reason that they think being a musician or actor is a good career: "I can't decide if I want to be a scientist like James Watson, a musician like Britney Spears, or an actor like Harrison Ford.", and (b) foreign immigration as a source of scientific talent: Science may be one of the lowest paid fields for high IQ people in the U.S., but it pays a lot better than most jobs in China or India.

Incidentally, I met Philip many years ago through a common friend who is a scientist at Harvard, at one of the many parties she hosted. (When I first wrote this post I thought she and Philip had been housemates, but she says that recollection is incorrect. I think I do remember his dog Alex, though.) At one of these parties I met Steve Pinker in the kitchen. After a long conversation about his research I remember thinking: gee, isn't that all kind of obvious? Don't you wish you understood Yang-Mills theory? I was still a kid, just like Albert Q. Mathnerd described below :-)

You might like to dismiss Greenspun's perspective on this subject, but keep in mind that the guy earned a math SB (at 18) and PhD in EECS at MIT and founded several software startups. So he's not entirely clueless about how the academic and real worlds work.

...Adjusted for IQ, quantitative skills, and working hours, jobs in science are the lowest paid in the United States.

This article explores this fourth possible explanation for the dearth of women in science: They found better jobs.

Why does anyone think science is a good job?

The average trajectory for a successful scientist is the following:

age 18-22: paying high tuition fees at an undergraduate college

age 22-30: graduate school, possibly with a bit of work, living on a stipend of $1800 per month

age 30-35: working as a post-doc for $30,000 to $35,000 per year

age 36-43: professor at a good, but not great, university for $65,000 per year

age 44: with young children at home (if lucky), fired by the university ("denied tenure" is the more polite term for the folks that universities discard), begins searching for a job in a market where employers primarily wish to hire folks in their early 30s

This is how things are likely to go for the smartest kid you sat next to in college. He got into Stanford for graduate school. He got a postdoc at MIT. His experiment worked out and he was therefore fortunate to land a job at University of California, Irvine. But at the end of the day, his research wasn't quite interesting or topical enough that the university wanted to commit to paying him a salary for the rest of his life. He is now 44 years old, with a family to feed, and looking for job with a "second rate has-been" label on his forehead.

...Consider someone taking the kind of high IQ and drive that would be required to obtain a tenure-track position at U.C. Berkeley and going into medicine. This person would very likely be a top specialist of some sort, earning at least $300,000 per year. Instead of being fired at age 44, our medical specialist would be near the height of her value to her patients and employer. Her experience and reputation would continue to add to her salary and prestige until she was perhaps 60 years old. [A woman who wanted to spend more time with her children can choose from a variety of medical careers, such as emergency medicine, that involve shift work and where a high salary can be earned with just two or three shifts per week. She could also work from home as a radiologist reading data transmitted via Internet.]

Consider taking the same high IQ and work ethic, going into business, and being put on the fast track at a company such as General Electric. Rather than being fired at age 44, this is about the time that she will be handed ever-larger divisions to operate, with ever-larger bonuses and stock options.

A top lawyer at age 44 is probably a $500,000 per year partner in a big firm, a judge, or a professor at a law school supplementing her $200,000 per year salary with some private work. ...

What about the excitement and fun of science?

Is life all about money and job security? What about excitement and fun? Isn't that a good reason to choose a job? Sure! I love every minute of my $8 per hour job as a helicopter instructor, but on the other hand I don't say that it is a great career and I can't understand why there aren't more women helicopter instructors.

Some scientists are like kids who never grow up. They love what they do, are excited by the possibilities of their research, and wear a big smile most days. Although these people are, by Boston standards, ridiculously poor and they will never be able to afford a house (within a one-hour drive of their job) or support a family, I don't feel sorry for them.

Unfortunately, this kind of child-like joy is not typical. The tenured Nobel Prize winners are pretty happy, but they are a small proportion of the total. The average scientist that I encounter expresses bitterness about (a) low pay, (b) not getting enough credit or references to his or her work, (c) not knowing where the next job is coming from, (d) not having enough money or job security to get married and/or have children. If these folks were experiencing day-to-day joy at their bench, I wouldn't expect them to hold onto so much bitterness and envy.

...The most serious concern is that the field that a youngster found fascinating at age 20 will no longer be fascinating after 20 or 25 years. If you have a narrow education and have been earning an academic salary, it is much tougher to change careers at age 45 or 50 than for someone who was in a job where the earnings are higher and begin at a younger age. A doctor who practices for 10 years can easily save enough to finance a switch to almost any other occupation. A successful lawyer can walk away after 15 or 20 years, commute to school from his oceanfront and town houses, and become a furniture maker (my friend's dad did this).

Why do American men (boys, actually) do it?

Pursuing science as a career seems so irrational that one wonders why any young American would do it. Yet we do find some young Americans starting out in the sciences and they are mostly men. When the Larry Summers story first broke, I wrote in my Weblog:

A lot more men than women choose to do seemingly irrational things such as become petty criminals, fly homebuilt helicopters, play video games, and keep tropical fish as pets (98 percent of the attendees at the American Cichlid Association convention that I last attended were male). Should we be surprised that it is mostly men who spend 10 years banging their heads against an equation-filled blackboard in hopes of landing a $35,000/year post-doc job?

Having been both a student and teacher at MIT, my personal explanation for men going into science is the following:

young men strive to achieve high status among their peer group

men tend to lack perspective and are unable to step back and ask the question "is this peer group worth impressing?"

Consider Albert Q. Mathnerd, a math undergrad at MIT ("Course 18" we call it). He works hard and beats his chest to demonstrate that he is the best math nerd at MIT. This is important to Albert because most of his friends are math majors and the rest of his friends are in wimpier departments, impressed that Albert has even taken on such demanding classes. Albert never reflects on the fact that the guy who was the best math undergrad at MIT 20 years ago is now an entry-level public school teacher in Nebraska, having failed to get tenure at a 2nd tier university. When Albert goes to graduate school to get his PhD, his choice will have the same logical foundation as John Hinckley's attempt to impress Jodie Foster by shooting Ronald Reagan. ...

Obama and race on campus

The melting pot still has a ways to go, even on elite university campuses. I suspect most liberal whites feel better about themselves thinking that they have a black friend, or voting for a black candidate. Apparently most black students don't feel they have even a single white friend.

It's still early, but at the moment I think I'm likely going to vote for Obama, partially because I think he's the smartest and most effective person in the race, but also because I can't take any more of Hillary's pandering. I actually respect McCain -- the biggest problem I have with him is probably his age.

WSJ: ..."It's much harder to be a white person and go to an all black party at Duke than vote for Obama, says Jessie Weingartner, a Duke junior. "On a personal level it is harder to break those barriers down."

Jazmyn Singleton, a black Duke senior agrees, After living in a predominantly white dorm freshman year, she lives with five African-American women in an all-black dormitory. "Both communities tend to be very judgmental," says Ms. Singleton, ruefully. "There is pressure to be black. The black community can be harsh. People will say there are 600 blacks on campus but only two-thirds are 'black' because you can't count blacks who hang out with white people."

The racial divisions among college students are striking both because of the fervor for Obama and the increasing diversity on campus. Colleges offer a unique opportunity for students to get to know each other in a relaxed atmosphere where many of the issues that often divide blacks and whites, like income and educational levels, are minimized amid the common goals of going to class, playing sports and going to parties.

About 10% of Duke students are African-American, compared to 4.5% two decades ago; they include many popular athletes as well as student leaders. The newly elected head of the graduate and professional student association is an African-American woman. Black and white students live together in the same group of dorms during freshman year, though they can join fraternities and sororities and select their roommates starting in sophomore year.

Like many colleges, Duke sponsors initiatives to address race relations on campus, an effort that gained added impetus following the widely publicized incident two years ago when white lacrosse players hired a black stripper to perform at a party and the woman then falsely accused several of the students of raping her.

...

But working or voting for an African-American running for president doesn't necessarily bridge differences -- on campus or, later, in the workplace. Following a recent discussion in one of his classes about the campaign, in which most students expressed support for Sen. Obama, Eduardo Bonilla-Silva, a Duke sociologist, asked his white students how many had a black friend on campus. All the white students raised their hands.

He then asked the black students how many of them had a white friend on campus. None of them raised their hands.

The more he probed, Mr. Bonilla-Silva says, the more he realized that the definition of friendship was different. The white students considered a black a "friend" if they played basketball with him or shared a class. "It was more of an acquaintance," recalls Mr. Bonilla-Silva.

Black students, by contrast, defined a friend as someone they would invite to their home for dinner. By that measure, none of the students had friends from the opposite race. Mr. Bonilla-Silva says when white college students were asked in series of 1998 surveys about the five people with whom they interacted most on a daily basis, about 68% said none of them were black. When asked if they had invited a black person to lunch or dinner recently, about 68% said "no." He says his own research and more recent studies show similar results.

Pop goes the housing bubble

Figure via Calculated Risk. Larger version here

I believe the outlines of the bust are becoming as visible as the bubble itself was to any astute observer a few years ago. But no bottom yet! If I had to guess, I'd say we are going to give back most of the integral over the curve from 1997 to 2007 or so, net of overall inflation during that period (say 20-30%). In other words, extend the blue trend line beyond the early nineties, and integrate your favorite curve minus this trend line from 1997-2007 to get the overvaluation. (Note the graph is of year over year price changes in nominal dollars, not absolute price.)

Or, just look at the figure below to see that prices might have to drop 30-40% to return to consistency with the long term trend.





As we've discussed before, house price increases tend to be quite modest if measured in real dollars. (Except, of course, for bubbles and special cases :-)

The Case-Shiller national index will probably be off close to 12% YoY (will be released in earlylate May). Currently (as of Q4) the national index is off 10.1% from the peak.

The composite 10 index (10 large cities) is off 13.6% YoY. (15.8% from peak)

The composite 20 index is off 12.7% YoY. (14.8% from peak)