Friday, October 24, 2008

Intellectual honesty: how much do we know?

As I have emphasized many times on this blog, when it comes to complex systems like society or economy (and perhaps even climate), experts have demonstrably little predictive power. In rigorous studies, expert performance is often no better than random. This is not surprising to anyone who has studied nonlinear dynamics and is free from ideological or professional bias (see below).

What is worse, experts are usually wildly overconfident about their capabilities. I'm often quite critical here of work done in fields like economics and social science, not because the results are unimpressive -- I acknowledge that the problems are hard and the systems under study intractable -- but because the researchers themselves often have beliefs whose strength is entirely unsupported by available data. They are usually unaware that biases (strong priors) shape their thinking, not scientific hypothesis testing. These biases come in many forms -- market fundamentalism, extreme belief in nature over nurture or vice versa, unrealistic faith in the powers of government, unrealistic faith in simple models, etc. (More from Robin Hanson and company here.)

It's refreshing to hear experts openly proclaim their lack of knowledge. Below are some comments from economist Arnold Kling, which would have gotten a physicist like me into hot water had I made them:
Economists pretending to have knowledge: ...I am shocked at the behavior of my fellow economists during this crisis. They are claiming to know much more than they do about causes and solutions. Rather than trying to understand and explain what is going on, they are engaged in a fierce battle over narrative.

...My main beef with economists is that standard macroeconomics does such a poor job of describing what is going on. The textbooks models are pretty much useless. Where in the textbooks is "liquidity preference" a demand for Treasury securities? Where in the textbooks does it say that injecting capital into banks is a policy tool?

Graduate macro is even worse. Have the courses that use representative-agent models solving Euler equations been abolished? Have the professors teaching those courses been fired? Why not?

I have always thought that the issue of the relationship between financial markets and the "real economy" was really deep. I thought that it was a critical part of macroeconomic theory that was poorly developed. But the economics profession for the past thirty years instead focused on producing stochastic calculus porn to satisfy young men's urge for mathematical masturbation.
More from Paul Romer of Stanford:
In the current crisis, the astonishing and unexpected consequences of the Lehman Brothers bankruptcy should serve as a similarly decisive data point. On the Thursday and Friday before Lehman filed for protection, I was at a conference on the financial crisis. Everyone there expected them to file on Monday. We repeated for each other all the fundamentalist arguments: "Everyone had been given time to prepare." "The courts handle bankruptcies all the time." None of us expected that putting Lehman through a court managed bankruptcy would be much different from arranging a forced sale of Bear Stearns.

We were all wrong. Within days, AIG was insolvent. Runs were developing on Goldman Sachs, Morgan Stanley, and the entire money market fund industry. Banks had stopped lending to each other in the Fed Fund market. Rates on Treasuries approached zero.
Here is one of the best academic papers I have seen on the credit crisis and the growth of the shadow banking system ("New Financial Architecture"). But note that the author James Crotty of U Mass Amherst is not considered a mainstream researcher in the field -- he's "heterodox"! It is amusing that in economics Crotty is considered a quasi-Marxist, whereas his views correspond quite strongly with those of actual financiers (at least when privately expressed).
This PERI Working Paper argues that the ultimate cause of the current global financial crisis is to be found in the deeply flawed institutions and practices of what is often referred to as the New Financial Architecture (NFA) – a globally integrated system of giant bank conglomerates and the so-called ‘shadow banking system’ of investment banks, hedge funds and bank-created Special Investment Vehicles. These institutions are either lightly and badly regulated or not regulated at all, an arrangement defended by and celebrated in the dominant financial economics theoretical paradigm – the theory of efficient capital markets. The NFA has generated a series of ever-bigger financial crises that have been met by larger and larger government bailouts.

After a brief review of the historical evolution of the NFA, this paper analyses its structural flaws: 1) the theoretical foundation of the NFA – the theory of efficient capital markets – is very weak and the celebratory narrative of the NFA accepted by regulators is seriously misleading; 2) widespread perverse incentives embedded in the NFA generated excessive risk-taking throughout financial markets; 3) mortgage-backed securities central to the boom were so complex and nontransparent that they could not possibly be priced correctly; their prices were bound to collapse once the excessive optimism of the boom faded; 4) contrary to the narrative, excessive risk built up in giant banks during the boom; and 5) the NFA generated high leverage and high systemic risk, with channels of contagion that transmitted problems in the US subprime mortgage market around the world.

Finally, as any trader knows, talk is cheap. Your opinion counts nothing unless you have skin in the game. By that accounting, it's George Soros and John Paulson who emerge as the real experts here -- they are each up billions of dollars after making smart bets on the crisis :-)

Obligatory disclaimer concerning string theory and speculative theoretical physics:
There is an old saying in finance: in the short run, the market is a voting machine, but in the long run it's a weighing machine. ...

You might think science is a weighing machine, with experiments determining which theories survive and which ones perish. Healthy sciences certainly are weighing machines, and the imminence of weighing forces honesty in the voting. However, in particle physics the timescale over which voting is superseded by weighing has become decades -- the length of a person's entire scientific career. We will very likely (barring something amazing at the LHC, like the discovery of mini-black holes) have the first generation of string theorists retiring soon with absolutely no experimental tests of their *lifetime* of work. Nevertheless, some have been lavishly rewarded by the academic market for their contributions.


Anonymous said...

On the question of prediction, can economists ever hope to be effective when economic systems are made of intelligent actors with their own economic notions, who are acting on their own expectations of the future? This makes for non-linearity of course, but of a peculiarly reflexive sort; aren't we moving into Turing halting problem territory, or rather, haven't we always been there?

Economies can be viewed as rule-governed, even with emergent rule patterns, but ultimately the actors (who, as you've reminded us, Steve, are essentially cunning apes) can choose to stop following the rules, can follow their own idiosyncratic interpretations of the rules, or can be overcome by fear and panic.

The best economists leaven their technical expertise with a large dose of common sense and insight into the human comedy (or tragedy, as the case may be). When economics becomes totalitarian in its scope, it comes unhinged. It can't explain human behavior so much as explicate the problems that humans set for themselves as they struggle to stay fed and housed and sustain some form of civilization on an ever-growing scale.

So much depends on the capacity of individuals to say "wait, stop, this is nuts" in face of immediate social and microeconomic pressures to go with the crowd.

Anonymous said...

I think it is worthwhile to segregate economists into those who are paid to attempt to understand the economy, and those who are paid to make money from the economy. Of course there is significant overlap, but the primary incentives are much different and, I would argue, matter. In some sense, this is analogous to distinguishing scientists and engineers.

In a world where one is judged by how much money one makes, it is easy to slip into believing that profitable trades validates the models being used to make those trades. Of course this is not necessarily the case. Recall that early steam engines were build based upon incorrect theories of heat.

In the physics community there is a big debate about the extent to which formulating theories which are (effectively) unable to be tested is consistent with the principles of scientific thought. Is such a debate present in the econ community?

Raymond Luxury Yacht said...

The most realistic school of economics I know is the Austrian school. Unfortunately, it too is considered "heterodox". But for those of us who aren't concerned with future prospects in the education field, it offers several benefits that other schools don't: A comprehensive theory of capital STRUCTURE that doesn't gloss over TIME, and doesn't treat people like aggregates, plus a theory of HOW entrepreneurship and competition work in a non-"equilibrium" state.

However, as a theory that is deducted from the axiom that humans act, it can't offer any quantitative predictions. This is only realistic. Austrian economics has always been candid about the fact that economics is a social science, and not a natural science. Hayek, the only Austrian economist to get the nobel prize understood this and showed that planned economies are impossible, because you can't "solve" the economic allocation problem like you can solve an equation.

The subjects under study in the natural and in the social sciences are fundamentally different. Humans aren't like planets. Humans act according to their own will. Planets don't act. They move according to the laws of physics, but they certainly don't act.
To use the methods of the natural sciences on humans and expect to formulate a quantitative law that governs human action, is absurd, for there is no such law.

Anonymous said...

Agree with the theme of your essay but wonder about your first sentence: Why do you say "perhaps even climate" and set this off in parentheses? Isn't climate as complex as economies and perhaps more so? And isn't our understanding and knowledge of climate as primitive as economists' understanding and knowledge of economies? Almost certainly this applies when it comes to predictions? I have an hypothesis about this choice of wording that I look forward to disproving.

Steve Hsu said...

Wow, very thoughtful comments. I was expecting to get beat up, so I am thrilled!

I'm actually a Hayekian in many ways. I particularly agree with the emphasis on information and information processing in the economy. ;-)

I separated climate because it doesn't have the reflexive character, mentioned already in the first comment, that economy and society do. I'm certainly more pessimistic than climate modelers that we understand what is happening, and only time will tell whether their predictive abilities are any better than those of other experts who predict the behavior of complex systems.

Anonymous said...

Economics science *as taught* in the United States is politically correct (meaning artificially deprecated, including problematic areas repeatedly identified by this blog.) Ur not imagining things. Neither is Kling. But look to actual theory and it's all there.

Luke Lea said...

At least Greenspan has had the decency to admit that he was wrong. For that we can be thankful. His opinion carries a lot of weight among market fundamentalists. Or in the words of Madison, if men were angels then government would be unnecessary.

Raymond Luxury Yacht said...

Sorry to keep returning to this, but I have to answer to Born Again Democrat:

As a market fundamentalist Alan Greenspan's opinions carry absolutely no weight with me. I'm not sure what other people you are grouping me with, but the free market enthusiasts I know all wish to abolish the Federal Reserve and the fiat money system.

Anonymous said...

I'll speak up for the econ tribe. First, as someone who has slogged his way through the courses Kling describes, it is true that the cutting edge of academic macro (which, by the way, is completely different from Keynesianism, IS-LM, and the stuff in the undergrad textbooks) has nothing to say about the crisis. Sometime around the 70s and 80s, academic macro was reinvented: instead of trying to come up with ad-hoc, hand-waving verbal arguments about national GDP, investment, savings, etc as the "old" economists did, they would build up the entire economy out of the behavior of individuals' decisions (i.e. the "microfoundations" of macro). Now, this isn't a bad idea, in fact this is the same thing that a lot of other schools also claim they want to do: as zlatko says, the Austrians claim to prove everything from logical principles (I don't think this is that different from mainstream econ, and runs into much of the same problems). However, it turns out that models with a single ("representative") agent are the only mathematically tractable ones, and through the quirks of the academic incentive system, the representative agent model has pretty much been the only game in town. Now there are some things this model can capture (e.g. national output) but some things that cannot be: in particular, credit does not exist in a system with only one agent. Which, as Stiglitz has pointed out, has resulted in the rather embarrassing situation where the cream of academic macro is pretty much irrelevant to the most interesting macroeconomic event of the past 50 years. Hopefully this also is an opportunity for some new economists to fix this situation...

Finally, as any trader knows, talk is cheap. Your opinion counts nothing unless you have skin in the game. By that accounting, it's George Soros and John Paulson who emerge as the real experts here -- they are each up billions of dollars after making smart bets on the crisis :-)
I don't agree with this at all. For a while, nearly every trader and hedge fund operator was raking in the dough; now many of them are losing money. Was there any difference between their opinions before and after, or do their opinions differ much from Soros or Paulson? Should we ask the pot-smoker Andrew Lahde about how to reform the banking system just because he was up 800%? I've said before that while academic economists were foolishly negligent of what was happening right under their noses, they weren't the ones who ran the Wall Street machine. Economists weren't the ones crowding into MBA and MFE programs. And while much fun has been poked at principles like equilibrium and "no-arbitrage", that didn't stop people from coding it up in pricing formulas and going out and selling securities based on it. Whether they believed in it or not, it was people with "skin in the game" who put it to work in the real world.

Steve Hsu said...

TC: thanks for the insightful comments. I was at least partially kidding when I wrote the following (hence the smiley face):

> Your opinion counts nothing
> unless you have skin in the game. ...

I think models with agents are fine, but it seems obvious that if the agents have finite information and information processing capabilities, and different priors (internal models that they carry around), the macro outcomes will depend sensitively on the distribution of agents, their internal states (which introduces path / history dependence), etc. Eventually you are back to dealing with sociological and psychological issues -- agents which are sufficiently realistic reintroduce all of the difficulties that one had in the first place!

Seth said...


Are you familiar with Christopher Sims' work on "Rational Inattention"?

Baby steps, but in a constructive direction.

Sabine Hossenfelder said...

You might also find this quote by Jeffrey Sachs interesting.

Blog Archive