Monday, April 07, 2008

The New Math

Alpha magazine has a long article on the current state of quant finance. It may be sample bias, but former theoretical physicists predominate among the fund managers profiled.

I've always thought theoretical physics was the best training for applying mathematical techniques to real world problems. Mathematicians seldom look at data, so are less likely to have the all-important intuition for developing simple models of messy systems, and for testing models empirically. Computer scientists generally don't study the broad variety of phenomena that physicists do, and although certain sub-specialties (e.g., machine learning) look at data, many do not. Some places where physics training can be somewhat weak (or at least uneven) include statistics, computation, optimization and information theory, but I've never known a theorist who couldn't pick those things up quickly.

Physicists have a long record of success in invading other disciplines (biology, computer science, economics, engineering, etc. -- I can easily find important contributions in those fields from people trained in physics, but seldom the converse). Part of the advantage might be pure horsepower -- the threshold for completing a PhD in theoretical physics is pretty high. However, a colleague once pointed out that the standard curriculum of theoretical physics is basically a collection of the most practically useful mathematical techniques developed by man -- the high points and greatest hits! Someone trained in that tradition can't help but have an advantage over others when asked to confront a new problem.

Having dabbled in fields like finance, computer science and even biology, I've come to consider myself as a kind of applied mathematician (someone who applies mathematical ideas to the real world) who happens to have had most of his training from working on physical systems. I suspect that physicists who have left the field, as well as practitioners of biophysics, econophysics, etc. might feel the same way.

Readers of this blog sometimes accuse me of a negative perspective towards physics. Quite the contrary. Although I might not be optimistic about career prospects within physics, or the current state of the field, I can't think of any education which gives a richer understanding of the world, or a greater chance of contributing to it.

...Finkelstein, who also grew up in Kharkov, has a Ph.D. in theoretical physics from New York University and a master’s degree in the same discipline from the Moscow Institute of Physics and Technology. Before joining Horton Point as chief science officer, he was head of quantitative credit research at Citadel Investment Group in Chicago.

Most of the 12 Ph.D.s at Horton Point’s Manhattan office are researching investment strategies and ways to apply scientific principles to finance. The firm runs what Finkelstein, 54, describes as a factory of strategies, with new models coming on line all the time. “It’s not like we plan to build ten strategies and sit on them,” he says. “The challenge is to keep it going, to keep this factory functioning.”

Along with his reservations about statistical arbitrage, Sogoloff is wary of quants who believe the real world is obliged to conform to a mathematical model. He acknowledges the difficulty of applying scientific disciplines like genetics or chaos theory — which purports to find patterns in seemingly random data — to finance. “Quantitative work will be much more rewarding to the scientist if one concentrates on those theories or areas that attempt to describe nonstable relationships,” he says.

Sogoloff sees promise in disciplines that deal with causal relationships rather than historical ones — like mathematical linguistics, which uses models to analyze the structure of language. “These sciences did not exist five or ten years ago,” he says. “They became possible because of humongous computational improvements.”

However, most quant shops aren’t exploring such fields because it means throwing considerable resources at uncertain results, Sogoloff says. Horton Point has found a solution by assembling a global network of academics whose research could be useful to the firm. So far the group includes specialists in everything from psychology to data mining, at such schools as the Beijing Institute of Technology, the California Institute of Technology and Technion, the Israel Institute of Technology.

Sogoloff tells the academics that the goal is to create the Bell Labs of finance. To align both parties’ interests, Horton Point offers them a share of the profits should their work lead to an investment strategy. Scientists like collaborating with Horton Point because it combines intellectual freedom with the opportunity to test their theories using real data, Sogoloff says. “You have experiments that can be set up in a matter of seconds because it’s a live market, and you have the potential for an amazing economic benefit.” ...

15 comments:

  1. Anonymous4:01 AM

    I've been trying to poke around finance models, and the contribution from physics is quite impressive. But I have a question: do physicists model strategic behavior, e.g. what are going to be the effects of the presence of competitors on our models? Or, what are the effects of mortgage models on the borrowers' behavior? These are the kinds of questions that economists are supposed to know about - too bad few were paying attention.

    ReplyDelete
  2. TC: a lot of physics models in finance assume (implicitly) efficient markets (e.g., in the form of stochastic fluctuations). The Alpha article talks about agent-based simulations, but that isn't really coming from physics per se.

    On the other hand, a guy I know (former collaborator) who worked on prepayment models for mortgages did include feedback (e.g., from general economic conditions) in predicting how individuals would behave.

    ReplyDelete
  3. Anonymous9:06 PM

    The article mentions a paper by Professor Lo which analyzed the August 2007 quant fund meltdown. In the paper, Professor Lo largely replicated the performance of quant hedge funds with a trivial model that bought yesterday's losers and shorted yesterday's winners.

    Based on this I wonder, if having the best minds work on these problems is really a waste. Can their theories be reduced to simple rules? My long running feeling is that the best quant funds excel at engineering disciplines such as real time software systems, database management, control theory, and operations research more than anything else.

    My comments are mainly with regard to the equity funds mentioned in the article.

    ReplyDelete
  4. Anonymous9:59 PM

    Physicists, or at least physics, contributes the models. I re-read a book on LTCM last week, and Merton is quoted as saying that he and other economists went over to the library and raided the physics section. His contribution to the Black-Scholes equation was a derivation from the theory of diffusion.

    One view of the present mess is that the models were based on historical data, but the nortgage market was contaminated by huge amounts of fraud by all parties to the transactions. Models that try to identify small inefficiencies on interest rates blow up in that situation. Arguably the intellectual prestige of physics and its past successes made people too self-confident.

    ReplyDelete
  5. Scott:

    You are right that many of the individual hedge fund strategies are quite simple at the end of the day, but it's quite tricky to back test a possible strategy and determine whether the statistical evidence for its succeess is adequate. (Or to determine when it may have stopped working.) You need someone who is super smart or insightful making those decisions.

    You are also right that a lot of the work in these quant funds goes into "plumbing" like software or even fast hardware to drive down latency. But that doesn't mean you don't benefit from having the smartest guys working on it. Coming up with a creative improvement, even in plumbing, can add a lot of value. *IF* I can get a really smart physicist focused on improving the plumbing (e.g., through gigantic remuneration), then I'd rather have that person than a less clever person who happens to have degrees in plumbing disciplines. (No offense to engineers; my dad was one :-) I saw this myself in my previous startup, which had about 10 physics PhDs working on software and encryption technology.

    Whether having all these smart people focused on gaming the markets is good for society: see the Charlie Munger post from a week or so ago.

    ReplyDelete
  6. Anonymous9:58 PM

    "amazing economic benefit."

    The amazing economic benefit being making fairly obscene shitloads of money for John Company as well as yourself.

    It leaves a bad taste in my mouth.

    ReplyDelete
  7. Anonymous2:32 AM

    Yes, that is the dictionary definition of "amazing economic benefit."

    I hear it tastes like pressed duck.

    ReplyDelete
  8. Anonymous1:29 AM

    I'm curious what you think of experimental physics as a discipline... you seem to be rather carefully distinguishing theoretical physics from just physics here.

    ReplyDelete
  9. I think many experimentalists, if forced to leave physics, would do just fine on Wall St. Fewer of them leave physics because the job situation isn't quite as bad for them as for theorists -- for example, there are more jobs for experimenters in industry than for theorists.

    The two populations are somewhat different, as the two activities require different skill sets. Theorists have more math chops and would probably pick up finance theory faster, on average. A commenter on my old post comparing GRE scores between different fields noted that if physics in general came out #1 by a small margin over CS and math, that there would be a big gap between theorists' scores and those in other disciplines. I think that is true.

    On the other hand it is possible that experimenters are more likely to have "normal" personality types, are less likely to be lost in the clouds all the time (i.e. autistic), and may be more practically effective. That would probably help them adjust to the realities of the corporate world and to have more psychological insight into markets.

    ReplyDelete
  10. Anonymous9:50 PM

    The ruling quant model of economics in the late 19th Cent was "equilibrium". Did they get that from physics or theories of chemical equilibrium? Chemistry envy? :-)

    Did thought experiments on gambling have any influence on physics? That's the only place I can think of that economic models influenced the hard sciences.

    ReplyDelete
  11. Anonymous1:29 PM

    Great post! I'm an undergrad interning at a hedge fund this summer...majoring in math/CS, but I often wish I did more physics (especially after reading your post!).

    Question: do you have any suggestions for finance resources to look at for someone from a theoretical background? I've been lurking on your blog, but I don't understand much =\. Or suggestions for physics stuff to look at that's particularly relevant to hedge funds?

    Thanks!

    ReplyDelete
  12. Hi anonymous,

    That's a complicated request. Here are some comments and suggestions. A lot of these books have been discussed already on this blog.

    1) complex/dynamical systems: see a review article in the field of econophysics (perhaps by H.E. Stanley), or the popular books by Didier Sornette or Per Bak, or even Gleick's book Chaos.

    2) path integrals and option pricing: I find this the best way to think about option pricing, but that might just be due to my background training. See the book by H. Kleinert for an overview (he has a chapter on finance; most of the book is physics applications)

    Standard textbooks: Hull, Merton, etc. See also the review articles "finance for beginners" by John Norstad referenced on this blog.

    3) market psychology, behavioral stuff, history:

    Inventing Money by Nick Dunbar (story of LTCM, history of derivatives)

    When Genius Failed by R. Lowenstein (LTCM) *

    Nassim Taleb's books

    Against the Gods: the story of risk by Peter Bernstein *

    Liar's Poker (life as a bond trader; Wall St. culture; a bit dated) by Michael Lewis *

    Derman's book, as well as How I Became a Quant (collection of interviews)

    A Camera, Not an Engine (history of academic finance, derivatives) by D. MacKenzie

    Hedgehogging (not great, but insights into the hedge industry) by Barton Biggs

    * = warning, author has incomplete technical understanding

    I'm sure I am leaving out a lot of good stuff, but perhaps others can help me out :-)

    ReplyDelete
  13. Anonymous4:51 PM

    If I could advise the anonymous undergraduate, I would say, "Solve every problem that you can with math. Model everything as best you can with math. Figure out how good it is. Figure out how to figure out how good it is. Be relentless and eventually, you'll understand what the math is telling you. Then take a shot at the market. And don't forget Kelly."

    ReplyDelete
  14. Anonymous6:20 PM

    Steve and David, thank you both! But who is this "Kelly" that David mentions...?

    (I want to say it's David himself (as in hi s last name is Kelly), meaning don't forget him and his advice when I strike gold ;), but I might just be totally confused.)

    ReplyDelete
  15. kelly = kelly criteria. How much of your total capital to risk given the advantage you have -- a bettor's rule of thumb. See the book Fortune's Formula:

    http://infoproc.blogspot.com/2005/11/fortunes-formula.html

    Discovered by Bell Labs physicist John Kelly.

    Good luck!

    ReplyDelete