Stanley: ...
When they analyzed these data–200 million of them–in exactly the same fashion that Bachelier had analyzed data almost a century earlier, they made a startling discovery. The pdf of price changes was not Gaussian plus outliers, as previously believed. Rather, all the data–including data previously termed outliers–conformed to a single pdf encompassing both everyday fluctuations and “once in a century” fluctuations. Instead of a Gaussian or some correction to a Gaussian, they found a power law pdf with exponent 4, a sufficiently large exponent that the difference from a Gaussian is not huge; however, the probability of a “once in a century” event of, say, 100 standard deviations is exp(10,000) for the Gaussian, but simply 108 for an inverse quartic law. If one analyzes a data set containing 200 million data in two years, this means there are only two such events–in two years!
Now which is better, the concept of “everyday fluctuations” which can be modeled with a drunkard’s walk, complemented by a few “once in a century” outliers? Or a single empirical law with no outliers but for which a complete theory does not exist despite promising progress by Xavier Gabaix of NYU’s Stern School of Management and his collaborators? Here we come to one of the most salient differences between traditional economics and the econophysicists: economists are hesitant to put much stock in laws that have no coherent and complete theory supporting them, while physicists cannot afford this reluctance. There are so many phenomena we do not understand. Indeed, many physics “laws” have proved useful long before any theoretical underpinning was developed . . . Newton’s laws and Coulomb’s law to name but two.
And all of us are loathe to accept even a welldocumented empirical law that seems to go against our own everyday experience. For stock price fluctuations, we all experience calm periods of everyday fluctuations, punctuated by highly volatile periods that seem to cluster. So we would expect the pdf of stock price fluctuations to be bimodal, with a broad maximum centered around, say, 13 standard deviations and then a narrow peak centered around, say, 50 standard deviations. And it is easy to show that if we do not have access to “all the data” but instead sample only a small fraction of the 200 million data recently analyzed, then this everyday experience is perfectly correct, since the rare events are indeed rare and we barely recall those that are “large but not that large”.
The same is true for earthquakes: our everyday experience teaches us that small quakes are going on all the time but are barely noticeable except by those who work at seismic detection stations. And every so often occurs a “once in a century” truly horrific event, such as the famous San Francisco earthquake. Yet when seismic stations analyze all the data, they find not the bimodal distribution of everyday experience but rather a power law, the GutenbergRichter law, describing the number of earthquakes of a given magnitude.
Pessimism of the Intellect, Optimism of the Will Favorite posts  Manifold podcast  Twitter: @hsu_steve
Tuesday, February 03, 2009
Power laws, baby
Thanks to reader GS for the link to this excellent article by statistical physicist Eugene Stanley. There's much more to the article, but I decided to excerpt this discussion of security price fluctuations and the observation that they are far from log normal.
Subscribe to:
Post Comments (Atom)
Blog Archive

▼
2009
(204)

▼
02
(21)
 Deprogramming the cult of the Efficient Market
 Trento talk
 Innovation competitiveness
 Macro graphs
 David X. Li
 Demand destruction
 Netbooks and tipping points
 Heritability of brain structure
 Race and IQ in Nature
 Bhide, innovation and basic research
 Designer babies
 The World's Greatest Economic Minds
 Robot Genius, PC Armor
 Ant algorithms
 Liam Casey on BBC
 On $500k a year
 US car sales fall below China’s for first time
 Comment moderation
 How they actually MAKE the money
 Power laws, baby
 Markets overshoot on the downside, too

▼
02
(21)
Labels
 physics (377)
 genetics (302)
 globalization (281)
 finance (268)
 brainpower (265)
 genomics (251)
 technology (237)
 american society (230)
 China (210)
 innovation (193)
 economics (183)
 ai (181)
 psychometrics (172)
 science (166)
 photos (162)
 psychology (158)
 machine learning (145)
 biology (142)
 travel (142)
 genetic engineering (130)
 universities (129)
 higher education (124)
 human capital (119)
 credit crisis (115)
 startups (113)
 iq (106)
 cognitive science (99)
 podcasts (98)
 autobiographical (88)
 political correctness (85)
 politics (85)
 careers (84)
 geopolitics (82)
 statistics (80)
 credit crunch (78)
 elitism (76)
 evolution (75)
 bounded rationality (74)
 quantum mechanics (74)
 gilded age (73)
 talks (72)
 income inequality (71)
 social science (71)
 genius (70)
 history of science (66)
 caltech (64)
 realpolitik (63)
 books (62)
 MSU (59)
 mma (57)
 sci fi (57)
 harvard (54)
 biotech (53)
 silicon valley (53)
 academia (51)
 mathematics (51)
 kids (50)
 education (49)
 bgi (48)
 history (48)
 intellectual history (48)
 cdo (45)
 derivatives (43)
 neuroscience (43)
 behavioral economics (41)
 jiujitsu (41)
 literature (41)
 physical training (39)
 video (38)
 computing (37)
 ufc (37)
 bjj (36)
 bubbles (36)
 film (36)
 mortgages (36)
 google (35)
 expert prediction (34)
 many worlds (34)
 affirmative action (33)
 hedge funds (33)
 economic history (32)
 nuclear weapons (31)
 race relations (31)
 security (31)
 black holes (30)
 quants (30)
 von Neumann (30)
 efficient markets (29)
 feynman (29)
 foo camp (29)
 movies (29)
 sports (29)
 music (28)
 singularity (26)
 entrepreneurs (25)
 housing (25)
 obama (25)
 subprime (25)
 berkeley (24)
 taiwan (24)
 conferences (23)
 epidemics (23)
 venture capital (23)
 athletics (22)
 meritocracy (22)
 quantum field theory (22)
 ultimate fighting (22)
 wall street (22)
 cds (20)
 internet (20)
 blogging (19)
 scifoo (19)
 gender (18)
 goldman sachs (18)
 new yorker (18)
 cryptography (17)
 dna (17)
 freeman dyson (17)
 smpy (17)
 treasury bailout (17)
 university of oregon (17)
 algorithms (16)
 japan (16)
 personality (16)
 privacy (16)
 autism (15)
 christmas (15)
 cosmology (15)
 happiness (15)
 height (15)
 oppenheimer (15)
 Fermi problems (14)
 fitness (14)
 les grandes ecoles (14)
 social networks (14)
 wwii (14)
 chess (13)
 government (13)
 hedonic treadmill (13)
 india (13)
 neanderthals (13)
 probability (13)
 russia (13)
 war (13)
 aspergers (12)
 blade runner (12)
 malcolm gladwell (12)
 net worth (12)
 nobel prize (12)
 nsa (12)
 philosophy of mind (12)
 research (12)
 Einstein (11)
 entropy (11)
 geeks (11)
 harvard society of fellows (11)
 string theory (11)
 television (11)
 Go (10)
 ability (10)
 art (10)
 climate change (10)
 cold war (10)
 football (10)
 italy (10)
 mutants (10)
 nerds (10)
 olympics (10)
 pseudoscience (10)
 complexity (9)
 crossfit (9)
 democracy (9)
 encryption (9)
 energy (9)
 eugene (9)
 flynn effect (9)
 france (9)
 james salter (9)
 pop culture (9)
 turing test (9)
 alan turing (8)
 alpha (8)
 data mining (8)
 dating (8)
 determinism (8)
 games (8)
 keynes (8)
 manhattan (8)
 pca (8)
 philip k. dick (8)
 qcd (8)
 quantum computers (8)
 real estate (8)
 robot genius (8)
 success (8)
 usain bolt (8)
 Iran (7)
 aig (7)
 ashkenazim (7)
 basketball (7)
 environmentalism (7)
 free will (7)
 fx (7)
 game theory (7)
 hugh everett (7)
 new york times (7)
 paris (7)
 patents (7)
 poker (7)
 simulation (7)
 tail risk (7)
 teaching (7)
 volatility (7)
 anthropic principle (6)
 bayes (6)
 class (6)
 drones (6)
 godel (6)
 intellectual property (6)
 markets (6)
 nassim taleb (6)
 noam chomsky (6)
 prostitution (6)
 rationality (6)
 academia sinica (5)
 bobby fischer (5)
 econtalk (5)
 fake alpha (5)
 global warming (5)
 information theory (5)
 iraq war (5)
 kasparov (5)
 luck (5)
 nonlinearity (5)
 perimeter institute (5)
 renaissance technologies (5)
 sad but true (5)
 software development (5)
 vietnam war (5)
 warren buffet (5)
 100m (4)
 Poincare (4)
 bill gates (4)
 borges (4)
 cambridge uk (4)
 censorship (4)
 charles darwin (4)
 creativity (4)
 demographics (4)
 hormones (4)
 humor (4)
 inequality (4)
 judo (4)
 kerviel (4)
 microsoft (4)
 mixed martial arts (4)
 monsters (4)
 moore's law (4)
 solar energy (4)
 soros (4)
 trento (4)
 200m (3)
 babies (3)
 brain drain (3)
 charlie munger (3)
 cheng ting hsu (3)
 chet baker (3)
 correlation (3)
 ecosystems (3)
 equity risk premium (3)
 facebook (3)
 fannie (3)
 feminism (3)
 fst (3)
 intellectual ventures (3)
 jim simons (3)
 language (3)
 lee kwan yew (3)
 lewontin fallacy (3)
 lhc (3)
 magic (3)
 michael lewis (3)
 nathan myhrvold (3)
 neal stephenson (3)
 olympiads (3)
 path integrals (3)
 risk preference (3)
 search (3)
 sec (3)
 sivs (3)
 society generale (3)
 supercomputers (3)
 systemic risk (3)
 thailand (3)
 alibaba (2)
 assortative mating (2)
 bear stearns (2)
 bruce springsteen (2)
 charles babbage (2)
 cloning (2)
 computers (2)
 david mamet (2)
 digital books (2)
 donald mackenzie (2)
 drugs (2)
 eliot spitzer (2)
 empire (2)
 exchange rates (2)
 frauds (2)
 freddie (2)
 gaussian copula (2)
 heinlein (2)
 industrial revolution (2)
 james watson (2)
 ltcm (2)
 mating (2)
 mba (2)
 mccain (2)
 mit (2)
 monkeys (2)
 national character (2)
 nicholas metropolis (2)
 no holds barred (2)
 offices (2)
 oligarchs (2)
 palin (2)
 population structure (2)
 prisoner's dilemma (2)
 skidelsky (2)
 socgen (2)
 sprints (2)
 twitter (2)
 ussr (2)
 variance (2)
 virtual reality (2)
 abx (1)
 anathem (1)
 andrew lo (1)
 antikythera mechanism (1)
 athens (1)
 atlas shrugged (1)
 ayn rand (1)
 bay area (1)
 beats (1)
 book search (1)
 bunnie huang (1)
 car dealers (1)
 carlos slim (1)
 catastrophe bonds (1)
 cdos (1)
 ces 2008 (1)
 chance (1)
 children (1)
 cochranharpending (1)
 cpi (1)
 david x. li (1)
 dick cavett (1)
 dolomites (1)
 dune (1)
 eharmony (1)
 escorts (1)
 faces (1)
 fads (1)
 favorite posts (1)
 fiber optic cable (1)
 francis crick (1)
 gary brecher (1)
 gizmos (1)
 greece (1)
 greenspan (1)
 hypocrisy (1)
 igon value (1)
 iit (1)
 inflation (1)
 information asymmetry (1)
 iphone (1)
 jack kerouac (1)
 jaynes (1)
 jazz (1)
 jfk (1)
 john dolan (1)
 john kerry (1)
 john paulson (1)
 john searle (1)
 john tierney (1)
 jonathan littell (1)
 las vegas (1)
 lawyers (1)
 lehman auction (1)
 les bienveillantes (1)
 lowell wood (1)
 lse (1)
 machine (1)
 mcgeorge bundy (1)
 mexico (1)
 michael jackson (1)
 mickey rourke (1)
 migration (1)
 money:tech (1)
 myron scholes (1)
 netwon institute (1)
 networks (1)
 newton institute (1)
 nfl (1)
 oliver stone (1)
 phil gramm (1)
 philanthropy (1)
 philip greenspun (1)
 portfolio theory (1)
 power laws (1)
 pyschology (1)
 randomness (1)
 recession (1)
 sales (1)
 singapore (1)
 skype (1)
 standard deviation (1)
 star wars (1)
 starship troopers (1)
 students today (1)
 teleportation (1)
 tierney lab blog (1)
 tomonaga (1)
 tyler cowen (1)
 venice (1)
 violence (1)
 virtual meetings (1)
 war nerd (1)
 wealth effect (1)
11 comments:
"So we would expect the pdf of stock price fluctuations to be bimodal"
This is incorrect. What "we would expect" is one distribution with ever changing parameters.
Distributions over short periods "we would expect" to be inconsistent with distributions over long periods.
This guy doesn't know what he's taking about.
I remember reading a good paper explaining the frequent occurence of power laws in many context. can't find it thoug..
I once tried to detect outliers to clean historic data, and the data that don't fit gaussian returns are overwhelming.
Even filtering for 0.0001 quartile turns you over lots of valid data points.
There are many assumption that are completely wrong in finance, not only statistically, but also economically.
The complete lack of use uncertainty for estimates is from my perspective a bigger problem.
Complex models needs to be calibrated (over stationary distribution..)
I really don't want to "insult" any of the experts, but why is that people care so much about the probability distributions of stock prices?
As every physicist knows, Einstein found it hard to grasp the randomness in Quantum Mechanics. If I remember it right, Einstein considered a probabilistic approach an exercise of "defeatism". In finance, people seem to be quite defeatist. A great example is the fantasy of gaussian fluctuations. Then along came Mandelbrot, Taleb and their homies with new fantasies such as power laws.
IMHO, going for random walks, power laws and the like is a simplistic approach. There are buying and selling orders being issued at various prices, all the time. A "good model" of stock price fluctuations should account for the structure of the process that leads to the formation of the prices. I rarely see academics working on highquality orderbook data, so I am not too surprised that finance has been (and shall always be) a cemetery of theories. In Physics, no one would be able to go for so long with such crude experimental data, but in Finance it seems that anything goes...
"I rarely see academics working on highquality orderbook data"
This would apply only to "highfrequency finance".
As someone looking at it from the econ side, I kind of agree that fancier distributions won't tell us about the mechanisms of these tail events  and the econophysicists themselves are part of the financial industry, therefore part of the system as a whole. All those mortgage borrowers taking on more debt than they could afford were a tail event  but they, in turn, were affected by the availability of cheap credit, made possible by Wall Street. I'd like to ask an econophysicist one day  can they model the effect that a newly developed formula for pricing mortgages has, on the mortgage borrowers themselves?
Soros has made the same comment. Whatever the theory, its application will invalidate it.
re:stochastix
"I really don't want to "insult" any of the experts, but why is that people care so much about the probability distributions of stock prices?"
From a practicioner pov, there are a few reasons. obviously, if you know the ditribution, you can make money, but that is not the main reason as this goal is clearly illusionary.
More importantly, with a credible distribution, you can justify making a business by assessing its "risk" whatever that means. note here that the credibility is more important than the reality.
"I rarely see academics working on highquality orderbook data"
The high quality data are not accessible to academics.
It requires a lot of means, an army of statisticians just to clean and maintain data.
Those means are out of reach to academics. Also if you know something in that area.. why be an academic ? It's not really universal knowledge as real science can be.
PS: awesome blog you have !
"More importantly, with a credible distribution, you can justify making a business by assessing its "risk" whatever that means."
The more risk can be quantified the more underwriting volume increases. The sellside and the buyside know their risks and can hedge. The more underwriting volume increases the faster econmoic growth.
I have a very superficial knowledge about things like this  but the following might be relevant in the context of power laws in empirical data:
http://www.cscs.umich.edu/~crshalizi/weblog/491.html
Av
stochastix
Do you know of a better deterministic theory which could replace the old quantum theory of the BohrSommerfeld quantization program, which does NOT involve probability?
Correct me if I'm wrong, but aren't the eigenvalues of the SE easily represented as the minima of an infinite dimensional polynomial?
In that case, the trajetory of a marble (the state of the sytem) rolling from one minimum to another would be incomputable giving the illusion of random transitions.
A (very) little knowledge is a dangerous thing.
Post a Comment