Stanley: ...
When they analyzed these data–200 million of them–in exactly the same fashion that Bachelier had analyzed data almost a century earlier, they made a startling discovery. The pdf of price changes was not Gaussian plus outliers, as previously believed. Rather, all the data–including data previously termed outliers–conformed to a single pdf encompassing both everyday fluctuations and “once in a century” fluctuations. Instead of a Gaussian or some correction to a Gaussian, they found a power law pdf with exponent 4, a sufficiently large exponent that the difference from a Gaussian is not huge; however, the probability of a “once in a century” event of, say, 100 standard deviations is exp(10,000) for the Gaussian, but simply 108 for an inverse quartic law. If one analyzes a data set containing 200 million data in two years, this means there are only two such events–in two years!
Now which is better, the concept of “everyday fluctuations” which can be modeled with a drunkard’s walk, complemented by a few “once in a century” outliers? Or a single empirical law with no outliers but for which a complete theory does not exist despite promising progress by Xavier Gabaix of NYU’s Stern School of Management and his collaborators? Here we come to one of the most salient differences between traditional economics and the econophysicists: economists are hesitant to put much stock in laws that have no coherent and complete theory supporting them, while physicists cannot afford this reluctance. There are so many phenomena we do not understand. Indeed, many physics “laws” have proved useful long before any theoretical underpinning was developed . . . Newton’s laws and Coulomb’s law to name but two.
And all of us are loathe to accept even a welldocumented empirical law that seems to go against our own everyday experience. For stock price fluctuations, we all experience calm periods of everyday fluctuations, punctuated by highly volatile periods that seem to cluster. So we would expect the pdf of stock price fluctuations to be bimodal, with a broad maximum centered around, say, 13 standard deviations and then a narrow peak centered around, say, 50 standard deviations. And it is easy to show that if we do not have access to “all the data” but instead sample only a small fraction of the 200 million data recently analyzed, then this everyday experience is perfectly correct, since the rare events are indeed rare and we barely recall those that are “large but not that large”.
The same is true for earthquakes: our everyday experience teaches us that small quakes are going on all the time but are barely noticeable except by those who work at seismic detection stations. And every so often occurs a “once in a century” truly horrific event, such as the famous San Francisco earthquake. Yet when seismic stations analyze all the data, they find not the bimodal distribution of everyday experience but rather a power law, the GutenbergRichter law, describing the number of earthquakes of a given magnitude.
Tuesday, February 03, 2009
Power laws, baby
Thanks to reader GS for the link to this excellent article by statistical physicist Eugene Stanley. There's much more to the article, but I decided to excerpt this discussion of security price fluctuations and the observation that they are far from log normal.
Subscribe to:
Post Comments (Atom)
Blog Archive

▼
2009
(204)

▼
02
(21)
 Deprogramming the cult of the Efficient Market
 Trento talk
 Innovation competitiveness
 Macro graphs
 David X. Li
 Demand destruction
 Netbooks and tipping points
 Heritability of brain structure
 Race and IQ in Nature
 Bhide, innovation and basic research
 Designer babies
 The World's Greatest Economic Minds
 Robot Genius, PC Armor
 Ant algorithms
 Liam Casey on BBC
 On $500k a year
 US car sales fall below China’s for first time
 Comment moderation
 How they actually MAKE the money
 Power laws, baby
 Markets overshoot on the downside, too

▼
02
(21)
Labels
 physics (321)
 finance (257)
 genetics (256)
 globalization (245)
 brainpower (232)
 technology (175)
 american society (158)
 economics (153)
 China (152)
 photos (150)
 psychometrics (147)
 science (146)
 genomics (131)
 innovation (131)
 psychology (126)
 travel (122)
 credit crisis (115)
 universities (107)
 higher education (106)
 human capital (105)
 biology (97)
 iq (91)
 startups (89)
 ai (86)
 genetic engineering (85)
 cognitive science (78)
 credit crunch (78)
 careers (75)
 gilded age (71)
 elitism (70)
 evolution (69)
 income inequality (69)
 autobiographical (67)
 statistics (60)
 books (59)
 quantum mechanics (59)
 machine learning (58)
 caltech (56)
 genius (54)
 social science (54)
 talks (50)
 sci fi (48)
 bgi (47)
 history of science (47)
 kids (47)
 mma (47)
 cdo (45)
 silicon valley (45)
 education (43)
 geopolitics (43)
 derivatives (42)
 mathematics (40)
 podcasts (40)
 bounded rationality (39)
 harvard (39)
 politics (39)
 behavioral economics (38)
 academia (37)
 intellectual history (37)
 political correctness (37)
 bubbles (36)
 mortgages (36)
 MSU (35)
 history (35)
 jiujitsu (34)
 literature (34)
 hedge funds (32)
 expert prediction (31)
 realpolitik (31)
 bjj (30)
 foo camp (30)
 physical training (30)
 film (29)
 quants (29)
 ufc (29)
 computing (28)
 efficient markets (28)
 many worlds (28)
 black holes (27)
 economic history (27)
 sports (27)
 entrepreneurs (25)
 subprime (25)
 google (24)
 housing (24)
 obama (24)
 singularity (24)
 taiwan (24)
 movies (23)
 race relations (23)
 von Neumann (23)
 berkeley (22)
 feynman (22)
 ultimate fighting (22)
 athletics (21)
 nuclear weapons (21)
 wall street (21)
 affirmative action (20)
 cds (20)
 neuroscience (20)
 scifoo (20)
 music (19)
 biotech (18)
 goldman sachs (18)
 internet (17)
 meritocracy (17)
 quantum field theory (17)
 security (17)
 treasury bailout (17)
 university of oregon (17)
 venture capital (17)
 conferences (16)
 freeman dyson (16)
 gender (16)
 smpy (16)
 autism (15)
 blogging (15)
 cosmology (15)
 cryptography (15)
 japan (15)
 oppenheimer (15)
 personality (15)
 algorithms (14)
 happiness (14)
 new yorker (14)
 hedonic treadmill (13)
 probability (13)
 aspergers (12)
 height (12)
 india (12)
 malcolm gladwell (12)
 net worth (12)
 nobel prize (12)
 television (12)
 wwii (12)
 christmas (11)
 dna (11)
 fitness (11)
 geeks (11)
 les grandes ecoles (11)
 neanderthals (11)
 string theory (11)
 blade runner (10)
 entropy (10)
 football (10)
 government (10)
 italy (10)
 mutants (10)
 nerds (10)
 olympics (10)
 social networks (10)
 ability (9)
 chess (9)
 dating (9)
 eugene (9)
 flynn effect (9)
 pseudoscience (9)
 Einstein (8)
 complexity (8)
 crossfit (8)
 determinism (8)
 encryption (8)
 harvard society of fellows (8)
 keynes (8)
 pca (8)
 philip k. dick (8)
 philosophy of mind (8)
 pop culture (8)
 real estate (8)
 research (8)
 robot genius (8)
 usain bolt (8)
 aig (7)
 alpha (7)
 art (7)
 ashkenazim (7)
 basketball (7)
 data mining (7)
 energy (7)
 free will (7)
 game theory (7)
 hugh everett (7)
 james salter (7)
 manhattan (7)
 poker (7)
 qcd (7)
 turing test (7)
 Go (6)
 alan turing (6)
 anthropic principle (6)
 environmentalism (6)
 france (6)
 fx (6)
 games (6)
 nassim taleb (6)
 nsa (6)
 privacy (6)
 prostitution (6)
 russia (6)
 success (6)
 tail risk (6)
 teaching (6)
 volatility (6)
 war (6)
 Fermi problems (5)
 academia sinica (5)
 bayes (5)
 bobby fischer (5)
 class (5)
 climate change (5)
 democracy (5)
 econtalk (5)
 luck (5)
 noam chomsky (5)
 paris (5)
 renaissance technologies (5)
 sad but true (5)
 software development (5)
 warren buffet (5)
 100m (4)
 Iran (4)
 Poincare (4)
 borges (4)
 cambridge uk (4)
 charles darwin (4)
 cold war (4)
 creativity (4)
 fake alpha (4)
 feminism (4)
 global warming (4)
 godel (4)
 hormones (4)
 humor (4)
 inequality (4)
 intellectual property (4)
 iraq war (4)
 kerviel (4)
 markets (4)
 microsoft (4)
 mixed martial arts (4)
 monsters (4)
 moore's law (4)
 nonlinearity (4)
 patents (4)
 perimeter institute (4)
 solar energy (4)
 soros (4)
 trento (4)
 video (4)
 vietnam war (4)
 200m (3)
 babies (3)
 bill gates (3)
 brain drain (3)
 censorship (3)
 charlie munger (3)
 chet baker (3)
 correlation (3)
 demographics (3)
 ecosystems (3)
 equity risk premium (3)
 facebook (3)
 fannie (3)
 fst (3)
 information theory (3)
 intellectual ventures (3)
 jim simons (3)
 judo (3)
 kasparov (3)
 language (3)
 lee kwan yew (3)
 lewontin fallacy (3)
 lhc (3)
 michael lewis (3)
 nathan myhrvold (3)
 neal stephenson (3)
 new york times (3)
 olympiads (3)
 path integrals (3)
 quantum computers (3)
 rationality (3)
 risk preference (3)
 search (3)
 sec (3)
 sivs (3)
 society generale (3)
 thailand (3)
 alibaba (2)
 assortative mating (2)
 bear stearns (2)
 bruce springsteen (2)
 charles babbage (2)
 cheng ting hsu (2)
 cloning (2)
 david mamet (2)
 digital books (2)
 donald mackenzie (2)
 drones (2)
 eliot spitzer (2)
 empire (2)
 exchange rates (2)
 freddie (2)
 gaussian copula (2)
 heinlein (2)
 industrial revolution (2)
 james watson (2)
 ltcm (2)
 magic (2)
 mating (2)
 mba (2)
 mccain (2)
 monkeys (2)
 national character (2)
 nicholas metropolis (2)
 no holds barred (2)
 offices (2)
 oligarchs (2)
 palin (2)
 population structure (2)
 prisoner's dilemma (2)
 simulation (2)
 skidelsky (2)
 socgen (2)
 sprints (2)
 supercomputers (2)
 systemic risk (2)
 variance (2)
 virtual reality (2)
 abx (1)
 anathem (1)
 andrew lo (1)
 antikythera mechanism (1)
 athens (1)
 atlas shrugged (1)
 ayn rand (1)
 bay area (1)
 beats (1)
 book search (1)
 bunnie huang (1)
 car dealers (1)
 carlos slim (1)
 catastrophe bonds (1)
 cdos (1)
 ces 2008 (1)
 chance (1)
 children (1)
 cochranharpending (1)
 cpi (1)
 david x. li (1)
 dick cavett (1)
 dolomites (1)
 drugs (1)
 dune (1)
 eharmony (1)
 epidemics (1)
 escorts (1)
 faces (1)
 fads (1)
 favorite posts (1)
 fiber optic cable (1)
 francis crick (1)
 frauds (1)
 gary brecher (1)
 gizmos (1)
 greece (1)
 greenspan (1)
 hypocrisy (1)
 igon value (1)
 iit (1)
 inflation (1)
 information asymmetry (1)
 iphone (1)
 jack kerouac (1)
 jaynes (1)
 jfk (1)
 john dolan (1)
 john kerry (1)
 john paulson (1)
 john searle (1)
 john tierney (1)
 jonathan littell (1)
 las vegas (1)
 lawyers (1)
 lehman auction (1)
 les bienveillantes (1)
 lowell wood (1)
 lse (1)
 mcgeorge bundy (1)
 mexico (1)
 michael jackson (1)
 mickey rourke (1)
 migration (1)
 mit (1)
 money:tech (1)
 myron scholes (1)
 netwon institute (1)
 networks (1)
 newton institute (1)
 nfl (1)
 oliver stone (1)
 phil gramm (1)
 philanthropy (1)
 philip greenspun (1)
 portfolio theory (1)
 power laws (1)
 randomness (1)
 recession (1)
 sales (1)
 singapore (1)
 skype (1)
 standard deviation (1)
 star wars (1)
 starship troopers (1)
 students today (1)
 teleportation (1)
 tierney lab blog (1)
 tomonaga (1)
 twitter (1)
 tyler cowen (1)
 ussr (1)
 venice (1)
 violence (1)
 virtual meetings (1)
 war nerd (1)
 wealth effect (1)
11 comments:
"So we would expect the pdf of stock price fluctuations to be bimodal"
This is incorrect. What "we would expect" is one distribution with ever changing parameters.
Distributions over short periods "we would expect" to be inconsistent with distributions over long periods.
This guy doesn't know what he's taking about.
I remember reading a good paper explaining the frequent occurence of power laws in many context. can't find it thoug..
I once tried to detect outliers to clean historic data, and the data that don't fit gaussian returns are overwhelming.
Even filtering for 0.0001 quartile turns you over lots of valid data points.
There are many assumption that are completely wrong in finance, not only statistically, but also economically.
The complete lack of use uncertainty for estimates is from my perspective a bigger problem.
Complex models needs to be calibrated (over stationary distribution..)
I really don't want to "insult" any of the experts, but why is that people care so much about the probability distributions of stock prices?
As every physicist knows, Einstein found it hard to grasp the randomness in Quantum Mechanics. If I remember it right, Einstein considered a probabilistic approach an exercise of "defeatism". In finance, people seem to be quite defeatist. A great example is the fantasy of gaussian fluctuations. Then along came Mandelbrot, Taleb and their homies with new fantasies such as power laws.
IMHO, going for random walks, power laws and the like is a simplistic approach. There are buying and selling orders being issued at various prices, all the time. A "good model" of stock price fluctuations should account for the structure of the process that leads to the formation of the prices. I rarely see academics working on highquality orderbook data, so I am not too surprised that finance has been (and shall always be) a cemetery of theories. In Physics, no one would be able to go for so long with such crude experimental data, but in Finance it seems that anything goes...
"I rarely see academics working on highquality orderbook data"
This would apply only to "highfrequency finance".
As someone looking at it from the econ side, I kind of agree that fancier distributions won't tell us about the mechanisms of these tail events  and the econophysicists themselves are part of the financial industry, therefore part of the system as a whole. All those mortgage borrowers taking on more debt than they could afford were a tail event  but they, in turn, were affected by the availability of cheap credit, made possible by Wall Street. I'd like to ask an econophysicist one day  can they model the effect that a newly developed formula for pricing mortgages has, on the mortgage borrowers themselves?
Soros has made the same comment. Whatever the theory, its application will invalidate it.
re:stochastix
"I really don't want to "insult" any of the experts, but why is that people care so much about the probability distributions of stock prices?"
From a practicioner pov, there are a few reasons. obviously, if you know the ditribution, you can make money, but that is not the main reason as this goal is clearly illusionary.
More importantly, with a credible distribution, you can justify making a business by assessing its "risk" whatever that means. note here that the credibility is more important than the reality.
"I rarely see academics working on highquality orderbook data"
The high quality data are not accessible to academics.
It requires a lot of means, an army of statisticians just to clean and maintain data.
Those means are out of reach to academics. Also if you know something in that area.. why be an academic ? It's not really universal knowledge as real science can be.
PS: awesome blog you have !
"More importantly, with a credible distribution, you can justify making a business by assessing its "risk" whatever that means."
The more risk can be quantified the more underwriting volume increases. The sellside and the buyside know their risks and can hedge. The more underwriting volume increases the faster econmoic growth.
I have a very superficial knowledge about things like this  but the following might be relevant in the context of power laws in empirical data:
http://www.cscs.umich.edu/~crshalizi/weblog/491.html
Av
stochastix
Do you know of a better deterministic theory which could replace the old quantum theory of the BohrSommerfeld quantization program, which does NOT involve probability?
Correct me if I'm wrong, but aren't the eigenvalues of the SE easily represented as the minima of an infinite dimensional polynomial?
In that case, the trajetory of a marble (the state of the sytem) rolling from one minimum to another would be incomputable giving the illusion of random transitions.
A (very) little knowledge is a dangerous thing.
Post a Comment