Wednesday, November 07, 2012

Hail to the quants, pundit fail

Pundit idiocracy: "Close race", "Too close to call", "Neck and neck". (I heard this all day long.)

Quants and data geeks: "Obama will win. Unlikely to be close."

From an earlier post High V, Low M:
high verbal ability ... is useful for appearing to be smart, or for winning arguments and impressing other people, but it's really high math ability that is useful for discovering things about the world -- that is, discovering truth or reasoning rigorously.

... The statistical techniques used to analyze data obtained in a messy, complex world require mathematical ability to practice correctly. In almost all realistic circumstances hypothesis testing is intrinsically mathematical.
See also Obama wins! and Expert Prediction. Scorecard of predictions here (accuracy highly correlated with M, not V ;-)

Who is this guy?
Xu Cheng, Moodys’ Analytics: Obama 303, Romney 235 (Note that this prediction was made back in February) “This prediction is tied to the Moody’s Analytics current baseline forecast for U.S. growth, which assumes that most states will continue to recover at slow to moderate speeds.”


38 comments:

Ron Narmelo said...

Unfortunately those who have the most power to make decisions that have immediate consequences have often made their way into power by virtue of their high V.
And if they do possess high M they do a great job of suppressing it in favor of the fame and fortune that comes along with the V component in certain domains ( especially in politics).
Recalling the presidential debates, I am amazed (in a bad way) by how readily individuals who seek to serve and influence an entire nation speak about important matters with an appalling disregard for the facts.

Can you imagine a physicist declaring at an important conference that he or she asserts x & y based on findings that upon review turn out to be inaccurate? Should the American nation condone alarming inaccuracies by presidential candidates because the latter have "charm" ? Argumentatively debating is one thing, twisting the facts is another; the two should be mutually exclusive in the presidential process.

I am very happy that we have a black president as it reflects a civil advancement from less civil times in the past... But what about a scientist president ?

Anonymous_IV said...

www.xkcd.com/1131 :-)

Liam Finley said...

In fairness, pundit George Will also called it pretty close, 321-217. Except that he got the two candidates switched around.

All in all, a bad night for hacks, but they're not going away. Just as the stock pickers and market gurus aren't going away from CNBC no matter how many times they get it wrong.

Paul said...

Wang seems to have been off a bit in his predictions for the House (claims gerrymandering may be responsible for this).

Rodrigo Guzman said...

not only that, but this wasn't exactly sophisticated machine learning on big data sets with complicated data. all it took to call this election correctly since ~last week (safe side) was to take simple averages of the latest poll data. so much innumeracy is so sad.

Stephen Hsu said...

Central Limit Theorem = black magic for our punditocracy (idiocracy).

Yan Shen said...

"But what about a scientist president ?"

You know, I wonder if there are any countries that might be characterized as technocracies and ruled by scientists/engineers as opposed to the typical lawyer/businessmen/politician types we have who run things in this country... And I also wonder what a comparison of the recent track record between these different kinds of governments might reveal about the effectiveness of each type of leadership...

tractal said...

.

SethTS said...

Indeed. Did you see this interactive graphic in the NYT?

http://elections.nytimes.com/2012/results/president/scenarios

All 2^9 paths are treated as equally likely, but this is SO much better than just reporting two numbers from a national poll.

SethTS said...

Um... is the answer to your riddle "China"? Steve posted once or twice on the idea of "performance legitimacy" used among Chinese technocrats as their excuse for avoiding democratic accountability.

LondonYoung said...

High M? question - the capital markets were down 2.5 % today on this result. What can we infer from that? Given that the result was not a surprise to the High M crowd, I was surprised at this violent reaction.

steve hsu said...

It's not just the High M crowd that moves markets ...

steve hsu said...

Nice graphic!

steve hsu said...

New technology, not yet well understood ... :-)

asdf asdf said...

Aren't high M low V people often artistic nerds with zero clue how people and the real world work.


Do we all need to rewatch that Simpsons episode when Mensa takes over.

tractal said...

Why bother posting all these interesting talent search studies which suggest both V and M are important to things like scientific research if you're going to maintain regardless that V is just fluff for show? To my (limited) knowledge the literature does not support any of these sweeping conclusions. I know some of your comments are just being snide but this is a research interest of yours, right?

BlackRoseML said...

I can't prove the central limit theorem.

"All 2^9 paths are treated as equally likely, but this is SO much better than just reporting two numbers from a national poll."

I knew considering each swing state as being statistically independent from each other would be an erroneous assumption, since factors affecting voter turnout for a given candidate in one state would likely exert a similar influence in another state. The focal point of my own analysis was the outcome of New Hampshire, but it was not concerned with accurately forecasting the outcome in individual states, just the result of the general Presidential election.

I posted this on Facebook


I don't vote in bourgeois elections as a Marxist-Leninist, but it would seem that the election would be decided by New Hampshire, despite the state only having four electoral votes: if Obama wins New Hampshire, the Bayesian probability that Obama wins is practically one (barring some miracle in Wisconsin, Michigan, Pennsylvania, or Iowa,and let's not forget that what happens in NH during this election is also correlated with the outcomes in other [swing] states [such as Nevada and Colorado that closes after NH].) [I omitted mention of Ohio, since Obama can win without it and NC, Virginia, and Florida, by winning NH, Nevada, Colorado, Iowa, Wisconsin, and Pennsylvania -- states that lean towards Obama in pre-election polling and in-trade bets.]


I will check at 6 - 6:30 PM PST to see the outcome in NH. Also, if either Virginia, NC, or Florida are too close to call during that time, regardless of the eventual final tabulation of the votes of those states, Obama would win, since it is likely the outcomes in mildly contestable states would also favor Obama. Romney needs to win those three states in order to have a fighting chance.

Christopher Chang said...

In fairness, a few pundit-types did explicitly state that, if the poll results were not systematically biased, Silver was right; they just hoped that, despite the media frenzy, the "likely voter" models of most polls were wrong and Democrat turnout would be low. Silver did acknowledge that this type of group error has happened in the not-so-distant past. But yes, most pundits were innumerate.

Terry Tao's comment: https://plus.google.com/u/0/114134834346472219368/posts/gnC9BZQyCTM

steve hsu said...

Go back to the original High V Low M post and look very carefully at what I wrote about V. High V High M and High V Low M are two very different phenotypes. See also the SMPY / SVPY papers.

"High verbal ability is useful for more than just impressing others -- it typically implies a certain facility with concepts and relationships between ideas -- but high V alone is a dangerous thing. The most confused people I meet in the Academy tend to be high V, low (modest) M types.

More on the V / M split in this longitudinal study of gifted children (SMPY / SVPY -- see esp. figure 4)."

LaurentMelchiorTellier said...

"Pundits" on every channel, if one seeks them out.

I feel uncertain about two assumptions, though:


1. Is the ability to spew punditry a good indicator of the V dimension? Despite the obvious, intuitive link (communication comes out of mouth in language format = verbal), I'd expect the correlation between confident salesmanship and V to be fairly modest. If you look at difficult V test items, such as fx. "verbal reasoning" test items, it seems likely that fast-talking, clever-faced, divisive-statement-making pundits would frequently do poorly on them - keeping in mind the close correlation between scores on arithmetic and verbal reasoning. Perhaps more often a case of "modest V, modest M, high narrative ability".

2. Are the pundits selected for slots on TV even supposed to predict or argue well? As opposed to having the ability to provoke the desired types of emotional narratives in the viewer (including the narrative "ah, this pundit doesn't know what he's talking about, I want to watch him some more because loathing is addictive"), in much the same way as Honey Boo Boo garners a viewership exponentially larger than Terry Tao. It's imho likely that the channels choose their "pundits", not with any illusion of making accurate predictions, but to compel primate eyeballs, in much the same way as compellingly flawed ideas and personalities are used in other TV shows. Narratives, sold as "news".

steve hsu said...

It's not just TV pundits. Look at David Brooks (or any NYTimes columnist), who clearly is a high V modest M type.

BlackRoseML said...

Could anyone define "modest" M ability, especially in the context of this thread. What minimum threshold mathematical and statistical competence is need to possess "modest M"? Does it require one to have an understanding of calculus, such as differential equations, and statistics, such as knowing about correlation, statistical independence, central limit theorem, probability distributions, and variance.

LaurentMelchiorTellier said...

I wish I hadn't looked. >_<

*
Cognizant of the respect which NYT journalism is supposed to command, I'm not certain that V need be very high to do opinion column journalism of this type either (David hasn't got anything on James). Other criteria may weigh heavier.

steve hsu said...

There's no hard threshold. Even someone who doesn't understand what a standard error estimate is can still appreciate "quant magic good" and thereby improve their own predictions about the future. Interestingly, my impression is that elite education at, e.g., HPS is good because many HVLM types who attend these schools will develop a healthy respect for quant magic (or respect for HVHM thinking). This doesn't always happen and some HVLM types are dismissive of quant methods throughout their entire lives. One positive thing about this election is that both candidates were people who could appreciate (if not perform) quant analysis.

Paul said...

"One positive thing about this election is that both candidates were people who could appreciate (if not perform) quant analysis."

This is true, but IMO it will still be a long, long time before "quant magic" is universally understood or embraced as it applies to a variety of problems. It's easy to support quant methods when those methods tell you what you want to hear (you're going to win an election, or you can win if you just do xyz). But the most useful applications of statistical analysis and the scientific method are just as likely (maybe more likely) to tell us things that are disturbing, disheartening, bleak, contrary to our pre-existing world view etc.

Wishful thinking is something we're all guilty of, to varying extents. I told everybody I knew that Obama was going to win and it wasn't going to be close; since none of my acquaintances were particularly enamored of Romney, they were agreeable to this, and I looked smart. Mostly, though, I keep my mouth shut, because it's not worth offending people's sensibilities with numbers.

Stephen Hsu said...

Just as personality factors are largely independent of cognitive factors, the characteristic of being "epistemologically careful" (someone who reasons cautiously, with minimal bias) is only imperfectly correlated with g. I know many physicists who fall in love with their theoretical constructs and lose the ability to reason dispassionately about whether they actually describe Nature. The problem is much, much worse in fields that study complex, strong-interacting systems. "All priors, All the time"!

tractal said...

I understood that you think having both high M and V is important to some kinds of achievement. There was, however, a mismatch between the rhetoric in your post and that conclusion. Whatever high V is, it behaves like a very important part of understanding the world. I agree that low M is going to probably lead to more confusion than low V (as in this case) but even then is it low "M", or just statistical illiteracy? George Will and other shamans are probably capable of learning at least enough quant to discard their own conclusions. Not necessarily a case of "low M" in the cognitive sense, so much as sheer ignorance. Election augery is another case where math helped us predict nature, but you framed it as another case where M cognitive abilities trump V, which is arguable. Especially considering most of the commentariate isn't really high anything. I should not have attacked your post like I did, but even if the pundits were representative of high V (which here would have to mean very high V, since they're going up against very high M quants) the reasoning would still be shaky.

As an aside, is it possible that academic HVLM types seem really confused to you because they work in unempirical fields? Not having to square anything with reality lets bias reign free. The most striking aspect of humanities intellectuals, to me, is that they are vehemently ideological and live in an echo chamber. Hard to say if its because low M, or just due to living in an intellectual environment that can afford to hold onto every prejudice and throw epistemic caution to the wind. Its a little hard to judge Gould, for instance. The man was a Marxist writing about sociobiology! There probably isn't enough M in the world to save a marxist from getting sociobiology importantly wrong.

Along similar lines its interesting that the extremely high V, very high M kids in the SMPY study tended to go into humanities fields, even though their math scores indicated exceptionally high ability in STEM. Do those guys just end up being the good social scientists and English Professors? I'm guessing quite a few of them end up manifesting the kind of thinking and attitudes you associate with HVLM. No one in the middle ages understood Bayesian probability. Neither does anyone in an English department, for the same reasons.

If I've been aggressive its because this "Yay M, Boo superficial V" has become a theme, but hasn't been accompanied by proportionate evidence or argument. You've written interesting things re: how accessing frontiers of knowledge require a high M threshold, and that this will only continue. But a lot of that discussion seems to play fast and loose with "M" as knowledge, and "M" as a specifically mathematical, non-verbal component of "G." I would never take issue with the idea that M knowledge is critical, and more "important" than textual analysis etc. The notion, however, that there are a bunch of glib HVLM types running about doing stupid things (M being synonymous with rigorous reasoning) and get away with it because V lets them look smart is a little much. Likely, "G" is the dominant factor here and elsewhere, verbal ability and mathematical ability are both important, and sometimes necessary, conditions for science and math achievement, and a lot of the "HVLM" phenomenon is just low G / bias. This post earned snark because the evidence at hand has almost no relationship to the theme pushed, and the theme itself is pretty petty (and probably an overstatement of the truth). Sometimes, though, that's the nature of a blog (writing about topical things, weaving in ideas when you can-->sometimes mismatch.)

BlackRoseML said...

I asked for the definition in this thread because we are primarily discussing the cognitive profiles of individuals of moderate intellectual caliber (that is people who are capable of being accepted into a Public Ivy or better); surely in the context of this thread, "high M" does not mean someone who cares in the low 600s of the SAT, even though that in the low 80th percentile of the general population. Tractal, I thank you for pointing out the distinction between "high M" and "statistical literacy", since the former is likely to be an innate cognitive trait, while the latter is an acquired competency that is definitely correlated, but most certainly, not synonymous with "high M". Indeed, I believe the cognitive "M" threshold of possessing "statistical literacy" is low enough that it does not require "high M". Also, I can envision a mechanism where verbal ability can facilitate the development of statistical competence. For instance, consider Hsu's example of standard error of measurement. It definitely requires one to notice the nuance between the observed mean of a given random sample taken from a larger group, and the "real" mean from the group, and how the "sample mean" approximates the "real mean". Realizing these nuances requires an aptitude to discern subtle, abstract concepts -- an aptitude that is likely the essence of V.

I agree with the crux of tractal's reasoning: lacking high M, but possessing high V, does not preclude one from genuinely comprehending statistical reasoning without any faith (belief in "quant magic"). I do not see the necessity for "high M" anyway, since interpreting poll results does not require high mathematics or the mental manipulation inherent in the more difficult SAT-M problems.

Yan Shen said...

"They did this based on emotional need."


That must explain why many smart people even are vehemently anti-HBD or believe in God, etc...

Yan Shen said...

"Just as personality factors are largely independent of cognitive factors..."

Is this necessarily true? I get the impression from looking around the world that people who are significantly skewed towards M often are shy, quiet, passive, meek math geeks who barely know how to stand up for themselves, while people skewed towards V have fiery, aggressive personalities(the stereotype of the quiet and self-effacing engineer versus say the boisterous and argumentative lawyer comes to mind).

I'm often amazed that the most vocal opponents of anti-Asian racism these days are Jewish Americans such as Lee Siegel rather than Asian Americans themselves, who are far more likely to offer up suggestions that Asian Americans should either accept the racist status quo in this country or return to the land of their origin...

Yan Shen said...

One point that has been noted by Half Sigma and others is that high M seems to be conducive to value creation, while high V seems to be conducive to value transference. Some might say that there's nothing smarter than being able to con society into paying you money that you did relatively little to deserve!

Odoacer said...

I am skeptical about how well technocrats can govern. They may be smart (and sometimes too confident), but they will be often wrong, just less often wrong than stupid people about certain things.

David Coughlin said...

And I will add, you can find discussions of the pullback all over the place. It is useful to think about it in the epistemological context. You start with a naive universe of possibilities, a space of parameters joined to a space of dynamical variables. Then you supposed relationships on the parameters, operate on the dynamical variables [pick any one of the 'middle' sets in the pullback construction] and get at a set of outcomes. Alternately, you can start with a set of dynamical outcomes, and work your way backwards, operating on the set of parameters, getting you back to an understanding of the universe of possibilites. I'm happy to just be able to pick right between have-an-idea and have-some-data. The virtuous circle of learning goes around and around that construction. I get especially annoyed when people dogmatically go one way or the other [and the have-an-idea dogmatists are the worst].

jeffhsu3 said...

How about Germany? Angela Merkel has a PhD in quantum chemistry and did her undergrad in physics.

ben_g said...

Seems like the problem may be more party identification than low M. The media is motivated to make everything look closer than it is, from sports to politics. And of course the party hacks will always predict a win for their side.

steve hsu said...

Sure, that's why left-leaning commentators kept saying it was "neck and neck", "too close to call" all through election day...

yulva said...

“This prediction is tied to the Moody’s Analytics current baseline
forecast for U.S. growth, which assumes that most states will continue
to recover at slow to moderate speeds.”

According to the BLS, in their latest revision, only 125,000 net new jobs were created during Obama's presidency.

http://www.huffingtonpost.com/2012/09/27/data-job-growth-obama_n_1919533.html

Hastings Banda said...

"but it's really high math ability that is useful for discovering things
about the world -- that is, discovering truth or reasoning rigorously."


Did this blog's author say this?


Really high math ability is useful for doing really high math and theoretical physics and ripping the face off muppets. That's it.


As a graduate in maths I wish it were more useful. For more read A Mathematicians Apology.

Blog Archive

Labels