You can see that there is significant predictive efficiency gained in going from 2 to 3 or 3 to 4 stars. For example, there are only about a third more 2 star recruits than 3 star recruits, but the latter are almost 10x as likely to be named All-American. On the other hand, the distinction between 4 and 5 star recruits seems a bit iffy to me. Only about 1 in 10 players at or above the 4 star level are designated 5 star, with the latter distinction raising their All-America odds by a bit less than 3. There's information there, but you pay a high price in selectivity.
It's not surprising that this system works. Coaches are heavily incentivized to win, and some ratings are provided by professionals who make their living scouting HS talent. But of course prediction is imperfect: there are plenty of 3 and 4 star recruits who outperform 5 stars. The issue is whether expected return from using the predictions is positive...
University admissions committees should be able to produce an analysis of this quality or better. If the objective function to be optimized is performance at the university, the data is readily available (see below). But one could also use criteria such as eventual net worth (major donor status), fame, notable achievements, etc. to assess quality of admissions decisions. Lots of assertions are made in this context (see comments on this NYTimes article), such as that high test scores are negatively correlated with leadership or interpersonal skills that impact later life. This might be true, but I've yet to find any careful analysis of the claim.
How much can we enhance odds ratios for becoming a millionaire / billionaire / STEM PhD / Nobel laureate by proper filtering of applicants?
See Data mining the university and Nonlinear psychometric thresholds for physics and mathematics for predictors of college performance by major as a function of HS GPA and SAT.
"...such as that high test scores are negatively correlated with leadership or interpersonal skills that impact later life."
ReplyDeleteIf one defines leadership as winning high-school popularity contests, research does show that higher test scores negatively correlate with leadership.
If I read the table correctly, the jump from 4 stars to 5 stars looks a little better when you consider players rather than selections (I found this a little bit surprising, I expected a higher rate of multiple selections from the higher rated players, how do 3-star recruits have the highest rate?!). "Odds" of being named All-American (ignoring multiple selections): 5-star 1/23, 4-star 1/77, 3-star 1/391, 2-star 1/1826, overall 1/306
ReplyDeleteI wonder how much (and what type of) effort is devoted to evaluating the "mispredictions" and refining the ratings.
P.S. I think they are giving probabilities rather than odds (AA/all rather than AA/(not AA)) so I did the same. It doesn't make that much of a difference for low numbers like these.
Kind of sad how athletics is more or less a complete meritocracy in this country, but academics has been corrupted by racist non-sense. Today, the NFL is roughly 70% African American, despite the fact that African Americans make up only 13% or so of the general population. No one seems to have a problem with that. I guess making sure that we're entertained by the best possible athletes is a worthwhile endeavor, but making sure that our schools admit people strictly on merit isn't worth anyone's time.
ReplyDeleteI think there's a more fundamental problem here with the way we define success.
ReplyDeleteSome have suggested that while mathematical ability is conducive to value creation, verbal ability is conducive to value transference. Many people become "successful" in life not because they create tangible value for society, but because they've managed to claw their way to the top of the corporate hierarchy. The free market rewards those most adept at transferring value to themselves. If there's a negative correlation between high test scores and future leadership potential, then it reflects the sad reality that the real class struggle in society is between the value creation class and the value transference class.
What if we fundamentally re-calibrated the way we define success? Rather than encouraging our children to aspire to be glib suits who create nothing, but merely own, what if we encouraged them to be the future generation of scientists or engineers?
Your shtick is tired and dumb. No one talks about race in sports for the same reasons no one talks about race in anything else. Noticing leads to questioning race egalitarian assumptions, so its suppressed by self-censorship+threats of social ostracism.
ReplyDeleteOT: Most universities would do a way better job admitting talent if they were scientific about the input-->output. The amazing thing is that they aren't doing it already, considering the first school to do it would be at a tremendous academic advantage overnight. There are too many vested interests everywhere: addcoms want their campuses to be vibrant pictures of diversity and they REALLY want to help bring about racial equality. Beyond that they're going to service their donor legacy's and favor students who are sporting 'budding activist progressive, helped build huts last summer' resumes. They are pursuing their objectives and there is no political will to hold them accountable, if that's even the right word.
If eventual net worth or fame are the criteria, cut to the chase and just choose those whose parents are very rich or famous already. No statistical study needed.
ReplyDeleteI've had anonymous commentators who sounded authentic claimed they'd worked on sophisticated models for the admissions and development departments at Ivy League-level schools that predict what kind of high school students are likely to turn into big donors as alumni. But, the models are very hush-hush.
ReplyDeleteMy impression from reading articles about big donors are that smart jocks, especially if they are legacies or their children are legacies, are the #1 givers.
Recent Duke admissions records are available via their sociology project on college life. Duke's president Brodhead is a Yalie, former Yale professor and former dean of Yale college, so I suspect that Duke's methodology is not dissimilar to Yale's. I doubt there are sophisticated models. There may be general conclusions taken from historical studies -- such as that athletes are more likely to succeed in business, legacies more likely to give, etc. Some of this is discussed in The Chosen in the context of Harvard admissions. The Duke scoring system is quite conventional.
ReplyDeleteI would also recommend The Game of Life by James Shulman and former Princeton president William Bowen. They emphasize hard data, and they dispute your contention that jocks are big donors, in addition to many other arguments for college athletics like increasing underrepresented-minority numbers, increasing donations during winning seasons, and athletic teams covering their own expenses.
ReplyDeleteI think that a huge part of the problem is that no one agrees (or maybe is willing to explicitly state) what the actual purpose of higher education is. Is it training students with skills that will be useful in future careers? Or is it to inspire the "leaders of tomorrow" or to "liberate students to explore, to create, to challenge, and to lead" or to help students "discharge the offices of life with usefulness and
ReplyDeletereputation" or some other wishy-washy statement? Or is it all just an extremely expensive way to accredit and network (as you have pointed out in earlier posts)? The curriculum (especially for traditional humanities majors) at pretty much every undergraduate institution is based on outdated ideas about the role of higher education and does not seem to be very closely related to any reasonable idea about what the goal of higher education should be. I'm not surprised that admissions criteria reflect the same central confusion.
Of course, technical majors and schools are much better about this, and correspondingly there are much sharper metrics that one could discuss optimizing. Still, even in the sciences, I think there are questions that aren't being given sufficient attention. For instance, speaking to friends from my alma mater (Harvey Mudd), it seems practicing engineers typically find less than 10% of what they learned in getting their engineering degree useful in their career (beyond getting the job in the first place). (I think that is a fairly high estimate.) Of course there are a number of caveats before drawing conclusions (for instance, whether having the broader context helps one understand the specialized skills used in a future career or if everyone is using a different 10%), but it raises the question: what is the college degree actually _for_? I think this question immediately informs what hypothetical function one should be trying to maximize.
(I should note that I think Harvey Mudd is an exceptionally good school, but I think its curriculum still reflects some of the central confusions about regarding the role of college education in the modern world.)
Why do you doubt the existence of such sophisticated methods, Steve? It doesn't require a conspiracy theory to think Duke's publicity acknowledged scoring system is supplemented using less-tasteful methods that aren't publically known.
ReplyDeleteIt's in the university's interest to keep these methods to themselves, and instead publicize a hard-to-analyze and so hard-to-criticize holistic approaches. Combined with the fact that some vagueness in the methods allows the university leaders to deceive themselves a bit, it all seems sensible.
Mr Hsu,
ReplyDeleteHow are things going at the Cognitive Research Lab?
My mother's cousin was an applied math prof at U Wash and in Who's Who. His daughter was denied admission to Stanford when less qualifieds were admitted. When Stanford tried to recruit him, he said FU.
ReplyDeleteThe bottom line is the US is shit.