Jaan Tallinn: Coronavirus, Existential Risk, and AI - Manifold Episode#42

Steve talks with Skype founder and global tech investor Jaan Tallinn. Will the coronavirus pandemic lead to better planning for future global risks? Jaan gives his list of top existential risks and describes his efforts to call attention to AI risk. They discuss AGI, the Simulation Question, the Fermi Paradox and how these are all connected. Do we live in a simulation of a quantum multiverse?

RATIONALITY

Jaan X-Risk Links

LessWrong

Slate Star Codex

Metaculus


ADDITIONAL RESOURCES

Transcript

Fermi Paradox — Where Are All The Aliens?

Is Hilbert space discrete?
https://arxiv.org/abs/hep-th/0508039


man·i·fold /ˈmanəˌfōld/ many and various.

In mathematics, a manifold is a topological space that locally resembles Euclidean space near each point.

Steve Hsu and Corey Washington have been friends for almost 30 years, and between them hold PhDs in Neuroscience, Philosophy, and Theoretical Physics. Join them for wide ranging and unfiltered conversations with leading writers, scientists, technologists, academics, entrepreneurs, investors, and more.

Steve Hsu is VP for Research and Professor of Theoretical Physics at Michigan State University. He is also a researcher in computational genomics and founder of several Silicon Valley startups, ranging from information security to biotech. Educated at Caltech and Berkeley, he was a Harvard Junior Fellow and held faculty positions at Yale and the University of Oregon before joining MSU.

Corey Washington is Director of Analytics in the Office of Research and Innovation at Michigan State University. He was educated at Amherst College and MIT before receiving a PhD in Philosophy from Stanford and a PhD in a Neuroscience from Columbia. He held faculty positions at the University Washington and the University of Maryland. Prior to MSU, Corey worked as a biotech consultant and is founder of a medical diagnostics startup.

Slate Star Codex Meetup -- Berkeley

I will be at this meetup later today:

BAY MEETUP 1/6 UPDATE

POSTED ON JANUARY 4, 2019 BY SCOTT ALEXANDER
Due to rain, we’re switching to holding the meetup indoors at 3045 Shattuck Ave, Berkeley, 94705. There will be several floors of space available for overflow, so hopefully it won’t be too crowded. Thanks to Claire, REACH, and Event Horizon for setting this up.

Time is still 3:30 PM on Sunday, 1/6. There’s also a Facebook event here.

For the unfamiliar, Slate Star Codex is one of the best blogs on the planet, with a large devoted following of rationalists. Scott is an incredibly talented writer and thinker, and I envy him his readership and commentariat :-)

Risk, Uncertainty, and Heuristics

Risk = space of outcomes and probabilities are known. Uncertainty = probabilities not known, and even space of possibilities may not be known. Heuristic rules are contrasted with algorithms like maximization of expected utility.

See also Bounded Cognition and Risk, Ambiguity, and Decision (Ellsberg).

Here's a well-known 2007 paper by Gigerenzer et al.

Helping Doctors and Patients Make Sense of Health Statistics

Gigerenzer G1, Gaissmaier W2, Kurz-Milcke E2, Schwartz LM3, Woloshin S3.

Many doctors, patients, journalists, and politicians alike do not understand what health statistics mean or draw wrong conclusions without noticing. Collective statistical illiteracy refers to the widespread inability to understand the meaning of numbers. For instance, many citizens are unaware that higher survival rates with cancer screening do not imply longer life, or that the statement that mammography screening reduces the risk of dying from breast cancer by 25% in fact means that 1 less woman out of 1,000 will die of the disease. We provide evidence that statistical illiteracy (a) is common to patients, journalists, and physicians; (b) is created by nontransparent framing of information that is sometimes an unintentional result of lack of understanding but can also be a result of intentional efforts to manipulate or persuade people; and (c) can have serious consequences for health. The causes of statistical illiteracy should not be attributed to cognitive biases alone, but to the emotional nature of the doctor-patient relationship and conflicts of interest in the healthcare system. The classic doctor-patient relation is based on (the physician's) paternalism and (the patient's) trust in authority, which make statistical literacy seem unnecessary; so does the traditional combination of determinism (physicians who seek causes, not chances) and the illusion of certainty (patients who seek certainty when there is none). We show that information pamphlets, Web sites, leaflets distributed to doctors by the pharmaceutical industry, and even medical journals often report evidence in nontransparent forms that suggest big benefits of featured interventions and small harms. Without understanding the numbers involved, the public is susceptible to political and commercial manipulation of their anxieties and hopes, which undermines the goals of informed consent and shared decision making. What can be done? We discuss the importance of teaching statistical thinking and transparent representations in primary and secondary education as well as in medical school. Yet this requires familiarizing children early on with the concept of probability and teaching statistical literacy as the art of solving real-world problems rather than applying formulas to toy problems about coins and dice. A major precondition for statistical literacy is transparent risk communication. We recommend using frequency statements instead of single-event probabilities, absolute risks instead of relative risks, mortality rates instead of survival rates, and natural frequencies instead of conditional probabilities. Psychological research on transparent visual and numerical forms of risk communication, as well as training of physicians in their use, is called for. Statistical literacy is a necessary precondition for an educated citizenship in a technological democracy. Understanding risks and asking critical questions can also shape the emotional climate in a society so that hopes and anxieties are no longer as easily manipulated from outside and citizens can develop a better-informed and more relaxed attitude toward their health.

Slate Star Codex on Superforecasting

Scott Alexander (Slate Star Codex) on Philip Tetlock's Superforecasting:

Book review
Highlighted passages.

I especially liked this passage that Scott highlights:

When hospitals created cardiac care units to treat patients recovering from heart attacks, Cochrane proposed a randomized trial to determine whether the new units delivered better results than the old treatment, which was to send the patient home for monitoring and bed rest. Physicians balked. It was obvious the cardiac care units were superior, they said, and denying patients the best care would be unethical. But Cochrane was not a man to back down…he got his trial: some patients, randomly selected, were sent to the cardiac care units while others were sent home for monitoring and bed rest. Partway through the trial, Cochrane met with a group of the cardiologists who had tried to stop his experiment. He told them that he had preliminary results. The difference in outcomes between the two treatments was not statistically signficant, he emphasized, but it appeared that patients might do slightly better in the cardiac care units. “They were vociferous in their abuse: ‘Archie,’ they said, ‘we always thought you were unethical. You must stop the trial at once.'” But then Cochrane revealed he had played a little trick. He had reversed the results: home care had done slightly better than the cardiac units. “There was dead silence and I felt rather sick because they were, after all, my medical colleagues.”

[ Cochrane Collaboration ]    [ Bounded Cognition ]

See also Medical Science?

New Yorker on behavioral economics

Elizabeth Kolbert reviews two recent books on behavioral economics. She even notes the negative consequences of "irrationality" for democracy.

NewYorker: ...Over the years, Tversky and Kahneman’s initial discoveries have been confirmed and extended in dozens of experiments. In one example, Ariely and a colleague asked students at M.I.T.’s Sloan School of Management to write the last two digits of their Social Security number at the top of a piece of paper. They then told the students to record, on the same paper, whether they would be willing to pay that many dollars for a fancy bottle of wine, a not-so-fancy bottle of wine, a book, or a box of chocolates. Finally, the students were told to write down the maximum figure they would be willing to spend for each item. Once they had finished, Ariely asked them whether they thought that their Social Security numbers had had any influence on their bids. The students dismissed this idea, but when Ariely tabulated the results he found that they were kidding themselves. The students whose Social Security number ended with the lowest figures—00 to 19—were the lowest bidders. For all the items combined, they were willing to offer, on average, sixty-seven dollars. The students in the second-lowest group—20 to 39—were somewhat more free-spending, offering, on average, a hundred and two dollars. The pattern continued up to the highest group—80 to 99—whose members were willing to spend an average of a hundred and ninety-eight dollars, or three times as much as those in the lowest group, for the same items.

This effect is called “anchoring,” and, as Ariely points out, it punches a pretty big hole in microeconomics. When you walk into Starbucks, the prices on the board are supposed to have been determined by the supply of, say, Double Chocolaty Frappuccinos, on the one hand, and the demand for them, on the other. But what if the numbers on the board are influencing your sense of what a Double Chocolaty Frappuccino is worth? In that case, price is not being determined by the interplay of supply and demand; price is, in a sense, determining itself.

...A few weeks ago, the Bureau of Economic Analysis released its figures for 2007. They showed that Americans had collectively amassed ten trillion one hundred and eighty-four billion dollars in disposable income and spent very nearly all of it—ten trillion one hundred and thirty-two billion dollars. This rate of spending was somewhat lower than the rate in 2006, when Americans spent all but thirty-nine billion dollars of their total disposable income.

According to standard economic theory, the U.S. savings rate also represents rational choice: Americans, having reviewed their options, have collectively resolved to spend virtually all the money that they have. According to behavioral economists, the low savings rate has a more immediate explanation: it proves—yet again—that people have trouble acting in their own best interests. It’s worth noting that Americans, even as they continue to spend, say that they should be putting more money away; one study of participants in 401(k) plans found that more than two-thirds believed their savings rate to be “too low.”

...Like neoclassical economics, much democratic theory rests on the assumption that people are rational. Here, too, empirical evidence suggests otherwise. Voters, it has been demonstrated, are influenced by factors ranging from how names are placed on a ballot to the jut of a politician’s jaw. A 2004 study of New York City primary-election results put the advantage of being listed first on the ballot for a local office at more than three per cent—enough of a boost to turn many races. (For statewide office, the advantage was around two per cent.) A 2005 study, conducted by psychologists at Princeton, showed that it was possible to predict the results of congressional contests by using photographs. Researchers presented subjects with fleeting images of candidates’ faces. Those candidates who, in the subjects’ opinion, looked more “competent” won about seventy per cent of the time.

When it comes to public-policy decisions, people exhibit curious—but, once again, predictable—biases. They value a service (say, upgrading fire equipment) more when it is described in isolation than when it is presented as part of a larger good (say, improving disaster preparedness). They are keen on tax “bonuses” but dislike tax “penalties,” even though the two are functionally equivalent. They are more inclined to favor a public policy when it is labelled the status quo. In assessing a policy’s benefits, they tend to ignore whole orders of magnitude. In an experiment demonstrating this last effect, sometimes called “scope insensitivity,” subjects were told that migrating birds were drowning in ponds of oil. They were then asked how much they would pay to prevent the deaths by erecting nets. To save two thousand birds, the subjects were willing to pay, on average, eighty dollars. To save twenty thousand birds, they were willing to pay only seventy-eight dollars, and to save two hundred thousand birds they were willing to pay eighty-eight dollars.

What is to be done with information like this? We can try to become more aware of the patterns governing our blunders, as “Predictably Irrational” urges. Or we can try to prod people toward more rational choices, as “Nudge” suggests. But if we really are wired to make certain kinds of mistakes, as Thaler and Sunstein and Ariely all argue, we will, it seems safe to predict, keep finding new ways to make them. (Ariely confesses that he recently bought a thirty-thousand-dollar car after reading an ad offering FREE oil changes for the next three years.)

Tyler Cowen and rationality

I recently came across the paper How economists think about rationality by Tyler Cowen. Highly recommended -- a clear and honest overview.

The excerpt below deals with rationality in finance theory and strong and weak versions of efficient markets. I believe the weak version; the strong version is nonsense. (See, e.g, here for a discussion of limits to arbitrage that permit long lasting financial bubbles. In other words, capital markets are demonstrably far from perfect, as defined below by Cowen.)

Although you might think the strong version of EMH is only important to traders and finance specialists, it is also very much related to the idea that markets are good optimizers of resource allocation for society. Do markets accurately reflect the "fundamental value of corporations"? See related discussion here.

Financial economics has one of the most extreme methods in economic theory, and increasingly one of the most prestigious. Finance concerns the pricing of market securities, the determinants of market returns, the operating of trading systems, the valuation of corporations, and the financial policies of corporations, among other topics. Specialists in finance can command very high salaries in the private sector and have helped design many financial markets and instruments. To many economists, this ability to "meet a market test" suggests that financial economists are doing something right. Depending on one's interpretation, the theory of finance makes either minimal or extreme assumptions about rationality. Let us consider the efficient markets hypothesis (EMH), which holds the status of a central core for finance, though without commanding universal assent. Like most economic claims, EMH comes in many forms, some weaker, others stronger. The weaker versions typically claim that deliberate stock picking does not on average outperform selecting stocks randomly, such as by throwing darts at the financial page. The market already incorporates information about the value of companies into the stock prices, and no one individual can beat this information, other than by random luck, or perhaps by outright insider trading.

Note that the weak version of EMH requires few assumptions about rationality. Many market participants may be grossly irrational or systematically biased in a variety of ways. It must be the case, however, that their irrationalities are unpredictable to the remaining rational investors. If the irrationalities were predictable, rational investors could make systematic extra-normal profits with some trading rule. The data, however, suggest that it is very hard for rational investors to outperform the market averages. This suggests that extant irrationalities are either very small, or very hard to predict, two very different conclusions. The commitment that one of these conclusions must be true does not involve much of a substantive position on the rationality front.

The stronger forms of EMH claim that market prices accurately reflect the fundamental values of corporations and thus cannot be improved upon. This does involve a differing and arguably stronger commitment to a notion of rationality.

Strong EMH still allows that most individuals may be irrational, regardless of how we define that concept. These individuals could literally be behaving on a random basis, or perhaps even deliberately counter to standard rationality assumptions. It is assumed, however, that at least one individual does have rational information about how much stocks are worth. Furthermore, and most importantly, it is assumed that capital markets are perfect or nearly perfect. With perfect capital markets, the one rational individual will overwhelm the influence of the irrational on stock prices. If the stock ought to be worth $30 a share, but irrational "noise traders" push it down to $20 a share, the person who knows better will keep on buying shares until the price has risen to $30. With perfect capital markets, there is no limit to this arbitrage process. Even if the person who knows better has limited wealth, he or she can borrow against the value of the shares and continue to buy, making money in the process and pushing the share price to its proper value.

So the assumptions about rationality in strong EMH are tricky. Only one person need be rational, but through perfect capital markets, this one person will have decisive weight on market prices. As noted above, this can be taken as either an extreme or modest assumption. While no one believes that capital markets are literally perfect, they may be "perfect enough" to allow the rational investors to prevail.

"Behavioral finance" is currently a fad in financial theory, and in the eyes of many it may become the new mainstream. Behavioral finance typically weakens rationality assumptions, usually with a view towards explaining "market anomalies." Almost always these models assume imperfect capital markets, to prevent a small number of rational investors from dwarfing the influence of behavioral factors. Robert J. Shiller claims that investors overreact to very small pieces of information, causing virtually irrelevant news to have a large impact on market prices. Other economists argue that some fund managers "churn" their portfolios, and trade for no good reason, simply to give their employers the impression that they are working hard. It appears that during the Internet stock boom, simply having the suffix "dot com" in the firm's name added value on share markets, and that after the bust it subtracted value.11

Behavioral models use looser notions of rationality than does EMH. Rarely do behavioral models postulate outright irrationality, rather the term "quasi-rationality" is popular in the literature. Most frequently, a behavioral model introduces only a single deviation from classical rationality postulates. The assumption of imperfect capital markets then creates the possibility that this quasi-rationality will have a real impact on market phenomena.

The debates between the behavioral theories and EMH now form the central dispute in modern financial theory. In essence, one vision of rationality -- the rational overwhelm the influence of the irrational through perfect capital markets -- is pitted against another vision -- imperfect capital markets give real influence to quasi-rationality. These differing approaches to rationality, combined with assumptions about capital markets, are considered to be eminently testable.

Game theory and the failed quest for a unique basis for rationality:

Game theory has shown economists that the concept of rationality is more problematic than they had previously believed. What is rational depends not only on the objective features of the problem but also depends on what actors believe. This short discussion has only scratched the surface of how beliefs may imply very complex solutions, or multiple solutions. Sometimes the relevant beliefs, for instance, are beliefs about the out-of-equilibrium behavior of other agents. These beliefs are very hard to model, or it is very hard to find agreement among theorists as to how they should be modeled.

In sum, game theorists spend much of their time trying to figure out what rationality means. They are virtually unique amongst economists in this regard. Game theory from twenty years ago pitted various concepts of rationality against each other in purely theoretical terms. Empirical results had some feedback into this process, such as when economists reject Nash equilibrium for some of its counterintuitive predictions, but it remains striking how much of the early literature does not refer to any empirical tests. This enterprise has now become much more empirical, and more closely tied to both computational science and experimental economics.

Computational economics and the failed quest for a unique basis for rationality:

Nonetheless it is easy to see how the emphasis on computability puts rationality assumptions back on center stage, and further breaks down the idea of a monolithic approach to rationality. The choice of computational algorithm is not given a priori, but is continually up for grabs. Furthermore the choice of algorithm will go a long way to determining the results of the model. Given that the algorithm suddenly is rationality, computational economics forces economists to debate which assumptions about procedural rationality are reasonable or useful ones.

The mainstream criticism of computational models, of course, falls right out of these issues. Critics believe that computational models can generate just about "any" result, depending on the assumptions about what is computable. This would move economics away from being a unified science. Furthermore it is not clear how we should evaluate the reasonableness of one set of assumptions about computability as opposed to another set. We might consider whether the assumptions yield plausible results, but if we already know what a plausible result consists of, it is not clear why we need computational theories of rationality.

As you can tell from my comments, I do not believe there is any unique basis for "rationality" in economics. Humans are flawed information processing units produced by the random vagaries of evolution. Not only are we different from each other, but these differences arise both from genes and the individual paths taken through life. Can a complex system comprised of such creatures be modeled through simple equations describing a few coarse grained variables? In some rare cases, perhaps yes, but in most cases, I would guess no. Finance theory already adopts this perspective in insisting on a stochastic (random) component in any model of security prices. Over sufficiently long timescales even the properties of the random component are not constant! (Hence, stochastic volatility, etc.)