Moore's Law and AI

By now you've probably heard that Moore's Law is really dead. So dead that the semiconductor industry roadmap for keeping it on track has more or less been abandoned: see, e.g., here, here or here. (Reported on this blog 2 years ago!)

What I have not yet seen discussed is how a significantly reduced rate of improvement in hardware capability will affect AI and the arrival of the dreaded (in some quarters) Singularity. The fundamental physical problems associated with ~ nm scale feature size could take decades or more to overcome. How much faster are today's cars and airplanes than those of 50 years ago?

Hint to technocratic planners: invest more in physicists, chemists, and materials scientists. The recent explosion in value from technology has been driven by physical science -- software gets way too much credit. From the former we got a factor of a million or more in compute power, data storage, and bandwidth. From the latter, we gained (perhaps) an order of magnitude or two in effectiveness: how much better are current OSes and programming languages than Unix and C, both of which are ~50 years old now?

HLMI = ‘high–level machine intelligence’ = one that can carry out most human professions at least as well as a typical human. (From Minds and Machines.)

Of relevance to this discussion: a big chunk of AlphaGo's performance improvement over other Go programs is due to raw compute power (link via Jess Riedel). The vertical axis is ELO score. You can see that without multi-GPU compute, AlphaGo has relatively pedestrian strength.

ELO range 2000-3000 spans amateur to lower professional Go ranks. The compute power certainly affects depth of Monte Carlo Tree Search. The initial training of the value and policy neural networks using KGS Go server positions might have still been possible with slower machines, but would have taken a long time.

DNA data deluge

NYTimes reports: DNA Sequencing Caught in Deluge of Data. Spooks and other scientists have similar problems. See also here.

NYTimes: BGI, based in China, is the world’s largest genomics research institute, with 167 DNA sequencers producing the equivalent of 2,000 human genomes a day.

BGI churns out so much data that it often cannot transmit its results to clients or collaborators over the Internet or other communications lines because that would take weeks. Instead, it sends computer disks containing the data, via FedEx.

“It sounds like an analog solution in a digital age,” conceded Sifei He, the head of cloud computing for BGI, formerly known as the Beijing Genomics Institute. But for now, he said, there is no better way.

The field of genomics is caught in a data deluge. DNA sequencing is becoming faster and cheaper at a pace far outstripping Moore’s law, which describes the rate at which computing gets faster and cheaper.

The result is that the ability to determine DNA sequences is starting to outrun the ability of researchers to store, transmit and especially to analyze the data.

... The cost of sequencing a human genome — all three billion bases of DNA in a set of human chromosomes — plunged to $10,500 last July from $8.9 million in July 2007, according to the National Human Genome Research Institute.

That is a decline by a factor of more than 800 over four years. By contrast, computing costs would have dropped by perhaps a factor of four in that time span.

The lower cost, along with increasing speed, has led to a huge increase in how much sequencing data is being produced. World capacity is now 13 quadrillion DNA bases a year, an amount that would fill a stack of DVDs two miles high, according to Michael Schatz, assistant professor of quantitative biology at the Cold Spring Harbor Laboratory on Long Island.

There will probably be 30,000 human genomes sequenced by the end of this year, up from a handful a few years ago, according to the journal Nature. And that number will rise to millions in a few years.

In a few cases, human genomes are being sequenced to help diagnose mysterious rare diseases and treat patients. But most are being sequenced as part of studies. The federally financed Cancer Genome Atlas, for instance, is sequencing the genomes of thousands of tumors and of healthy tissue from the same people, looking for genetic causes of cancer. ...

Here's a slide I sometimes use in talks.

The exponential curve for genome sequencing

Below is an update on progress towards less expensive gene sequencing. At the moment you can have your genome sequenced for $350k, but we might hit the $1k mark within just a few years. This progress is funded by a combination of taxpayer and venture capital dollars. The rate of technological advance would slow to a snail's pace without sophisticated capital markets, intellectual property rights and plain old human greed and ambition.

For a cost per base pair curve extending up to 2005, see here. As the cost nears $1k per genome we will see a tremendous explosion in detailed genetic data across all major population groups.

NYTimes: A person wanting to know his or her complete genetic blueprint can already have it done — for $350,000.

But whether a personal genome readout becomes affordable to the rest of us could depend on efforts like the one taking place secretly in a nondescript Silicon Valley industrial park. There, Pacific Biosciences has been developing a DNA sequencing machine that within a few years might be able to unravel an individual’s entire genome in minutes, for less than $1,000. The company plans to make its first public presentation about the technology on Saturday.

Pacific Biosciences, or PacBio, is just one entrant in a heated race for the “$1,000 genome” — a gold rush of activity whose various contestants threaten to shake up the current $1-billion-a-year market for machines that sequence, or read, genomes. But the company has attracted some influential investors. And some outside experts say that if the technology works — still a big if — it would represent a significant advance.

“They’re the technology that’s going to really rip things apart in being that much better than anyone else,” predicted Elaine R. Mardis, the co-director of the genome center at Washington University in St. Louis.

If the cost of sequencing a human genome can drop to $1,000 or below, experts say it would start to become feasible to document people’s DNA makeup to tell what diseases they might be at risk for, or what medicines would work best for them. A DNA genome sequence might become part of each newborn’s medical work-up, while sequencing of cancer patients’ tumors might help doctors look for ways to attack them.

To spur such advances, the federal government has awarded about 35 grants totaling $56 million to companies and universities for development of technology that could put the $1,000 genome sequence within reach. PacBio has received $6.6 million from that program.

The nonprofit X Prize Foundation, meanwhile, is offering $10 million to the first group that can sequence 100 human genomes in 10 days, for $10,000 or less per genome. Six companies or academic groups — although not PacBio — have signed up for the competition so far.

Computerized sequencing machines use various techniques to determine the order of the chemical units in DNA, which are usually represented by the letters A, C, G and T. Humans have three billion such units, or six billion if one counts the second copy of each chromosome pair.

The industry has long been dominated by Applied Biosystems, which sold hundreds of its $300,000 sequencers to the publicly financed Human Genome Project and to Celera Genomics for their sequencing of the first two human genomes, which were announced in 2000. But two newcomers — Solexa and 454 Life Sciences — have already started to cut into Applied Biosystems’ sales with machines that are faster and less costly per unit of DNA sequenced. Solexa is now owned by Illumina and 454 Life Sciences by Roche.

Applied Biosystems, which is a unit of Applera, recently started selling its own new type of sequencer, which it obtained by buying Agencourt Personal Genomics for $120 million in 2006. Helicos BioSciences, a newly public company, announced its first order on Friday. It has said its machine might be able to sequence a human genome for $72,000, with further improvements to come.

“We can look somebody in the eye and say, ‘This instrument is going to get you to the $1,000 genome,’ ” said Steve Lombardi, the president of Helicos, which is based in Cambridge, Mass.

Intelligent Bio-Systems, a privately held company in Waltham, Mass., says it will introduce a machine by the end of the year that might reduce the cost of a genome to $10,000. Other contenders include the privately held companies NABsys of Providence, R.I., VisiGen Biotechnologies of Houston and Complete Genomics of Mountain View, Calif.

Some contestants say that they might try for the X Prize as early as next year and that the $1,000 genome is as little as three years away. But other experts are more conservative. ...

Hardware vs Software

OK, call me biased, but the kind of physical science behind hardware advances seems a bit, well, harder than writing a new OS or application. In fact, if I think about the main drivers behind the information revolution of the last 20 years, I'd give much more credit to hardware advances than to the concurrent advances in software.

Think about it -- state of the art OSes aren't that different from the original BSD or Unix flavors, whereas the flash memory in my iPod is the equivalent of warp drive to someone from 1985! I don't see a factor of 10^6 improvement in anything related to software, whereas we've achieved gains of that scale in processors, storage and networking (bandwidth).

Meanwhile, funding for research in physical science has been flat in real dollars during my scientific career. Go figure!



From the Times, an article about IBM's work on "racetrack" storage, which may be key to the continued exponential growth in storage capacity.

I'm no experimentalist, but what they're doing sounds hard!

The tech world, obsessed with data density, is taking notice because Mr. Parkin has done it before. An I.B.M. research fellow largely unknown outside a small fraternity of physicists, Mr. Parkin puttered for two years in a lab in the early 1990s, trying to find a way to commercialize an odd magnetic effect of quantum mechanics he had observed at supercold temperatures. With the help of a research assistant, he was able to alter the magnetic state of tiny areas of a magnetic data storage disc, making it possible to store and retrieve information in a smaller amount of space. The huge increases in digital storage made possible by giant magnetoresistance, or GMR, made consumer audio and video iPods, as well as Google-style data centers, a reality.

Mr. Parkin’s new approach, referred to as “racetrack memory,” could outpace both solid-state flash memory chips as well as computer hard disks, making it a technology that could transform not only the storage business but the entire computing industry.

“Finally, after all these years, we’re reaching fundamental physics limits,” he said. “Racetrack says we’re going to break those scaling rules by going into the third dimension.”

His idea is to stand billions of ultrafine wire loops around the edge of a silicon chip — hence the name racetrack — and use electric current to slide infinitesimally small magnets up and down along each of the wires to be read and written as digital ones and zeros.

His research group is able to slide the tiny magnets along notched nanowires at speeds greater than 100 meters a second. Since the tiny magnetic domains have to travel only submolecular distances, it is possible to read and write magnetic regions with different polarization as quickly as a single nanosecond, or one billionth of a second — far faster than existing storage technologies.

If the racetrack idea can be made commercial, he will have done what has so far proved impossible — to take microelectronics completely into the third dimension and thus explode the two-dimensional limits of Moore’s Law, the 1965 observation by Gordon E. Moore, a co-founder of Intel, that decrees that the number of transistors on a silicon chip doubles roughly every 18 months.