Baidu, baby, Baidu

Although Google sits astride the English (and Indo-European)-language Web like a colossus, their market share in Chinese is less than 25 percent and hasn't grown at all in recent years, even under the lead of celebrity hire Kai Fu Lee (who recently left Google to run his own venture fund). Search in China is dominated by Baidu, which appears to be Google's only rival in any language (eat your heart out, Quaero and Agence de l'innovation industrielle :-)

Given that China has the largest number of internet users (over 300 million and still growing very fast) and generates more search queries than the US, this is an interesting strategic development.

For a discussion of Baidu by founder Robin Li, watch this talk. The first 40 minutes or so will be old hat to any startup veteran, but the last 20 minutes is fairly interesting -- Li makes some interesting comments about the future of innovation in China. He notes that the Chinese language search index triples in size each year, whereas in English and other languages it only grows about 50% per year.






Here's a goofy idea for a sci fi story: a conversation or encounter between two big early AIs, one which evolved out of Google and its Indo-European corpus of data + "Mechanical Turk" / user input, and the other which evolved from Baidu and its primarily Sinic corpus and input. Would they be different in a fundamental way?

WolframAlpha: W|A

If you are attuned to the geek universe, you've probably already heard the rumors about WolframAlpha, the "computational knowledge engine" from Steve Wolfram, the creator of Mathematica. The service is scheduled to launch this weekend, and media hype has already reached a frenzy -- see, e.g., this NYTimes article.

Curious to find out more, I emailed a former theoretical physicist who works (remotely) for Wolfram Research but sits just down the hall from me. Years ago I had lobbied for an office for him because, although he isn't primarily in physics research any more, he's a brilliant guy and was bound to end up doing something interesting. (Wanting to take advantage of the opportunity to work remotely at Wolfram he had researched the best places to live and ended up choosing Eugene.) Little did I know he's been leading the W|A project from here! He stumbled into my office yesterday to answer my questions and tell me that he's knee deep in reviewing code commits and getting ready for the launch :-)

My experience has confirmed over and over again that big brains tend to do interesting things. Whenever I meet someone who is "scary smart" (there are not so many in the world), I keep an eye out for what they do later in life.

Here is what AI researcher Doug Lenat wrote about W|A:

...At its heart is a formal Mathematica representation. Its inference engine is basically a large number of individually hand-engineered scripts for tapping into data which he and his team have spent the last several years gathering and "curating". For example, he has assembled tables of historical financial information about countries' GDP's and about companies' stock prices. In a small number of cases, he also connects via API to third party information, but mostly for realtime data such as a current stock price or current temperature. Rather than connecting to and relying on the current or future Semantic Web, Alpha computes its answers primarily from his own curated data to the extent possible; he sees Alpha as the home for almost all the information it needs, and will use to answer users' queries.

In an important sense, Alpha is a logical extension of Mathematica: it extends the range of types of information for which significant power can be gained by manually, and exhaustively, enumerating a large set of cases: airplane designs, cities, currencies, etc. I.e., Alpha extends what Mathematica has done previously for things like chemical compounds, geometric surfaces, topological configurations, arithmetic series, trigonometric ratios, and equations. In the new cases, as Mathematica did in those abstract math cases, Alpha excels at not just retrieving the stored data but performing various appropriate numeric calculations on the data, and displaying the results in beautiful graphs and easily comprehended tables for the user.

The resulting mosaic covers a large portion of the space of queries that the average person might genuinely want to ask, in the course of their day. The interface is not exactly natural language, but can be treated by the user as though it were -- just as users of browsers can treat them as though they parsed sentences even though they don't. A better way to think of it is a DWIMM ("do what I might mean"), so if you type in something like "gdp France / Germany", it calculates and returns a graph of the relative fraction of France's annual GDP to Germany's GDP, over the last 30 years or so. If you just type in "gdp", it looks up your local host and (in my case) displays the GDP of the USA over the last 30 years, plus various pieces of information about what gross domestic product is, from a mathematical formula perspective but not from a semantic one. It does not have an ontology, so what it knows about, say, GDP, or population, or stock price, is no more nor less than the equations that involve that term.

See also this video of a lecture by Wolfram about W|A.

Two book reviews

I've had both of these on my shelf for some time, but haven't found time to write detailed reviews.


Google's PageRank and Beyond, Langville and Meyer (Princeton University Press).

Written by two math professors, this is the best technical account I could find of search algorithms. The math (mainly linear algebra and a little graph theory) is accessible and introduced in a self-contained way in a separate chapter. The coverage isn't limited to beautiful algorithmic ideas (the primary one being to find the dominant eigenvector of the matrix representing the graph of hyperlinks) -- the discussion includes nitty gritty details like how to treat dangling nodes, how to accelerate computations, etc. There's also a running historical summary of Google's development up to and including the IPO.

If you're wondering why I have this book, it's not just academic curiosity -- the PageRank algorithm in its basic form can be understood pretty quickly from overviews available online. I'm interested in understanding the current state of the art and the possibility of improvements ;-)


An Engine, Not a Camera, D. MacKenzie (MIT Press)

This is the best history of modern finance and options pricing theory I have yet read. MacKenzie has a sufficient understanding of the theory and of the subtle sociological issues involved (strangely, he is not an economist but a sociologist). Figures like Mandelbrot (the mathematician), Thorp (perhaps the real inventor of Black-Scholes) and Osborne (a physicist) appear along with better known economists like Samuelson, Fama, Miller, Sharpe, Black, Scholes, Merton, etc. The section on Mandelbrot and Levy distributions is especially good, as is the account of LTCM. The title is from Milton Friedman, who (controversially) characterized economic theory as an "engine to analyze the world, not a photographic reproduction of it".