"Hadoop wildly overhyped" and other database arcana

Michael Stonebreaker talk on the current state of database architecture and technology, entitled The Traditional RDBMS Wisdom is (Almost Certainly) All Wrong. This field was pretty dead for a long time, but big changes are afoot now. Readers might be familiar with Streambase, one of his startups.

A friend summarizes the talk as follows

1. Relational databases are going through large transformations (not just as a threat from the non relational no-sql DBs, but the internal model that Oracle, MySQL itself use is breaking apart and need to be rewritten).

2. The current relational model has effectively already broken apart into two solutions (in large companies)- Huge datasets for after the fact statistical analysis (ie- Walmart trying to analyze a recent sale, etc) have already moved over to column grouped data (as opposed to row records), which lets you stream through a given column of data quickly. Real time data on the other hand, is kept at the terabyte range, and the whole database is loaded into memory. Although this speeds up things significantly, the next bottlenecks appear around thread locking, etc- Fixing this is an area of current research, but the speedup gains are ultimately orders of magnitude faster than the old relational database model, and so will probably all change soon.

3. The speaker described Hadoop as being terrible at everything except embarrassingly parallel multi machine computations. He mentioned that Google itself doesn't use mapreduce anymore and is probably amused at all the attention that it is getting in the world.

There were some other interesting points in the talk, but the main takeaway is that we probably are in for some huge changes in the next decade.